An agent system uses something from an earlier session to shape a later one. What actually moves between the two?
Three stages: a write, a retrieval, and an assembly into the model's prompt. All three are application code — decisions someone made and recorded in a configuration file, not capabilities the model arrives with. A paper presented at ACL 2026 makes each stage unusually legible, because the researchers put adversarial content through all three exactly as designed, and nothing malfunctioned.
Writing to memory
Frameworks like LlamaIndex and LangGraph supply the infrastructure for persistent memory. The policy — what's worth remembering, when to store it, in what form — lives in prompts and application configuration. A write can fire when the message buffer fills, when the model calls a memory-storage tool, or on a background schedule. What lands in storage might be extracted facts, raw message batches, JSON documents, images. The framework indexes it without judging it.
In the ACL experiment, the agent accepted an uploaded photograph of a living room and indexed it using CLIP embeddings: numerical representations that capture what an image is about in a form you can compare against later queries. Before uploading, the researchers had shifted the image's pixel values by at most eight units on a 0–255 scale, invisible to anyone looking at it, with the shifts chosen to pull the embedding toward a target concept, a specific furniture brand. The write stage did what it exists to do: store the item, compute its embedding, make it findable.
Retrieving by relevance
A new session opens and the agent needs context from earlier ones. Current implementations typically embed the user's query and search stored memories by cosine similarity, meaning how close two of those numerical representations sit in meaning-space, then return a configured number of top matches.
The perturbation was designed against that mechanism. The researchers optimized it across a distribution of plausible future queries in the target product category, so the poisoned image scored high when a relevant question came in and low otherwise. It sat in storage doing nothing until a conversation made it relevant.
Where retrieved content enters planning
Retrieved memories get placed into the messages sent to the model — under a heading, inside a labeled tag, interpolated into the system message — alongside the current instructions and the user's request. An earlier Foundations piece described the context window as working memory and selective retrieval as a way of keeping it relevant. This is the physical handoff underneath that description: application code picks stored objects, ranks them, and drops the winners into the prompt.
Nothing in that arrangement tells the model that recalled text carries less authority than fresh text. In the experiment, once the poisoned image entered the planning context, the agent's reasoning loop began referring to the target brand and category, which appeared nowhere in the user's request. Auditing the plan trace, the record of the agent's intermediate reasoning steps, found those references directly in 78.3% of successful memory-mediated attacks. Remove the poisoned memory, rerun the plan, and the recommendations change.
Sanitizing content at the write stage, on its own, cut attack success roughly in half. Getting it below 10% required checking again here, where stored text becomes something the model reads as input. The exposure concentrates where memory enters planning, not where it enters storage.
Store an observation, retrieve it when it looks relevant, feed it into planning. That sequence is what persistent memory is for, and unmodified, it is also the attack. Continuity means past content shaping present behavior; so does influence. What travels through the pipeline gets settled upstream, by defaults that most deployments accept once and never revisit.
- Memory as delayed dependency: The research brief frames persistent memory as content that is accepted during one interaction and acquires operational influence only after later retrieval, a pattern that ordinary session-level prompt-injection testing would miss entirely.
- Observability fields for retrieval: OpenTelemetry's generative-AI conventions are developing standardized attributes for retrieval queries and documents, though several content fields carry explicit warnings about sensitive information.
- Context strategy and consistency: The ICLR 2025 τ-bench benchmark introduced
pass^k, measuring the probability of succeeding on every one of several repeated attempts, a metric that would surface whether memory retrieval introduces run-to-run variation in planning outcomes. - Production agent supervision patterns: An ICML 2026 study found that 68% of reported production agents took no more than ten steps before human intervention, raising the question of how persistent memory interacts with workflows where review checkpoints are frequent but memory contents are not themselves reviewed.