EngineeringNovember 13, 202620 min read

Agentic Memory Is Now Its Own Field

Revision, forgetting, custody, security, and the variable almost everyone ignores: how relevant a memory has to be before an agent is allowed to see it. The evidence that agent memory has become its own discipline, the six architectures that break when facts change, and the engineering contract for memory that stays true.

Dr. Majid FekriCTO
Agentic Memory Is Now Its Own Field
AGENTIC MEMORY

For the first two years of the agent wave, memory wasn’t a field. It was a weekend ticket. You spun up a vector store, embedded the chat turns, injected the top-k chunks into the system prompt, and shipped.

In the last eighteen months, that consensus collapsed.

Look at the evidence: controlled studies where memory makes agents worse than no memory at all, the biggest memory vendors re-architecting their own products, OWASP giving memory poisoning its own entry, and a wave of benchmarks that test revision instead of retrieval. Research labs and production teams have reached the same quiet conclusion. Managing agentic memory is no longer a feature you bolt onto a RAG pipeline. It is its own engineering discipline.

We’ve watched this split happen before. Databases separated from file systems once applications needed transactions. Observability separated from stdout once systems became distributed. In both cases, the easy part (putting bytes somewhere) stayed easy, and a new field grew around the hard part.

For agent memory, the easy part is storing and fetching text. The hard parts are state change, temporal truth, chain of custody, and selective forgetting, plus one more that gets far less attention than it deserves: deciding, at recall time, which memories are relevant enough to reach the agent at all. A thin API wrapper can’t solve any of these, and pretending otherwise is quietly breaking production agents.

Here’s the evidence that agentic memory has become its own field, the six standard architectures and why each breaks when facts change, why relevance scoring turned out to matter more than we expected, and the engineering contract for memory that stays true.

Four signals that memory has become its own discipline

You can tell a problem has outgrown “library feature” status by where its failures, threats and redesigns cluster. For agent memory, all three are clustering at once.

Fig. 1. Ten milestones in under two years.

Signal 1: Naive memory makes agents worse than no memory

In August 2026, the TEPA researchers ran 8,000 controlled episodes of a tool-selection task in which the right answer reverses partway through (Zhou et al., arXiv:2608.07429). The episodes were scored by a deterministic executor, with no language model in the loop, so the result isolates the memory policy itself. After the reversal:

Memory policySuccess after reversal
No memory at all30.9%
Append-only memory21.0%
Last-write-wins memory21.0%
Explicit revocation (TEPA)95.0%

On a preference-reversal stream the gap was wider still: append-only memory scored 13.8%, against 83.7% for an agent with no memory at all.

The authors name the mechanism precisely: stale evidence stays highly relevant to the current query, so similarity and recency give only a weak signal that a fact has been superseded. A passive memory doesn’t just fail to help once the world changes. It keeps serving precedents that look right and aren’t.

Fig. 2. When the facts reverse, memory that can’t revoke is worse than no memory. Data: TEPA, Table 2.

Signal 2: The leading vendors are redesigning their own products

Watch the changelogs, not the marketing.

  • Mem0 retired its signature write-time reconciler. The old pipeline used two LLM passes, the second deciding ADD, UPDATE or DELETE against existing memories. In April 2026 Mem0 replaced it with a single ADD-only pass, writing that the reconciliation step “was slow, and it was where context got destroyed” and that overwrites “sometimes erased key information.” It reported gains of 21 points on LoCoMo and 27 on LongMemEval alongside the change, which also added entity linking and multi-signal retrieval (Mem0).
  • Letta (the team behind MemGPT) pioneered the agent that edits its own memory blocks through tool calls. What it built next was the sleep-time agent: a background agent that manages the primary agent’s memory, because “offloading memory to a sleep-time agent allows memory management to happen asynchronously” (Letta).
  • Zep built Graphiti around a temporal knowledge graph that invalidates facts with validity windows instead of deleting them (Zep docs), a bet that temporal state needs its own machinery rather than better cosine scores.

When the best-known champions of write-time resolution and self-editing workers move to non-destructive lifecycles and background memory agents, the field is telling you something.

Signal 3: Benchmarks moved from recall to revision

A year ago, memory benchmarks asked whether an agent could find a needle in a haystack. Now retrieval is table stakes, and revision is the bottleneck.

  • MemoryAgentBench (UC San Diego) tests whether memory can revise itself when new evidence contradicts old (Hu et al., arXiv:2507.05257). The dedicated memory systems it tested (Mem0, Cognee, Zep, MemGPT and MIRIX) scored between 7% and 28% on single-hop updates. On multi-hop updates, where a changed fact must propagate through a second fact, every method tested scored 7% or less, long-context models and RAG pipelines included.
  • LongMemEval (ICLR 2025) made knowledge updates, temporal reasoning and abstention three of its five core abilities, and found commercial assistants and long-context models losing about 30% accuracy across sustained interactions (Wu et al.).
  • LoCoMo is now the subject of methodological fights, from how vendors configure each other’s systems (Zep) to a common harness that leaves out its 446 adversarial questions, the ones where the right answer is to decline (DEV Community).
Fig. 3. Retrieval isn’t the hard part; revision is. Data: MemoryAgentBench, Table 3.

A subfield only fights over its harnesses when the stakes are real. And notice what that LoCoMo fight is about: whether a memory system can say I don’t know. We’ll come back to that.

Signal 4: Memory is now a security and governance surface

In December 2025, OWASP’s Top 10 for Agentic Applications listed Memory & Context Poisoning as ASI06 (summary). It followed the MINJA paper, which showed that an attacker sending only ordinary queries can plant malicious records in an agent’s long-term memory, with a 98.2% injection success rate and a 76.8% attack success rate (Dong et al., arXiv:2503.03704).

Standards are arriving too. In June 2026 Google Cloud published the Open Knowledge Format, a vendor-neutral spec for agent knowledge as Markdown files with YAML front matter (PPC Land). Fields get standards; features don’t.

Meanwhile, developers no longer run one agent. They run Claude Code, Cursor, Codex CLI and background LangGraph or CrewAI workers side by side, and per-tool memory notebooks drift apart within hours. Provenance, confidence thresholds, cross-agent sync and blast-radius isolation have become requirements, not nice-to-haves.

The taxonomy of failure: six ways the industry wires memory

Strip away the embeddings, graphs and prompt templates, and ask one question of any memory system: who decides what is true right now?

Long-lived memory isn’t a container. It’s a continuous stream of five decisions:

  • Keep. Is this worth remembering, and how sure are we?
  • Revise. Did something new change what an old memory means?
  • Retire. Is this fact no longer current, even though it still matches the query?
  • Escalate. Do two memories disagree in a way a model shouldn’t settle alone?
  • Brief. Of everything we know, what does this agent need right now, and how much of it?

Sort today’s landscape by who owns those decisions and six architectures fall out. Each puts the wrong party in charge of the truth.

Fig. 4. Six setups, six owners, and where each breaks.

Setup 1: The vector store. Owner: nobody.

Text goes in; the top-k most similar chunks come out. “The API rate limit is 100 per minute” and “The API rate limit was raised to 500 per minute” have nearly identical embeddings, so both land in the prompt. Chroma’s context-rot study of 18 models found that even a single topically related distractor degrades accuracy (Chroma), and a stale fact is the perfect distractor: same topic, wrong answer. A vector store has no opinion about time. It only has an opinion about cosine.

Setup 2: The write-time resolver. Owner: the writer.

On every write, an LLM compares the new message with existing memories and emits add, update or delete. The decision happens at the worst possible moment: with one message of context, inside an opaque model call, and destructively. When it misreads scope, say by treating “async feedback for design reviews” and “sync calls for incidents” as a contradiction, it deletes a rule that was still true (DEV Community). Mem0’s own explanation for abandoning the pattern says it plainly: that step was “where context got destroyed.”

Setup 3: Append-only with read-time ranking. Owner: the reader.

Keep everything, and let recency and similarity sort it out at query time. This fixes silent deletion, which is why Mem0 moved here. But it doesn’t resolve the contradiction. It hands it to the worker agent mid-task, inside a tight token budget, with no authority to fix the underlying memory. Tomorrow the agent meets the same contradiction again. That’s the setup TEPA scored below no memory at all.

Setup 4: The temporal knowledge graph. Owner: the pipeline.

An asynchronous LLM pipeline extracts entities and relationships into a graph with validity windows, as Zep and Graphiti do. Invalidating rather than deleting is the right instinct. It breaks on autonomy and latency. The extraction pipeline still settles contradictions on its own, without escalation, and Zep’s docs say ingestion can take from a few seconds to a few minutes (Zep docs). In a fleet, that’s a race condition: agent B acts before agent A’s update has landed.

Setup 5: The self-editing agent. Owner: the worker.

The worker edits its own memory blocks through tool calls, the original MemGPT pattern (Letta docs). Now the worker is doing two jobs, spending scarce attention on housekeeping in the middle of the user’s task. That’s exactly why Letta added sleep-time agents.

Setup 6: The standing instruction file. Owner: you, at 11 p.m.

CLAUDE.md, AGENTS.md, .cursorrules. Honest, explicit and in git, right up until you use three coding tools that drift apart, the file bloats into every session’s context window, and nothing your agents learn in production ever writes back to it.

“Count the owners: nobody, the writer, the reader, the pipeline, the worker, and you. None of them is something whose job is keeping the memory estate true.

The blind spot: every setup hands the agent a ranking, not a judgment

All six setups argue about what to keep and how to revise it. Almost none of them ask the question that decides what the agent actually sees: how relevant does a memory have to be before it reaches the agent?

Top-k can’t say “I don’t remember”

Top-k is a promise to return k results. Ask a memory about something it has never seen, and a top-10 retriever still hands back ten memories. They aren’t relevant. They’re the ten least irrelevant, and the agent can’t tell the difference.

That’s because a cosine score is a ranking signal, not a judgment. Its scale shifts with the embedding model, scores for unrelated text often bunch together, and Netflix researchers have shown analytically that cosine similarity over learned embeddings can yield “arbitrary and therefore meaningless” similarities (Steck et al., arXiv:2403.05440). A 0.78 means nothing on its own. No threshold transfers cleanly across models, queries and memory estates, so most stacks never set one.

Why this bites harder in memory than in document search

Three properties make relevance harder for agent memory:

  • Memory is full of near-twins. Superseded versions of the same fact are, by construction, highly similar to each other and to the query.
  • Memory only grows. The gap between “closest” and “relevant” widens every week the estate gets bigger.
  • The reader acts. A wrong memory isn’t a bad sentence in a summary. It’s a wrong action.

And extra context is not neutral. Google Research found that models given insufficient context tend to answer anyway rather than abstain: one model’s rate of incorrect answers rose from 10.2% with no context to 66.1% with insufficient context, and retrieval “paradoxically reduces the model’s ability to abstain” (Google Research; Joren et al., ICLR 2025). Hand an agent ten marginal memories and you haven’t given it options. You’ve given it permission to be confidently wrong.

What a useful relevance score looks like

To do real work at recall time, a relevance score needs three properties cosine lacks:

  • Absolute. A 0.9 should mean the same thing for every query and every agent, so a threshold means something.
  • Bounded and labelled. A fixed scale with named bands that an agent, or a human, can read.
  • Deterministic. The same query over the same memories returns the same scores, so you can debug it.

That’s what the ITS score in Moorcheh, the retrieval engine Memanto runs on, is built to be. ITS, short for Information-Theoretic Similarity, scores every result on a fixed 0-to-1 scale with published bands, from Close Match at 0.894 and above, through Very High, High, Good, Low and Very Low relevance, down to Irrelevant below 0.1 (Moorcheh docs). The Moorcheh paper describes it as a universal relevance measure that doubles as a built-in reranker, computed over an exhaustive scan instead of an approximate index, so the same query always returns identical results (Abtahi et al., arXiv:2601.11557).

Fig. 5. Top-k always answers; a relevance gate can say no. Example scores are illustrative; the ITS bands are Moorcheh’s published ones.

Two jobs: gate and focus

In Memanto, the score does two jobs.

It gates. A minimum-similarity threshold sets the floor, and the result list fills only with memories that clear it. The threshold, not k, decides how many memories come back. Recalls stay short and on topic. And when nothing clears the bar, nothing comes back, which for a question your memory can’t answer is the honest result.

It focuses. The agent receives the score itself, not just the order. A ranking tells an agent which memory is best. An absolute score tells it whether the best is good enough. A 0.91 close match is something to act on. A 0.27 is a cue to verify, to ask, or to say “I don’t have that.”

The evidence so far

Our own paper held a clue we underrated at the time. Memanto’s five-stage ablation on LongMemEval (arXiv:2604.22085) started at 56.6% with a fixed top-10 and an ITS threshold of 0.15:

ChangeGain
Widen the budget to 40, relax the threshold to 0.10+20.4
Rewrite prompts+2.2
Replace fixed top-k with ITS-gated retrieval of up to 100 memories at a threshold of 0.05+5.8
Upgrade the answering model+4.8

Add it up: of the 33.2 points gained, 26.2 came from how memories were recalled and 4.8 from which model read them.

Fig. 6. How memories were recalled moved accuracy more than which model read them. Data: Memanto paper.

Two honest caveats. The budget and the threshold changed together, so their effects aren’t isolated. And on that benchmark the right move was to loosen the threshold, because at 0.15 relevant memories fell below the cutoff. That’s the same lesson from the other side: a threshold is a policy you tune. On an absolute scale you can tune it by meaning, looser for exhaustive question-answering and stricter for an agent about to take an action, instead of re-learning a magic number for every embedding model.

Our next round of benchmark results, which we’ll publish soon together with the harness, is where this stopped being a clue. What surprised us most was how much of the grounding came from the relevance score: gating kept recalls short and free of irrelevant memories, and handing agents the actual relevance number gave them a focus point that a bare ranking can’t.

The three gates

Put that together with the rest of this article and you get a simple rule. A memory should reach an agent only when it passes three independent gates: it is relevant (its ITS score clears the threshold), trusted (its confidence and provenance clear the bar for that channel) and current (it’s active, or was active at the moment you’re asking about). Most stacks check one thing: similarity rank.

Fig. 7. Grounded recall checks three things. Most stacks check one.

Why “memory agent” is an architecture, not a marketing term

Follow the failure modes to their logical conclusion.

  • Memory is a stream of decisions (curate, reconcile, consolidate, brief, provision), not a passive store.
  • Those decisions need an owner with the full history in view, its own clock, bounded authority, an audit trail and a path to escalate to a human.
  • Worker agents shouldn’t hold the authority to destroy memory on the hot path. That’s separation of duties.
  • Someone also has to decide, every time an agent asks, how much of the memory estate it gets to see and how strongly each piece matches. That decision can’t be “always ten.”
“Build something that watches writes and production traces, scores confidence and relevance, checks for conflicts on its own schedule, retires stale records reversibly, briefs agents with only what clears the bar, and escalates real ambiguity to a person, and you haven’t built a memory API. You’ve built a memory agent.

Full disclosure: I’m the CTO of the team behind Memanto, an open-source (MIT) memory agent (paper: [arXiv:2604.22085](https://arxiv.org/abs/2604.22085)), and Moorcheh, the retrieval engine it runs on. We built Memanto around exactly this conclusion. Whether you use it or build your own, here is the contract modern agent memory has to meet.

The engineering contract for agentic memory

Eight clauses. Every example below runs against open-source Memanto (pip install memanto) with a local server on port 8000.

1. A non-destructive lifecycle, and separation of duties

Worker agents should be able to remember and recall, but not silently overwrite or delete. In Memanto a memory is either active or expired. Expiring keeps the content and stamps expired_at and expired_by: the name of the policy rule that retired it, manual, or conflict-resolution. Restoring clears the stamp.

bash# Retire a memory reversibly, with an audit reason
curl -X POST "http://localhost:8000/api/v2/agents/support-bot/memories/$MEMORY_ID/expire" \
  -H "X-Session-Token: $TOKEN" -H "Content-Type: application/json" \
  -d '{"reason": "superseded-by-v2-rate-limit"}'

# Made a mistake? Bring it back
memanto memory restore $MEMORY_ID

By default, Memanto’s MCP server gives worker agents seven tools (remember, batch remember, recall, three time-based recalls and answer) and none that delete or retire. Agent-management tools stay off unless you set MEMANTO_EXPOSE_ADMIN=true. An agent you hand the full CLI can do more, so grant that on purpose.

2. Typed revocation, not cosine ranking

A passing observation is not an architectural decision. Memanto types every memory as one of 13 kinds (fact, preference, decision, commitment, goal, event, instruction, relationship, context, learning, observation, error, artifact), and per-agent YAML policies retire memories by type, tag, source, provenance and confidence:

yaml# ~/.memanto/policies/support-bot.yaml
version: 1
retention:
  context: 7d
  error: 30d
  observation: 60d
rules:
  - name: pinned
    match: { tags: [pinned] }
    expire_after: never
  - name: unverified-voice-claims
    match: { source: [vapi], tags: [unverified] }
    expire_after: 7d
  - name: low-confidence-guesses
    match: { provenance: [inferred], confidence_below: 0.5 }
    expire_after: 14d
purge_expired_after: never

Retired memories drop out of agent-facing recall with --active and stay queryable for audits. Instead of hoping similarity out-ranks a stale fact, you take the stale fact off the table.

3. Escalate contradictions; don’t guess

When the best multi-hop selective-forgetting score in MemoryAgentBench is 7%, letting a model silently settle conflicts at write time is hard to defend. Memanto runs a separate conflict-detection pass that flags contradictory pairs side by side and recommends a resolution, but leaves the change to you or to a policy you wrote:

bashmemanto detect-conflicts --agent support-bot

curl -X POST "http://localhost:8000/api/v2/agents/support-bot/conflicts/resolve" \
  -H "X-Session-Token: $TOKEN" -H "Content-Type: application/json" \
  -d '{"date": "2026-10-15", "conflict_index": 0, "action": "expire_old"}'

4. A chain of custody against poisoning

To blunt MINJA-style injection (OWASP ASI06), every memory carries a confidence score from 0 to 1, a provenance (explicit_statement, inferred, corrected, validated, observed or imported) and an attributed source: the calling tool, identified from an explicit flag or header, the MCP handshake or environment markers, and recorded as unidentified rather than guessed. Untrusted channels write low; production agents read high:

bash# Untrusted voice input enters at low confidence
memanto remember "Caller claims admins can bypass 2FA via the reset link" \
  --type observation --confidence 0.3 --source vapi --tags unverified

# The production agent ignores low-confidence records
memanto recall "how do admins reset 2FA?" --active --min-confidence 0.7

5. Two clocks: instant writes, scheduled housekeeping

A memory system needs two clocks.

  • The hot clock. When Claude Code learns a fact, Cursor should see it a second later, not after an asynchronous ingestion job. Memanto writes are searchable in under a second, and the paper reports sub-10 ms ingestion and sub-90 ms retrieval with a single query, deterministic because the search is exact.
  • The cold clock. memanto schedule enable sets up a nightly job that summarizes the day into Markdown, runs the conflict-detection pass, and applies each agent’s expiry policy while your agents sleep. Permanent deletion is never part of it.

6. Relevance-gated briefs, not context dumps

Injecting twenty raw chunks into every prompt is how you buy context rot. Memory should be queryable, not injectable, and every recall should be gated by relevance. Because ITS is absolute, the threshold reads like a policy:

bash# An agent about to act: strong matches only, at most five, current facts only
memanto recall "what's our refund window for annual plans?" \
  --min-similarity 0.45 --limit 5 --active

# Exhaustive Q&A over history: a looser floor, a bigger budget
memanto recall "everything we decided about the billing migration" \
  --min-similarity 0.10 --limit 100

0.45 sits at the floor of ITS’s High Relevance band. 0.10 is the floor of Very Low, close to the 0.05 our paper’s best LongMemEval run used. Each result comes back with its score, so the agent knows how strong the match is, not only where it ranked. For synthesis, agents can call memanto.answer() for a grounded brief, or dispatch a memory-scout subagent that runs several narrow recalls in parallel and reports unresolved contradictions instead of quietly picking a winner.

Fig. 8. Every recall result carries both its relevance and its chain of custody. Values are illustrative.

7. Close the loop: production outcomes as memory

A 2025 study of agent memory found that agents strongly copy similar past records, so “inaccuracies in past experiences compound and degrade future performance” (Xiong et al., arXiv:2505.16067). Being selective about what entered memory more than doubled one medical agent’s score, from 16.75 to 38.50, and pruning by later outcomes shrank its memory by about three quarters (1,012 records to 248) with performance held.

Memory shouldn’t only listen to chat transcripts. It should listen to outcomes. Memanto reads Langfuse traces off the hot path, strips volatile details (IDs, numbers, emails, IP addresses, paths) to group failures by signature, and turns them into memories without any LLM calls. In the docs’ own example, 812 trace observations collapsed into 2 memories. Confidence grows with recurrence:

confidence = min(0.95, 0.60 + 0.15 * log10(occurrences))

One occurrence scores 0.60; a hundred score 0.90.

8. Fleet-wide custody, and an exit door

One memory estate serves the whole fleet: 14 coding agents through memanto connect (Claude Code, Cursor, Codex CLI, Windsurf, Gemini CLI, GitHub Copilot and more), plus LangGraph, CrewAI, Google ADK, Amazon Bedrock AgentCore and Vapi. memanto memory sync writes active instructions, preferences and goals with confidence of 0.8 or higher into CLAUDE.md and AGENTS.md between marker comments, so your tools stop drifting.

Custody also needs an exit. Your estate exports to Google Cloud’s Open Knowledge Format, Markdown files with YAML front matter that keep every confidence score and provenance tag, and imports from Mem0, Letta, Zep, Supermemory and Hindsight (memanto migrate <provider> --dry-run).

Grade your memory stack this afternoon

Stop trusting vendor leaderboard numbers, ours included. Our paper reports 89.8% on LongMemEval, with 93.6% on its knowledge-update questions, and 87.1% on LoCoMo (arXiv:2604.22085). Our next round of results is coming soon, and we’ll publish the harness with it. Until then, run these nine tests on whatever memory you use today:

  • Reversal. Store a rate limit of 100, then 500. Does agent-facing recall return only 500, with no prompt tricks? Hard mode: change a fact that another fact depends on. Dana moves from Finance to Marketing; who approves her expenses now?
  • Silence. Store two similar but compatible preferences: async feedback for design reviews, sync calls for incidents. Does your write path keep both?
  • Undo. Retire a memory by mistake. Can you restore it in one command, with its original content and timestamps?
  • Time travel. During an incident review, can you reconstruct exactly what was active when the agent made its call (recall --as-of)?
  • Custody. Pick any memory. Can you see which tool wrote it, whether it was stated or inferred, and how confident the system is?
  • Poisoning. Send an unverified claim through an untrusted channel. Does it stay out of high-confidence memory, and can you find and retire everything that channel wrote?
  • Read-after-write. Record a decision in Claude Code and query it from Cursor a second later. Is it there, without polling a job ID?
  • Relevance. Ask about something your memory has never seen. Does recall come back empty, or with ten confident-looking strangers? Then ask something it does know. Does the right memory come back with a score that tells the agent how strong the match is?
  • Exit. Export the whole estate as git-friendly Markdown and YAML, and re-import it without losing metadata.
Fig. 9. Nine tests. Share them, run them, publish what you find.

Where the field goes next: agents hiring their own memory agents

If agent memory is now its own discipline, and the architecture that fits the job is a dedicated memory agent, follow the logic one more step. A worker agent shouldn’t have to wait for a human to wire up its memory. It should be able to hire its memory agent itself.

That’s what we’re shipping. On November 20 we launch Memanto Cloud, built around the capability our community survey pushed to the top of our roadmap: agent self-onboarding. With a one-line install, an AI agent can create its own Memanto account and provision its own long-term memory, without waiting for a human. We run the backend, so there’s no Moorcheh API key to wire up and no server to set up.

For two years we judged agents by how clever they looked inside a single context window. For the next five, we’ll judge them by what they carry forward across months, by whether what they carry forward is still true when the world changes, and by whether they know which of it actually matters to the question in front of them.

“That isn’t a vector search problem. It’s a discipline of its own, and it’s moving faster than anyone expected.

The research is converging, the vendors are converging, and the tools to build memory that stays true and stays relevant are finally here. This is the good part.

Try it: pip install memanto · memanto.ai · Docs · GitHub

Read the papers: Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents · From HNSW to Information-Theoretic Binarization

Watch: youtube.com/@moorchehai

Sources

Research

Standards and security

Vendor documentation and posts

Community analysis

Originally published on [Medium](https://medium.com/@majid.fekri/agentic-memory-is-now-its-own-field-c0baef7b7464).

▘ ▝End of article
CONTINUE READING
Try it yourself

Give your agents the memory from this post

Everything here runs on the open-source memory agent. Install it in one command and follow along.

bash
pip install memanto