总览 / 概念 / Contextual Agentic Memory is a Memo, Not
↗ 源 Markdown
概念
safety alignment evaluation benchmark reasoning
创建2026-05-05
更新2026-05-05
阅读量级12 分钟
来源2604.27707
概念导读

The paper argues that all current agentic memory systems implement lookup, not memory. This is a category error with provable consequences:

  1. Core Problem Definition
  2. Theoretical Analysis
  3. The Experience Compression Spectrum
  4. Formal Framework: Change θ vs Change C
  5. Memory Taxonomy
  6. Theorem 1: Compositional Sample Complexity Separation
  7. Capacity Bound for Multi-Fact Integration (Appendix A)
  8. Four Structural Limitations

Contextual Agentic Memory is a Memo, Not True Memory

arXiv: 2604.27707v1 [cs.AI] | Date: 2026-04-30 Authors: Binyan Xu (CUHK), Xilin Dai (Zhejiang U), Kehuan Zhang (CUHK)

1. Core Problem Definition

The paper argues that all current agentic memory systems implement lookup, not memory. This is a category error with provable consequences:

  • Retrieval generalizes by similarity to stored cases (exemplar-based cognition)
  • Weight-based memory generalizes by applying abstract rules to novel inputs (rule-based cognition)

Conflating the two produces agents that: 1. Accumulate notes indefinitely without developing expertise 2. Face a provable generalization ceiling on compositionally novel tasks 3. Are structurally vulnerable to persistent memory poisoning

The paper draws on Complementary Learning Systems (CLS) theory from neuroscience: the brain pairs fast hippocampal exemplar storage with slow neocortical weight consolidation during sleep. Current AI agents implement only the hippocampal half.

2. Theoretical Analysis

2.1 The Experience Compression Spectrum

Memory, skills, and rules lie on a single spectrum differing only in compression ratio:

Raw traces (low compression, high fidelity)
  → Natural-language skills (medium compression, actionable)
    → Parameterized rules (high compression, generalizable)

Key insight: Episodic traces belong in external store (fast, temporary); skills can live in context or weights (bridge); rules must be in weights (slow, generalizable). Current systems implement the entire spectrum as context-engineering (C-engineering).

2.2 Formal Framework: Change θ vs Change C

Every technique that changes an LLM agent's output falls into one of two categories:

  1. Change θ: Modify model weights via pre-training, fine-tuning, RL, or gradient-based updates → changes P(X|θ)
  2. Change C: Inject content into context window via prompting, RAG, MCP, skill files, scratchpads → conditions on P(X|θ, C)

Critical asymmetry: θ-compression is generative (model can recombine weight-encoded rules for unseen inputs); C-compression is retrieval-based (model can only use what is explicitly in context).

2.3 Memory Taxonomy

Type Substrate Persists Updated by Generalizes
Working Context window Session only Token generation Limited by L
Episodic External store Cross-session Read/write ops Exemplar-based
Semantic Model weights Permanent Pre-training Rule-based
Experiential Model weights Permanent Fine-tuning/CL Rule-based

The "Experiential" row is systemically absent from all deployed systems.

2.4 Theorem 1: Compositional Sample Complexity Separation

Setup: - Let F = {f₁, ..., fₖ} be base concepts, ⊕: F × F → Y be a composition operator - Agent observes n labeled compositional examples D = {(fᵢ, fⱼ, ⊕(fᵢ, fⱼ))} - M_R = retrieval-based memory (frozen model + top-K retrieval) - M_P = parametric memory (fine-tuned weights θ*)

Assumption 1 (Bounded in-context composition): The frozen model, given K demonstrations of ⊕ from D, achieves accuracy ᾱ < 1 on held-out pairs.

Definition: Compositional Generalization Capacity CGC(M, D) = accuracy over all concept pairs drawn uniformly from (F choose 2).

Theorem: For any target generalization level 1 - δ with δ < 1 - ᾱ:

  1. Retrieval lower bound: M_R requires n_R ≥ (1-δ-ᾱ)/(1-ᾱ) × (k choose 2) = Ω(k²) stored examples
  2. Parametric upper bound: If ⊕ belongs to hypothesis class H with VC dimension d, n_P = O((d + log(1/δ))/δ) examples suffice
  3. Separation: n_R/n_P = Ω(k²/d). For d = O(k), ratio = Ω(k); for d = O(1), ratio = Ω(k²)

Corollary: sup CGC(M_R, D) < sup CGC(M_P, D) whenever n_P ≤ n < n_R.

Proof sketch: For a stored pair (fᵢ, fⱼ) ∈ S, retrieval answers correctly. For a novel pair, the frozen model processes K retrieved examples but achieves accuracy at most ᾱ. Solving CGC ≥ 1-δ yields the Ω(k²) bound on n_R. Full proof and constructive modular-arithmetic example in Appendix B.

Remark 2 (Context-window independence): The separation is independent of context window size K. Increasing K may raise ᾱ marginally, but the Ω(k²) coverage requirement remains as long as ᾱ < 1. The bound also applies when the retrieval policy is learned (MemRL-style systems).

2.5 Capacity Bound for Multi-Fact Integration (Appendix A)

For any task requiring integration of m > K mutually dependent facts: - sup Acc(M_R, T_m) has a strict upper bound when m exceeds the retrieval budget K - Weight-based memory has no such hard bound

3. Four Structural Limitations

3.1 Definitional: Exemplar-Based Lookup Cannot Extrapolate

Retrieval generalizes by similarity to stored cases. Rule-based cognition generalizes by applying abstract principles extracted from, but no longer dependent on, stored cases. Current agentic memory implements exemplar-based cognition in perpetuity.

3.2 Structural: The Generalization Gap (Theorem 1)

Formal proof above. The gap grows on tasks requiring transfer to held-out question types.

3.3 Dynamic: C-Engineering Cannot Develop Expertise

Each session begins from the same frozen weights. A Reflexion agent that accumulates thousands of verbal self-critiques is still running the same frozen model. "Verbal reinforcement learning" is not learning — the model's weights remain unchanged.

3.4 Security: Persistent Memory Poisoning

Agentic memory structurally converts transient prompt injection into persistent compromise. Injected content propagates across all future sessions via the external store.

4. Evidence

4.1 Mechanistic Support

  • ROME (Meng et al., 2022): Factual associations stored in specific MLP weight layers
  • MEMIT (Meng et al., 2023): Edited thousands of facts via weight modification
  • ParamMem (Yao et al., 2026): Encoding agent reflections into weights outperforms storing them externally
  • Geva et al. (2020): FFN layers act as key-value memories
  • Yao et al. (2024): Knowledge localized to fact memory units in FFN neurons; SFT updates only attention routers, not knowledge circuits
  • Ye et al. (2025): 90% of SFT parameter updates contribute nothing to knowledge enhancement

4.2 Empirical Support

  • Ovadia et al. (2024): RAG excels at rare-entity recall but cannot improve compositional reasoning beyond base model capacity; fine-tuning improves reasoning systematically
  • Yang et al. (2026): Fine-tuning achieves highest accuracy on multi-hop queries requiring novel compositional combinations
  • Yao et al. (2026): Parametric storage outperforms external storage, with the gap growing on transfer to held-out question types
  • SCAN (Lake & Baroni, 2017) and COGS (Kim & Linzen, 2020): Weight-based models outperform retrieval-only on systematic compositional splits

5. Co-existence Architecture Proposal

The paper proposes a dual-path architecture:

  1. Hippocampal path (fast): External store for episodic traces, recent context, reversible operations
  2. Neocortical path (slow): Weight consolidation for abstract rules, skills, and principles extracted from experience

Consolidation mechanisms: - Targeted fine-tuning of knowledge circuits (FFN layers) - Experience replay with gradient-based updates - Periodic distillation from episodic store to weights

6. Call to Action

  • System builders: Implement consolidation paths alongside retrieval
  • Benchmark designers: Evaluate on compositionally novel tasks, not just seen distributions
  • Continual learning community: Re-engage with the agentic setting

7. Critical Analysis

Strengths: - First formal proof (Theorem 1) establishing a quantitative sample-complexity separation between retrieval and parametric memory - Grounded in neuroscience (CLS theory) and cognitive science (Chi et al., 1981) - Addresses four alternative views (Alternative 1-5) systematically - Mechanistic interpretability evidence strengthens the theoretical claims

Weaknesses: - Theorem 1 assumes ᾱ < 1; for broadly general operators, ᾱ → 1 and the separation vanishes - No empirical experiments in the paper — purely theoretical analysis - The co-existence architecture is proposed but not implemented or evaluated - Consolidation mechanisms (targeted fine-tuning) are mentioned but not specified in detail - Does not address the practical cost of weight consolidation vs retrieval

Open Questions: - Can hybrid architectures achieve the best of both worlds in practice? - What is the optimal consolidation schedule (analogous to sleep in CLS)? - How to identify which knowledge should be consolidated vs kept in external store? - Can the generalization gap be closed with better retrieval engineering (e.g., learned retrieval policies)?