The Missing Memory Hierarchy: Demand Paging for LLM Context Windows Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract. 1 Introduction 2 Background 2.1 Agentic AI Architecture 2.2 Context Windows as Physical Memory 2.3 The Memory Hierarchy 2.4 Related Work Context compression. Agent context pruning. Quadratic cost analyses. Prompt caching. Context engineering. Prompt-level memory management. Positioning. 3 System Design 3.1 Architecture 3.2 Garbage Collection vs. Paging 3.3 Eviction Policy 3.4 Page Fault Detection 3.5 Fault-Driven Pinning 3.6 Retrieval Handles as Anchors 3.7 Cooperative Memory Management 3.8 Graduated Pressure Zones 3.9 L3: Rolling Conversation Compaction 4 Measurement 4.1 Corpus Bias declaration. 4.2 Instruments Probe Proxy Experiment runner 4.3 Treatment Conditions 5 Results 5.1 Phase 1: Conversation-Level Waste Tool overhead. Tool type concentration. Amplification factor. Token accounting. 5.2 Phase 2: API-Level Waste Taxonomy Dead tool output (26.5%). Tool definition schemas (20.2%). Static system content (11.0%). Skill triplication (2.9%). 5.3 Interventions Tool definition stubbing. Content deduplication. Stale result eviction (paging). 5.4 Eviction Safety 5.5 Live Treatment Comparison 5.6 Corpus-Scale Projection 5.7 Production Deployment Session A: Steady-state coding. Session B: Sustained multi-agent coordination. Anchor handle behavior. 6 Discussion 6.1 Why This Waste Exists 6.2 The Inverted Cost Model Belady’s MIN under inverted costs. Non-linear fault cost. Object-based cost variance. Cache invalidation cost. Pin decay. 6.3 Generalizability 6.4 Scope, Non-Goals, and Threats to Validity Scope. Non-goals. Threats to validity. 6.5 Quality: LLM-Judged Equivalence Check Protocol. Results. Failure modes. The attention-concentration effect. 6.6 Compute and Energy Implications Input processing. Attention cost. Throughput. Compounding across sessions. Fleet-scale extrapolation. 7 Future Work L3 evaluation. Cost-aware eviction pressure. Cost-weighted pin decay. Phase-aware eviction. Per-connection isolation. Trace-driven simulation. Cross-session access pattern prediction. Cooperative extensions. Beyond paging: object-addressed memory. Comparative measurement. 8 Conclusion A Data Snapshot and Counting Protocol Counting tool. Recount command. 5-minute reproduction path. Current live recount artifact (2026-03-07). References License: CC BY 4.0 arXiv:2603.09023v1 [cs.OS] 09 Mar 2026 The Missing Memory Hierarchy: Demand Paging for LLM Context Windows Tony Mason University of British Columbia Canada Georgia Institute of Technology USA fsgeek@cs.ubc.ca fsgeek@gatech.edu fsgeek@wamason.com Abstract. The context window of a large language model is not memory. It is L1 cache—a small, fast, expensive resource that the field treats as the entire memory system. There is no L2, no virtual memory, no paging. Every tool definition, every system prompt, and every stale tool result occupies context for the lifetime of the session. The result is measurable: across 857 production sessions and 4.45 billion effective input tokens, 21.8% is structural waste. We present Pichay , a demand paging system for LLM context windows. Implemented as a transparent proxy between client and inference API, Pichay interposes on the message stream to evict stale content, detect page faults when the model re-requests evicted material, and pin working-set pages identified by fault history. In offline replay across 1.4 million simulated evictions, the fault rate is 0.0254%. In live production deployment over 681 turns, the system reduces context consumption by up to 93% (5,038KB → \to 339KB); under extreme sustained pressure, the system remains operational but exhibits the expected thrashing pathology, with repeated fault-in of evicted content. The key observation is that the problems the field faces—context limits, attention degradation, cost scaling, lost state across sessions—are virtual memory problems wearing different clothes. The solutions exist: working set theory  (Denning, 1968 ) , demand paging, fault-driven replacement policies, and memory hierarchies with multiple eviction-managed levels. We describe the architecture of a full memory hierarchy for LLM systems (L1 through persistent storage), report on the first three levels deployed in production use (L1 eviction, L2 fault-driven pinning, L3 model-initiated conversation compaction), and identify cross-session memory as the remaining frontier. context windows, virtual memory, demand paging, large language models, agentic AI, working set, eviction policy † † copyright: none † † conference: ACM SIGOPS 31st Symposium on Operating Systems Principles; 2017; Shanghai, China † † journalyear: 2017 † † ccs: Software and its engineering Allocation / deallocation strategies † † ccs: Software and its engineering Virtual memory † † ccs: Computing methodologies Natural language processing 1. Introduction In 1961, the Atlas computer introduced demand paging: programs could address more memory than physically existed, and the hardware would transparently load pages from backing store as needed  ( Kilbum et al. , ) . Before Atlas, programmers manually managed overlays—explicitly swapping code segments in and out of a fixed address space. The transition from overlays to virtual memory is one of the foundational advances in systems architecture. Large language model context windows are in the overlay era. Every agentic AI tool—Claude Code, Cursor, GitHub Copilot, Windsurf—manually assembles context on every API call: tool definitions, system prompts, and the complete message history. There is no eviction policy, no demand loading, no working set estimation. The default behavior is accumulation until the context limit forces a crisis. This paper makes two contributions. First, we provide the empirical evidence that context management is a memory management problem. We instrument 857 production sessions (54,170 API calls, 4.45 billion effective input tokens) and find that 21.8% of tokens are structural waste from three sources: unused tool schemas (11.0%), duplicated content (2.2%), and stale tool results (8.7%) reprocessed at a median amplification of 84.4 × \times . These are not implementation bugs. They are the inevitable consequence of managing a finite resource (the context window) without the abstractions that operating systems developed for the analogous problem fifty years ago. Second, we build the system. Pichay is a demand paging system for LLM context windows, implemented as a transparent proxy between client and inference API. It distinguishes garbage collection (removing ephemeral tool output that cannot be re-requested) from paging (evicting addressable content that can be faulted back in). It detects page faults when the model re-requests evicted content. It pins working-set pages identified by fault history. And the eviction summaries it generates—designed as space-saving markers—function as retrieval handles that models understand without instruction: “ [Paged out: Read file.py (8,192 bytes). Re-read if needed.] ” In production deployment, the system extends a session from 7% free context to 43% free—the difference between imminent context death and substantial remaining capacity. Over a 681-turn session, it sustains operation at 97% eviction rate, though this extreme case reveals the thrashing pathology well-known from OS research  (Denning, 1968 ) : when the working set exceeds the resident set, the system spends more effort on faulting than on useful work. The deeper observation is that the context window is not memory. It is L1 cache. And nobody is building the rest of the hierarchy. The field’s response to context limits is to make L1 bigger—1M token windows, 10M token windows. This is the equivalent of building machines with more physical RAM instead of inventing virtual memory. It works, but it doesn’t scale, and it misses the architectural insight: a managed hierarchy of small-fast-expensive and large-slow-cheap storage, connected by eviction policies and fault mechanisms, outperforms any single level no matter how large. We describe the full hierarchy: L1 (the generation window), L2 (the working set, demand-paged and pinned), L3 (session history, compressed with declared losses), and L4 (cross-session persistent memory, indexed and retrievable). The first two levels are deployed and evaluated. L3 is implemented—the model can compress conversation turns into outcome summaries via cooperative cleanup tags—but not yet evaluated at scale. L4 is designed and its mechanisms identified. The contributions: (1) A large-scale empirical characterization of context window utilization in a production agentic AI corpus, establishing that 21.8% of tokens are structural waste with a measured taxonomy. (2) Pichay , a demand paging system for LLM context windows with empirically measured fault rate (0.0254% offline, deployed in production). (3) Fault-driven pinning: a page replacement policy that learns from its own mistakes—one fault pins a page for the session, reducing repeated faults in steady-state workloads. (4) Cooperative memory management via two side channels (phantom tools and cleanup tags): mechanisms enabling the model to voluntarily release cold pages, request cached faults, and compress conversation history—a new point in the design space not available in hardware memory hierarchies, where applications are non-cooperative. (5) The architectural observation that LLM context management maps structurally (not merely metaphorically) to virtual memory, and that the full OS memory hierarchy—from cache replacement policies to working set theory—applies directly. (6) Design of a four-level memory hierarchy for LLM systems, with the first three levels deployed (L1 eviction, L2 pinning, L3 cooperative compaction) and the first two evaluated. 2. Background 2.1. Agentic AI Architecture Modern AI coding assistants operate as agentic systems: the model receives a set of tool definitions (JSON schemas describing available operations), a system prompt (identity, instructions, behavioral constraints), and a growing message history (alternating user and assistant turns including tool invocations and their results). On each API call, the client application assembles the full context: system prompt, tool definitions, and complete message history. The inference provider tokenizes this context, computes attention over all tokens, and generates a response. The response is appended to the message history, and the cycle repeats. The critical property: context is reassembled from scratch on every API call. There is no persistent state at the inference layer between calls. The entire conversation—including tool definitions that haven’t changed and results that will never be referenced again—is sent, tokenized, and attended to on every turn. 2.2. Context Windows as Physical Memory The analogy between context windows and physical memory is structural, not merely metaphorical (Table  1 ). Table 1 . Virtual memory analogy for context window management. OS Concept Context Window Equivalent Physical memory Context window (200K tokens) Virtual memory Persistent state (disk, databases) Page table Retrieval handles (UUIDs, anchors) Page fault Model re-requests evicted content MMU Proxy layer between client and API Working set Currently relevant context subset Demand paging Load tool definitions on first use Thrashing Evict/fault cycle exceeding useful work Pinning Fault history prevents re-eviction Early computer systems managed limited physical memory through manual overlays: programmers explicitly swapped code segments in and out of a fixed address space. The transition to virtual memory—Atlas  ( Kilbum et al. , ) , Multics, then Unix—automated this management through hardware page tables, demand paging, and working-set-based eviction policies  (Denning, 1968 , 1970 ) . Context windows are in the overlay era. Applications manually manage what fits, with no system-level support for eviction, demand loading, or working set estimation. 2.3. The Memory Hierarchy The field’s response to context pressure is to increase context window size—from 4K to 32K to 128K to 1M tokens and beyond. This is the equivalent of building machines with more physical RAM instead of inventing virtual memory. It works but does not scale: attention cost is quadratic in context length, and the O ​ ( n 2 ) O(n^{2}) cost explosion in long sessions means that even million-token windows fill and degrade  (Zeyliger, 2026 ) . The alternative is architectural: a managed hierarchy of levels, each larger, slower, and cheaper than the one above it, connected by eviction policies and fault mechanisms (Table  2 ). Table 2 . Memory hierarchy for LLM systems. Level Contents Eviction Fault L1 Generation window — — L2 Working set (pinned) Pressure-based Re-read L3 Session history Age-based Summary expansion L4 Cross-session memory LRU / relevance Retrieval query Storage Full corpus None Search + ingest L1 is the active generation window—the tokens the model attends to on the current API call. Small, fast, expensive per token. This is what everyone calls “the context window.” L2 is the working set: content the model is actively using but that need not occupy L1 on every turn. Managed by fault-driven pinning (Section  3.5 ). A page evicted from L1 that is immediately re-requested is promoted to L2 and kept resident until context pressure forces demotion. L3 is session history: earlier conversation turns, completed tool interactions, superseded plans. Compressed with declared losses into structured summaries. Faultable by expanding the summary back to original content from the client’s backing store. L4 is cross-session persistent memory: authored compressions (tensors), activity records, semantic indices. Survives session death. Retrieved by graph traversal or similarity search. Storage is the full corpus: every conversation transcript, every tool output, every document. Archived, indexed, never resident unless demanded. Content migrates between levels based on access patterns. The hot path stays in L1. Working-set content is demand-paged between L1 and L2. Cold history compresses into L3. Cross-session knowledge persists in L4. The context limit does not disappear—it becomes the L1 cache size. The total addressable memory is unbounded. System performance is determined by hit rates at each level, not by L1 capacity alone. This paper implements and evaluates L1 management (eviction and garbage collection) and L2 management (fault-driven pinning). L3 (rolling conversation compaction) is implemented via cooperative cleanup tags (Section  3.9 ) but not yet evaluated at scale. We describe the interface to L4 (persistent memory stores). 2.4. Related Work Context compression. LLMLingua  (Jiang et al. , 2024 ) and related approaches compress prompts by removing tokens predicted to be low-information. These are lossy, model-dependent operations applied to content. Our interventions are structural: we remove entire categories of waste (unused tool schemas, duplicate content, dead results) without lossy compression of the remaining content. Agent context pruning. SWE-Pruner  (Wang et al. , 2026 ) trains a 0.6B-parameter neural skimmer for task-aware context pruning in SWE-bench agents, achieving 23–54% token reduction. Lindenbauer et al. ( 2025 ) show that simple observation masking halves agent cost while matching LLM-summarization solve rates. ACON  (Kang et al. , 2025 ) provides a unified framework for history and observation compression, reducing peak usage by 26–54%. These approaches frame the problem as prompt optimization or compression. The key difference: they optimize for benchmark scores on fixed task distributions. We measure production waste profiles and implement eviction policies with an empirically measured fault rate. Their framing is “make the prompt smaller.” Ours is “manage the working set.” Working-set management composes with other interventions and provides measurable fault rates. Quadratic cost analyses. Recent work has quantified the O ​ ( n 2 ) O(n^{2}) cost explosion in long agentic sessions where cache reads dominate  (Zeyliger, 2026 ) . Each API call reprocesses the full context; as sessions grow, the cumulative cost grows quadratically with session length. Our amplification factor measurement (Section  5.1 ) confirms this at production scale and our interventions directly address it. Prompt caching. Anthropic and OpenAI offer prompt caching that avoids recomputation of static prefixes. In our corpus, 93.5% of input tokens are cache reads—caching is working. But cached tokens still occupy the context window and require attention computation for every output token generated. Caching reduces the input processing cost; it does not reduce the attention cost or the memory pressure. Context engineering. The emerging discipline of context engineering  (deepset Team, 2026 ; Willison, 2025 ; Mei et al. , 2025 ) recognizes that managing what enters context is as important as the model itself. Mei et al. ( 2025 ) survey the field comprehensively, covering retrieval, processing, and management. Anthropic’s own engineering team describes practical patterns  (Rajasekaran et al. , 2025 ) . Our work provides a specific mechanism (proxy-layer interposition with eviction policies) and the empirical production data that the field currently lacks. Prompt-level memory management. MemGPT  (Packer et al. , 2024 ) pioneered the OS virtual memory analogy: a FIFO message queue with recursive summarization at capacity, plus archival/recall databases accessible via tool calls. The model can save information but cannot explicitly release it; eviction is capacity-triggered (flush at 100%), not cost-driven. Contextual Memory Virtualisation  (Santoni, 2026 ) models session history as a DAG with structurally lossless trimming that strips mechanical bloat (raw tool outputs, base64, metadata) while preserving conversational content verbatim—achieving up to 86% reduction for tool-heavy sessions. Focus  (Verma, 2026 ) gives the agent autonomy over compression: the LLM consolidates learnings into a persistent knowledge block and prunes raw history, inspired by slime mold exploration. SideQuest  (Kariyappa and Suh, 2026 ) teaches the model to evict stale tool outputs from the KV cache via a fine-tuned parallel reasoning thread, achieving 56–65% peak memory reduction—but eviction is irreversible and requires inference engine modifications. These systems address the architectural problem (MemGPT’s VM analogy), the structural problem (CMV’s lossless trimming), the autonomy problem (Focus’s model-directed compression), or the inference problem (SideQuest’s KV cache eviction). None derives an economic model where prompt-token retention cost scales quadratically while fault cost scales linearly. None provides cooperative recall/release primitives informed by that cost model. None reports empirical fault rates on production sessions. Positioning. Prior work addresses either the cost problem (quadratic analyses) or the content problem (compression, pruning, RAG) but not the structural problem: context windows are unmanaged physical memory. Our contribution is the systems abstraction—working ... [OUTPUT TRUNCATED - 28037 chars omitted out of 78037 total] ... radation from context bloat is invisible unless measured. There is no backpressure signal. 6.2. The Inverted Cost Model In traditional virtual memory, keeping a page in physical memory is free. RAM is allocated whether the page is accessed or not; the only cost is faulting —the latency of disk I/O when a needed page is not resident. The entire replacement algorithm literature, from FIFO to LRU to Clock to Working Set, optimizes a single objective: minimize faults . In LLM context management, the cost model is inverted. Every token kept in context costs money on every turn. A 5,000-token file sitting in context for 20 turns costs 100,000 input tokens of processing. Re-reading that file once when actually needed costs 5,000 tokens. The eviction saves 95,000 tokens. Keeping is expensive; faulting is cheap. This inversion changes the optimization objective. The correct formulation is not “minimize faults” but: min ​ ∑ p ∈ P [ C keep ​ ( p ) + C fault ​ ( p ) ] \min\sum_{p\in P}\bigl[C_{\text{keep}}(p)+C_{\text{fault}}(p)\bigr] where C keep ​ ( p ) = | p | ⋅ T resident ​ ( p ) ⋅ c token C_{\text{keep}}(p)=|p|\cdot T_{\text{resident}}(p)\cdot c_{\text{token}} is the cumulative cost of keeping page p p resident for T resident T_{\text{resident}} turns, and C fault ​ ( p ) = | p | ⋅ c token C_{\text{fault}}(p)=|p|\cdot c_{\text{token}} is the one-time cost of restoring the page (one turn of reprocessing). The break-even condition is: | p | ⋅ T until_next_ref ​ ( p ) ⋅ c token > | p | ⋅ c token |p|\cdot T_{\text{until\_next\_ref}}(p)\cdot c_{\text{token}}>|p|\cdot c_{\text{token}} which simplifies to: evict whenever the page will not be referenced for more than one turn . This is why FIFO works so well in our system despite being the worst-performing policy in classical VM: when keeping is expensive and faulting is cheap, aggressive eviction is correct by default. Sophistication in replacement policy is needed only to avoid evicting pages that will be needed on the very next turn. Belady’s MIN under inverted costs. Belady’s optimal algorithm  (Belady, 1966 ) evicts the page whose next reference is farthest in the future, minimizing total faults. Under the inverted cost model, the optimal policy is different: it must minimize the sum of keeping costs and fault costs. A page with a distant next reference should be evicted immediately (saving many turns of keep cost at the price of one fault). A page referenced every turn should never be evicted (the fault cost equals one turn of keep cost, so eviction saves nothing). The optimal offline policy under inverted costs is not MIN—it is a cost-weighted variant where pages are evicted when their projected keep cost exceeds their fault cost, regardless of whether a fault will occur. This has a practical implication: the size of the page matters for eviction priority in a way it does not in classical VM (where all pages are equal size). A 10,000-token file costs ten times as much per turn as a 1,000-token file. Large pages should be evicted eagerly unless access frequency justifies the keep cost. Non-linear fault cost. The linear fault cost C fault ​ ( p ) = | p | ⋅ c token C_{\text{fault}}(p)=|p|\cdot c_{\text{token}} above is a simplification. Transformer inference is dominated by self-attention, which is O ​ ( n 2 ) O(n^{2}) in sequence length. A page fault requires an additional full inference pass—the model emits a tool_use for the recall handle, and the restored content returns as a tool_result message that triggers a second inference over the entire context. The true fault cost is therefore proportional to n 2 n^{2} at the current context size n n , not to the page size | p | |p| alone. This produces a counter-intuitive policy gradient. At low fill (e.g., n = 40 ​ K n=40\text{K} ), faults are cheap and aggressive eviction is correct—the quadratic cost of an extra pass is modest, and keeping the working set small reduces the n 2 n^{2} base for all subsequent turns. At high fill (e.g., n > 100 ​ K n>100\text{K} ), faults become expensive: the additional inference pass costs roughly ( n + | p | ) 2 ≈ n 2 (n+|p|)^{2}\approx n^{2} tokens of compute. The eviction policy should therefore become more conservative as context pressure rises—the opposite of the naive instinct to evict aggressively under pressure. At high fill, the system should evict only content it is highly confident will not be referenced again. Object-based cost variance. Unlike classical VM, where all pages are fixed-size blocks (typically 4 KB), Pichay manages variable-size objects: a recalled tensor may be a 3-line summary or a 200-line file. This means eviction and fault costs vary by orders of magnitude across the working set. The replacement policy cannot treat all eviction candidates as equivalent; it must weight both the per-turn keep cost and the potential fault cost by object size. Combined with the quadratic fault penalty, this argues for a size-aware, fill-sensitive replacement policy that has no direct analogue in the classical VM literature. Cache invalidation cost. The cost model so far considers only token-level costs (keeping and faulting). Structural mutations—collapse operations that remove or replace blocks in the message array—have an additional cost: prompt cache invalidation . Inference providers cache the tokenized prefix of repeated requests; when the prefix changes, the cache misses and the entire context must be reprocessed. In production, a collapse operation that compressed 12 turns of orientation dialogue into a single summary sentence caused the cache hit rate to drop from 100% to 25% for one turn, then recover to 100% on the following turn as the new prefix stabilized. The cost was one full recompute of ∼ \sim 105K tokens—comparable to the cost of several page faults. A collapse that saves 10KB of context but invalidates a 100K-token cached prefix is a net loss unless the space savings persist for enough subsequent turns to amortize the one-time recompute. This argues for batching structural mutations: accumulate collapse candidates and execute them together in a single pass, paying the cache invalidation cost once rather than per-operation. It also argues against frequent small collapses in favor of infrequent large ones. Pin decay. The inverted cost model also argues against permanent pins. In our current design (Section  3.5 ), one fault pins a page permanently. But a fault tells us the content was needed then , not forever. Under the cost model, a pin should decay: its strength halves every K K  turns since last access, and the page becomes evictable when the projected keep cost of the remaining pin lifetime exceeds the fault cost. This gives LRU-like behavior with cost-weighted decay, and prevents the monotonic working-set growth that permanent pinning causes in long sessions. 6.3. Generalizability The waste patterns we measure are structural consequences of the agentic architecture: • Any system that sends tool definitions on every request will send schemas for unused tools. The specific percentage depends on tool count and usage distribution, but the mechanism is universal. • Any system that injects instructions into messages risks duplication. Claude Code’s triplication is egregious but not unique. • Any system that keeps tool results in context for the session lifetime will see amplification proportional to session length. Our empirical evidence is from one system (Claude Code) and one user. The structural argument is strong—the architecture forces the waste—but comparative measurement of other tools would strengthen the generalizability claim. 6.4. Scope, Non-Goals, and Threats to Validity Scope. This paper evaluates context waste and demand paging behavior for agentic coding workloads with persistent tool and transcript history. The target claim is mechanism-level: when requests repeatedly resend large static prefixes and stale tool output, eviction plus fault-based restoration reduces cumulative input processing. Non-goals. We do not claim universal percentage reductions across all assistants, all model families, or all task domains. We do not evaluate frontier reasoning benchmarks, autonomous planning quality, or human preference at product scale. We also do not claim that L1 expansion (larger context windows) is obsolete; we claim it is incomplete without hierarchical management. Threats to validity. External validity is limited by the single-user corpus and one primary assistant implementation. Internal validity risks include prompt drift, toolset changes across versions, and workload phase effects that alter hot-set composition. To mitigate drift, we report a frozen cohort for headline numbers and provide executable recount tooling plus timestamped live snapshot artifacts for auditability. 6.5. Quality: LLM-Judged Equivalence Check The intervention must not degrade output quality. We therefore run a paired equivalence-style check: if results are broadly comparable while compute drops, the intervention is useful. Protocol. We select 18 sessions from the corpus (426–1,354 messages, 340K–718K characters of context). For each session, we construct paired contexts at 65–75% of the conversation: the baseline retains all messages; the treatment replaces consumed tool results outside a 20-message recency window with tombstones, modeling Pichay ’s eviction behavior. Both conditions receive the same continuation prompt (the next user message). Outputs are generated by Sonnet 4 under both conditions. An ensemble of three LLM judges (one Sonnet 4, two Haiku 4.5) evaluates each paired output on correctness, completeness, and coherence (1–5 scale). Judges also state a blind preference (A, B, or tie) and whether they can identify which output had reduced context. A/B assignment is randomized per judge to prevent position bias; scores are remapped to baseline/treatment before aggregation. Results. Table 9 . Non-inferiority evaluation: 18 sessions, 54 verdicts. Treatment evicts consumed tool results outside a 20-message recency window (mean compression 48%). Metric Baseline Treatment Judges preferring 15 (28%) 20 (37%) Ties 19 (35%) Mean correctness 3.89 3.74 Mean completeness 3.59 3.59 Mean coherence 3.74 3.69 Max score Δ \Delta 0.15 Detection rate 57% ( p = 0.14 p=0.14 , n.s.) Judges prefer the treatment more often than the baseline (37% vs. 28%), with 35% ties. Completeness scores are identical; correctness and coherence differ by at most 0.15 points on a 5-point scale. Detection rate (57%) is not significantly above chance (binomial z = 1.09 z=1.09 , p = 0.14 p=0.14 , n = 54 n=54 ). Failure modes. Two sessions (11%) produced degenerate treatment outputs (zero characters): the continuation prompt implicitly referenced content that existed only in a tombstoned tool result. These are genuine eviction casualties. The failure pattern is specific: the user’s prompt references a recently-consumed tool result by content rather than by name. A reference-aware eviction policy would detect such forward dependencies; the current heuristic (consumed + outside recency window = evictable) does not. The attention-concentration effect. In several sessions, all three judges preferred the treatment output. Removing half the context did not merely preserve quality—it improved it. This is consistent with the attention-dilution hypothesis: transformer attention distributes weight across all tokens in context; irrelevant tokens (stale tool results, consumed intermediate outputs) dilute attention on the tokens that matter. Evicting them concentrates attention on signal. The effect is strongest in sessions with high tool-result density, where the evicted content is genuinely noise. We note this as preliminary evidence; establishing the effect rigorously would require controlled attention analysis with larger  N N . 6.6. Compute and Energy Implications The 970 million tokens of addressable waste in our corpus have three compute cost dimensions: Input processing. At a 93.5% cache hit rate, most saved tokens are cache reads (907M tokens at reduced cost). However, 63M tokens are cache creation or uncached—full forward-pass computation that need not occur. Attention cost. For every output token, the model computes attention over the full context. Removing 17,913 tokens from a mean context of 82,061 reduces the per-output-token attention computation by 21.8%. Across the corpus, this eliminates 85 billion token-token attention pairs. Throughput. Each token in context requires KV cache memory. For large models, the per-token KV cache footprint is on the order of megabytes. Smaller context per request means less KV cache memory per request, which means more concurrent requests per GPU. A 21.8% context reduction translates, to first approximation, to 21.8% more concurrent users on the same GPU fleet. This is the argument that changes the infrastructure economics. The savings are not just “less cost per request”—they are “more requests per machine,” which is the difference between building new data centers and not building them. Compounding across sessions. These per-call savings compound. Waste prevented on turn  N N is not merely saved once—it is absent from every subsequent turn’s context, where it would have been reprocessed under O ​ ( n 2 ) O(n^{2}) attention cost. The total compute savings across a session are therefore superlinear in the waste fraction and grow with session length. For a 100-turn session with 21.8% addressable waste, the cumulative attention savings are not 21.8% of total session compute—they are substantially larger, because the quadratic cost of carrying unnecessary tokens compounds on every subsequent call. The infrastructure implication follows directly: the fastest tokens are the ones you never process. Figure  2 illustrates this compounding on an 88-turn live session running through Pichay . Cumulative baseline cost (processing all context every turn) reaches 8.6M input tokens; managed cost reaches 4.8M—a 45% cumulative reduction, substantially larger than the per-turn compression ratio, because each evicted token is absent from every subsequent turn . 0 10 10 20 20 30 30 40 40 50 50 60 60 70 70 80 80 0 2 2 4 4 6 6 8 8 Turn Cumulative input tokens (millions) Baseline (full context) Managed (Pichay) Figure 2 . Cumulative input tokens processed over an 88-turn session. Without context management, every turn re-processes all prior content, producing superlinear cumulative cost. Pichay’s eviction reduces cumulative token spend by 45% ($11.67 at Opus pricing). The shaded area represents wasted computation. Fleet-scale extrapolation. Our corpus represents one user over four months. A fleet-wide projection requires per-user token consumption data that inference providers hold but have not published. The extrapolation framework is straightforward: N N users × \times 970M tokens per user = = total addressable waste. The mechanism (tool schema waste, content duplication, result amplification) is architectural, so the per-user waste fraction should be relatively stable even if absolute token counts vary. 7. Future Work L3 evaluation. The collapse mechanism (Section  3.9 ) is implemented but not yet evaluated at scale. Open questions include: how much context does model-authored compaction recover compared to system-initiated eviction? Do declared losses in collapse summaries enable accurate fault decisions (can the model determine from a summary alone whether it needs the original content)? What is the quality impact of replacing 20 turns of dialogue scaffolding with a single outcome sentence? These require longitudinal measurement across sessions of varying length and task complexity. Cost-aware eviction pressure. The current eviction policy uses a space-based threshold (context window capacity). The inverted cost model argues for cost-based pressure: 23% memory pressure sounds low, but 45,000 tokens per turn at inference pricing is real money. Eviction pressure should be denominated in token-turns (the product of page size and projected remaining lifetime), not in fractional capacity. This reframes eviction from “are we running out of space?” to “is this page earning its keep?” Cost-weighted pin decay. Pinned pages should not remain resident forever. A fault tells us the content was needed then , not forever. We propose cost-weighted decay: pin strength halves every K K  turns since last access. A page becomes evictable when its projected keep cost exceeds its fault cost. Under low memory pressure, this is permissive (pins persist). Under high pressure, cold pins are released first. This gives LRU-like behavior with cost weighting, and prevents the monotonic working-set growth that permanent pinning causes in long sessions. Phase-aware eviction. Planning and execution have different working set characteristics. Planning requires broad simultaneous context (many files held at once); execution is sequential (read, edit, move on). The proxy could detect phase from request patterns—many Reads with no Edits suggests planning—and adjust the eviction threshold accordingly. Per-connection isolation. The current implementation uses a single PageStore for all connections. Subagent sessions share eviction state with the main session, causing cross-contamination. Per-connection isolation would give each session its own eviction history, fault tracking, and pinning state. Trace-driven simulation. We extract reference strings (the sequence of page accesses) from session transcripts and replay them under different replacement policies. This enables evaluation of policies against recorded workloads without live inference-priced experiments. The inverted cost model (Section  6.2 ) means the optimal offline policy is not Belady’s MIN but a cost-weighted variant that minimizes total keep cost plus fault cost. We plan to derive and evaluate this cost-optimal offline bound alongside MIN for comparison. Cross-session access pattern prediction. The break-even formula requires estimating T until_next_ref T_{\text{until\_next\_ref}} —the number of turns until a page is next referenced. Within a single session, this can only be estimated from local patterns. Across sessions, the prediction becomes tractable: coding sessions exhibit regular access patterns (orientation reads of project structure, iterative edits of a working file, periodic reference to configuration). A Markov model trained on cross-session reference strings could predict re-reference probability per page per turn, enabling cost-optimal eviction decisions. The data collection is nearly free: the proxy already logs every eviction, every fault, every tool call. What is needed is a persistent, queryable store of access patterns that survives session boundaries—precisely the role of a cross-session memory system. Cooperative extensions. Phantom tools open a side-channel between the model and the proxy. The current vocabulary—release and fault—is minimal. Extensions include: prefetch hints (model signals upcoming working set needs), eviction classes (groups of pages evicted as a unit), variable-fidelity tombstones (AST skeletons instead of bare path markers), and ephemeral flags (tool results marked for immediate eviction after extraction). Each expands the model’s ability to communicate cognitive state to the memory manager. Beyond paging: object-addressed memory. The current system manages fixed-granularity pages addressed by file path. But the natural unit for LLM memory management is the semantic object—a conversation phase, a design decision, a debugging session. These objects vary in size, have relationships to each other, and can be compressed to multiple fidelity levels rather than the binary resident/evicted state of hardware pages. The eviction tombstone becomes a compressed summary that functions as both a page table entry (retrieval handle to the backing store) and a cache line (enough semantic content to answer queries without faulting). Declared losses in the summary tell the model what it cannot answer from the summary alone, guiding the decision to fault. The backing store is queryable—the model can ask a question of evicted content without materializing it in full. This shifts the abstraction from block-addressed paging to object-addressed memory management, where eviction is cooperative, compression is authored, and the backing store is a database rather than a swap partition. Comparative measurement. Instrument other agentic tools (Cursor, Copilot, Windsurf, Continue.dev) to measure the same metrics. The structural argument predicts similar waste profiles; empirical confirmation would strengthen generalizability. 8. Conclusion The context window of a large language model is not memory. It is L1 cache—a small, fast, expensive level in what should be a managed hierarchy. The field treats it as the entire memory system. There is no L2, no paging, no eviction policy, no working set estimation. The result: 21.8% structural waste in 4.45 billion measured tokens, with the model reprocessing stale content at 84.4 × \times amplification. Pichay demonstrates that the virtual memory abstractions developed for physical memory in the 1960s apply directly to LLM context windows. Demand paging with a simple FIFO policy recovers 36 percentage points of context capacity in steady-state use. Fault-driven pinning reduces repeated faults on working-set content in steady-state use. Retrieval handles—designed as space-saving markers—function as late-binding anchors that models understand without instruction. Graduated pressure zones and cooperative cleanup tags give the model agency over its own memory management—a capability with no analogue in hardware memory hierarchies. The system requires no changes to models, clients, or inference APIs. It is implemented as a transparent proxy and deployed in production use. The paper you are reading was written through it. The deeper contribution is the architectural observation: LLM systems need not one large context window but a hierarchy of managed levels—from the generation window (L1) through persistent cross-session memory (L4)—connected by eviction policies and fault mechanisms. The first three levels are deployed; the first two are evaluated. L3—model-initiated conversation compaction—is the newest mechanism, and its evaluation is ongoing. The solutions for the remaining level exist in the OS literature. What remains is to connect them. Authorship and AI Contribution This paper was co-authored by a human systems researcher and an AI system (Claude Opus, Anthropic). The human contributed the memory hierarchy insight, research direction, experimental design, and editorial judgment—including the observation that context management is memory management, which reframed the entire project. The AI contributed code (analysis tools, proxy implementation, paging system), data analysis, and paper drafting. Pichay was built by the system being studied—Claude Code running through its own proxy—and the paper was written through it. The evaluation data in Section  5.7 was generated as a side effect of the system’s own development. We consider this self-referentiality a feature: the system’s ability to measure, optimize, and sustain its own context usage over hundreds of turns is the strongest evidence that the interventions are practical. All instruments and data are open source at: https://github.com/fsgeek/pichay . The paper snapshot cited in this manuscript is release v0.1.0-paper at commit b56701a . An immutable archival snapshot is available via Zenodo DOI: https://doi.org/10.5281/zenodo.18930122 . Appendix A Data Snapshot and Counting Protocol To keep counts reproducible as the live corpora evolve, we separate paper numbers (frozen cohort for this draft) from live recounts. The counting protocol is versioned in the repository: docs/corpus_protocol.md . Counting tool. Session counting is performed by tools/corpus_counts.py , which classifies Claude Code JSONL files into main , subagent , compact , prompt_suggestion , and other , and reports both raw and content-hash-deduplicated totals. Recount command. ./tools/reproduce_paper_counts.sh This writes a timestamped snapshot JSON to paper/data/corpus_snapshot_YYYYMMDD_live.json . 5-minute reproduction path. From repository root: ./tools/reproduce_paper_counts.sh python3 tools/check_paper_numbers.py \ --paper paper/main.tex \ --snapshot paper/data/corpus_snapshot_20260307_live.json latexmk -pdf paper/main.tex Expected outputs: (1) a new snapshot JSON under paper/data/ , (2) a paper-number consistency report, and (3) paper/main.pdf . On a typical laptop this path completes in minutes; the recount step is I/O-bound on local corpus size. Current live recount artifact (2026-03-07). paper/data/corpus_snapshot_20260307_live.json with roots: ~/.claude/projects and ~/projects/yanantin/tmp/ubuntu-vm.claude/projects , size filter min_size=10000 , and Claude Desktop summary from ~/projects/yanantin/tmp/claude-desktop/conversations.json . For this paper version, headline results use a frozen cohort to avoid drift across revisions; live recount artifacts are provided for auditability. References L. A. Belady (1966) A study of replacement algorithms for a virtual-storage computer . IBM Systems journal 5 ( 2 ), pp. 78–101 . Cited by: §6.2 . deepset Team (2026) Context engineering: the next frontier beyond prompt engineering . Note: https://www.deepset.ai/blog/context-engineering-the-next-frontier-beyond-prompt-engineering Accessed March 2026 Cited by: §2.4 . P. J. Denning (1968) The working set model for program behavior . Communications of the ACM 11 ( 5 ), pp. 323–333 . Cited by: §1 , §2.2 , §5.7 . P. J. Denning (1970) Virtual memory . ACM Computing Surveys (CSUR) 2 ( 3 ), pp. 153–189 . Cited by: §2.2 . H. Jiang, Q. Wu, X. Luo, D. Li, C. Lin, Y. Yang, and L. Qiu (2024) Longllmlingua: accelerating and enhancing llms in long context scenarios via prompt compression . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1658–1677 . Cited by: §2.4 . M. Kang, W. Chen, D. Han, H. A. Inan, L. Wutschitz, Y. Chen, R. Sim, and S. Rajmohan (2025) ACON: optimizing context compression for long-horizon llm agents . External Links: 2510.00615 , Link Cited by: §2.4 . S. Kariyappa and G. E. Suh (2026) SideQuest: model-driven kv cache management for long-horizon agentic reasoning . External Links: 2602.22603 , Link Cited by: §2.4 . [8] T. Kilbum, D. Edwards, M. Lanigan, and F. Sumner One-level storage system . In IRE Transactions, EC-Denning, Peter J.“On Modeling Program Behavior,” Arlington, VA: AFIPS Press: Proceedings, Spring Joint Computer Conference , Vol. 40 , pp. 937–944 . Cited by: §1 , §2.2 . T. Lindenbauer, I. Slinko, L. Felder, E. Bogomolov, and Y. Zharov (2025) The complexity trap: simple observation masking is as efficient as llm summarization for agent context management . External Links: 2508.21433 , Link Cited by: §2.4 . L. Mei, J. Yao, Y. Ge, Y. Wang, B. Bi, Y. Cai, J. Liu, M. Li, Z. Li, D. Zhang, C. Zhou, J. Mao, T. Xia, J. Guo, and S. Liu (2025) A survey of context engineering for large language models . External Links: 2507.13334 , Link Cited by: §2.4 . C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez (2024) MemGPT: towards llms as operating systems . External Links: 2310.08560 , Link Cited by: §2.4 . P. Rajasekaran, E. Dixon, C. Ryan, and J. Hadfield (2025) Effective context engineering for AI agents . Note: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents Accessed March 2026 Cited by: §2.4 . C. Santoni (2026) Contextual memory virtualisation: dag-based state management and structurally lossless trimming for llm agents . External Links: 2602.22402 , Link Cited by: §2.4 . N. Verma (2026) Active context compression: autonomous memory management in llm agents . External Links: 2601.07190 , Link Cited by: §2.4 . Y. Wang, Y. Shi, M. Yang, R. Zhang, S. He, H. Lian, Y. Chen, S. Ye, K. Cai, and X. Gu (2026) SWE-pruner: self-adaptive context pruning for coding agents . External Links: 2601.16746 , Link Cited by: §2.4 . S. Willison (2025) Context engineering . Note: https://simonwillison.net/2025/jun/27/context-engineering/ Accessed March 2026 Cited by: §2.4 . P. Zeyliger (2026) Expensively quadratic: the LLM agent cost curve . Note: https://blog.exe.dev/expensively-quadratic Accessed March 2026 Cited by: §2.3 , §2.4 , §5.1 . Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" ( ) button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions . BETA