▌EVOKE

Long-running LLM agent sessions outgrow the physical KV cache budget within a few turns. EVOKE evicts low-relevance blocks under budget pressure and recovers them recompute-free via a custom save/restore primitive in a forked llama.cpp: 20–32× faster than re-prefilling the same tokens.

Demos

Qwen 2.5 7B, pure attention. A 14-turn session with a 1024-token budget. A fact is planted at turn 1 (favorite number = 4242), 12 unrelated knowledge questions fill the session, and at turn 14 the fact is probed. The session survives 68 evictions and 10 recoveries, and the model recalls “4242”.

Qwen 3.5 9B, hybrid Mamba/Attention + mrope, thinking-mode. Same demo at a 2048-token budget (thinking traces run longer). The model emits a <think>...</think> trace each turn; with EVOKE_SUPPRESS_THINKING_STRIP=1 the server keeps the trace in the returned content so cached state stays aligned with what the client echoes back, and no session resets fire. 20 evictions, 4 recoveries, fact recalled.

What it actually is

• Two new C++ primitives in a forked llama.cpp: llama_kv_block_save and llama_kv_block_load. They serialise a position range’s K/V tensors to a host buffer and splice them back with per-cell RoPE re-anchoring, with no llama_decode call.
• A third C primitive llama_attn_capture_* that taps per-head softmax attention weights from one or more chosen layers (up to 16) into a host buffer once per decode. Used by the relevance scorer to learn what the model is actually attending to.
• A per-layer residual capture (llama_set/get_embeddings_layer_inp) feeding a workspace probe: a ridge probe distilled from the model’s Jacobian-lens workspace readout scores blocks by content at prefill time, one dot product per position. Probes and the offline validation pipeline live in the companion j-space repo.
• A Python policy layer (evoke/manager.py, evoke/scorer.py, evoke/attention_scorer.py) that drives eviction under a watermark policy via a multi-signal scorer (model attention + harness priority + task-focus coherence + recency) and routes recovery through three pluggable backends: discard, breadcrumb, or kv_restore (the recompute-free splice).
• An OpenAI-compatible chat-completions server that exposes EVOKE as a stateful endpoint. The persistent KV cache survives across requests; only the new tail of each prompt is decoded.
• Cross-architecture coverage: pure attention with standard RoPE (Qwen 2.5, Llama 3), hybrid Mamba/Attention (Qwen 3.5), MoE attention with mrope and thinking mode (Qwen 3.6 35B-A3B).

How does the system know what’s relevant?

Five signals, combined into a per-block score in [0, 1]. Lowest scores get evicted first when the cache exceeds budget.

• The workspace probe (new, and the headline result). Attention-history selectors can only rank blocks the model has already attended to. The workspace probe is forward-looking: it predicts, from content alone at prefill time, whether a block holds the kind of workspace representation later reasoning reads from. On pre-registered offline gates it ranks the task-critical block at fact-AUC 0.891 vs SnapKV’s 0.622 (Qwen 2.5 7B), replicated 0.865 vs 0.563 on Qwen3-8B and 0.939 vs 0.541 on Qwen3-4B with zero hand-tuning; under masked eviction at a 25% budget it keeps the fact answerable in 35/36 episodes vs 25/36 and 14/36; on the live agent bench it passes every budget with recovery disabled while H2O fails all three, at -0.6% measured decode overhead.

• The model’s own attention. A second softmax for one or more chosen transformer layers runs alongside the main attention path, writing per-head softmax weights to a host buffer once per decode. The scorer maintains a sliding window of recent attention mass per block (last 64 decode steps, EWMA decay 0.95). Blocks the model is actually attending to score high. This is the strongest single signal — the truest answer to “what’s relevant right now.”

• Harness-supplied priority tags. A coding harness like opencode or Claude Code can set evoke_priority (a float multiplier) and evoke_pinned (boolean, excluded from eviction entirely) on each chat request. Useful when the harness knows things the model can’t see: a file read is the central artifact of the current task; a tool scratch output is one-shot. Defaults to 1.0 / false.

• Task-focus coherence. The scorer tracks a single task-focus embedding that updates via EMA on new user messages but snaps to the new message when a topic shift is detected (cosine drop below 0.3) or signaled by the harness via evoke_task_boundary=true. Blocks from a prior task lose their coherence score in one pass instead of decaying over five turns.

• Recency, sink protection, source-type floors. Stability priors: prevent thrashing on a single attention spike; protect StreamingLLM-style sink tokens; give USER and ASSISTANT turns a floor so the conversation backbone isn’t evicted before document content.

Final score: min(priority * (w_jlens·workspace + w_attn·attn + w_rec·recency + w_coh·coherence) / Σw, 1.0) lifted by a source-type floor (USER blocks 0.6, ASSISTANT blocks 0.5 by default) and with pinned-block protection.

Latency

Measured on Qwen 2.5 7B, RTX 4070 Ti SUPER, Flash Attention enabled. kv_block_load is the EVOKE recovery path; re-prefill is the cost of re-encoding the same tokens via llama_decode.

Block (tokens)	save (ms)	load (ms)	re-prefill (ms)	speedup
20	1.10	0.48	11.90	25×
40	1.61	0.70	13.78	20×
160	4.69	1.50	32.60	22×
640	16.37	4.34	118.36	27×
1280	31.90	7.25	232.18	32×

The gap widens linearly with block size: re-prefill is O(tokens × model_FLOPs), load is O(tokens × bytes).

Verified end-to-end

The mechanism has been run, measured, and stress-tested against a real coding agent.

• Live opencode session against Qwen 3.5 9B (hybrid Mamba/Attention + thinking, budget = 2048). 250 cumulative evictions, 4 smart-recoveries, active_tokens held near 1414 (within budget) while cached_tokens grew to 32 902. The agent’s conversation was 23× larger than what was actually held in GPU at any moment.
• Three model families verified at the primitive level (Qwen 2.5 7B, Qwen 3.5 9B, Qwen 3.6 35B-A3B). Full server-side evaluation is on Qwen 2.5 7B; cross-architecture latency and scorer ablations are explicit limitations in the paper.

Status

Research prototype, targeting both a working system and a paper draft. The mechanism is verified end-to-end across three model families. Latest milestone: the workspace eviction signal (v2 line), validated on three pre-registered gates across two models and folded into the paper with figures. Previously closed: tools-aware Jinja chat template (so tool-using turns no longer trigger session resets), multi-session pool with state swap on a custom X-EVOKE-Session header, iSWA dual-cache support in the fork primitives, multi-layer attention capture (up to 16 layers per decode).

License: Apache-2.0 on the policy layer; MIT-licensed on the forked llama.cpp work.