We need to forget — not Jev, not summaries

The past was a lie; memory has no return; every spring gone by could never be recovered.

— Gabriel García Márquez, One Hundred Years of Solitude

Intro

Jev is hot right now. fast-jev-compaction uses it for context compaction: instead of asking an LLM for a summary, it asks Jev to judge every tool call and its result — keep or drop. The repo got an official shout-out and nearly 5k stars in days. I liked the idea and wrote pi-jev to try this in pi.

It felt great at first: compaction finished almost the instant I hit /compact, no more waiting. But then I worried about what it might be throwing away. I looked through the repo — no evaluation of compaction quality anywhere; unit tests use a fake Jev; the only quantitative output is size reduction. It only cares about compressing hard, not compressing right. So one weekend afternoon I ran my own eval on ~2,000 local pi sessions.

TL;DR

1. Jev compaction ≈ blindly dropping every tool result — and the latter is free. At the default threshold (0.5), Jev kept 0 of 9,471 tool results. Lowering the threshold preserved some, but no better than random — often worse. Feeding it result content and recent user prompts changed nothing.
2. Jev isn’t wrong: tool results barely need keeping. Of ~8M tokens of tool output, only ~3.6–7.5% is ever used again. You can drop it all; the model just re-runs a tool call when it needs something.
3. LLM summaries waste both time and tokens. Compaction doesn’t need an LLM: drop tool outputs, keep the call records — zero cost, instant. Downstream behavior is nearly identical to an LLM-written summary.

1. Fast, loveable — and it drops every tool result?

Jev is a model API from TypeSafe. Not a chat model — a decision interface: send a state plus questions, get answers. Three question types: noul returns a 0–1 probability for a claim, choice picks one option, score rates on a scale.

Compaction uses noul + score, three questions per tool call:

const { answers } = await jev.decide({
  model: "typesafe/jev-1.13",
  state,      // the history being compacted
  questions: toolCalls.flatMap(call => [
    { type: "noul",  claim: "this call is worth keeping" },       // → 0~1
    { type: "noul",  claim: "its full result is worth keeping" }, // → 0~1
    { type: "score", claim: "how stale is this result" },         // → graded
  ]),
});

// keepThreshold defaults to 0.5
toolCalls.forEach((call, i) => {
  const { keepCall, keepResult } = answers[i];
  if (keepResult >= 0.5)    keep(call, call.result);            // keep as-is
  else if (keepCall >= 0.5) keep(call, head(call.result, 300)); // keep first 300 chars
  else                      drop(call);                         // drop entirely
});

In practice Jev compaction is second-fast; pi’s built-in LLM summary takes minutes. Over an order of magnitude faster — love it.

Evaluating it has an inherent problem: “which results should stay” has no ground truth. A summary can be read by a human; nobody can adjudicate 9,471 results.

My answer: let the future judge. pi session files are append-only — compaction never deletes the raw history — so I replayed it: cut compaction points exactly where pi’s own trigger logic would, using pi’s prepareCompaction so the segment handed to Jev matches production bit for bit. With a 128k window: 91 compaction points across 43 projects, 9,471 tool calls, ~8M tokens of results.

“Was a result used?” is judged by string matching, no judge model: pick distinctive strings from each result (multi-segment paths, camelCase identifiers, hex ids) and check whether the agent later produced them itself before seeing any new information. If it did, it’s working from memory.

Round one, immediate surprise: at the default 0.5 threshold, Jev keeps nothing. Of 9,471 results, only 0.3% scored ≥0.3 on “worth keeping as-is” — none reached 0.5. This isn’t an artifact of my labels; it’s Jev’s raw probability output. My only 4 real Jev compactions in history look the same: one of them dropped 101 of 103 calls, 306K chars → 26K, zero results kept.

So the thing I loved for being fast wasn’t selecting at all — it’s drop-everything with a 300-char head start. Is dropping everything actually costly?

The accounting says no. Summary size plus tokens later re-fetched: Jev 19.1k, random-drop at the same size 19.0k, keep-everything 105k, theoretical optimum (keep only what’s used) 18.4k. Dropping everything is nearly optimal. But 208 calls labeled “used later” were all dropped. Whose fault? Easiest target: the 0.5 threshold.

2. Is 0.5 too high? Lower it — after auditing the ruler

Before touching the threshold: audit the labels. “208 useful” comes from rules written on the spot; check the rules before convicting Jev.

Sample 50 calls: 25 labeled useful, 25 useless. Shuffled, blind, read each in its original context, verdict first, answer after. Four verdicts: Y = used unique content from memory; D = used, but a copy exists elsewhere; R = re-read later, but would have re-read anyway even without compaction; N = unused.

Both rules fail. “Re-run” is a bad signal: 0 of 10 truly needed memory — read returns lines with hash anchors, edit requires fresh anchors, agents re-read files that are still in context anyway. Re-reading isn’t loss. “Recall” points the right way but attributes too widely: 12 of 15 did use the info, but only 5 used content unique to that result; the rest had copies elsewhere. Good news in the other direction: of the 25 “useless”, zero were unique-content uses that got missed.

Rewrote the rules: dropped the re-run rule, required recalled strings to be unique. “Useful” falls from 208 to 61 — 0.6% of calls, 3.6% of result tokens. Full picture by strictness:

Now sweep keepThreshold down and count how many of the 61 truly-useful calls survive:

At 0.14 — barely compressing — they tie. But that’s not compression: 43% of points compress less than 15% and fall back to pi’s default summary. Once it actually compresses, Jev preserves less than half of what random does. “Only-Jev-dropped” events: 12 vs 2 the other way, sign test p≈0.01 (events cluster across 33 points, so discount the significance, but the direction is clear).

So it’s not the threshold: lower it, and Jev still doesn’t keep the right things. Is its judgment just random? One defense remains untried: it never saw the evidence.

3. Is the context misconfigured? Enrich it and retry

I read the actual requests sent to Jev. The punchline: it never sees a single tool result’s content.

The state contains a note that dropped content can always be re-read, the last 3 user messages, and the history — with each call’s arguments truncated to 1000/200/60 chars and results reduced to one line: “success, N chars (omitted)”. The most recent ~20k tokens that pi keeps aren’t sent either. To judge “keep this result?”, it has what was called, how big it was, and where the conversation stands. Flying blind.

Enrichment experiment: attach up to 300 chars of each result’s head, include pi’s last 6 kept messages, set the “goal” to the latest user question. The extra content is budgeted by compressing the base history, so total request size matches production; ~5,000 tokens total for result heads. To rule out the boring bug, I verified 31/31 requests actually contained the new content.

Result: nothing changed. Ranking AUC 0.507 → 0.510, paired delta +0.003, CI ±0.07. The probability distribution is nearly identical, still no result above 0.5; at matched summary size it still doesn’t beat random. Information shortage isn’t the main cause — precisely, improvements above 0.07 AUC are excluded; smaller ones are undetectable with 29 points. The only untried lever is phrasing: the prompt line “dropped content can always be re-read” may itself push Jev toward dropping everything.

Feeding it information didn’t help. What remains is embarrassing.

4. Jev ≈ random? So my money was wasted?

Lay out the evidence: ranking AUC 0.507 (base), 0.510 (enriched), 0.501 (random) — and 0.5 is a coin flip. At real compression it preserves half of what random does. The extra requests, the two seconds, the fractions of a cent per compaction bought a coin flip. Yes, mostly wasted.

But not entirely — the experiments brought back something more valuable: after dropping, nothing collapsed.

L2 behavioral replay: find the moment the agent first used a dropped item from memory, swap the context for the compacted version, have another model take one step from there, and compare full history / Jev-compacted / random-keep. Model: deepseek-v4.1-flash. (Side note: my first pick was v3.2; I idly asked why not flash, checked, and flash is cheaper with a longer context.)

Across 61 events, I manually read every Jev-condition output: 41% refetched the info another way — re-reading files, re-searching, re-executing; 51% didn’t need it at that step and did something reasonable; 7% stalled with no output, concentrated in 2 contexts; 2% were artifacts (an 83-token result that never triggered truncation). Zero cases of confidently fabricating from vague memory.

“Names appearing out of thin air” — the hardest fabrication evidence — averages 0.03 per replay. Typical behavior: the original agent said “I just wrote this function, I’ll answer without tools”; the compacted model doesn’t trust that and reads the code first. The cost is a few extra tool calls, not errors. Even the full-history control re-fetches 25% of the time — agents just love re-reading, as section 2 already showed.

Jev and random-keep behave identically in 51 of 61 events; for the 12 “only-Jev-dropped” events, a control run with the content present behaves nearly the same as without.

So yes, the money was wasted — but selection turns out to be a problem that doesn’t exist. Drop things, and the model fetches them itself. Do we still need Jev? And if not, how does plain dropping compare to LLM summaries?

5. Drop vs. summary: summaries waste time and tokens

Since selection needs no intelligence, write a zero-cost rule, “mask”: keep all user messages, assistant text, call names and arguments; results keep only ok/error, size, and the first 300 chars; short results and non-refetchable tools (subagents, web fetches, ask_user) are kept whole. No model calls, no latency.

This isn’t new. Anthropic’s context editing clear_tool_uses does exactly this — clear old results, keep call records; JetBrains published a paper showing simple observation masking halves cost at SWE-bench resolve rates comparable to LLM summaries. My data adds one more vote.

But one gap remained: I had never compared against pi’s built-in LLM summary. On the 32 compaction points with confirmed memory use, four schemes head-to-head, metric = how many of the 60 truly-used-later calls survive:

mask keeps 12 more than the others; paired interval +17.4pp, lower bound 8.1. pi’s LLM summary compresses hardest (3.2k tokens) but preserves no more concrete content than random dropping. In its defense: string matching under-credits prose summaries, which can paraphrase. But behavior doesn’t lie: same 61 events, one-step replay under each scheme — refetch 15–18%, other tool calls 75–78%, invented names near zero — indistinguishable. The only pure-text answers (2 events, LLM-summary condition) read as grounded design opinions, not guesses.

The only differences left are cost and character: mask is free and instant; Jev adds a few 2-second requests per compaction and selects worse than random; pi’s LLM summary pays a full LLM call over the entire history each time (~$0.015, minutes of latency, one failure in 33 runs).

A free rule and a costly, slow LLM summary are indistinguishable under behavioral replay. When we summarize, we are wasting time and tokens at once.

Epilogue: one afternoon’s ledger

5pm to 10pm, five experiments, ~$3.8 total, under the $5 credit. The opening question was “does Jev select well?”; the closing answer is “there’s nothing worth selecting” — with LLM summaries dragged down too.

Looking back, the recurring character wasn’t Jev but the ruler. The 208 “useful” calls were an echo of the labels; the 46% refetch rate was an artifact of cd prefixes; “the model got stuck” was max_tokens truncation; the 0.81 “prefer big results” baseline was label bias toward big files; even “keep-everything still loses 9%” turned out to be a real truncation bug in the product. Treat every surprise as a measurement error first; only what still stands after the check deserves to be a conclusion. That sentence is probably the most valuable thing the $3.8 bought.

Boundaries: one person’s workflow, one model version (typesafe/jev-1.13), single-step replays. “No difference detected” ≠ “no difference” — only the obvious tier is excluded. But the magnitude is hard to flip. Tool output looks like memory that needs careful curation; it’s actually an external drive you can re-read anytime. Márquez wrote that every spring gone by can never be recovered. In an agent’s context, as long as the call records remain, spring can simply be re-executed.

The eval code lives in pi-jev’s eval/ directory, runs on local sessions, and ships outside the npm package; every number is reproducible for free from cached answers.