every harness we integrate with z0intelligence gets the same question: does the integration make the frontier model do less work, measured in billed cost per verified task, without making it worse? tonight it was claude code's turn.
the question
we only counted what claude code itself reports as usage, and we only counted an answer if a programmatic verifier agreed.
how we measured
- claude code 2.1.285, headless, sonnet (plus one haiku arm), pinned source commits, paired arms on the same task instance, arms interleaved per rep.
- a z0 plugin writes per-message usage receipts into tokenomics. after one fix (the
Stophook fires before the final message reaches the transcript, so we sweep again atSessionEnd) the receipts equal claude code's billed usage on all four token fields. - current-work questions drift as repos change — two answer keys went stale during the night because of our own pushes — so the final held-out set runs against pinned fixture clones and a frozen transcript subset, with keys computed before any run.
the prefix is the bill
on short tasks, almost none of the cost is the task. about 10.3k tokens of claude code's core prompt are shared across sessions; everything after it — user hooks' output, skill listings, mcp — is rewritten on every cold session and re-read on every request. the cache is scoped to directory and startup git snapshot, so every fresh worktree starts cold: stock claude code cost about 3x more cold than warm on the same task ($0.126 vs $0.0425).
a lean launch profile (project settings only, explicit mcp, no skill listing) cut billed cost a median 63.8% on cold sessions and 27.2% on warm ones (15/15 pairs cheaper), with verified quality unchanged (20/20 both arms). z0 now ships it as z0int claude-code launch --profile lean, plus warm to place work where the prefix is still cached.
a state packet helps tools
the z0 state packet reconstructs "what is true now" from git, docs and local session history, with a source pointer on every claim, and injects it at session start. on a held-out question set frozen before any run, packet plus normal tools answered 14/14 versus 12/14 for raw tools, with half the tool calls and 30% fewer input tokens. on its own, without tools, it scored 8/14 — it is an aid, not a replacement.
packet-only also scored 0/4 on the injected-abstention items. the manifest reports that failure. a compact packet is not permission to guess when evidence is absent.
the wins multiply
| pinned held-out (22 trials/cell) | correct | total cost |
|---|---|---|
| stock + raw tools | 21/22 | $2.186 |
| stock + packet | 22/22 | $1.815 |
| lean + raw tools | 20/22 | $1.063 |
| lean + packet | 22/22 | $0.861 |
the packet factor was 0.83 / 0.81 and the lean factor 0.49 / 0.47 regardless of the other lever; the product predicts 0.404 of the baseline and we observed 0.394. the manifest reports lean + packet cost 61% less than stock claude code with raw tools, at equal or better correctness. on the drifting dev set the same composition read -74%; we report the held-out number.
what did and did not work
- observationpack on short tasks (after nvlabs/sol-pi): claude code has no seam to rewrite history later, so our port packs large bash output immediately. on short tasks it rarely triggered — sonnet already pipes big output through
tail, and claude code natively spills anything over ~30k characters — and billed cost did not move (paired medians -1.3% and +0.3%). - but not on long sessions. on a long recall task (twelve full-file outputs, eight over the pack threshold, then seven detail questions answered without re-reading sources) it cut total billed cost 30.3% ($4.159 to $2.900; median paired -26.7%, 9/12 cheaper, wilcoxon p=0.0068) with 12/12 verified in both arms. we extended the sample twice after looking (n=3, n=6), which we disclose; the p-value still clears a 0.05/3 bar. this task is deliberately output-heavy. it also composes with the lean profile: on the same task, lean + observationpack cost a further median 38.2% less than lean alone (6/6 pairs cheaper, wilcoxon p=0.031, quality 6/6 both). verdict: opt-in for long sessions, off for short ones.
- task-conditioned skill exposure: hiding irrelevant skill descriptions saves ~5.7k prefix tokens, but the lexical selector had 3/30 recall failures and was not promoted.
- routing to a cheaper model: haiku without the packet was worse and more expensive than sonnet (16/22, $1.230 vs 20/22, $1.063) because it explored more; with the packet it was 27% cheaper but lost two answers. no router. the packet helped haiku more than sonnet, which is the more interesting finding.
what this does not prove
- small samples: 5–11 questions per set, 2–4 reps, one host, one night.
- the savings are on short tasks and current-work questions; long coding sessions are dominated by different terms.
- costs are claude code's reported
total_cost_usdon a subscription, not an invoice. - the imported summary contains the lean and observationpack runs. the packet and model-routing figures above are source-reported in the manifest and readme, not re-derived from that summary by this harvest.
- the dev-set numbers carry overfit risk; the pinned held-out set is the headline.
frozen study sources
- [studies/claude-code-savings-v0/README.md](https://github.com/kvnloo/z0evals/blob/948ad207ea5d9b280934a8f959df2acb82579361/studies/claude-code-savings-v0/README.md) - [studies/claude-code-savings-v0/manifest.yaml](https://github.com/kvnloo/z0evals/blob/948ad207ea5d9b280934a8f959df2acb82579361/studies/claude-code-savings-v0/manifest.yaml) - [studies/claude-code-savings-v0/data/summary.json](https://github.com/kvnloo/z0evals/blob/948ad207ea5d9b280934a8f959df2acb82579361/studies/claude-code-savings-v0/data/summary.json)receipts
this post is rendered from posts/claude-code-savings-v0.md. the frozen study, its manifest and its pinned sources live under studies/claude-code-savings-v0 when it has been imported, or on its study pull request when it has not. the research log lists the study's status and links.