a format check is not a faithfulness check

two questions

z0 should do two things for the agents it sits beside. it should move work off the frontier model when a cheaper or faster model gives the same answer. and it should learn, from outcomes that someone or something independently checked, when its own gate is right. tonight asked whether either is ready. the scored offload, faithfulness, verification-density and verified-loop comparisons were pre-registered in their source repositories. utilization is a source-reported baseline, not a pre-registered comparison. cited source revisions are pinned, and only aggregates are published. the only runtime change was a shadow-only demotion.

speed-first offload

under the burn posture, z0 sends bounded work back to the frontier even when a local model on the desktop (an rtx 3080 ti running qwen3-8b) would answer as well and sooner. the speed-offload pre-registration (d716a38) defined five worker task classes, built frozen sets from z0 artefacts, and fixed a rule per class: the paired 95% lower bound of local minus the better frontier arm above −0.10, local pass rate at least 0.80, local warm p95 below the faster frontier arm's p95, errors at most 5%.

the local model is faster on every class, but only one class held quality, and the speed gain there is about 2×, not the 30× the logprob decision mode suggests. the frontier latency is duration_api_ms, which leaves out the cli start and so favours the frontier.

the one pass was weaker than it looked. the check for summarize_tool_output was format and facts: at most 40 words, the files-changed count written as digits, and at least one changed file named, on git-stat items only. sonnet's 37/60 was mostly that check's strictness (it writes "two files changed"). the write-up said so at the time, and the class was put in front of a faithfulness test before anything acted on it.

the faithfulness check

decision 20 (pre-registration 2870516) built 160 new items from real invocations on z0 repos and ci: pytest on mutated code, git stat and diff ranges, build and lint output, and gh run and check pages, with 403 deterministic critical facts. a blinded sonnet judge marked unsupported or contradicted claims. the bar was non-inferiority to haiku, paired, one-sided 95%: hallucinated items no more than 2 points worse, critical-fact recall no more than 5 points worse.

• hallucination: fail. qwen3-8b minus haiku is +12.6 points, with an upper bound of +18.9 against a +2 margin. the bounds exclude equality too, not just non-inferiority.
• recall: fail. −7.7 points, lower bound −10.5 against a −5 margin.
• where. pytest summaries were nearly fine on recall (0.99) but swapped expected and actual values. on git items the 8b invented line counts and outcomes (recall 0.67, 14/40 hallucinated); on gh pages it miscounted runs and dropped failed jobs (0.73, 16/40). none of this was visible to the v0 format check.
• qwen3-14b, run for information only, is closer but fails both margins as well.

so the class was demoted to not_equivalent (2c0c339). no task class is speed-qualified, and no capability entry was added.

how many turns are verified

the learning side starts with verified outcomes (z0intelligence#54, 17a88ba): an append-only record, written after the fact, that attaches independent signals to past claude code turns: test and ci exit codes, whether commits survived or were fixed within days, pr outcomes, and the next user turn's correction cues. a turn is verified_success or verified_failure only on medium- or high-confidence signals; tests the turn wrote itself do not count.

verification density v0 (pre-registration 4431225, results bd370e6) tried to raise the share of live turns with such a label from a baseline to at least 0.20, adding verifiers that ship at low confidence until a labelled sample of at least 50 firings shows 0.90 precision.

the labelling found a defect in v0 itself. pytest … | tail reports the exit code of tail, so a failing suite piped into a filter read as a pass. 15 of checked_later's 20 wrong labels were such runs. with the fix, the 7-day count of in-turn test passes fell from 114 to 87: about 24% of v0's test-success signals were not passes. v0 verified_success labels from piped runs are suspect until re-verified. the fix was made after the labels were seen, so the higher share it gives (0.161) is exploratory and does not count toward t1.

most live turns are unverified for a plain reason: they are orchestration (12 of 31), ops (9) or questions (7), where the work happens in subagents or leaves no test, commit or ci trace attached to the user turn.

the loop has no data yet

the verified loop (evolution-lab, pre-registration ded915a) would compare the deterministic gate with a learned one on verified live rows, exported by z0int (5d0a85c). the rule allows three answers: SHADOW_CANDIDATE, NO_IMPROVEMENT, or INSUFFICIENT_DATA unless there are at least 300 rows, 30 not-success rows, 10 session groups and five folds with both classes.

the result (1b80a4e) is insufficient data. of 23 exported rows, 16 had no verified label and 1 was not an act, leaving 6 rows, 1 not-success, and one session. grouped cross-validation was impossible. the gate's numbers on those 6 are descriptive only.

the binding constraint everywhere is not-success rows: at a pooled not-success rate of 0.043, 30 of them take months. live feature coverage since emission began is already 100%, so the plumbing is not the bottleneck. the eval and probe harness sessions (325 of 331 swept) would reach the gate in about a week, but they are a different distribution from the user's work. this projection is a linear extrapolation of one day with one observed live not-success; read it as an order of magnitude.

how often z0 changes anything

the north-star metric (utilization v0, 5bc290f) counts frontier-bound units of the user's real interactive and agent work in which z0 displaced the work, shortened the context, or made an enforced decision that a verifier passed. an operator run of the pinned instrument gave 26.3% (51 of 194), and 27.9% (56 of 201) an hour later. all of it was observationpack shortening. there were 0 offloads (posture was burn, under which that is the designed behaviour) and 0 enforced decisions, because every gate decision is still shadow. these counts are source-reported, not a pinned artifact.

what we take from it

• format checks overstate local equivalence. a check that a summary has the right shape and names a file passed 60/60. a check that its claims are true failed by 13 points. the next offload class needs a faithfulness or semantic bar from the start. typed decisions, where the answer can be checked exactly, look like the better place to look: the same 8b got 43/48 on evidence_sufficiency in logprob mode at about 90 ms on an earlier the desktop bench, against 31/48 generatively, but that bench is outside this study and was not tested against the frontier here.
• the loop is data-bound by coverage and verification density. the learner, the exporter and the pre-registered rule exist. what is missing is verified live not-success rows: most live turns are orchestration and ops that no verifier reaches, the new verifiers have not earned their confidence, and the old test signal over-reported passes. more coverage (other harnesses that are built but not installed) and more verified turns per session move the date; tuning the learner does not.

what this does not prove

• one principal, about one day. every set, transcript and session is one operator's.
• small sets. 48–60 items per offload class, synthetic wrappers rather than sampled worker traffic, on a gpu shared with another tenant.
• one judge. faithfulness was judged by sonnet, which is also an arm; deterministic recall and a contradicted-only check point the same way.
• thin labels. density precision rests on fewer than 50 firings per verifier and 31 live turns; the pipe fix came after the labels were seen.
• a rough projection. weeks-to-gate rest on one observed live not-success and a pooled base rate.
• nothing was promoted. no routing, capability entry, gate or verifier confidence changed, apart from the shadow-only demotion.