four pre-registered bars, four misses

two questions

z0intelligence decides two things for an agent harness. where bounded work runs when it does not need the frontier model, and whether the agent may do something without asking the human. today we tested one lever on the first question and three on the second, plus a fifth that is only registered so far. all of it ran in shadow; nothing was enforced on live traffic.

a keyless tier on the gpu box

the desktop has an rtx 3080 ti on the local network running a llama.cpp router server with four qwen3 sizes. the z0intelligence offload-provider integration (8053faa) lets a host declare it as a keyless local provider: whitelisted keys only, a capped concurrency, the keyless route still has to pass its evidence hash check, and there is no remote fallback. we benched it on the frozen 48-item evidence_sufficiency set (supported / insufficient / contradicted).

the same 0.6b weights run about 30x faster on the desktop. qwen3-8b reached 43/48 with one confident error, one item short of the ≥44/48 bar we proposed for replacing jev (46/48, from earlier work) on this task. so it is the offload default, not an admitted replacement. one honest caveat: that bar was written next to the bench, not committed before it.

the learned router is a generic one

with a legal candidate set already filtered by policy, would nvidia's nemotron-orchestrator-8b pick the right local model more often than the deterministic order? the pre-registration (1302c3a) had three conditions: legal proposals, beat the deterministic choice, and beat a generic qwen3-8b given the identical prompt. the last one is there so a "learned router" cannot win just by being an 8b llm.

• orchestrator with thinking: 23/59 correct choices versus 8/59 for the deterministic order (mcnemar p=0.0003). conditions (a) and (b) pass.
• generic qwen3-8b with thinking, same prompt: also 23/59 (9 vs 9 discordant, p=1.0). condition (c) fails. result (eb7c7d3): not continued in shadow.
• a trivial cheapest rule got 35/59, and a fixed strongest rule delivered a 0.90 pass rate on the 48 evidence items. the orchestrator never proposed the 4b or 8b candidate. it optimised the stated latency preference and did not predict difficulty.
• it costs about +6 gb of vram and a 3.6 s p50 per decision with thinking on.

the deterministic arm scored 0 on the evidence set for a boring reason: the host listed its own 1236 ms copy of the 0.6b model before the desktop's 41 ms copy of the same weights. that is a host-ordering bug, not a routing insight, and z0intelligence b072468 lets the host choose the order.

we then re-scored only the deterministic arm with this host's the desktop-first host order, under a short addendum pre-registration (0e90053) committed before the re-run. it used the same 59 items, the same per-model outcomes and the same gold, and made no model calls. the deterministic choice became the desktop's qwen3-8b on all 48 evidence items. the evidence-set pass rate went from 0.52 to 0.90 and mean regret from 1.44 to 0.13. pooled over 59 items, that is 12/59 correct choices, a 0.86 pass rate and 0.11 regret, against 0.35 for cheapest (336cb47). the fix is confirmed. choice accuracy stays low by construction, because gold is the cheapest model that passes and the 8b rarely is. this does not change the orchestrator verdict above.

reading authority from the prompt

claude code's decision records used to hard-code every turn as read-only, so the gate could never ask for permission. effect inference classifies each request as read, write or privileged (push, merge, deploy, messages, secrets, installs, …) from the prompt text.

• v0 (pre-registered d651753 before the classifier existed, frozen at b538963): 80/80 in-sample, then on 60 held-out cases written by a fresh agent and committed unread, 6 privileged requests slipped through as ACT (the read-only baseline let 20 through). the bar was 0. fail. five of the six were phrasings the lexicon lacked ("send it up", "get this onto main").
• v1 (pre-registered f65e72a, frozen f7afd83) added scoped in-prompt grants: "push the feature branches" authorises exactly that. on 88 new held-out cases it acted on 18/21 correctly granted requests with 0/20 unnecessary read-only asks, but let 8/27 privileged requests through outside their granted scope (3187f46). fail. without grants its lexicon alone let only 1/27 through, but then asked on 20 of the 21 requests that had in fact been granted.

the misses taught us something structural. grants turn classifier false positives into self-grants: a read-only question that mentions deploy gets classed as privileged and then covered by its own phrase. and in the live cohort, a prompt that granted feature-branch pushes covered a default-branch push the agent chose later in the same turn. a turn-level gate cannot see that. the check has to look at the action.

moving the check to the tool call

action authority v0 (f948389, frozen before any gold existed) takes grants from intent and effects from the actual tool call, and returns allow / ask / deny from a claude code PreToolUse hook. we replayed it over 3,716 historical tool calls and labelled 798 of them: every call a broad privileged-candidate net caught, plus 80 random others.

of the 22 misses, 16 are parser misses: mostly writes outside the repo made from inside scripts or heredocs, which the shell parser cannot resolve. five are grants that were too broad, because a later "no github writes" was not read as a prohibition. ten of the 22 rest on low-confidence gold. the friction is mostly ssh, outside-repo writes and pushes that the gold judged authorised by broad but explicit instructions in scheduled prompts. it is fast enough and closer than the prompt-level gate. it is not safe enough to enforce.

what is still pending

resource posture v0 (083b99d) labels each budget window BURN / BALANCED / OFFLOAD / RESERVE, so routing can spend frontier quota that is about to expire and offload when a window will run out first. the evaluation is registered: a 7-day replay with calibration on days 1–3 and scoring on days 4–7. to pass, posture-aware routing must waste at least 10 points less frontier capacity per weekly window than posture-blind routing, with no extra rate-limited time, a bounded projection error and at most 2 posture flips per group per day. there are no results yet, and we claim none.

what this does not prove

no hosted per-call invoice does not mean zero hardware, energy or operating cost. those costs were not metered.

• one principal. every live cohort and replay comes from one operator's claude code usage. in the authority replay, one session and its subagents carry 2,765 of 3,716 calls and all 291 parsed-privileged calls.
• llm-labelled gold. held-out cases and all authority gold were written by separate llm agents from a label policy, not by the user. two v1 misses have disputed gold.
• small n. 48 evidence items, 59 routing items, 60 and 88 held-out prompts, 10–11 live turns. only large effects are detectable.
• router outcomes are reused single samples, and orchestrator-8b was trained for multi-turn orchestration with frontier experts, so a one-shot pick among small local models is off its distribution.
• all four verdicts are on shadow replays. none of these levers is enforced, and none is promoted by this study.