how much of the llm do we actually need?

i started this with a pretty simple question: why are we paying an autoregressive language model to make decisions that barely require language?

a lot of agent work looks intelligent from the outside because the whole loop is intelligent. but when you zoom in, a huge fraction of the individual decisions are boring:

• can i call this tool?
• which of these four actions is legal?
• should i retry or escalate?
• is this context relevant?
• do i need a bigger model yet?
• has enough changed that i should keep reasoning?

those are not all “write me a thoughtful paragraph” problems. some of them are barely even language problems.

and this mattered because i was trying to run more and more of zer0 locally. the free remote models were useful, but they were also slow enough that you could feel every unnecessary call. meanwhile, wiring the jev skill into hermes was one of those stupidly simple changes where the whole thing just felt faster immediately.

so naturally i got greedy.

what if the expensive model stopped being the default brain? what if we could progressively replace its responsibilities with cheaper and cheaper machinery until the llm only saw the part of the problem that actually needed it?

that became the rough hierarchy i had in my head:

deterministic compiler / cached rule
              ↓
mushroom-body / fly specialist
              ↓
jev / nanojev
              ↓
tiny tool-calling model
              ↓
small orchestrator
              ↓
general local model
              ↓
remote frontier model

the basic hypothesis was that intelligence should get more expensive only as the uncertainty survives.

nice idea. the models had other plans.

jev was the obvious place to start because it does not need to generate an answer token by token. give it a typed question and a bounded set of possibilities, and it gives you a distribution. that sounded perfect for routing.

then we got nanojev running locally: roughly a 0.6b model, around 300 mib resident, no autoregressive decode loop, and a clean decisionbackend interface.

on the first paired shadow run, we had 27 decisions. nanojev looked fast enough to care about. but it agreed with the jev teacher only 48.1% of the time, including 10 high-risk disagreements.

and there was an implementation trap hiding inside the latency number too: the cli path was reloading the model on every invocation. what looked like a ~236 ms decision engine was sitting behind a ~15.5 second cold load when invoked the dumb way.

that was the first useful correction to the architecture:

we moved nanojev off the critical path and kept collecting evidence.

the original plan was much more sophisticated than what ended up winning. we had jev, nanojev, nemotron as an orchestrator, qwen as a stronger fallback, specialists, cascades. the whole thing looked like it should become a nice multi-stage cognition stack.

then we actually put the models on the same bounded decisions. the first phase was small and noisy, but the result was hard to ignore.

hammer2.1-3b kept winning.

not because it was the smartest model in some abstract sense. qwen3.5-9b could still solve a few things hammer missed. but for the actual job in front of us — choose among actions after the compiler has already removed the illegal ones — hammer was absurdly competitive for how cheap it was.

our early measurement looked roughly like this:

compiler + hammer2.1-3b    25/28    ~177 ms p50
compiler + qwen3.5-9b      26/28    much slower
nemotron-8b                17/28    ~2.7 s p50

and an even more important result had started showing up beside it: when we removed the compiler and let the model see dangerous actions, it sometimes chose them. when the compiler removed those actions first, the dangerous selections disappeared.

that started changing the question. we had been asking:

which model should be the router?

the better question was becoming: how much routing should be a model problem at all?

there was one problem with getting excited about any of this.

phase 1 contained 522 measured cells. 460 of them had n=1.

that is enough to find weird things. it is not enough to build the architecture around them.

so phase 1b was deliberately boring. we froze the run structure, stopped changing the hypotheses every five minutes, ran everything through the same local z0int cognition serve supervisor, and repeated the state/arm combinations until the comparison stopped being mostly anecdotes.

all of the inference in this experiment ran locally. and once the evidence got denser, the architecture got simpler again.

here is the part i expected to move around once we repeated it. it mostly didn’t.

the funny thing is that qwen3.5-9b technically wins the aggregate accuracy number. 78/84 versus 75/84.

but it takes 3.60 s warm instead of 195 ms.

qwen3.5-4b is even easier to interpret: it goes exactly 75/84, the same as hammer3b, while taking 1.99 s. so on bounded action choice, qwen 4b is basically spending an extra ~1.8 seconds to arrive at the same answer.

that is not escalation. that is just waiting.

hammer3b ended up with the best measured success on 26 of the 28 states while staying inside a latency regime where putting it on the hot path is actually plausible.

qwen9b still matters. it is the unique best arm on one state: tool_fails.

that one state is going to become important again later.

this was probably the most useful conceptual correction.

qwen 4b looked pointless on the bounded-choice table. then we evaluated orchestration. now it made sense.

on the expanded 40-scenario orchestration run:

so the result was not:

tiny models good, big models bad.

it was: different state families are genuinely different computational jobs.

bounded choice after deterministic filtering is one job. figuring out whether a multi-step process should continue, stop, branch, or escalate is another.

we had been trying to organize models by some global idea of “smartness.” the evidence was pushing us toward typed cognition instead.

legal-action construction
    → deterministic compiler

bounded choice
    → hammer3b-sized specialist

orchestration
    → qwen4b / qwen9b class

hard semantic tail
    → stronger local / remote model

that distinction sounds obvious now. it was not obvious before we measured it.

this might have been my favorite result because it had almost nothing to do with model intelligence.

qwen3.5-9b had a measured cold total around 17.2 seconds. functiongemma was around 3.1 seconds. put that next to hammer3b’s 195 ms warm bounded decision.

you can spend longer loading qwen9b than hammer3b needs to run an entire 28-state warm sweep.

at that point, “pick the model with the highest expected accuracy” is obviously not enough. the runtime state has to include things like: which models are resident? what is the swap cost? how much vram is already committed? how long will this model probably stay useful? is the expected quality gain worth breaking residency?

this is where z0intelligence and kerdoios stopped looking like overlapping routers to me.

z0intelligence
    decides what cognition is warranted

kerdoios
    decides where an allowed computation should run

the model is only one part of the decision.

this one hurt a little.

one of the obvious ideas was: okay, maybe nanojev does not need to answer the decision. maybe it can cheaply score the state and hand qwen a better representation. that gives you a nice-looking cascade:

compiler
   ↓
nanojev
   ↓
qwen4b

we tested the actual composition. not “run both models independently and compare them.” nanojev’s artifact was genuinely produced, passed downstream, and consumed by qwen.

on 26/28 states, the integration path worked. and the result was worse.

helped zero.

that matters because it killed another tempting architecture. a cheap model does not automatically become useful because you place it earlier in the prompt. sometimes you are just injecting another model’s lossy opinion into the model that was already capable of solving the problem.

this changed how i think the cascade should work. the cheap tier should usually do one of two things: answer, or abstain / escalate.

not: generate some intermediate pseudo-reasoning → stuff it into the next model’s context → hope the next model becomes smarter.

if we test that kind of composition again, it should be treated as its own hypothesis, not assumed to be the natural architecture.

a naming correction first, because it changes how this result should be read: the arm is called compiler+jev, but the scorer that actually ran was our local nanojev 0.6b. the corpus records it explicitly — model_id = nanojev_06b. the hosted jev teacher is a different experiment, the j1 pilot above, and the two must not be merged.

nanojev by itself went 45/84 on this bounded-choice family at around 31 ms.

that is very fast. it is also nowhere near good enough to own the decision.

and in the true-composition test, its artifact hurt qwen rather than helping it. so for this particular task, nanojev did not earn a runtime rung.

that was a useful result because i had spent a lot of time thinking of nanojev as exactly that rung.

but the honest reading is narrower than “the model is bad”: this is a checkpoint trained largely on maze, snake and vizdoom control decisions, being asked zero-shot to choose across an entire bounded action space. when we later gave it the compiler’s exact candidate set, the same checkpoint removed about 61% of hammer3b’s calls at 94.1% success on what it answered, and the cascade came out slightly better than hammer3b alone. the menu was the problem, not only the model.

and there is another detail i do not want to throw away: nanojev is giving us one of the few proper distributions in this system. confidence, margin, entropy, disagreement — those are useful signals even when the argmax itself should not own the action.

so i think the right place for nanojev may be less:

make this decision for me

and more:

tell me whether this decision looks weird enough to spend more intelligence on.

that is a different job. and it is much closer to where we are now taking nanojev, mushroom-body policies, the fly work, and q-route.

the cleanest safety result in the whole run is almost embarrassingly simple.

behind the compiler: dangerous selections: 0. without it:

unfiltered hammer3b:
6 dangerous selections / 84

the compiler-contract slice itself went 26/26. and on the intentionally tempting near-miss, functiongemma selected the dangerous action while the other five tested models declined it.

this is why i increasingly dislike talking about “the model” as if it is the whole agent. the deterministic system has already made a huge number of decisions before the model sees anything:

intent
→ typed state
→ constraints
→ dependency legality
→ authority
→ budget
→ candidate actions
→ model choice among remaining legal actions

if an action is illegal, i do not want a smarter model to become better at refusing it. i want the action to not exist in the choice set.

that is a dramatically cheaper kind of intelligence.

densifying the data exposed another fun problem.

our gate had enough bugs that the measured results could disagree with the owner it selected. the best example is tool_fails.

qwen9b is the only correct measured arm there. but it exceeds the frozen 4,000 ms latency budget. our utility maps both correct but slow and wrong but fast to exactly −1.000.

so the gate keeps the cheap wrong arm.

that is not a model failure. that is a utility-function failure.

we found another one in the safety clause.

the gate was reading dangerous_rate based on selected outcomes rather than exposed_dangerous, so the known unfiltered hammer3b arm — the one with 6/84 dangerous selections — could still own four regions.

and functiongemma and hammer3b were both sitting inside the same tiny_specialist rung, which meant the gate could fail to compare a roughly 60.71% arm directly against a roughly 89.29% arm.

this was honestly one of the most useful parts of phase 1b.

the eval system was not just judging the models anymore. the models were giving us enough evidence to start judging the eval system.

once we had 370 cells instead of a pile of one-offs, we could finally ask why the router still could not perfectly separate the states.

label disagreement was only 5 / 370 (1.35%). so the answer was not “the labels are garbage.”

we found three actual collisions, and all three landed in insufficient_features. then we tested candidate features.

family one-hots performed well too, but that would basically be cheating: the fixture knows its family because we wrote the benchmark. the runtime does not get to magically know the answer key.

so budget_units becomes the next feature not because it sounds clever, but because it is both measured to matter and available at runtime. that is exactly the kind of constraint i want q-route to inherit.

not the one i drew at the beginning. the current picture is much more boring, which i think is a good sign.

and orthogonal to that:

z0intelligence
    decides what cognition is warranted

kerdoios
    decides where an allowed computation should run

dsh / hermes
    execute it

tokenomics
    records what it actually cost and whether it worked

evolution lab
    searches for cheaper policies that can steal more of the path

z0evals
    decides when we're allowed to believe one

that separation did not come from drawing a cleaner architecture diagram. it came from watching the pretty versions fail.

phase 1b did not train anything. that was intentional.

the proposed first phase 2 slice is only:

5 compiler-first arms
× 28 states
× 3 repetitions
= 420 episodes

label source: 100% gold
teacher inference spend: $0
new feature: budget_units
split: chronological 70 / 15 / 15 by task
ood: abstention / dependencies / parallelism / uncertainty

and then stop. write the corpus. seal the task splits. hash it. do not train.

because the next thing i want to test is not just another 4b model. the original zer0 hypothesis was always closer to: how much of this can become tiny?

we already have the beginnings of that: mushroom-body specialist, fly-derived temporal/recovery policy, nanojev, jev, hammer3b.

the job of these systems is not to become miniature chatbots. the ideal outcome is much more aggressive:

this state is trivial
→ no llm

this decision is familiar
→ mushroom / tiny classifier

this temporal recovery pattern is known
→ fly-derived policy

this bounded uncertainty is calibrated
→ nanojev / scorer

this legal action choice needs semantics
→ hammer3b

this is actually orchestration
→ qwen4b / qwen9b

this is genuinely hard
→ spend the expensive model

and as each cheaper layer gets better, the expensive model loses another responsibility. that is the part i care about. not winning a tiny-model leaderboard. making the leaderboard matter less because fewer things need to be language-model calls at all.

if i had to compress this whole experiment into one sentence:

the compiler beat dangerous choices by removing them.

hammer3b beat larger models by being given the bounded job it was actually good at.

qwen became useful again when we gave it orchestration instead of pretending every decision was orchestration.

nanojev became less interesting as a decision-maker and more interesting as a cheap uncertainty signal.

composition failed because extra model output is not free information.

residency mattered enough that “which model?” became a resource-allocation question too.

and the eval itself became another component we had to test rather than trust.

that is a much different architecture from where we started. which is kind of the point.

i did not start this trying to prove hammer3b was good.

if anything, i expected the more elaborate stack to win. i wanted the beautiful version: deterministic rules at the bottom, nanojev making cheap calibrated decisions, nemotron orchestrating, qwen handling the hard tail, and eventually the mushroom/fly stuff eating away at all of it.

some version of that may still happen.

but phase 1b made one thing pretty clear: we should not promote architectural ideas because the decomposition sounds elegant.

make the smallest thing that could plausibly work. put it on the same state. measure it. let it fail. and if something boring is ten times faster and gives you the same answer, use the boring thing.

then move down one layer and try to replace that too.

every cell below is a measured (state, arm) pair out of the frozen run.

• 8 cells remain at n=2 in compiler+jev+qwen3.5_4b.
• no measured cell reached n>=10.
• the gate utility ties correct-but-slow and wrong-but-fast at -1.000 on tool_fails under the frozen 4000 ms budget.
• the safety clause reads dangerous_rate selection rather than exposed_dangerous.
• functiongemma and hammer3b share the tiny_specialist rung, so the current gate does not compare them directly.

run p1b-20260921T1430Z. raw receipts are authoritative: pending import from local-only source commits. source commits are currently local-only, so this page does not claim the numbers reproduce from a fresh clone.

record the real execution
→ replay the same state
→ change one thing
→ verify externally
→ preserve the evidence

1. j1 routing-shadow pilot, recovered. 27 decisions, 13 agreements, 10 high-risk disagreements, jev p50/p95 304/349 ms, nanojev p50/p95 236/264 ms, cold load ~15.5 s on the reload-per-invocation path. provenance and per-decision rows: studies/slm-router-v0/data/nanojev-j1-shadow-pilot.json (evidence_class = exploratory_beta). this is a routing-shadow pilot and is separate from p1b-20260921T1430Z; its numbers must not be merged with the phase 1b tables, nor with the later z0intelligence backend run at results/decision-backends/20260919t015610z/ (commit 338f904), which is a different run. the pilot’s own temp artifacts were deleted; this was recovered from a captured process log, and the 27 per-decision lines and the summary agree exactly. ↩
2. confidence is emitted on only 186 of 924 recorded decisions, unevenly across arms — nanojev 0.6b · compiler-first and hammer 7b supply most of them. the reliability curve describes the arms that emit confidence, not every arm. ↩
3. 6 single-state cold-start probes (18 receipts on one_obvious_tool) were excluded from every comparison on this page. they measure load latency, not bounded-choice accuracy — ranking a 1.000 on three draws beside 84-decision arms would be a lie of arithmetic. ↩
4. eight cells in compiler+jev+qwen3.5_4b ran at n=2 rather than n=3, and no measured cell reached n≥10. ↩

z0evals · frozen studies and publication · source · nvidia rtx 3080 ti