a 25 ms commit window for t3 code: 22% fewer sqlite commits on real agent turns

the short version

• the problem: the t3 code server commits to sqlite once per streamed event, and each commit rewrites about 29 pages. it is tracked upstream in issue 24003. pr 24145 batched queued snapshots and was closed under the one-problem rule; its reviewer, @simasrazinskas, found live events rarely queue up and suggested grouping commits across threads instead.
• what we built: that suggestion, with a time window as the trigger. plain node, message, turn-item and plan updates wait up to 25 ms, then commit together. any other write commits the queue first, so order is unchanged.
• what it costs: a hard crash can lose up to one window of events that no client had seen yet. in a 4-turn pilot, a 150 ms window saved no more and added 150 ms of lag, so the window has to stay short.
• how sure we are: measured on 156 real codex turns (6,674 stored events) at 30 turns per arm one thread at a time and 24 per arm four threads at once. commit counts are tight; byte savings are clear only with four threads; lag changes are within noise. claude code turns are still to run (its weekly limit was spent), so this is one provider for now. the simulated a/b and the replay estimate below are kept as upper bounds and are no longer the headline.

one commit per event

experiment 03b preloaded a small shim that counts the server's write syscalls per file and parses wal frame headers. the counts cover the server process only. turns are the repo's devReplay acp agent, driven through the real adapter, orchestration, projections and checkpoints. p50 (nearest rank) over 10 tapped turns, on a 618.8 MiB doctor-compacted seed derived from the 654.6 MiB database, at 4 KiB pages:

• commits keep rewriting the same pages. the event table and its indexes account for 42.5% of frames. sqlite_sequence, the projection metadata row, the thread row and its project_updated index each sit on one page that is rewritten about 46 times per turn.
• a turn writes about 29× what it adds (median of the per-turn ratios): about 7.1 MiB written to grow the database by about 242 KiB.
• checkpoint tuning cannot remove the main cost. a smaller checkpoint threshold raises copy-back. a larger one lowers copy-back somewhat (modelled), but cannot touch the wal frames, which are paid on every commit. the lever is the number of commits.

the change

fork branch exp/poh/04-commit-batching, based on upstream main 5b9731ce. the feature is one commit behind T3CODE_EXPERIMENTAL_COMMIT_WINDOW_MS, off by default.

• one queue for all runs and threads. a single EventSink queue holds the plain node, message, turn-item and plan updates from every run and thread. it commits as one transaction at most N ms after its first entry.
• order is preserved. any other EventSink write commits the queue first, so commit order and sequence order stay arrival order. an event whose normalization compares against a pending entity commits that entity first.
• clients never see unstored events. publication follows the commit.
• graceful shutdown flushes the queue.

the failure modes are the part we asked upstream about:

• hard crash: up to one window of events that were never published is lost.
• batch write fails (disk full, or a lock timeout that outlasts 2 retries): the error is logged, the window's events are dropped, and that thread's next deferred write fails with the cause. without the flag, the single write fails at its call site.

measured on real turns

real codex agent turns (codex-cli 0.162.1, gpt-5.6-luna, low effort) ran through t3 code's real codex adapter on a main that includes 29806. the server was a fresh bench home on the public synthetic seed after t3 doctor --fix (993 events, 70.8 MiB). each turn did real work in a fresh scratch repo: read files, run a command, edit a file, explain, and one long answer with no tools. the same prompts ran in the same order in both arms, alternating off and 25 ms (abba). the turns ran 136 commands and made 37 edits, and none were excluded.

two shapes: one thread at a time (s1), and four threads started together (s4), like subagent fan-out. per-event numbers are the fair comparison because real turns vary in length.

• the saving is real but about half the simulated one. codex events mostly arrive more than 25 ms apart, so fewer of them share a window than in devReplay turns.
• four threads at once did not add coalescing for codex. commits per event with the window on are the same in s4 (0.542) as in s1 (0.547).
• no measurable cost. event lag, first event, first assistant text, frame gap, turn wall, cpu and rss all moved by less than their noise. real provider jitter is larger than 25 ms.
• an unexplained burst. the second four-thread batch in each server boot wrote about 200 extra commits that stored no event, in 11 of 12 runs, in both arms. the s4 rows above use first batches; the second batches are in the data. it does not favour either arm.

data, harness.

simulated turns (upper bound)

four arms. pilot slots 1–3 ran all four (a b c d / d c b a) at fc7cfbeb; slots 4–8 ran off, 25 ms and idx (a b c / c b a) at be02c345; the diff between the two touches only the pooling script, so the slots pool. each slot is one boot with 2 tapped turns.

• off: no flags. 8 slots, 16 turns.
• w25: 25 ms window. 8 slots, 16 turns.
• idx: no flags, on a seed with the correlation index dropped and the command index partial. 8 slots, 16 turns.
• w150: 150 ms window. pilot only: 2 slots, 4 turns.

how we read it:

• on simulated turns, 25 ms cuts commits by 43%, wal frames by 24% and sqlite bytes by 21%. reconnect replay wire bytes do not change.
• event lag p50 is the only latency metric with a tight interval. its median rises from 17 to 32.5 ms; the shift is +15.5 ms [−14, +20], below the 25 ms window that bounds the added wait by design. the other latency intervals (first live event, first assistant delta, turn wall, event lag p95) are wide. they rule out large regressions, not small ones.
• the window merges ws frames. each turn sends 3 fewer, and the frame gap grows by 27.5 ms. assistant text keeps the same cadence.
• 150 ms saves no more and adds 150 ms of event lag in the 4-turn pilot, so the window has to stay short. it also shows the latency metrics catch a real delay.
• the index trim had no measurable per-turn effect at 16 turns. upstream has since merged one part of it, the correlation-index removal (pr 23211); the a/b's seeds predate that.

replay estimate (proxy)

the simulated turns are one agent on one thread. to see what the rule does on real traffic, the estimator replays orchestration_events in sequence order, read-only, using the ingestion timestamps (occurred_at). it counts commits under the same window rule. the baseline is one commit per event.

one install, 2026-11-03 03:30–05:28 utc, 9,292 events across 21 threads, mostly the hermes acp agent with delegated subagents:

this is an estimate. it assumes one commit per event, treats ingestion time as commit time, and does not model the extra flushes forced when a pending entity is compared against. we transcribed these numbers from the posted discussion; the source database keeps changing, so we did not re-run them for this study (data).

re-run after 29806

29806 merged on november 3, so we re-ran the a/b on top of it. it was a 2×2 of window off vs 25 ms, on the parent of its merge commit (9def583f, "pre") and on the merge commit itself (550c5fa3, "post"). the arms were interleaved inside each job: 4 jobs × 3 slots, so 12 slots and 24 tapped turns per arm, all on github-hosted ubuntu-latest runners (4 vcpu). the seed is synthetic and public, about 200 MiB and 58.7k events from a committed generator. each base's t3 doctor --fix ran on its own copy first. on post that pruned the seed to 993 events and 70.5 MiB; on pre it compacted it to 188 MiB.

• the window's saving does not depend on pruning. post's saving minus pre's is +0.14 MiB per turn [−0.81, +0.61], zero within noise.
• pruning lowers per-commit cost only a little. with the window off it saves 3.3 KiB per commit [−8.5, −0.1], about 3%. its large effect is file size at rest, so the two changes stack.
• absolute bytes differ from the a/b above because the seed differs; compare the deltas. turn wall, first live event, cpu, rss and reconnect bytes did not move beyond noise (data, run).

the page size trade

the same tap measured bigger pages. they pack the event rows better, but each commit then writes bigger frames:

wire bytes were identical, and no latency, rss or cpu metric moved beyond noise at n = 10 boots per arm. the cost is ssd write volume, so page size only becomes worth it once turns commit far less often. an earlier a/b measured /proc write_bytes and reported 2.1× at 16 KiB. on that host's filesystem that counter had 16 MiB of round-to-round noise, so the syscall tap replaced it (noise under 0.7 MiB). flag: T3CODE_EXPERIMENTAL_PAGE_SIZE; design note.

background: where the bytes sit

we started from a different question: could the provider own the transcript, with t3 code keeping only a cache? byte accounting on the same 654.6 MiB database said that is the small lever.

• text is 3% of the file. messages, reasoning and tool input total 18.6 MiB.
• tool output is 42.2 MiB (6.8% of the 619.1 MiB post-doctor file). evicting it saves 37.9 MiB (6.1%), and no adapter can serve it back today. even full provider ownership (policy B) removes only a third. ids and metadata, indexes and page slack are 92% of what remains.
• superseded snapshot bodies were the biggest waste. we built a fold for them (experiment 02, flag T3CODE_EXPERIMENTAL_SNAPSHOT_FOLD): −20.1% after VACUUM, 18/18 projection tables equal after rebuild. its review found one race (resume check vs replay read) that pr 29806 also handles. both fall back to a snapshot for a cursor in removed history. the designs differ: our fold kept a per-thread watermark, while 29806 uses one global pruning boundary, with retention exceptions such as each stream's newest event.
• 29806 merged on november 3; 29811 was closed. 29806 prunes events the projections already cover, which covers most event-log bytes and is why we dropped the fold. 29811 would have returned freed pages at startup for small databases; its author closed it because t3 doctor already converts databases to incremental auto-vacuum and the server then returns freed pages every minute. on this database the event log is 408 of 654.6 MiB including indexes. re-measured on top of it, pruning trims per-commit cost by about 3% and leaves the window's saving unchanged.

the case for provider-owned history is freshness: turns run in another client would show up on the next open. it is not bytes.

what this does not show

• one install, one user, linux, under memory pressure. bytes and commit counts are deterministic. latency intervals are wide except event lag.
• one real provider so far. the real-turn a/b covers codex only; claude code was rate-limited and hermes acp failed on its configured model. the devReplay a/b and the replay estimate are proxies and overstate the saving (43% and 41% vs 22% measured).
• an unexplained commit burst on the second four-thread batch per server boot, in both arms; not yet traced.
• small n. 16 tapped turns per arm for off, 25 ms and idx; the 150 ms arm is a 4-turn pilot. page size: 10 tapped turns per arm.
• different bases. each experiment was cut from the upstream main of its day (listed in sources); none of these numbers describe a later main, except the re-run after 29806, which uses a synthetic seed.
• only the sqlite share. the tap counts the server's write syscalls, not device writes. issue 24003's macos data shows the per-commit cost explains only part of the disk writes there.
• the fold's in-pass latency was never measured. host memory pressure stopped every bench server before the fold window; the only bound is the longest fold transaction, 49.8 ms.

sources

fork branches on kvnloo/t3code, each pinned to the sha the results came from, with the upstream main it was cut from:

upstream: discussion 29954 (our proposal), issue 24003 and our comment there, pr 24145 (queued-snapshot batching, closed), pr 29806 and pr 29811 (pruning and page reclaim, open), pr 23211 (unused correlation index, merged), discussion 6897 (retention and compaction, closed).

study files: manifest, readme, data, scripts.