Faster runs — a two-tier read cache, merged step gates, and agent health metrics
Measured across recent multi-step tasks, 64% of elapsed time sat between steps rather than in them — median 12 seconds per gap, and one nine-step task issued 79 step invocations. This release attacks that directly: repeated reads inside one run now hit a cache that survives across instances, and the four sequential checks that ran at every batch boundary became one hop with the two decisions running in parallel.
What shipped
- A two-tier read cache — an in-memory tier plus a durable one, so a repeated read inside a single run is served from cache even when the run spans separate serverless invocations. Previously the cache was one map per instance, and an identical query could run three times in one task with nothing able to tell you it had.
- Write-safe invalidation — a marker is written before a mutation and entries are stamped with the read's issue time, so a read already in flight when a write lands is never served afterwards.
- Four durable hops merged into one at each batch boundary, with the two decision-makers running in parallel rather than in sequence — latency becomes the longer of the two rather than their sum, for the same tokens and the same decisions.
- Agent health metrics and an /insights panel, so planner timing, cache hit rates and critic behaviour are visible numbers rather than log lines.
- Per-step timing split into preparation versus dispatch, plus the provider call's own duration on a cache miss — which is what makes the between-steps overhead measurable instead of inferred.
Why it was worth doing
Reliability work usually shows up as fewer failures. This one shows up as time: an agent that takes noticeably longer than it should feels unreliable even when every step succeeds, and "it's slow" is impossible to fix without knowing which part is slow. Splitting the timing was the prerequisite; the cache and the merged gate were what the measurement pointed at.
Frequently asked questions
- Does the read cache risk serving stale data?
- No. A marker is written before any mutation and cached entries carry the time their read was issued, so a read that was already in flight when a write landed is never served afterwards. Cached reads are also scoped to a single run.
- Did merging the step gates change any decisions?
- No — the same two deciders run with the same prompts and produce the same decisions. They now run in parallel inside one durable hop instead of sequentially across four, so the latency is the longer of the two rather than their sum.