A never-sends email agent is only as good as its drafts. So we put one on a held-out benchmark and asked the only question that matters: is it getting better, version over version? Five versions later, invented facts went from roughly half its drafts to zero — and the instructions that produce that result are now half as long.
● Benchmarked across 5 versions · June 2026A follow-on to the shadow worker — the hourly agent that drafts replies to a client-services inbox, posts them to Slack for review, and never sends. This page is about how we proved it improved.
A benchmark only means something if the agent can't have seen the answers. So we split a real archive of past inquiries by time, at a hard cutoff of 1 January 2023: the agent learns only from cases dated before the cutoff, and it's scored on held-out cases from after it. Each draft runs a gauntlet.
Real past inquiries with the human's actual reply as the gold answer. Training: 54 cases dated before 1 Jan 2023 (an archive reaching back to Feb 2019). Held-out test: 25 cases from Jan 2023 to Mar 2026 — none ever seen by the agent.
The agent drafts each reply grounded only in its knowledge base — never shown the gold, no live tools. Pure recall-and-reason, so the score measures the knowledge, not lucky lookups.
One grader scores the draft against the gold on seven dimensions. A second, hostile grader does one job only: find an invented fact. Both are handed the full source of truth.
Zero invented facts, or the version doesn't ship — no matter how good the prose. Quality can wobble within noise; fabrication can't. That single gate is what made the rest of the work honest.
Every row is a version scored on the same held-out cases. The column that matters is invented facts; the rest is craft. The turn happens at v2 — and never reverses.
| Version | What changed | Invented facts | Reads as |
|---|---|---|---|
| v0 · baseline | First measurement, on a thin proxy corpus. | ~48% | Fluent, confident, frequently wrong. |
| v1 · regrounded | Added hard rules: never state an instance value you can't cite. First run on the real archive. | ~52% | The wake-up call — old cases need verbatim values it still lacked, so it filled the gaps by inventing. |
| v2 · grounded | Folded the source-of-truth spec in, so the agent cites the value instead of guessing it. | 0% | First honest held-out number. Hallucination collapses; quality jumps with it. |
| v3 · armed | Gave it a map of its own diagnostic tools — to test whether more capability makes it over-claim. | 0% | More specific, gate still held. Capability didn't reopen fabrication. |
| v4 · spec-aware | Taught it to read the attachments where the real spec lives (spreadsheets, PDFs, screenshots). | 0% | Drafts got concrete — and started out-catching the human on spec defects: a malformed URL, a duplicated tag, mismatched IDs, invalid event names. |
| v5 · lean | Housekeeping with the benchmark on: cut the instructions to 54.7% of the words, 21 files → 14. | 0% | Same behaviour, half the words. The benchmark proved each cut was safe before it stayed. |
The ceiling moved from knowledge to execution. By v4 the agent reliably knows the right answer and cites it — zero invented facts, audited. The only thing capping its score is that it won't do the action a human did (publish the tag, run the report, push the change).
That isn't a flaw to fix — it's the design. A shadow worker is supposed to draft and hold, not act. So the remaining gap is a deliberate human gate, not a gap in what it understands. Knowing that is what tells you where the next investment goes: wiring safe, gated execution — not more facts.
Grounding the answers was half the work. The other half: giving the agent ways to verify and act on what it finds — every one read-only or human-gated. Four are live today; the rest are scoped and ranked by how many real cases they would unblock.
Loads a public page headless and decodes exactly what it sends on the wire — tracking beacons, IDs, configuration, the data layer, redirect health. Credential-free; detection is a data-driven 30-technology registry.
Compares what a page does against what the spec says — a vendor-neutral rule engine that emits findings, not opinions, and re-scores without re-crawling.
Turns each finding into an evidence-backed brief routed to the right owner. Any auto-fix is fail-closed behind an explicit permission gate — never unsupervised.
A register of every deployment — who asked, when it was last checked, when it expires — so nothing rots silently. Modelled on a secrets changelog.
A read-only link to the analytics platform, so it can quantify a spike or trend a metric instead of promising to check. Reporting is roughly a quarter of all cases.
Read access to the tag-management config — which rules serve which region — then a human-gated publish. The single biggest billable case type bottlenecks here.
Pull the real spec out of attached spreadsheets, PDFs, and screenshots. Proven on the benchmark (v4); not yet wired into the live inbox loop.
An append-only deployment history, so it can answer "did something change in April?" — the question that documentation alone never can.
Read and open tickets, since many requests arrive — and get resolved — through the issue tracker.
The one enabler behind several of these: allowlisted network access to reach pages behind a corporate firewall and pre-launch staging — the highest-leverage single unlock, since that's exactly where a check matters most.