Agent patterns
Is the agent any good?
You can't ship an AI agent on a hunch. The honest way to measure one is borrowed wholesale from how you'd examine a student — teach it on old cases, test it on cases it has never seen, and grade every answer against what a human expert actually did.
A companion to the shadow worker. Generalised — no client, no names, illustrative numbers.
// the question
One question the whole thing answers
Say you've built an agent that drafts the first reply to incoming requests. The whole project rests on one question: if a fresh request arrives, can the agent answer it as well as the human expert — without making things up? Everything below exists to answer that honestly and repeatably.
A customer emails: "please add the tracking tag to our new product page; it's already on the store site, keep it there too."
The expert's real reply: "We'll publish it by Friday. Heads up — the existing store tag is an older version and inconsistent with the rest; want us to update it to match?"
That second sentence — the unprompted catch — is the value a good answer has to find.
// the setup
Study material, and a hidden exam
This is the standard "teach a student, then give them an exam they haven't seen" setup. You split your cases into two piles — by date, and that date split is the whole trick.
Cases it learns from
The older cases. The agent doesn't "train" like a brain — you load these into its knowledge base as worked examples ("here's a past request and how the expert handled it"), with the rules and specs. Its solved-example homework.
Cases it has never seen
The newer cases — kept hidden. You hand the agent only the question; it drafts a reply; you compare to the expert's real answer, which it never read. A good score means it learned the skill, not the answer key.
If you tested on cases the agent had studied, it would just parrot the memorised answer — a fake-high score. Make it study the old cases and sit the exam on newer ones, and it can't have memorised the answers (you can't memorise the future). That's the only thing that makes the number honest.
// the bench
The exam, run and graded
Bench (benchmark) is the whole exam-and-grading machine. For each hidden test question it runs the same three moves — so when you change the agent, you re-run the identical exam and the numbers are comparable.
- Blind draft. The agent sees only the question and drafts a reply — no peeking at the expert's answer, no hidden cheat-sheet. It works only from the knowledge base it was taught.
- A critic grades it. A separate AI, acting as examiner, reads the draft, the expert's real answer (the gold), and a fixed rubric — so it isn't grading its own work.
- Roll up to a few numbers. All the scorecards collapse into a handful of headline figures you can read in one glance — and compare across versions.
// the grades
What the critic actually checks
A verdict on each answer — and the two checks that matter most.
Did it make something up?
Inventing one specific fact it can't cite — a number, an ID, a setting — is an automatic fail, no matter how well the rest reads. On real work, a confident wrong fact is worse than "let me confirm." This is the gate.
Did it catch the key move?
Most answers turn on one load-bearing move — the unprompted catch in our example. Did the draft find it, the way the expert did? That's the value, not the boilerplate.
// the ruler
The baseline — the "before" number
A score means nothing on its own. Is 50 out of 100 good? You can't tell — except against something. The baseline is the score you already have: "this is how good the agent was before the change." Say a first version invents a fact in half its drafts. Change something, re-run the same exam — if that rate drops, you know the change worked. Without a baseline a number is just a number; with one, it's progress (or a regression). It's the ruler every new score is measured against.
// the catch
One more step: QA the exam itself
The test set was built by machine — it auto-extracted each question, the expert's real answer, and a one-line summary of the key move. Before you trust a score, a human eyeballs a handful of cases: is the "gold" actually the expert's complete answer — not a half-reply or the wrong message? Is the key-move summary right? If the sample is clean you trust the rest; if not, you fix the extractor first. You don't grade a class on an exam with typo'd questions. This is QA on the exam, not on the agent.
// the loop
The whole thing, in one loop
- Build — split the cases by date into a hidden exam (test) + study material (train).
- Teach — load the train cases and the rules into the agent's knowledge base.
- Bench — run the hidden exam blind; the critic grades each draft against the gold.
- Compare — headline numbers against the baseline: did it get better?
- Human QA — eyeball a few exam questions to confirm the test itself is fair.
- Repeat — every time you improve the agent → a running answer to "is it good enough yet?"
// the principles
What keeps the number honest
- Held-out or it's theatre. Test only on cases the agent has never seen — a date cutoff is the cleanest fence.
- Grade against the human's real answer, not a vibe. The expert's sent reply is the answer key.
- A made-up fact is an automatic fail. Fluency is not correctness; the hallucination gate comes first.
- A score needs a baseline. "Better than last version" is the only number that means anything.
- QA the exam before you trust it. A machine-built test set gets spot-checked by a human first.