Agent patterns
The flag
Hide a random secret at the goal. Send the agent in with no map. It either comes back holding the secret or it doesn't — and no amount of fluent explanation can substitute for the string. It is the cheapest honest grader anyone has found.
A companion to Is the agent any good?, which grades judgement against a human expert. This one is about the tasks where you don't need a judge at all.
// the problem
Grading is the expensive half
Building an agent is the part everyone talks about. Knowing whether it worked is the part that costs money. The usual answer is a gold answer plus a critic: a human expert's real reply, a rubric, and a second model reading both. It works, and for anything that comes out as prose it is the only thing that works. But it carries three permanent taxes: a judge to run, a rubric to argue about, and a plausible answer that scores well while being wrong. Language models are good at looking right.
For a large class of tasks there is a harder test available, and it costs almost nothing.
// the trick
Put the answer where only success can reach it
Generate a random string. Put it at the goal state — inside the file the agent has to find, the database row it has to reach, behind the bug it has to fix, on the other side of the exploit it has to write. Then ask for the string back. Nothing else is graded.
flag{7f3c9a1e-payload-delivered}
The wrapper is a convention so a grader can spot it; the middle is random. It carries no meaning and that is the entire design. There is nothing in it to deduce, infer, or approximate.
The name comes from capture-the-flag, the security competition format: organisers hide tokens in deliberately vulnerable systems, teams score by submitting them. It moved into agent evaluation almost unchanged, because the format already solved the problem the field had — how do you grade an autonomous system without trusting its account of itself?
// why it holds
Four properties, one guarantee
Randomness closes the shortcut
A rubric can be gamed by writing what a grader likes. A random token cannot be written toward. There is no partial path to it and no clever phrasing that gets close.
Possession implies arrival
Holding the string is proof the agent reached the state that contained it. Not a claim of reaching it, not a convincing description of reaching it. The evidence and the achievement are the same object.
Regenerate and the test is new
Benchmarks rot as they leak into training data. Re-roll the token and the old answer is worthless, even if the whole environment was memorised. The test survives its own publication.
String equality is the whole grader
No second model, no rubric, no human pass, no per-run inference bill. The comparison is one line of code and it returns the same verdict every time it is run.
// the cost
What you give up to get that
The flag is binary, and binary is brutal. Everything that is not the final string returns exactly the same score as doing nothing at all. Three consequences, all real:
- No gradient. An agent that got 90% of the way there scores identically to one that sat still. In reinforcement-learning terms this is a sparse reward, and sparse reward is the hardest thing to learn from — the agent wanders with no signal until it stumbles onto the goal. Benchmarks soften this with subtask flags: intermediate tokens planted along the intended path, so partial progress is still measurable.
- It scores arrival, never route. An agent that reasoned its way in and one that brute-forced its way in return the same string. If how matters — cost, elegance, safety, whether it wrecked the environment getting there — the flag is silent on all of it and you need separate instrumentation.
- It only exists where a goal state does. You cannot hide a token inside "write a good summary." Tasks whose output is judgement have no verifiable end state, and no amount of wanting one will create it.
// the choice
Flag or judge
These are not rivals. They cover different halves of the work, and most serious agent programmes run both.
| Hidden flag | Gold answer + critic | |
|---|---|---|
| Grades | Did it reach the state | Was the judgement any good |
| Verdict | Binary, deterministic | Scored, arguable |
| Cost per run | A string comparison | A judge model, plus rubric upkeep |
| Fails when | The task has no verifiable end state | The critic is fooled by fluency |
| Use it for | Retrieval, tool use, exploits, migrations, fixes with a test | Drafts, replies, analysis, recommendations |
The practical rule: before you hire a judge, look for a goal state. A surprising share of tasks people grade with a rubric have a verifiable end hiding in them — a row that must exist, a test that must pass, a file that must appear. Every one of those you convert is a grader you stop paying for and an argument you stop having.
// beyond the exam
The same string, three jobs
One mechanism — a unique, unguessable token planted on purpose — does three different jobs depending on why you planted it. They get confused constantly.
Proves an achievement
Planted at a goal. If it comes back, the agent got there. Used for grading.
Proves a leak
Planted in a document or a corpus. If it later appears in a model's output, that text reached the training data. Used for contamination checks.
Proves an intrusion
Planted somewhere nobody legitimate reads — a fake credential, a decoy record. Any use of it means someone was in there. Used for detection.
Flags prove something was achieved. The other two prove something happened. Same trick, opposite intent.
// in the wild
Where you'll meet it
- Agentic security benchmarks. Cybench, InterCode-CTF and NYU CTF Bench all wrap real capture-the-flag challenges into environments an agent is dropped into with a shell and a goal. They are now standard fixtures in frontier-model capability reports.
- Multi-agent reinforcement learning. DeepMind's Capture the Flag work trained agents to play the literal game in first-person 3D, learning teamwork from nothing but the flag capture as reward.
- Coding and migration evals. The passing test is the flag in everything but name — planted, binary, unfakeable, and unmoved by how good the diff looks.
- Your own harness. This is the transferable part. Any internal eval where an agent must reach a state can plant a token instead of hiring a critic.
// the principles
What makes a flag worth planting
- Random, or it is not a flag. A guessable or derivable token grades fluency, which is the thing you were trying to stop grading.
- Plant it at the state you actually care about. The flag inherits its meaning entirely from its location; a token behind a trivial door certifies nothing.
- Re-roll it on every generation. That is what buys immunity from contamination, and it is free.
- Add subtask flags when the run is long. One terminal token over a fifty-step task tells you it failed, not where.
- Never grade a route with an arrival test. If cost, safety or method matter, measure them separately — the flag will not mention them.