Most agent-eval work targets the scary tail. A flattened LLM-as-a-judge is wrong on the median trajectory. Meet the agent judge that investigates before it scores.
Everyone hardens their eval harness for the scary tail. The jailbreak, the prompt injection, the one-in-a-hundred edge case. Meanwhile the flattened LLM-as-a-judge is quietly wrong on the median session: the boring support conversation that ran a hundred tool calls, looked fine, and wasn't.
Flattening a long session into one prompt can hide failures across turns, even on the typical input. The fix is a different shape of evaluation.
What production agent traffic actually looks like
Open your traces. If you run agents in production, the median session is not a two or three turn chat. It is a long tool-using trajectory: the agent calls a search, reads a record, calls another tool, retries after a token expires, loops back, and does it again forty times before it answers.
This is not the tail. This is the median. One of our design partners is an insurance voice-agent team running production agentic systems, and their real pain is not the exotic jailbreak. It is context rot: sessions that spiral into a hundred-plus tool calls, token-expiry retries, and loops that no single response ever reveals. The scary edge case gets attention because it is scary. The long boring session gets a green check because nobody looks past the final answer.
That gap is the whole problem. The place your judge is most confident is the place it has the least evidence.
Why flattening fails on the median
Here is how most "LLM-as-a-judge" setups work under the hood. Take the session, serialize it into one big block of text, paste it into a prompt, ask a model "was this good?" The session gets truncated to fit the context window. The tool calls become flat strings the model cannot execute or verify. Cross-turn state (what the agent knew at turn 5 versus turn 40) collapses into a wall of text with no way to navigate it.
A judge in that position cannot verify a stateful tool action. It cannot check whether the agent looped. It cannot tell whether turn 30 contradicted turn 8. It can only pattern-match on how the final response reads. On a two-turn chat that is often enough. On a hundred-turn trajectory it is guessing.
External work points the same direction. Be precise about what it does and does not say.
A preprint case study of a deployed multi-turn ordering agent (Zhang et al., arXiv:2606.10315) found its built-in judge caught well under a quarter of human-confirmed problems, 2 of 9 patterns (22%) in one batch, and missed the cross-turn state failures. The authors trace this to a scoring rubric with no category for those failures. That is one agent in one study, not a general law. It still shows a real multi-turn agent passing a judge while failing across turns.
There is also a single-author benchmark, BabelJudge (arXiv:2606.22329), that extends its tests to agent trajectories and measures three things worth tracking: tool accuracy, hallucination detection rate and trajectory-length bias. It is an early preprint, but those measures match what you see when a judge has to take in a long trajectory in one pass.
The objection you are already forming: "not another dashboard, not another eval tool. Process beats tools." Fair. This is not a metrics dashboard, and it is not a new chart to stare at. It is automated error analysis at scale: it surfaces the first upstream failure in each trajectory it reviews, the way a careful engineer would if they had time to read every session. It does not remove human judgment. It points it at the right place instead of at a haystack. The problem we are describing is not "you lack a tool." It is "the judge you already run is blind to the median session." That is a capability gap, not a UI gap.
The tell: our own eval miner outran our judge
We did not arrive at this from theory. We arrived at it because a system we already run started producing evals a response-only judge cannot score, even in principle.
We run an eval-mining engine that reads production traces and proposes new eval rubrics: failure modes it sees recurring that no one wrote a rule for. This is also how we fight eval staleness, the slow decay where a hand-written eval set stops matching production the week behavior shifts. Some of what it proposes are trajectory-level rubrics. One rubric type it emits is "repeated identical tool call loop detection." Think about what it would take to score that rubric. You have to look across the whole sequence of tool calls and decide whether the agent got stuck. A judge that only sees the final response has no way to answer it. There is nothing in the response to look at. The evidence lives in the trajectory the judge never gets to see.
A clean, live example from the mining engine: on a claims-assistant agent, it surfaced a failure-mode cluster we internally call "renewal, no actionable steps," clustered from 39 real production sessions around one failure the rule-based strategies had missed entirely. It is a genuine, recurring quality problem, and scoring it well means reasoning about what the agent did across the session, not grading the last message.
This was the aha. The evaluator we already run implies a judge we did not yet have. Our miner had outrun our judge. So we stopped flattening.
(For the "so you built eval mining, cool" crowd: mining traces into evals and auto-proposing fixes are increasingly table stakes, and the incumbents ship versions of both. Useful, and we run mining too. But it is support here, not the headline. The headline is the judge.)
Evaluation as an investigation
The shift is this. Stop treating evaluation as a single verdict emitted from a single flattened prompt. Treat it as an investigation of the trajectory: search for evidence, retrieve precedent, verify claims, then rule.
We credit prior work here explicitly. The agent judge framing already published in the field (queryable trajectories, adaptive rubrics, environment verification) is prior art we build on. The core idea that an agent-as-a-judge should query the trajectory rather than swallow it whole comes from that work, and it is right. Our divergence comes later.
Here is the shape of what we built.
The agent judge investigates the trajectory: a free, code-computed digest becomes the map, a reasoning-tier coordinator plans reads, cheap-tier readers compress evidence, and one investigation emits every rubric verdict at once.
First, a trajectory digest. This is pure code, zero LLM tokens. It computes the shape of the session up front: tool statistics, loop signals, a segmented timeline, a turn index. It is the map. It costs nothing and it exists before any model is called.
Second, a coordinator on the reasoning tier. It reads the digest and plans the investigation: what to look at, what to pull, what to check. It does not grade blindly. It decides where the evidence is, then goes and gets it, and it emits all the rubric verdicts at the end.
Third, read-only tools the coordinator investigates with: get_events, search_in_session, similar_labeled_cases (retrieval against human-labeled precedent), get_tool_stats. A verify_action tool is reserved for the verification leg. These are read-only by design. The judge investigates; it does not act.
Fourth, readers on a cheap tier (Haiku-class or a small model). They compress evidence for the coordinator. They never judge. Their only job is to turn raw events into a summary the reasoning tier can act on, cheaply.
The economics matter, because "run a whole investigation per session" sounds expensive. One investigation amortizes across all rubrics. The cost of scoring N rubrics is roughly one investigation plus N cheap emissions, not N full runs. And cost control lives inside the judge (routing, amortization, prompt caching, budgets, distillation later), not as a cheap flattened gate bolted in front of it.
Verdicts land in eval_results tagged judge_model='agent-judge'. Everything downstream (KPIs, mining, the UI) is unchanged, which means the new judge can run in shadow next to the old one and be compared on the same label set before it takes over anything.
What's built, and where the honest gap is
This is shipped. Agent judge v1 landed and has run in shadow mode since July 2026, writing agent-judge verdicts alongside the existing eval-runner. When it wins on the label set, it flips to primary. Trajectory rubrics (the ones a response-only judge cannot score at all) route to it immediately, because there is no honest alternative.
The full system has three legs, and we want to be straight about which are done:
Search is live. A ClickHouse event graph plus pgvector for retrieval. The judge can query the trajectory and pull similar labeled cases.
Adaptation is live. The mining engine (disagreement, anomaly, and clustering strategies) plus a trust layer where mined candidates never become live config until a human approves them. Nothing auto-promotes.
Verification is the phase-behind gap. This is the leg that would let the judge re-execute or confirm a tool action against the customer's own environment via connectors. It is designed, the
verify_actiontool slot is reserved, and it is not built yet. If you need the judge to independently verify that a stateful action really happened, we are not there. We would rather tell you that now than have you find it later.
Even with two of three legs, the shipped judge already scores the trajectory rubrics a flattened judge cannot touch at all. That is the part that is real today.
Why investigation beats per-step scoring
Almost everyone now says "trajectory evaluation." Look under the hood and much of it is per-step rubric scoring: chop the trajectory into steps, score each step, aggregate. That is better than flattening the whole thing into one prompt. It is still scoring, not investigating. It cannot decide, mid-evaluation, that the interesting thing is a contradiction between turn 8 and turn 30 and go pull both.
The difference is a judge that plans, queries, retrieves precedent, and then rules. That is the durable difference, and it is the one thing that does not commoditize the week after you ship it.
And to name our divergence from the prior art plainly: Published agent-judge loops refine the rubric. Ours refines the agent and specializes the judge through retrieval against the customer's own labeled cases, so The judge can retrieve your labeled cases when reviewing similar sessions.. Same starting insight, different thing we do with it.
Where this goes
The agent judge is the product. It is a judge that investigates the trajectory the way you would if you had time to read every session, and that gets more specialized to your domain the longer it runs.
We will write more on how the judge specializes to your domain and how we keep its cost down.
If you run agents in production and you have ever shipped a green check on a session nobody actually read, we should talk. Follow the series, or reach out if you want to point this at your own traces.