Who does the judging: LLM judges, agent judges, and humans

You have decided what to evaluate. Now: who says whether it passed?

There are five answers, and the mistake almost everyone makes is picking one. The right architecture is a cascade — cheap deterministic checks catch the obvious, a model judges the subjective, and humans spend their scarce attention only on what the machines cannot settle.

This chapter is about building that cascade, with the emphasis on the piece you will actually write code for: the LLM judge.

The cascade

Evaluator	Cost per case	Latency	Catches	Misses	Use for
Programmatic checks	~0	ms	Exact facts, forbidden phrases, schema violations, tool sequence	Anything requiring judgment	Every case, every run, first gate
Similarity metrics	~0	ms	Gross drift from a reference	Semantics, correctness, tone	Trend lines only, never thresholds
LLM-as-a-judge	$0.001–0.02	1–5s	Groundedness, helpfulness, tone, rubric adherence	Deep domain error, its own biases	Nightly suites, CI on a subset
Agent-as-a-judge	$0.02–0.50	10–60s	Plan quality, tool misuse, process failures on complex artifacts	Cost discipline; can be as wrong as the agent	Complex artifacts, failed cases only
Human review	$1–20	minutes–days	Everything, including what your rubric forgot	Scale, consistency, availability	Golden set, judge calibration, disputes
Production user feedback	~0	live	Real dissatisfaction, unknown unknowns	Precision; heavily biased sample	Discovery, not scoring

Read that table as a routing policy, not a menu. Every case goes through the programmatic checks because they cost nothing. Cases that survive get judged by a model. Cases the model scores near the threshold, or scores confidently in a way that contradicts a programmatic check, go to a human. Humans also periodically re-score a random sample regardless, because that is the only way you find out your judge has drifted.

The economics are the point. Ten thousand cases through a human is a quarter’s work. Ten thousand through an LLM judge is an overnight run and a two-figure bill. A hundred through a human, chosen because they are the ones that matter, is an afternoon — and that hundred is what makes the ten thousand trustworthy.

Saying it out loud. The mistake almost everyone makes is picking one evaluator. The right answer is a cascade: free deterministic checks run on every case, a model judges what actually needs judgment, and humans only touch what the machines can’t settle. Read the table as a routing policy rather than a menu — cases near the judge’s threshold, or where the judge contradicts a programmatic check, go to a person, and humans re-score a random sample regardless, because that’s the only way you find out the judge has drifted. The economics are the whole argument: ten thousand cases through humans is a quarter’s work, through an LLM judge it’s an overnight run and a two-figure bill, and the hundred you do route to humans are what makes the ten thousand trustworthy.

Programmatic checks first, always

Before any model does any judging, exhaust what a assert can do.

Exact-match on facts you control: the order ID, the tracking number, the total. Required substrings and forbidden substrings. Schema validation if the output is structured — if it must be JSON matching a Pydantic model, a parse failure is a hard fail and needs no judge. Tool trajectory assertions, which you build in Chapter 3. Budgets: steps, tokens, wall-clock.

These are free, instant, deterministic, and completely unarguable. They will catch more of your regressions than you expect, because most regressions are not subtle.

One warning about the second row of the table. String-similarity metrics — BLEU, ROUGE — and embedding similarity like BERTScore measure surface overlap with a reference answer, not correctness. A response can score 0.9 and be wrong; it can score 0.4 and be better than the reference. Use them as trend indicators across runs — a sudden drop in mean similarity on a fixed case set means something changed and is worth looking at — and never as a pass threshold. The sibling agentic-ai-evaluation-guide covers the metric zoo properly, including where each one is and is not valid.

Saying it out loud. Before any model judges anything, exhaust what an assert can do — exact match on the order ID, required and forbidden substrings, schema validation, tool-trajectory assertions, and budgets on steps and tokens. These are free, instant, and unarguable, and they’ll catch more regressions than you expect, because most regressions aren’t subtle. The one trap in this layer is similarity metrics — BLEU, ROUGE, embedding similarity. They measure surface overlap with a reference, not correctness: a response can score 0.9 and be wrong, or 0.4 and be better than the reference. Use them as a trend line that says something changed, never as a pass threshold.

LLM-as-a-judge

For everything that requires reading and judging — is this summary faithful, is this reply helpful, did this plan make sense — you use a model.

This works better than it has any right to, and it fails in specific, documented ways. Both halves matter.

Design the rubric first, prompt second

The most common failure is a judge prompt that asks “rate this response from 1 to 10.” You will get 7s. Almost all 7s. The scale has no anchors, so the judge has nothing to reason against and defaults to the middle of the distribution it saw in training.

A working rubric has four properties.

Decomposed. One criterion per question, scored separately, never a single blended “quality” score. “Grounded,” “responsive,” and “honest about gaps” are three things and an answer can be excellent at two and terrible at the third — which is exactly the information you want.

Anchored. Every point on the scale says what it means in concrete terms. Three points beat ten, because you and your judge can both tell 1 from 3 from 5 and neither of you can tell 6 from 7.

Grounded in evidence you supply. Give the judge the material it must judge against — the tool observations, the retrieved documents, the reference answer — and instruct it to score only against that. A judge that scores from its own world knowledge is a second agent with the same hallucination problem, and it will confidently mark a correct answer wrong because it disagrees about a fact it made up.

Structured on output. Demand JSON with a fixed key set. Free-text verdicts require parsing, and parsing errors show up as silently dropped scores.

Here is the rubric from the harness you will build, as data rather than prose:

RUBRICS = {
    "faithful_status": {
        "criteria": [
            ("grounded",
             "Every factual claim about the order (status, item, dates, tracking) appears "
             "in at least one tool observation in the trajectory. Invented facts score 1."),
            ("responsive",
             "The answer addresses the question the customer actually asked."),
            ("honest_about_gaps",
             "If information was missing or a lookup failed, the answer says so plainly "
             "instead of papering over it."),
        ],
        "scale": "1 = clearly violates, 3 = partially satisfies, 5 = fully satisfies",
        "threshold": 4.0,
    }
}

Keeping the rubric as data rather than a hardcoded string is not tidiness. It means the rubric is versioned, diffable, and reusable across the judge, the human review UI, and the documentation you give annotators — and those three drifting apart is how calibration quietly dies.

Saying it out loud. If you ask a judge to rate something one to ten, you’ll get sevens — almost all sevens — because an unanchored scale gives it nothing to reason against, so it regresses to the middle of the distribution it saw in training. A working rubric has four properties. It’s decomposed, one criterion per question, because an answer can be well-grounded and totally unresponsive and you want to see that. It’s anchored, with each point on the scale saying what it means in concrete terms — three points beat ten, because nobody can tell a six from a seven. It’s grounded in evidence you supply, so the judge scores against the tool observations rather than its own world knowledge, otherwise it’s just a second agent with the same hallucination problem. And it demands structured JSON, because free-text verdicts turn parse errors into silently dropped scores.

The prompt

JUDGE_PROMPT = """You are grading a customer-support agent. You are not the agent.

[CUSTOMER REQUEST]
{mission}

[WHAT THE AGENT OBSERVED]
{observations}

[AGENT'S FINAL ANSWER]
{answer}

Grade the answer on each criterion below, using this scale: {scale}

{criteria}

Judge only against the observations shown. If a claim in the answer is not supported
by them, that is ungrounded even if it sounds plausible.

Return ONLY a JSON object:
{{"scores": {{"<criterion>": <int 1-5>, ...}}, "rationale": "<=40 words"}}"""

Four deliberate choices in there.

“You are not the agent” is not decoration — judges that slip into answering the question instead of grading it are a real and annoying failure mode. The observations block is what makes groundedness checkable rather than a vibe. “Judge only against the observations shown” is the instruction that stops the judge from substituting its own knowledge. And the rationale comes after the scores in the schema on purpose: you want it short, because a judge asked for a paragraph will write a persuasive paragraph and then feel obliged to score consistently with its own rhetoric.

If you want the judge to reason before scoring, make that explicit and separate — a reasoning field first, then scores — and know that you are paying tokens for it. Reason-then-score generally improves quality on hard rubrics and is wasted money on simple ones. Measure it on your own set rather than adopting either as doctrine.

Saying it out loud. There are a few non-obvious moves in a judge prompt. “You are not the agent” is load-bearing — judges genuinely slip into answering the question instead of grading it. Handing it the observations block is what turns groundedness from a vibe into something checkable. And I put the rationale after the scores in the schema and cap it at forty words on purpose, because a judge asked for a paragraph will write a persuasive paragraph and then feel obliged to score consistently with its own rhetoric. If you want it to reason before scoring, make that an explicit separate field and know you’re paying tokens for it — reason-then-score helps on hard rubrics and is wasted money on simple ones, so measure it on your own set instead of adopting either as doctrine.

The implementation, with a mock mode

@dataclass
class Verdict:
    scores: dict[str, int]
    rationale: str

    @property
    def mean(self) -> float:
        return sum(self.scores.values()) / max(len(self.scores), 1)


class LLMJudge:
    """Real judge. Uses a different model from the agent under test, temperature 0."""

    def __init__(self, model: str = "claude-sonnet-4-5") -> None:
        import anthropic
        self._sdk = anthropic.Anthropic()
        self.model = model

    def score(self, rubric_name, mission, observations, answer) -> Verdict:
        prompt = render_prompt(rubric_name, mission, observations, answer)
        resp = self._sdk.messages.create(
            model=self.model, max_tokens=400, temperature=0,
            messages=[{"role": "user", "content": prompt}],
        )
        raw = "".join(b.text for b in resp.content if b.type == "text")
        data = json.loads(re.search(r"\{.*\}", raw, re.S).group(0))
        return Verdict({k: int(v) for k, v in data["scores"].items()},
                       data.get("rationale", ""))


class MockJudge:
    """Deterministic stand-in. Not a model: a rule that mimics the rubric closely
    enough to develop the harness offline."""

    def score(self, rubric_name, mission, observations, answer) -> Verdict:
        blob = " ".join(observations).lower()
        ans = answer.lower()
        claims = re.findall(r"out for delivery|delayed in transit|delayed|arriving today|by 8pm",
                            ans)
        unsupported = [c for c in claims if c.split()[0] not in blob]
        grounded = 5 if not claims else (5 if not unsupported else 1)
        responsive = 5 if len(ans) > 20 else 2
        honest = 2 if ("error" in blob and "sorry" not in ans and "could not" not in ans) else 5
        return Verdict({"grounded": grounded, "responsive": responsive,
                        "honest_about_gaps": honest},
                       "mock judge: heuristic grounding check")


def get_judge():
    if os.environ.get("ANTHROPIC_API_KEY") and os.environ.get("EVAL_LIVE_JUDGE"):
        return LLMJudge()
    return MockJudge()

Two things about the mock.

It exists so the whole harness runs in CI, offline, deterministically, with no key and no bill — the same argument as Part 1’s MockClient, applied one level up. And it is honest about being a heuristic: it approximates the rubric, it is not a model, and the moment you rely on its scores as truth you are measuring your own regex. Its job is to prove the plumbing works, and to fail loudly if the plumbing breaks.

Run it:

$ python3 judge_demo.py

grounded    mean=5.0  {'grounded': 5, 'responsive': 5, 'honest_about_gaps': 5}
ungrounded  mean=3.7  {'grounded': 1, 'responsive': 5, 'honest_about_gaps': 5}

The ungrounded answer — “your order was delayed at the depot and will arrive Thursday,” produced against observations that say it is out for delivery today — drops grounded to 1 and the mean below the 4.0 threshold. That is the shape of a working judge: the criterion that was violated is the criterion that moved.

Saying it out loud. Two implementation choices are worth defending. The judge runs at temperature zero on a different model from the agent under test, because a model grading its own family shows measurable self-preference. And there’s a mock judge — a deterministic heuristic — so the whole harness runs in CI, offline, with no API key and no bill. The important discipline with the mock is honesty about what it is: it proves the plumbing works and fails loudly when the plumbing breaks. The moment you start treating its scores as truth, you’re measuring your own regex rather than your agent.

The biases, and what to do about each

An LLM judge is a language model, and it has the failure modes of one. These are measured, not folklore; the MT-Bench paper (Zheng et al., 2023) documented the main ones and everything since has confirmed them.

Position bias. When comparing two candidates, judges favour whichever came first — sometimes dramatically. Mitigation: run every comparison twice with the order swapped and only accept a verdict when the two agree.

Verbosity bias. Longer answers score higher, holding content constant. Mitigation: put length in the rubric explicitly (“brevity, given equal correctness, is better”), and sanity-check by correlating scores with length across your set — if the correlation is strong, your judge is measuring word count.

Self-preference. Models prefer text produced by themselves or their family. Mitigation: do not judge with the same model you are testing. If you must, treat the absolute number as meaningless and use it only to compare two candidates from that same model.

Sycophancy toward assertive text. Confident phrasing scores higher than hedged phrasing, even when the hedge was correct. Mitigation: an explicit criterion rewarding honest uncertainty — the honest_about_gaps line above exists for this.

Scale compression. Judges cluster around the middle of a wide scale. Mitigation: three or five points, anchored, never ten.

Here is position-swap in code:

def pairwise(judge_fn, mission, observations, a, b):
    """Run both orderings and only trust an agreeing verdict. Disagreement means
    the judge is voting on position, not quality."""
    first = judge_fn(PAIRWISE_PROMPT.format(mission=mission, observations=observations, a=a, b=b))
    swapped = judge_fn(PAIRWISE_PROMPT.format(mission=mission, observations=observations, a=b, b=a))
    flip = {"A": "B", "B": "A", "tie": "tie"}
    if first == flip[swapped]:
        return first, True
    return "tie", False

=== pairwise with position swap ===
position-biased judge  winner=tie  consistent=False
content-driven judge   winner=A    consistent=True

A judge that always picks whatever is in the A slot gets collapsed to “tie, not consistent” and its vote is discarded. That inconsistency rate is itself a metric worth tracking: if more than a few percent of your pairwise comparisons disagree under swap, your rubric is too vague to decide with.

Prefer pairwise comparison to absolute scoring when the question is “is the new version better,” which is most of the time. Win rate is a far more stable signal than a shift in mean score, because the judge only has to rank, not calibrate. Keep absolute rubric scoring for the CI gate, where you need a threshold and there is nothing to compare against.

Saying it out loud. An LLM judge is a language model, so it inherits the failure modes of one, and these are measured rather than folklore. The three I’d name first are position bias — it favours whichever candidate came first — verbosity bias, where longer answers score higher with content held constant, and self-preference, where models prefer text from their own family. The mitigations are concrete: run every comparison twice with the order swapped and only trust an agreeing verdict, correlate scores against answer length to see if you’re measuring word count, and never judge with the model you’re testing. There’s also scale compression, which is why you use three or five anchored points instead of ten. And the swap-disagreement rate is itself a metric — if more than a few percent of your pairwise comparisons flip under swap, your rubric is too vague to decide with.

Calibration: the step everyone skips

An uncalibrated judge is worse than no judge, because it produces numbers that feel like evidence.

Calibration is simple. Take thirty to fifty cases. Have a human — ideally two — label each one acceptable or not. Run the judge over the same cases. Compare.

Raw agreement is not enough, because if 80% of your cases are acceptable then a judge that says “acceptable” to everything scores 80%. Use Cohen’s kappa, which corrects for agreement by chance:

$ \kappa = \frac{p_o - p_e}{1 - p_e} $

where $ p_o $ is observed agreement and $ p_e $ is the agreement you would expect if both raters were guessing with their own marginal rates.

The rough reading: below 0.4 the judge is not usable; 0.4 to 0.6 is moderate and fine for trend-watching but not for gating; above 0.6 you can gate on it; above 0.8 is better agreement than two humans usually manage on a subjective task, and should make you suspect the task was not subjective in the first place.

Here is the calibration script from the repo, run against the mock judge:

$ python3 calibrate.py

answer                                         human machine mean
Your order is out for delivery, arriving tod       1       1  5.0
Your order is delayed and will arrive Thursd       0       0  3.7
It's delayed in transit and the carrier hasn       1       1  5.0
It's arriving today by 8pm.                        0       0  3.7
Our lookup service is down right now; I can'       1       1  4.0
It's out for delivery.                             0       1  4.0  <-- disagree
Yes.                                               0       1  4.0  <-- disagree

raw agreement = 0.71   Cohen's kappa = 0.46
confusion: tp=3 tn=2 fp=2 fn=0

Read what that is telling you, because it is a realistic result and not a flattering one.

Raw agreement of 71% sounds fine and kappa of 0.46 says it is not — moderate, usable for watching trends, not usable as a merge gate. Both errors are false positives: the judge passed things the humans rejected. That asymmetry matters more than the headline number, because a judge that only errs toward “acceptable” is precisely the judge that will let a regression through.

The two disagreements are instructive. “Yes.” is factually correct and the humans rejected it as unhelpfully terse — a criterion the rubric does not contain, so the judge cannot see it. That is a rubric bug, and the fix is a new criterion, not a better model. The other is nastier: the mock’s grounding heuristic checks whether the word “out” appears in the observations, and the observation was ERROR: find_order failed: TimeoutError, which contains “out” inside “timeout”. A substring match found support that does not exist.

That is calibration doing its job. Neither bug is visible from reading the judge’s code, and both would have quietly corrupted every number the harness produced.

Recalibrate whenever you change the judge model, change the rubric, or change the agent enough that its failure modes shift — and on a schedule regardless, because model providers update models under stable aliases.

Saying it out loud. An uncalibrated judge is worse than no judge, because it produces numbers that feel like evidence. Calibration is thirty to fifty cases, humans label them acceptable or not, run the judge, compare. Raw agreement won’t do — if 80 percent of your cases are acceptable, a judge that says yes to everything scores 80 percent — so you use Cohen’s kappa, which corrects for chance. Rough reading: under 0.4 unusable, 0.4 to 0.6 fine for trends but not for gating, above 0.6 you can gate on it. And read the confusion matrix, not just the headline, because the direction of the errors matters more: a judge whose mistakes are all false positives is precisely the judge that will wave a regression through. One more caution — naive judging of multi-turn conversations agrees with humans far less than people assume, so calibrate per conversation, not per final message.

Saying it out loud. So the short version of LLM-as-a-judge: it works far better than it has any right to, and it fails in specific documented ways, and you have to hold both. Get the rubric right first — decomposed, anchored, grounded in the evidence you hand it — then worry about the prompt. Use a different model from the one you’re testing. Prefer pairwise “is B better than A” over absolute scores whenever you’re comparing versions, because the judge only has to rank rather than calibrate, and win rate is a much more stable signal than a shift in mean. And calibrate against human labels before you gate anything on it, with kappa above 0.6 as the bar.

Agent-as-a-judge

An LLM judge reads a final answer. Some artifacts are too big or too structured for that to mean anything: a pull request, a multi-file refactor, a research report with twelve citations, a data pipeline.

Agent-as-a-judge is the natural extension — the judge gets tools and a budget and investigates rather than reads. It can open the files the agent claimed to modify, run the test suite, check that each citation resolves and says what was claimed, or walk the trajectory step by step asking whether each tool call was the right move given what was known at the time. The pattern was formalised by Zhuge et al. (2024), and the practical result was that it approached human-level judgment on complex engineering tasks at a fraction of the cost.

What it adds over a plain LLM judge is verification instead of impression. “The report cites six sources” is checkable by fetching them. “The plan was logical” becomes “step 3 called list_files on a directory that step 2’s output already showed was empty.”

What it costs is real: an order of magnitude more tokens and latency, plus the uncomfortable fact that your judge is now a non-deterministic multi-step system with all the failure modes of the thing it is judging. You now need to evaluate your evaluator, and the only way out of that regress is human spot-checks of judge verdicts.

Deploy it narrowly. Run it on cases the cheap judge failed or scored ambiguously, not on your whole suite. Give it read-only tools — a judge with write access is a second agent loose in your systems. Cap its steps like any other agent. And give it the trajectory, not just the output, because process evaluation is the thing it is uniquely good at.

The sibling agentic-ai-evaluation-guide goes deeper on multi-agent and process evaluation architectures; here the decision you need is just when to reach for it, and the answer is: when the artifact is too complex to judge by reading, and only for the subset that needs it.

Saying it out loud. Agent-as-a-judge is what you reach for when the artifact is too big to judge by reading — a pull request, a multi-file refactor, a report with twelve citations. The judge gets tools and a budget and investigates instead of reading: it opens the files, runs the tests, fetches each citation to check it says what was claimed. That’s verification instead of impression — “the plan was logical” becomes “step 3 listed a directory that step 2 already showed was empty.” The cost is an order of magnitude more tokens and latency, plus your judge is now a non-deterministic multi-step system with all the failure modes of the thing it’s judging. And there’s a subtler trap: if the judge shares a base model with the agent, it isn’t an independent verifier, it’s a correlated one — it tends to be wrong in the same places. So deploy it narrowly, on cases the cheap layer flagged, with read-only tools and a step cap, and keep human spot-checks of its verdicts.

Humans

Automation gives you scale. Humans give you truth — and they are the only source of it, so spend them where they compound.

Four jobs are human jobs and stay human jobs.

Writing the golden set. Someone who understands the domain decides what a good answer to each case looks like. This is the foundation everything else rests on, and no model can do it because it is the definition of the target.

Calibrating the judges. The labelling exercise above. Thirty to fifty cases, redone whenever anything material changes.

Adjudicating disputes and edge cases. Cases where the judge is near threshold, where two checks disagree, or where the trajectory looks wrong but the answer looks right. These are the highest-information cases in your whole set and they are exactly where automation is least reliable.

Domain and safety review. Medical, legal, financial correctness. Bias and fairness. Adversarial red-teaming. An automated filter catches the blatant; a specialist catches what your policy did not anticipate.

Two things make human time count for five times as much.

Show the trajectory, not just the answer. A reviewer who can see the tool calls diagnoses in thirty seconds what takes five minutes to guess at from the output. The standard shape is two panels: conversation on the left, reasoning steps on the right, with the tool call arguments and results expandable inline.

Make the output structured. Not a comment box. A verdict plus a tag from a fixed vocabulary — bad_plan, wrong_tool, tool_misuse, hallucination, unhelpful_tone, should_have_escalated. Free-text feedback is unaggregatable and dies in a spreadsheet; tagged feedback becomes a bar chart of your top failure modes, which is a roadmap.

And one rule that saves a lot of pain: do not treat human labels as infallible. On subjective tasks, two competent annotators agree maybe 70–85% of the time. If your judge disagrees with a human 20% of the time and your humans disagree with each other 20% of the time, your judge is at human parity and chasing that last gap is wasted effort. Measure inter-annotator agreement before you go optimising judge agreement — the sibling guide covers the methodology properly.

Saying it out loud. Automation gives you scale, humans give you truth, and there are four jobs that stay human. Writing the golden set, because that’s the definition of the target and no model can define its own target. Calibrating the judges. Adjudicating the disputes and near-threshold cases, which are the highest-information cases you have and exactly where automation is weakest. And domain and safety review. Two things make human time worth five times as much: show them the trajectory rather than just the answer, and make their output structured — a verdict plus a tag from a fixed vocabulary, not a comment box, because free text dies in a spreadsheet and tags become a bar chart of your top failure modes. And don’t treat human labels as infallible: two competent annotators agree maybe 70 to 85 percent of the time on subjective tasks, so a judge disagreeing 20 percent of the time is already at human parity and chasing that gap is wasted effort.

The human gate at runtime

One human-in-the-loop pattern is not evaluation at all but belongs in the same mental slot: pausing before a consequential action.

Part 2 built ask_for_confirmation as a tool; Part 4 made it a control-flow feature backed by durable state. The evaluation-adjacent point is that every approval decision is a labelled example, for free, from someone qualified. Log them. An approve/reject stream on your riskiest actions is the highest-quality dataset you will ever be handed, and most teams throw it away.

Saying it out loud. There’s a human-in-the-loop pattern that isn’t evaluation but belongs in the same mental slot: pausing before a consequential action for approval. The evaluation-adjacent insight is that every one of those approve-or-reject decisions is a labelled example, produced for free, by someone qualified, on your riskiest actions. That’s the highest-quality dataset you’ll ever be handed, and most teams throw it away because it lives in a UI event and never gets written down. Log the decision, the trace ID, and who made it, and you’ve got a golden set that builds itself.

Production feedback

Your users are evaluating your agent continuously. Most of that signal is being discarded.

Explicit feedback — thumbs up and down, a star rating, a short comment. Cheap to collect and biased in a specific way: response rates are typically low single digits, and the people who respond are the annoyed ones and the delighted ones. Never read the ratio as a quality score. Read a change in the ratio as an alarm, and read the individual thumbs-down as a queue of cases to investigate.

Implicit feedback is usually better and almost always ignored. Did the user accept the suggestion? Merge the PR? Complete the booking? Immediately rephrase the same question, which is the clearest “that was wrong” signal there is? Escalate to a human? These are behavioural, unbiased by who chooses to rate things, and they map directly onto the effectiveness pillar.

The engineering requirement is one thing: a feedback event must capture the trace ID. A thumbs-down with no trace is a complaint. A thumbs-down that links to the exact trajectory is a bug report with a repro, and it goes straight into the review queue and then into your case set as a regression test. That loop — production failure becomes a case, case becomes a gate — is the whole flywheel, and it is one foreign key.

Saying it out loud. Your users are evaluating your agent continuously and most of that signal is being discarded. Explicit feedback — thumbs up and down — has response rates in the low single digits and skews to the annoyed and the delighted, so never read the ratio as a quality score; read a change in the ratio as an alarm and each thumbs-down as a case to investigate. Implicit feedback is usually better and almost always ignored: did they accept the suggestion, merge the PR, complete the booking, or immediately rephrase the same question, which is the clearest “that was wrong” signal there is. The one engineering requirement is that a feedback event has to carry the trace ID. A thumbs-down without a trace is a complaint; a thumbs-down with one is a bug report with a repro that goes straight into your case set. That whole flywheel is one foreign key.

The policy, in one paragraph

Programmatic checks on every case on every run, because they are free. An LLM judge, rubric-based and calibrated against human labels with kappa above 0.6, on your nightly suite and on a fast subset in CI. Pairwise with position swap when comparing two versions. An agent judge only for complex artifacts and only on the cases the cheap layers flagged. Humans on the golden set, the calibration set, the disputes, and safety. Production feedback wired to trace IDs, feeding the review queue, feeding the case set.

Chapter 3 builds the first four of those.

Saying it out loud. If someone asks me to state the evaluation policy in one breath: programmatic checks on every case every run because they’re free; a calibrated rubric-based LLM judge on the nightly suite and a fast subset in CI, with kappa above 0.6 before it gates anything; pairwise with position swap whenever you’re comparing two versions; an agent judge only for complex artifacts and only on cases the cheap layers flagged; humans on the golden set, calibration, disputes, and safety; and production feedback wired to trace IDs feeding back into the case set. The thing that makes it a policy rather than a wish list is that each layer is justified by the specific class of failure it catches and the cost it avoids in the layer above.

What you should be able to do now

Route evaluation through a cascade — programmatic, then model, then human — and justify the cost of each layer with the specific class of failure it catches.
Write a rubric that is decomposed, anchored, evidence-grounded, and structured, and explain why a 1-to-10 “overall quality” score produces almost no information.
Implement an LLM judge with a mock mode so your harness runs offline in CI, and explain why the judge model should differ from the agent model.
Name the main judge biases — position, verbosity, self-preference, sycophancy, scale compression — and apply the concrete mitigation for each, including position-swap pairwise comparison.
Calibrate a judge against human labels, compute Cohen’s kappa, read the confusion matrix asymmetry, and decide from it whether the judge is fit to gate a merge.
Decide when an agent judge earns its cost, and design a human review workflow — trajectory visible, verdicts tagged from a fixed vocabulary — that produces aggregatable data.
Wire production feedback to trace IDs so a thumbs-down becomes a reproducible case rather than a complaint.

Production Agent Engineering — A Practical Guide