Mini-project 2: build a reflection agent from scratch

Time to write the thing that checks its own work.

By the end of this chapter you will have a working reflection agent in about two hundred and fifty lines of Python, with no framework anywhere in it. It runs offline, with no API key, using a scripted model — so you can execute every snippet here immediately — and it swaps to a live Claude call by replacing one object.

The structure is the same as the ReAct chapter. We build the simplest thing that could work, watch it fail, and add exactly the piece that fixes that failure. Five versions. The failures are the curriculum, and in this chapter one of the failures is the pattern itself: there is a large and unflattering research literature on whether reflection works at all, and v4 is where we confront it instead of pretending it does not exist.

If you want the conceptual treatment of reflection — what it is, where it sits in a design, how to talk about it in an interview — that lives in the companion guide’s pattern page on reflection. This chapter assumes you have the idea and wants you to have the code.

Setup:

mkdir reflection-agent && cd reflection-agent
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic

Everything below lives in one file, reflect.py, which you grow as you go.

The shape we are aiming at

Reflection means the system does not ship its first attempt. A generator produces a draft, a critic evaluates that draft against criteria, and the critique drives a revision. Repeat until something says stop.

That is four moving parts — generate, evaluate, revise, terminate — and each has a failure mode that is invisible until you have written it yourself. The version sequence below is organized around those four.

One structural decision up front. Every model call goes through a tiny interface:

class Model(Protocol):
    def complete(self, *, system: str, messages: list[dict]) -> str: ...

Two implementations: a scripted fake for development and tests, and the real SDK.

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Callable, Protocol


@dataclass
class ScriptedModel:
    """Offline stand-in. Returns canned replies in order, and records every
    (system, messages) pair it was called with so tests can assert on them."""

    script: list[str]
    calls: list[dict] = field(default_factory=list)

    def complete(self, *, system: str, messages: list[dict]) -> str:
        self.calls.append({"system": system, "messages": [dict(m) for m in messages]})
        if not self.script:
            return "(script exhausted)"
        return self.script.pop(0)



class AnthropicModel:
    def __init__(self, model: str = "claude-sonnet-4-5") -> None:
        import anthropic
        self._sdk = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY
        self.model = model

    def complete(self, *, system: str, messages: list[dict]) -> str:
        resp = self._sdk.messages.create(
            model=self.model, max_tokens=2048, system=system, messages=messages,
        )
        return "".join(b.text for b in resp.content if b.type == "text")

The calls list is the part that earns its keep. A reflection agent’s entire behaviour is determined by what message list each call receives, and the bugs in this chapter are all bugs in message-list construction; recording every call lets you print exactly what the critic saw, which turns an argument about design into an assertion about data.

Be clear about what a fake can and cannot prove. It proves control flow: which call happens in what order, what history each one receives, when the loop exits, what happens when something throws. It cannot prove anything about how a real model behaves — where this chapter makes a claim about model behaviour, it cites research rather than a mock, and says which is which.

v1: the naive loop

The obvious implementation, and the one most tutorials show. Keep one message history. Ask for a draft, then ask for a critique, then ask for a revision, in the same conversation.

SHARED_SYSTEM = """You are a careful technical writer.
You draft text, critique your own drafts, and revise them."""

CRITIQUE_TURN = "Critique the draft above. List concrete, actionable problems."
REVISE_TURN = ("Revise the draft, addressing every point in the critique. "
               "Output the full revised text.")


def reflect_v1(model, task: str, rounds: int = 3) -> tuple[str, list[dict]]:
    messages: list[dict] = [{"role": "user", "content": task}]
    draft = ""
    for _ in range(rounds):
        draft = model.complete(system=SHARED_SYSTEM, messages=messages)
        messages.append({"role": "assistant", "content": draft})
        messages.append({"role": "user", "content": CRITIQUE_TURN})

        critique = model.complete(system=SHARED_SYSTEM, messages=messages)
        messages.append({"role": "assistant", "content": critique})
        messages.append({"role": "user", "content": REVISE_TURN})
    return draft, messages

Fourteen lines, and it runs. Give it a small writing task and a script:

TASK = ("Write a two-sentence release note for a feature that lets users "
        "export reports as CSV.")

model = ScriptedModel([
    "You can now export reports as CSV.",
    "Problem: no mention of where the button is. Problem: passive, no user benefit.",
    "You can now export any report as CSV from the Reports toolbar.",
    "This is much improved and addresses my earlier points. Looks good.",
    "You can now export any report as CSV from the Reports toolbar.",
    "Agreed, this is strong. No further changes needed.",
])
draft, messages = reflect_v1(model, TASK, rounds=3)

for m in model.calls[3]["messages"]:          # calls[3] is the second critic call
    print(f"  {m['role']:9s} | {m['content'][:58]}")

Actual output:

  user      | Write a two-sentence release note for a feature that lets
  assistant | You can now export reports as CSV.
  user      | Critique the draft above. List concrete, actionable proble
  assistant | Problem: no mention of where the button is. Problem: passi
  user      | Revise the draft, addressing every point in the critique.
  assistant | You can now export any report as CSV from the Reports tool
  user      | Critique the draft above. List concrete, actionable proble

What v1 gets wrong

Read the roles in that transcript, because the defect is right there in the fourth column.

Every assistant turn is attributed to the same speaker: the drafts are assistant, and so are the critiques. There is exactly one persona in this conversation and it is doing both jobs, which means the model being asked to critique is critiquing itself, in a transcript where it can see everything it has already said about this text. That is not a producer and a critic. That is one voice talking to itself, and it has two consequences.

The critic is conditioned on its own prior critiques. When the second critic call runs, Problem: no mention of where the button is is sitting in its context as something it said, and the most coherent continuation of “I raised these problems, then the text changed to address them” is “good, that is fixed.” The critic is no longer evaluating the draft against the requirements; it is evaluating the draft against its own prior complaint, which the generator was explicitly instructed to satisfy.

The generator is conditioned on the critiques as its own words too. It wrote both. Ask a model to argue against something it just argued for, inside one transcript, and you are fighting the strongest prior in the system.

Here is the shape that produces. Swap the fixed script for a fake whose reply is a function of the history it receives, with a policy that encodes the hypothesis directly — if my own earlier critique is visible in this history, stay consistent with it — so what you are watching is the loop’s structure permitting the collapse, not evidence that a real model collapses:

def anchored(system: str, messages: list[dict]) -> str:
    if messages[-1]["content"] != CRITIQUE_TURN:            # asked for a draft
        n = sum(1 for m in messages if m["content"] == REVISE_TURN)
        return DRAFTS[min(n, len(DRAFTS) - 1)]              # three canned drafts
    prior = sum(1 for m in messages if m["content"] == CRITIQUE_TURN) - 1
    if prior == 0:
        return "Problem: does not say where the export lives. Problem: no benefit stated."
    return "This addresses the points I raised earlier. I have no further objections."

Actual output:

call 0 (writer): You can now export reports as CSV.
call 1 (critic): Problem: does not say where the export lives. Problem: no user benefit s
call 2 (writer): You can now export any report as CSV from the Reports toolbar.
call 3 (critic): This addresses the points I raised earlier. I have no further objections
call 4 (writer): You can now export any report as CSV from the Reports toolbar.
call 5 (critic): This addresses the points I raised earlier. I have no further objections
call 6 (writer): You can now export any report as CSV from the Reports toolbar.
call 7 (critic): This addresses the points I raised earlier. I have no further objections

Round one did work. Rounds two and three were four model calls that produced a byte-identical draft and a critique that was pure agreement — the loop converged to self-congratulation and then kept paying for it.

You should be suspicious of a demonstration whose conclusion was written into the fake; that is why it is labelled. What the fake genuinely shows is that nothing in v1’s control flow prevents this, that the loop has no way to notice it, and that a fixed round count keeps spending regardless. Whether real models do it is an empirical question with real answers, and we get to them in v4.

The fix for the structural half is not a better prompt. It is a different message list.

v2: two histories, mirrored

This is the key design decision in the chapter, and it is the one thing the public course implementation this chapter is calibrated against gets exactly right.

Keep two message histories.

The generation history is the conversation the writer is having: the requirements as user, its drafts as assistant, the critiques arriving as user feedback. The reflection history is the conversation the critic is having: the drafts arriving as user content to review, its critiques as assistant.

The same text appears in both with the roles swapped: what the generator emitted as assistant is inserted into the critic’s history as user, and what the critic emitted as assistant is inserted into the generator’s history as user.

GEN_SYSTEM = """You write and revise content for the user.
When the user gives you a critique, output a complete revised version — not a diff,
not a commentary. The full text, every time."""

CRITIC_SYSTEM = """You are a meticulous reviewer.
The user will show you drafts of a piece of work. The requirements are:

{requirements}

List concrete, actionable problems with the draft, each one naming the requirement
it violates. Do not praise. Do not rewrite the draft yourself."""


def reflect_v2(model, task: str, rounds: int = 2) -> str:
    gen_history: list[dict] = [{"role": "user", "content": task}]
    reflect_history: list[dict] = []
    critic_system = CRITIC_SYSTEM.format(requirements=task)

    draft = ""
    for _ in range(rounds):
        draft = model.complete(system=GEN_SYSTEM, messages=gen_history)
        gen_history.append({"role": "assistant", "content": draft})
        reflect_history.append({"role": "user", "content": draft})       # role swap

        critique = model.complete(system=critic_system, messages=reflect_history)
        reflect_history.append({"role": "assistant", "content": critique})
        gen_history.append({"role": "user", "content": critique})        # role swap

    return draft

Four appends, two of them crossing over. That is the whole idea.

Print what each side saw on its second call:

=== what the WRITER saw on its second call ===
  user      | Write a two-sentence release note for a feature that lets us
  assistant | You can now export reports as CSV.
  user      | Problem: does not say where the export lives. Problem: no us

=== what the CRITIC saw on its second call ===
  user      | You can now export reports as CSV.
  assistant | Problem: does not say where the export lives. Problem: no us
  user      | You can now export any report as CSV from the Reports toolba

system prompts used: ['You are a meticulous reviewer.', 'You write and revise content for the user.']

Look at the two blocks side by side: they contain the same three pieces of text and assign opposite roles to all of them. Each call now sees a history in which it is consistently the assistant, its counterpart is consistently the user, and its own system prompt describes exactly one job.

Why this works

Three separate mechanisms, worth naming separately because they are usually blurred together.

Role consistency. Chat models are trained on transcripts where assistant is one coherent speaker. In v1 that role held two contradictory jobs, so every turn was slightly out of distribution. Here each conversation has one speaker doing one thing.

Persona separation. The generator’s system prompt says produce; the critic’s says find problems and do not praise. Instructions that contradict each other inside one prompt get averaged; instructions in separate calls do not.

Blindness to its own past agreement. The critic’s history contains its previous critiques — which it needs, so it does not repeat itself — but not v1’s scaffolding: the “now critique this” instructions, the revision requests, its own drafts labelled as its own speech. The critic sees drafts as someone else’s work, and you got that by moving strings between two lists rather than by writing a cleverer prompt.

None of this makes the critic correct. It makes the critic independent-ish, which is a smaller claim and the only one the mechanism supports — independence from your own previous statements is not competence, and v4 is about the difference.

What it costs

Two histories cost roughly twice the tokens of one, because both grow and both carry the same content.

Write the per-round cost as $c_g + c_c$ — one generator call plus one critic call — and the total for $n$ rounds is

$$C(n) = \sum_{i=1}^{n} \left( c_g(i) + c_c(i) \right)$$

where each $c(i)$ grows roughly linearly in $i$, because history number $i$ contains everything from rounds $1 \dots i-1$. That makes total spend $O(n^2)$ in the number of rounds, not $O(n)$: four rounds is not twice as expensive as two, it is closer to four times. Hold that number when someone suggests raising the iteration cap — it is why the caps in every serious implementation are two or three.

The second cost is duplication: the same draft text is now in two places, and if you compact one history you must think about the other independently. Which brings us to two real bugs.

Two bugs the reference implementation has, and you should not

The reference gets the two-history split right and then loses on the details, and both details are failure modes of this design specifically.

The critic never sees the requirements. Its reflection history is seeded with only the critic system prompt; the user’s task is never added to it. The critic therefore reviews a draft with no statement of what the draft was supposed to do, and its only option is to infer the requirements from the draft itself — exactly the circularity the pattern exists to break, since a critic that infers the goal from the artifact cannot detect that the artifact pursued the wrong goal. The fix is the {requirements} slot in CRITIC_SYSTEM above: put the task in the critic’s system prompt, where truncation cannot reach it.

Truncation silently evicts the task from the generator too. The reference caps both histories at three messages with a “keep the first message fixed” queue, which is a reasonable instinct — the $O(n^2)$ growth is real and something has to bound it. But the fixed first message is the system prompt and the user’s task is the second, so the task is the first thing evicted. Run the reference’s own history class and watch:

after round 0 gen = ['SYS', 'gen0', 'crit0']
               refl = ['CRITIC_SYS', 'gen0', 'crit0']
after round 1 gen = ['SYS', 'gen1', 'crit1']
               refl = ['CRITIC_SYS', 'gen1', 'crit1']

TASK is gone after the very first round. From round two onward the writer is revising a text it can no longer see the purpose of, steered only by the latest critique — which is how a reflection loop drifts off-spec while every individual step looks locally reasonable.

The general rule: anything that must survive to the last round belongs in the system prompt, not in the message list. The message list is the part you are allowed to compact. The requirements are not.

v3: a stopping rule

v2 still runs a fixed number of rounds, and a fixed count is wrong in both directions at once.

It is too many when the first draft was already fine — you pay $c_g + c_c$ per round to hear “still good”, and each round is a chance for the model to change something that did not need changing. It is too few when the work needed five passes and got two, and the loop returns a draft with known unfixed problems as if it were finished.

The fix has two halves and you need both.

Half one: let the critic say it is done. Give it a token to emit, and check for it.

STOP = "NO_FURTHER_OBJECTIONS"

CRITIC_SYSTEM = """You are a meticulous reviewer.
The user will show you drafts of a piece of work. The requirements are:

{requirements}

List concrete, actionable problems, each naming the requirement it violates.
If and only if you have no substantive objection left, reply with exactly this
one line and nothing else:

""" + STOP


def is_stop(critique: str) -> bool:
    """Strict: the token must be the entire reply, not merely present in it."""
    return critique.strip() == STOP

Half two: a hard cap the model cannot influence.

def reflect_v3(model, task: str, *, max_rounds: int = 4):
    gen_history: list[dict] = [{"role": "user", "content": task}]
    reflect_history: list[dict] = []
    critic_system = CRITIC_SYSTEM.format(requirements=task)
    draft, rounds, stopped_early = "", 0, False

    for rounds in range(1, max_rounds + 1):
        draft = model.complete(system=GEN_SYSTEM, messages=gen_history)
        gen_history.append({"role": "assistant", "content": draft})
        reflect_history.append({"role": "user", "content": draft})
        print(f"[{rounds}] draft:    {draft[:70]}")

        critique = model.complete(system=critic_system, messages=reflect_history)
        print(f"[{rounds}] critique: {critique[:70]}")
        if is_stop(critique):
            stopped_early = True
            break

        reflect_history.append({"role": "assistant", "content": critique})
        gen_history.append({"role": "user", "content": critique})

    print(f"--- stopped after {rounds} round(s), "
          f"{'critic satisfied' if stopped_early else 'budget exhausted'}")
    return draft

Two scripted runs, one for each exit:

### critic runs out of objections
[1] draft:    You can now export reports as CSV.
[1] critique: Problem: does not say where the export lives.
[2] draft:    You can now export any report as CSV from the Reports toolbar, so fina
[2] critique: NO_FURTHER_OBJECTIONS
--- stopped after 2 round(s), critic satisfied

### critic that is never satisfied
[1] draft:    Draft 0.
[1] critique: Problem: still not right (0).
[2] draft:    Draft 1.
[2] critique: Problem: still not right (1).
[3] draft:    Draft 2.
[3] critique: Problem: still not right (2).
--- stopped after 3 round(s), budget exhausted

Twenty scripted replies available, three rounds executed. The cap held.

Why “let the model decide when to stop” is dangerous alone

The stop token is a request, not a guarantee, and everything that can go wrong with it is a normal Tuesday.

An unappeasable critic never emits it. Prompt a model to hunt flaws and it will find flaws in a haiku, because “list concrete problems” has no natural fixed point. Without a cap this is an infinite loop that spends money at an accelerating rate, since each round’s context is larger than the last.

A lenient critic emits it immediately. You pay for the machinery, get single-pass quality with extra steps, and — worse — get a system that reports it was reviewed. False assurance is more expensive than no assurance.

An oscillating pair never converges: the generator fixes complaint A in a way that reintroduces B, the critic complains about B, the generator reintroduces A. Neither side is malfunctioning. The loop is orbiting.

And the token can be spoofed by ordinary prose. This is the one that bites, and the reference implementation has it: it tests if "<OK>" in critique, a substring check. A critic writing a perfectly sensible sentence that mentions the token trips it:

### the substring trap
naive  in-check : True
strict ==-check : False

The critique in that test was "Problem: the note never says what the file is for. When that is fixed I will reply NO_FURTHER_OBJECTIONS." — an explicit refusal to approve, read by the substring check as approval. Use equality on the stripped reply, or parse a structured field. Never in.

The design rule: the model proposes termination, the orchestrator disposes. The cap is the only exit that cannot be talked out of, so it has to exist — and running out of rounds is a normal operating condition, not a crash. The caller gets the best draft available plus an honest flag saying it never passed.

What v3 gets wrong

We now have a loop that terminates, has clean role separation, and stops when the critic is satisfied. We still have no idea whether any of it improves the output.

Every signal in this system is the model’s opinion of the model’s work, so the critic’s approval is not evidence. Whether that kind of approval is worth anything is not a matter of taste — it has been studied, and the answer is uncomfortable.

v4: make it measurable

What the evidence actually says

The tutorial version of this pattern asserts that reflection improves quality. The literature is considerably more careful, and if you are going to spend two to four times the tokens on a loop, you should know what the loop is and is not known to buy.

The optimistic result came first. Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback (NeurIPS 2023), had a single model generate, critique and refine with no extra training, and reported roughly a 20% absolute improvement on average across seven tasks with GPT-3.5, ChatGPT and GPT-4 (https://arxiv.org/abs/2303.17651). That paper is why the pattern spread.

The corrective came a few months later, and it is the paper to know. Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024), isolated intrinsic self-correction — correction using no external feedback, no oracle, no tools, nothing but the model’s own judgement — and found that “LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction” (https://arxiv.org/abs/2310.01798).

Their methodological point is the one worth internalizing. Several earlier results, they argue, used the ground-truth label to decide when to stop correcting: the loop kept going while the answer was wrong and halted when it became right. That is an oracle, and as they put it, if you already have the ground truth there is little reason to run the model at all. Remove the oracle, re-run the same setups, and the direction flips. Their numbers with self-correction and no oracle:

Every column goes down or sideways. The CommonSenseQA collapse is the loudest, but the mechanism is clearest in their breakdown of what self-correction changed on GSM8K with GPT-4: after two rounds, 90.5% of answers were unchanged, 8.0% went from correct to incorrect, and 1.5% went from incorrect to correct. The loop was five times more likely to break a right answer than to fix a wrong one — the sentence to remember when you are tempted to add another round.

Two related results point the same way. Stechly, Marquez and Kambhampati found on graph colouring that GPT-4 was no better at verifying a solution than at producing one, and — the sharper finding — that “the correctness and content of the criticisms… seems largely irrelevant to the performance”, with apparent gains coming from correct answers happening to appear among sampled candidates rather than from critique doing work (https://arxiv.org/abs/2310.12397). Valmeekam et al. found that in planning, “self-critiquing appears to diminish plan generation performance” relative to an external verifier, driven by a notable rate of false positives from the LLM verifier — a critic that approves broken plans (https://arxiv.org/abs/2310.08118).

Kamoi et al.’s survey in TACL is the balanced synthesis, and its three findings are effectively the design spec for v4 (https://aclanthology.org/2024.tacl-1.78/): feedback from prompted LLMs rarely enables successful self-correction outside specific task types; self-correction is effective when reliable external feedback is available; and large-scale fine-tuning can teach the capability.

The middle finding is the load-bearing one, and it is what the successful systems were doing all along. Reflexion (Shinn et al., 2023) reflects on task feedback signals from an environment rather than on introspection (https://arxiv.org/abs/2303.11366). CRITIC (Gou et al., ICLR 2024) has the model verify and revise its output by interacting with tools, and concludes that external feedback is essential for meaningful self-improvement (https://arxiv.org/abs/2305.11738).

So the honest statement of the pattern is not “reflection improves quality.” It is:

Reflection grounded in an external signal reliably improves quality. Reflection grounded in introspection alone is not reliably better than one pass, and on some tasks is measurably worse.

Everything v4 does follows from that sentence.

Reflect on tool output, not opinion

Change the task to something with an un-negotiable check. The requirements:

SPEC = textwrap.dedent("""\
    Write a Python function `parse_duration(text: str) -> int` that converts a
    duration string like "1h30m" into a number of seconds. It must accept any
    combination of hours, minutes and seconds in that order, each part optional,
    and raise ValueError on anything it cannot parse. Output only the code.""")

And the external signal — a test suite, executed in a separate process:

TESTS = textwrap.dedent("""\
    cases = [("90s", 90), ("5m", 300), ("2h", 7200),
             ("1h30m", 5400), ("1h30m15s", 5415)]
    for text, want in cases:
        got = parse_duration(text)
        assert got == want, f"parse_duration({text!r}) == {got!r}, want {want!r}"
    for bad in ["", "abc", "10x"]:
        try:
            parse_duration(bad)
        except ValueError:
            pass
        else:
            raise AssertionError(f"parse_duration({bad!r}) should have raised ValueError")
    print("all 8 checks passed")""")


@dataclass
class Verdict:
    ok: bool
    report: str


def run_tests(code: str, tests: str, timeout: float = 10.0) -> Verdict:
    """Execute candidate code against a test suite in a separate process.
    Never raises: a crash, a syntax error and a timeout are all just verdicts."""
    with tempfile.TemporaryDirectory() as tmp:
        path = Path(tmp) / "candidate.py"
        path.write_text(code + "\n\n" + tests + "\n")
        try:
            proc = subprocess.run([sys.executable, str(path)],
                                  capture_output=True, text=True, timeout=timeout)
        except subprocess.TimeoutExpired:
            return Verdict(False, f"TIMEOUT: tests did not finish in {timeout}s")
    if proc.returncode == 0:
        return Verdict(True, proc.stdout.strip())
    tail = (proc.stderr.strip().splitlines() or ["(no output)"])[-3:]
    return Verdict(False, "TESTS FAILED\n" + "\n".join(tail))

Note the same rule the ReAct chapter established for tools, applied to the verifier: it never raises. A syntax error, an exception at import time, an infinite loop that hits the timeout — all of them are Verdict(False, ...) with a report the model can read. The verifier’s failure modes are observations, not crashes.

Note also that run_tests executes model-written code. A subprocess with a timeout is enough for a book; production wants a container with no network and no filesystem access outside a scratch directory. “We exec the model’s output” is a sentence that should always be followed by “in a sandbox, and here is the sandbox.”

Now the same loop, with the verdict standing in for the critique:

VERIFIER_GEN_SYSTEM = """You write Python. The user gives you a spec, then gives you
the output of running your code against a test suite. Fix what the tests report.
Output only code."""


def reflect_verified(model, spec: str, *, max_rounds: int = 4):
    gen_history = [{"role": "user", "content": spec}]
    for r in range(1, max_rounds + 1):
        draft = strip_fences(model.complete(system=VERIFIER_GEN_SYSTEM,
                                            messages=gen_history))
        gen_history.append({"role": "assistant", "content": draft})
        verdict = run_tests(draft, TESTS)
        print(f"[{r}] verifier: {verdict.report.splitlines()[0][:72]}")
        if verdict.ok:
            return draft, True, r
        gen_history.append({"role": "user", "content": verdict.report})
    return draft, False, max_rounds

There is no reflection history here at all, because there is no critic to hold one: the critique is the process output.

Run the introspective loop and the verified loop on the same first draft — a parse_duration that only matches the 1h30m form:

### introspective critic (v3 machinery, code task)
[1] critic says: NO_FURTHER_OBJECTIONS
shipped code passes tests? False
TESTS FAILED
    return int(m.group(1)) * 3600 + int(m.group(2)) * 60
               ^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'

### verifier-driven loop, same first draft
[1] verifier: TESTS FAILED
[2] verifier: TESTS FAILED
[3] verifier: all 8 checks passed
verified=True after 3 round(s)

The scripted critic approving broken code is, again, scripted — it is the false positive Valmeekam et al. measured, reproduced so you can watch what it does to the control flow. What is not scripted is everything to the right of the verifier: those verdicts are real Python running real tests. The difference in what the two loops can do is therefore structural rather than stipulated — the introspective loop has no mechanism that could have caught AttributeError: 'NoneType', and the verified loop has none that could have shipped past it.

The quality of the feedback is the other half. Here is what the second draft — which fixed the pattern but let the empty string through — got back:

TESTS FAILED
  File "/tmp/tmpch0i6_ud/candidate.py", line 21, in <module>
    raise AssertionError(f"parse_duration({bad!r}) should have raised ValueError")
AssertionError: parse_duration('') should have raised ValueError

Compare that to a plausible LLM critique of the same code: “consider handling more edge cases.” One names the input, the expected behaviour and the actual behaviour; the other is a coin flip. This is the real reason external signals work better, and it is more mundane than the epistemology: verifier output is specific, and specific feedback drives specific revisions.

Ground truth is also cheap — a test run costs milliseconds and no tokens, an LLM critique costs a full model call — so the layering follows: run the free objective check first, and spend the critic call only on drafts that already passed it.

Measuring the delta

The number that tells you whether your loop is worth keeping is the difference between first-draft quality and final quality on a suite you did not tune against. Everything else is anecdote.

Here is the smallest harness that produces it: three scripted model behaviours — one that improves under feedback, one that gets stuck, one that starts correct and regresses — run through the same agent twice, once with the verifier and once without.

--- critic only -----------------------------------
  improves   first_draft_passes=False final_passes=False rounds=2  stalled: feedback repeated
  stuck      first_draft_passes=False final_passes=False rounds=2  stalled: draft repeated
  regresses  first_draft_passes=True  final_passes=False rounds=2  stalled: feedback repeated
  first-draft pass rate 1/3   final pass rate 0/3   delta -1

--- verifier + critic -----------------------------
  improves   first_draft_passes=False final_passes=True  rounds=3  budget: rounds exhausted
  stuck      first_draft_passes=False final_passes=False rounds=2  stalled: draft repeated
  regresses  first_draft_passes=True  final_passes=True  rounds=3  budget: rounds exhausted (returned last verified draft)
  first-draft pass rate 1/3   final pass rate 2/3   delta +1

Read the caveat first: the behaviours are ones I scripted, so these numbers measure my scripts, not any model. The harness is the deliverable, not the result.

What the harness makes visible is the thing a single run cannot: a negative delta is possible. The regresses row is the Huang et al. finding in miniature — a correct first draft, a critic that found something to say anyway, a revision that broke it — and in the critic-only column it costs you the one case you had. Without this table you would have seen three runs that each looked busy and productive, and concluded the loop was working.

The verifier column recovers that case through one small piece of engineering rather than better judgement: it remembers the last draft that passed and refuses to hand back a later one that did not. That is _finish in v5. A loop that can only ever return its final draft has no way to decline a regression, and given the numbers above, declining regressions may be the most valuable thing it does.

v5: assemble

The complete agent. Budget, trace, layered checking, stall detection, error handling.

class BudgetExceeded(RuntimeError):
    pass


class ModelUnavailable(RuntimeError):
    pass


@dataclass
class Round:             # one iteration, as recorded in the trace
    n: int
    draft: str
    feedback: str
    source: str          # "verifier" | "critic"
    accepted: bool
    seconds: float


@dataclass
class Check:             # what the checking layer concluded about one draft
    feedback: str
    source: str
    accept: bool
    passed_verifier: bool


@dataclass
class Result:
    output: str
    verified: bool
    stop_reason: str
    rounds: int
    trace: list[Round] = field(default_factory=list)


def _digest(text: str) -> str:
    return hashlib.sha1(" ".join(text.split()).encode()).hexdigest()[:12]


class ReflectionAgent:
    def __init__(self, model: Model, *, verifier: Verifier | None = None,
                 max_rounds: int = 4, max_calls: int = 12,
                 max_chars: int = 200_000, verbose: bool = True) -> None:
        self.model = model
        self.verifier = verifier
        self.max_rounds = max_rounds
        self.max_calls = max_calls
        self.max_chars = max_chars
        self.verbose = verbose
        self.calls = 0
        self.chars = 0

    def _log(self, *parts):
        if self.verbose:
            print(*parts)

    def _complete(self, system: str, messages: list[dict]) -> str:
        if self.calls >= self.max_calls:
            raise BudgetExceeded(f"call budget of {self.max_calls} exhausted")
        size = len(system) + sum(len(m["content"]) for m in messages)
        if self.chars + size > self.max_chars:
            raise BudgetExceeded(f"context budget of {self.max_chars} chars exhausted")
        last: Exception | None = None
        for attempt in (1, 2):
            self.calls += 1
            self.chars += size
            try:
                return self.model.complete(system=system, messages=messages)
            except Exception as exc:              # noqa: BLE001 — deliberate
                last = exc
                self._log(f"    model call failed ({type(exc).__name__}: {exc}); "
                          f"{'retrying' if attempt == 1 else 'giving up'}")
        raise ModelUnavailable(str(last))

    def _check(self, draft: str, reflect_history: list[dict],
               critic_system: str) -> Check:
        """Cheapest and most objective signal first."""
        if self.verifier is not None:
            try:
                verdict = self.verifier(draft)
            except Exception as exc:              # noqa: BLE001
                verdict = Verdict(False, f"VERIFIER ERROR: {type(exc).__name__}: {exc}")
            if not verdict.ok:
                return Check(verdict.report, "verifier", False, False)
        critique = self._complete(critic_system, reflect_history)
        return Check(critique, "critic", critique.strip() == STOP,
                     self.verifier is not None)

    def run(self, requirements: str, *,
            post_process: Callable[[str], str] = str.strip) -> Result:
        gen_history = [{"role": "user", "content": requirements}]
        reflect_history: list[dict] = []
        critic_system = CRITIC_SYSTEM.format(requirements=requirements)
        trace: list[Round] = []
        draft, best = "", None
        seen_drafts: set[str] = set()
        seen_feedback: set[str] = set()

        for n in range(1, self.max_rounds + 1):
            t0 = time.monotonic()
            try:
                draft = post_process(self._complete(GEN_SYSTEM, gen_history))
            except (BudgetExceeded, ModelUnavailable) as exc:
                return self._finish(draft, best, f"aborted: {exc}", n - 1, trace)

            gen_history.append({"role": "assistant", "content": draft})
            reflect_history.append({"role": "user", "content": draft})

            try:
                chk = self._check(draft, reflect_history, critic_system)
            except (BudgetExceeded, ModelUnavailable) as exc:
                return self._finish(draft, best, f"aborted: {exc}", n, trace)

            if chk.passed_verifier:
                best = draft
            trace.append(Round(n, draft, chk.feedback, chk.source, chk.accept,
                               time.monotonic() - t0))
            self._log(f"[{n}] draft {_digest(draft)} -> {chk.source}: "
                      f"{chk.feedback.splitlines()[0][:60]}")

            if chk.accept:
                return Result(draft, self.verifier is not None, "accepted", n, trace)
            if _digest(draft) in seen_drafts:
                return self._finish(draft, best, "stalled: draft repeated", n, trace)
            if _digest(chk.feedback) in seen_feedback:
                return self._finish(draft, best, "stalled: feedback repeated", n, trace)
            seen_drafts.add(_digest(draft))
            seen_feedback.add(_digest(chk.feedback))

            reflect_history.append({"role": "assistant", "content": chk.feedback})
            gen_history.append({"role": "user", "content": chk.feedback})

        return self._finish(draft, best, "budget: rounds exhausted",
                            self.max_rounds, trace)

    def _finish(self, draft, best, reason, n, trace) -> Result:
        """Never ship a regression: if an earlier draft passed the verifier and
        this one did not, hand back the one that passed."""
        if best is not None and best != draft:
            return Result(best, True, reason + " (returned last verified draft)",
                          n, trace)
        return Result(draft, best is not None and best == draft, reason, n, trace)

Six things in there are worth naming.

Three budgets, not one. max_rounds bounds iterations, max_calls bounds model invocations — not the same number, since a retry costs a call and not a round — and max_chars bounds context growth, which given the $O(n^2)$ accumulation is the one that actually protects you on a long run. A retry deliberately consumes budget: a flapping upstream should exhaust your allowance and stop, not retry forever inside a loop that thinks it is on round two.

Retries are two attempts, then abort. The bare except Exception is deliberate, for the same reason it was in the ReAct chapter. But unlike a tool failure, a model failure cannot be handed back to the model as an observation — there is nothing left to hand it to — so after the second attempt it becomes a ModelUnavailable and the run ends with whatever it had.

Verifier failures are observations. _check catches everything the verifier throws and turns it into VERIFIER ERROR: ..., which flows into the generator’s history like any other feedback. Your sandbox being down should degrade the loop, not crash it.

Stall detection on both sides. Whitespace-normalized digests of drafts and of feedback: an identical draft means the generator is not moving, identical feedback means the critic is not moving, and either way another round is pure spend. This is the exit that catches the v1 self-congratulation collapse and the oscillation case, and it costs eight lines.

_finish never ships a regression. The direct engineering response to the 8%-versus-1.5% number.

Everything lands in the trace. Each Round carries the draft, the feedback, which component produced it, whether it was accepted, and how long it took. In production this becomes a span rather than a print, but the fields do not change — and source is the field you will group by when someone asks whether the LLM critic is earning its call.

The entry point:

def build(offline: bool = True) -> ReflectionAgent:
    model = ScriptedModel(DRAFTS + [STOP]) if offline else AnthropicModel()
    return ReflectionAgent(model, verifier=lambda d: run_tests(d, TESTS))


if __name__ == "__main__":
    import os
    offline = not os.environ.get("ANTHROPIC_API_KEY")
    print(f"--- mode: {'offline scripted' if offline else 'live'} ---")
    result = build(offline).run(SPEC, post_process=strip_fences)
    print(f"\nverified={result.verified} rounds={result.rounds} "
          f"reason={result.stop_reason!r}")
    print(result.output)

$ python3 reflect.py

Actual output:

--- mode: offline scripted ---
[1] draft a80ec4a08992 -> verifier: TESTS FAILED
[2] draft ec1ac667f94b -> verifier: TESTS FAILED
[3] draft 3d1058f341c0 -> critic: NO_FURTHER_OBJECTIONS

verified=True rounds=3 reason='accepted'
import re

_PATTERN = re.compile(r"(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?")

def parse_duration(text):
    m = _PATTERN.fullmatch(text.strip())
    if m is None or not any(m.groups()):
        raise ValueError(f"cannot parse duration: {text!r}")
    h, mi, s = (int(g or 0) for g in m.groups())
    return h * 3600 + mi * 60 + s

Read the source column: verifier, verifier, critic. The critic was consulted exactly once, on the only draft that had already passed the tests, so two of the three rounds cost zero critic tokens. That is the layering doing its job.

And the remaining exits, each provoked with a scripted failure:

### stall detection
[1] draft a80ec4a08992 -> verifier: TESTS FAILED
[2] draft a80ec4a08992 -> verifier: TESTS FAILED
-> verified=False reason='stalled: draft repeated' rounds=2

### transient model failure, recovered
[1] draft a80ec4a08992 -> verifier: TESTS FAILED
    model call failed (ConnectionError: upstream 529); retrying
[2] draft ec1ac667f94b -> verifier: TESTS FAILED
[3] draft 3d1058f341c0 -> critic: NO_FURTHER_OBJECTIONS
-> verified=True reason='accepted'

### persistent model failure, aborted
    model call failed (ConnectionError: upstream 529); retrying
    model call failed (ConnectionError: upstream 529); giving up
-> verified=False reason='aborted: upstream 529'

### verifier itself explodes
[1] draft a80ec4a08992 -> verifier: VERIFIER ERROR: OSError: sandbox unavailable
[2] draft 3d1058f341c0 -> verifier: VERIFIER ERROR: OSError: sandbox unavailable
-> verified=False reason='stalled: feedback repeated'

### call budget
[1] draft 5177643b63c1 -> verifier: TESTS FAILED
[2] draft 654e2ba2c831 -> verifier: TESTS FAILED
[3] draft e365637d167f -> verifier: TESTS FAILED
-> verified=False reason='aborted: call budget of 3 exhausted'

Five failure classes, five clean exits, zero crashes, and in every case a stop_reason a human can act on. Note the fourth: a broken sandbox produced the same error twice, the feedback-stall detector noticed, and the loop stopped instead of burning its budget re-running a verifier that was never going to work. That was not designed for — it fell out of a generic stall check, which is the nice thing about generic stall checks.

Set ANTHROPIC_API_KEY and run it again. Same code path, real model, and the drafts will be its own.

What this version still gets wrong

An honest inventory, because the gap between this and production is the rest of the book.

The verifier defines correctness, and it is yours. verified=True means “passed eight assertions I wrote.” A draft that satisfies the tests and violates the spec passes. Everything the tests do not cover is unexamined, and the loop will happily optimize into exactly those gaps — the generator is being trained, within a run, on the signal you gave it. Your test suite is now part of your prompt, and it is the part with teeth.

Context grows quadratically and nothing compacts it. Both histories only ever append. max_chars stops the bleeding by aborting; it does not summarize, window, or drop the fossil record. The standard cure — carry the requirements, the latest draft, and the latest critique, drop the middle — is not implemented here, and if you implement it, remember the reference implementation’s bug: never let compaction touch the requirements.

No cost or latency budget. Rounds, calls and characters are bounded; dollars and seconds are not. There is no wall-clock timeout, so a slow model call hangs the agent, and the verifier’s own timeout only covers the subprocess.

The LLM critic is uncalibrated. We have no idea what its false-positive rate is. The fix is a planted-defect suite: drafts with known flaws that the critic must catch, run as a test, with its catch rate tracked over time. Without that, “the critic approved it” is a sentence with no known meaning — and Valmeekam et al.’s false positives are exactly what it would be measuring.

One critic, one axis. Real review is multi-dimensional — correctness, style, security, performance — and a single critic asked for all of them at once will pick whichever it noticed first. Separate critics per axis, run in parallel, with the orchestrator merging their findings, is the next step and it is not in this code.

Regression protection only covers what the verifier checks. _finish can decline a regression on the tests. It cannot detect that round three made the prose worse, because nothing measures the prose.

The stall detector is exact-match. Two drafts differing by one whitespace-normalized character are “different”. A near-duplicate loop sails past it. Real stall detection wants a similarity threshold, and choosing that threshold is an empirical question.

No cross-run learning. Every run starts blank. The critiques are the most valuable byproduct this system produces — a map of your generator’s recurring weaknesses — and we throw all of them away at the end of run(). Mining them and folding the recurring ones into the generator’s standing instructions is how the loop stops being a per-request tax and starts being a flywheel.

No evaluation beyond three scripted cases. The delta harness in v4 is the right shape and the wrong size. You need a real suite, held out, with the first-draft-versus-final delta computed on every change to the prompts, the critic, or the cap. Given the literature, this is not optional diligence — it is the only thing standing between you and a loop that quietly makes your output worse at three times the price.

What you should be able to do now

• Build a generate-critique-revise loop from scratch, with the generator and the critic as separate model calls with separate system prompts.
• Construct and maintain two mirrored message histories, role-swapping each side’s output into the other’s input, and explain what that buys and what it costs.
• Recognize the failure mode of a single shared history — one assistant persona doing both jobs, conditioned on its own prior verdicts — by reading the roles in a printed transcript.
• Implement a stop token correctly: strict equality on the stripped reply, never a substring check, always behind a hard cap the model cannot influence.
• Name the three exits — accepted, stalled, budget exhausted — and treat all three as normal operating conditions that return an honestly flagged result.
• State what the research actually supports: intrinsic self-correction is unreliable and can degrade correct answers; self-correction grounded in reliable external feedback works.
• Wire an external verifier into the loop as the primary signal, make its failures observations rather than exceptions, and layer the expensive LLM critic behind it.
• Measure the first-draft-versus-final delta on a suite, and understand that a negative delta is a real possible outcome that only measurement will reveal.
• Keep the last verified draft and refuse to ship a regression.

Further reading

• Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback (NeurIPS 2023): https://arxiv.org/abs/2303.17651
• Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024) — the negative result and the oracle-label critique: https://arxiv.org/abs/2310.01798
• Kamoi et al., When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (TACL 2024): https://aclanthology.org/2024.tacl-1.78/
• Stechly, Marquez and Kambhampati, GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems: https://arxiv.org/abs/2310.12397
• Valmeekam et al., Can Large Language Models Really Improve by Self-critiquing Their Own Plans?: https://arxiv.org/abs/2310.08118
• Gou et al., CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (ICLR 2024): https://arxiv.org/abs/2305.11738
• Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning (NeurIPS 2023): https://arxiv.org/abs/2303.11366
• Pan et al., Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Self-Correction Strategies — a taxonomy of training-time, generation-time and post-hoc correction: https://arxiv.org/abs/2308.03188
• Anthropic, “Building effective agents” — the evaluator-optimizer workflow is this pattern: https://www.anthropic.com/engineering/building-effective-agents
• Anthropic Messages API reference, for the system / messages shapes used above: https://platform.claude.com/docs/en/api/messages