Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Production System 2: a deterministic writing workflow

The last chapter built an agent that decides what to do.

This one builds a system that is not allowed to.

The task: turn a brief — a title, an audience, a handful of sourced facts — into a house-format technical document with prose, a diagram, and a working code example. Your team publishes forty of these a quarter. They must all look the same. The code in them must run. The numbers in them must be sourced. And when someone re-runs the pipeline on the same brief six months from now, it must produce the same document.

An agent is the wrong tool for this, and knowing why is the point of the chapter.

By the end you will have a workflow of about six hundred lines: an evaluator-optimizer loop with three real stopping conditions, an executable rubric, a typed-state graph, a standardized multi-media output where the code block is validated by running it, and an architecture where every seam is injectable. It runs offline, it is covered by twelve tests, and it produces byte-identical output across runs.

Setup:

mkdir -p writer && cd writer
python3 -m venv .venv && source .venv/bin/activate
pip install pytest            # langgraph only for the last section

Why not an agent

Start with the argument, because it is the reusable part.

An agent is the right shape when the path cannot be known in advance. The research agent in Chapter 1 did not know which sources existed, so it had to look, read, and decide what to do next based on what it found. No fixed pipeline could have expressed that.

The writing task is the opposite. The path is known: draft, check, fix what failed, check again, publish or hold. Every document takes that path. The only thing that varies is how many times the middle two steps repeat.

When the path is known, giving control to a model costs you four things and buys you nothing.

Reproducibility. Two runs of the same brief through an agent give two different documents. Two runs through this workflow give the same bytes, which means a diff in your repository is a real change and not sampling noise.

Auditability. When a document is wrong you want to know which step produced the error. A workflow has named steps. An agent has a trajectory, and “the model decided to skip the diagram” is not a defect you can fix.

Cost predictability. This workflow’s cost is bounded by its lap budget, exactly. An agent’s cost is bounded by whatever cap you set, and it will find a way to reach it.

Testability. Every node here is a function you can call with a fixture and assert on. That is why the test suite runs in a second and a half.

The rule of thumb, stated plainly: use an agent when the sequence of steps is data-dependent; use a workflow when only the number of repetitions is. An evaluator-optimizer loop is the smallest amount of “agentic” behaviour that buys you real quality — the system decides when to stop, and nothing else.

v1: contracts before behaviour

The whole system is built on one module that contains no logic at all.

writer/contracts.py:

@dataclass(frozen=True)
class Fact:
    """One input datum the document is allowed to assert. Nothing else is."""
    id: str
    text: str
    source: str


@dataclass(frozen=True)
class Brief:
    """The complete, explicit input. Two identical briefs must produce two
    identical documents — that is the contract this whole system exists to keep."""
    slug: str
    title: str
    audience: str
    facts: tuple[Fact, ...]
    max_words: int = 320
    require_diagram: bool = True
    require_code: bool = True


@dataclass(frozen=True)
class Document:
    """The artifact under construction: prose, one diagram, one code sample."""
    title: str
    summary: str
    sections: tuple[tuple[str, str], ...] = ()     # (heading, body)
    mermaid: str = ""
    code: str = ""
    code_expect: str = ""                          # expected stdout of `code`
    cited: tuple[str, ...] = ()                    # Fact ids referenced

Everything is frozen and everything is a tuple. A revision does not mutate a document; it returns a new one via dataclasses.replace. That is not stylistic fussiness — it is what lets the loop keep every lap’s document around, compare them, and publish the best one rather than the last one.

The critique side is where the design gets opinionated:

@dataclass(frozen=True)
class Issue:
    """One rubric violation, addressed to whoever must fix it."""
    check: str
    severity: str          # "blocker" | "major" | "minor"
    detail: str
    fix_hint: str = ""

    WEIGHTS = {"blocker": 1.0, "major": 0.4, "minor": 0.15}

An issue carries a fix_hint because a critique that says “the diagram is wrong” produces another wrong diagram. “Declare every node with a label before using it in an edge” produces a fixed one. This is the same principle as tool error messages in Part 2: the consumer of your error text is whoever has to act on it, and they need an instruction, not a diagnosis.

Severity is three levels with weights, not a 1-to-10 score. Ten-point scores from an LLM judge cluster on 7 and 8 and carry no information, as Part 5, Chapter 2 showed at length. Three levels with a clear definition each — blocker means unpublishable, major means a reviewer would send it back, minor means a nit — are things two people can agree on.

v2: the rubric, as executable checks

"""The rubric, as executable checks.

A rubric that lives in a prompt is a suggestion. A rubric that lives in functions
returning Issues is a gate. Every check here is deterministic: same document in,
same issues out, no model involved.
"""

Six checks. Structure, length, citations, diagram, code, audience. Two are worth reading in full.

Citations enforce the same rule Chapter 1 enforced, in a much cheaper way, because here the facts are given up front:

def check_citations(brief: Brief, doc: Document) -> list[Issue]:
    """Every number is sourced, and every source cited actually exists."""
    known = {f.id for f in brief.facts}
    issues = []
    prose = " ".join([doc.summary, *(b for _, b in doc.sections)])
    for sentence in re.split(r"(?<=[.!?])\s+", prose):
        if NUMBER.search(sentence) and not CITE.search(sentence):
            issues.append(Issue("citations", "blocker",
                                f"unsourced number: {sentence.strip()[:60]}",
                                "attach the [F#] of the fact this number came from"))
    for fid in CITE.findall(prose):
        if fid not in known:
            issues.append(Issue("citations", "blocker", f"cites unknown fact {fid}",
                                "cite only facts supplied in the brief"))
    return issues

Code is the check that justifies the whole architecture:

def run_python(code: str, *, timeout: float = 10.0) -> tuple[bool, str]:
    """Execute a snippet in a subprocess and capture stdout. No network, no input."""
    with tempfile.TemporaryDirectory() as tmp:
        path = Path(tmp) / "snippet.py"
        path.write_text(code)
        try:
            proc = subprocess.run([sys.executable, str(path)], capture_output=True,
                                  text=True, timeout=timeout, cwd=tmp)
        except subprocess.TimeoutExpired:
            return False, f"timed out after {timeout}s"
    if proc.returncode != 0:
        return False, proc.stderr.strip().splitlines()[-1] if proc.stderr else "non-zero exit"
    return True, proc.stdout.strip()


def check_code_runs(brief: Brief, doc: Document) -> list[Issue]:
    """The check that makes this workflow worth building: the code is executed."""
    if not doc.code.strip():
        return []
    ok, out = run_python(doc.code)
    if not ok:
        return [Issue("code", "blocker", f"snippet failed: {out}",
                      "fix the snippet until it exits 0")]
    if doc.code_expect and out != doc.code_expect.strip():
        return [Issue("code", "blocker",
                      f"stdout {out!r} does not match documented output {doc.code_expect.strip()!r}",
                      "make the code and the documented output agree")]
    return []

Two separate failures, and the second is the valuable one. Code that crashes is embarrassing but obvious. Code that runs and prints something different from what the article says it prints is the error that survives review and wastes a reader’s afternoon. Nothing but execution catches it.

A word on safety: this runs generated code. A subprocess with a timeout and a temp working directory is the right floor, and it is not a sandbox. If the generator is a model reading untrusted input, run this in a container with no network and a read-only filesystem — the machinery from Part 6, Chapter 7 is exactly what you want, pointed at a snippet instead of an agent.

The diagram check does structural validation of Mermaid without rendering it, and it is where this chapter earns its keep, because the first two versions of it were wrong.

Version one declared nodes only at the start of a line:

MERMAID_NODE = re.compile(r"^\s*([A-Za-z][\w]*)\s*[\[\(\{]")

Mermaid lets you declare a node inline anywhere — B[Brief] --> G[Generate draft] declares both — so a perfectly good diagram was reported as having undeclared nodes forever. Version two fixed that with findall over the whole line, and the same diagram still failed, because the edge regex could not cope with two things real Mermaid does constantly:

    B[Brief] --> G[Generate draft]      # source carries a label
    E -->|blockers| R[Revise]           # edge carries a label

The first did not match at all; the second matched with "blockers" as the target node. Version three:

MERMAID_EDGE = re.compile(r"""
    ^\s*([A-Za-z]\w*)                        # source id
    (?:\s*[\[\(\{][^\]\)\}]*[\]\)\}])?         # optional label on the source
    \s*[-.=]{1,3}[->.=]*>?\s*                # the arrow, in its several spellings
    (?:\|[^|]*\|\s*)?                        # optional edge label
    ([A-Za-z]\w*)                            # target id
""", re.X)

'    B[Brief] --> G[Generate draft]'          -> ('B', 'G')
'    G --> E[Evaluate]'                       -> ('G', 'E')
'    E -->|blockers| R[Revise]'               -> ('E', 'R')
'    R --> E'                                 -> ('R', 'E')
'  A-->B'                                     -> ('A', 'B')
'  A -.-> C{Choice}'                          -> ('A', 'C')
'  X === Y'                                   -> ('X', 'Y')
'  flowchart TD'                              -> None

The lesson is bigger than a regex. A validator that is wrong is worse than no validator, because the loop obediently spends its entire lap budget trying to satisfy a check that cannot be satisfied, and the run ends in “held for review” with a document that was fine. Validators need tests. There are two in the suite for this one — a dangling edge that must fail, and a labelled-edge diagram that must pass — and both exist because both were broken.

Note also what this check honestly does not do: it does not render. Rendering Mermaid needs the Mermaid CLI and a headless browser (https://github.com/mermaid-js/mermaid-cli), which is a fine thing to add in CI and a bad dependency for a unit test. Structural validation catches the failure that actually happens — an edge pointing at a node nobody declared, which renders as a mystery box.

v3: the loop

Generate, evaluate, revise. Twenty lines:

def optimize(brief: Brief, generator: Generator, *,
             convergence: Convergence | None = None,
             rubric: Sequence[Check] = RUBRIC,
             on_lap: Callable[[Lap], None] | None = None) -> Run:
    conv = convergence or Convergence()
    run = Run()
    doc = generator.draft(brief)
    while True:
        critique = evaluate(brief, doc, rubric)
        lap = Lap(n=len(run.laps) + 1, score=critique.score, critique=critique, doc=doc)
        run.laps.append(lap)
        if on_lap:
            on_lap(lap)
        verdict = conv.verdict(run.laps)
        if verdict:
            run.stopped = verdict
            return run
        doc = generator.revise(brief, doc, critique)

Both generator and rubric are parameters. The loop has no idea whether the generator is a model or a template, and no idea which checks exist. That is what “dependency injection” means in Python — passing the collaborator in rather than importing it — and it is the entire reason the tests can run a two-check rubric against a deliberately stubborn writer.

The generator is a Protocol with two methods:

class Generator(Protocol):
    def draft(self, brief: Brief) -> Document: ...
    def revise(self, brief: Brief, doc: Document, critique: Critique) -> Document: ...

The offline implementation, TemplateWriter, is deliberately imperfect. Its first draft has no diagram, unsourced numbers, and a code snippet that raises NameError. Its revise fixes one severity class per lap:

    def revise(self, brief: Brief, doc: Document, critique: Critique) -> Document:
        """Fix one severity class per lap, blockers first.

        Fixing a blocker often creates a lesser issue — adding a diagram creates a
        dangling edge, adding a section blows the word budget — so the evaluator has
        to run again after every class. That is why this is a loop and not a pipeline.
        """
        for severity in ("blocker", "major", "minor"):
            batch = [i for i in critique.issues if i.severity == severity]
            if batch:
                for issue in batch:
                    doc = self._apply(brief, doc, issue)
                return doc
        return doc

That docstring is the argument for the loop’s existence, and it is not hypothetical — you will watch it happen in the run below. Fixes interact. The diagram this writer adds when told “no diagram” contains a dangling edge; the second section it adds when told “too few sections” pushes the document over its word budget. A pipeline that generated once, critiqued once, and revised once would publish a document with a broken diagram and call it done.

v4: knowing when to stop

Three conditions, and getting them right took three attempts.

@dataclass
class Convergence:
    """When to stop. All three conditions are needed; any one alone misbehaves."""
    target: float = 1.0          # good enough to publish
    max_laps: int = 6            # hard ceiling on cost
    # a lap that reports exactly the issues the previous lap reported fixed nothing

    def verdict(self, laps: list[Lap]) -> str | None:
        last = laps[-1]
        if last.score >= self.target and not last.critique.blockers:
            return "converged"
        if len(laps) >= self.max_laps:
            return "lap-budget"
        if len(laps) >= 2 and last.critique.fingerprint == laps[-2].critique.fingerprint:
            return "no-progress"
        return None

Attempt one measured progress as a change in score, where score = max(0, 1 - penalty). The first run stopped after two laps with stopped=no-progress and published the draft:

  lap 1: score 0.00  issues=structure(major); structure(blocker); citations(blocker); citations(blocker); code(blocker)
  lap 2: score 0.00  issues=structure(major); diagram(major); diagram(minor); code(blocker)
  stopped=no-progress published=False

Look at those two lines. Lap 2 fixed three blockers. It made enormous progress. And the score did not move, because five weighted issues and four weighted issues both clamp to zero. The metric you report to humans and the metric you use for control are not the same metric, and conflating them here cost a run that was working fine.

Attempt two added an unclamped penalty and compared that instead. Better, and still wrong:

    @property
    def penalty(self) -> float:
        """Total weighted issue mass. Unclamped, so progress stays visible."""
        return sum(i.weight for i in self.issues)

A run with a two-check rubric traded one blocker for a different blocker — a snippet that crashed became a snippet whose output disagreed with the document — and the penalty was identical on both laps, so the loop declared no progress and quit one lap before it would have converged.

Attempt three compares the identity of the critique:

    @property
    def fingerprint(self) -> tuple:
        """Identity of a critique. Two laps with the same fingerprint made no progress,
        which is a stronger signal than a score that happens not to have moved."""
        return tuple(sorted((i.check, i.detail) for i in self.issues))

Different issues means something changed, even when the arithmetic did not move. Identical issues means the generator is stuck, and one more lap will produce the same result at the same price.

The three conditions catch three different things, and you need all three: target is success, max_laps is cost control, fingerprint is a stuck generator. Drop the fingerprint check and a stubborn model burns the full budget every time. Drop max_laps and a generator that oscillates between two flawed versions runs forever.

v5: the graph

The loop is one node in a slightly larger workflow, because “we produced a document” is not the same as “we published one”.

The engine is Part 4, Chapter 2’s, cut to what this needs — static edges, conditional edges, typed state, a superstep cap:

@dataclass
class State:
    """Typed workflow state. Nodes return partial updates; the engine merges."""
    brief: Brief
    doc: Document | None = None
    critique: Critique | None = None
    run: Run | None = None
    artifact: str = ""
    published: bool = False
    log: list[str] = field(default_factory=list)

log is the one reduced field: nodes return {"log": ["render"]} and the engine extends rather than overwrites. Everything else is last-write-wins, which is safe here because no two nodes ever write the same field.

The composition root wires four nodes:

        graph = Graph()
        graph.add_node("optimize", node_optimize)
        graph.add_node("render", node_render)
        graph.add_node("verify", node_verify)
        graph.add_node("quarantine", node_quarantine)
        graph.set_entry("optimize")
        graph.add_conditional_edge("optimize", lambda s: "render")
        graph.add_edge("render", "verify")
        graph.add_conditional_edge(
            "verify", lambda s: Graph.END if s.run.stopped == "converged" else "quarantine")
        graph.add_edge("quarantine", Graph.END)

quarantine is the node that makes this a production system rather than a demo. A document that did not converge is not thrown away and not published: it is stamped with a banner naming exactly what failed, and handed to a human.

        def node_quarantine(state: State) -> dict:
            return {"published": False,
                    "artifact": ("> **HELD FOR REVIEW** — this document did not clear the "
                                 f"rubric ({state.critique.render()}).\n\n" + state.artifact),
                    "log": ["quarantine"]}

The verify node in front of it re-checks the rendered file, not the object graph:

def verify_artifact(markdown: str) -> list[str]:
    """Re-check the rendered file, not the object graph.

    Everything upstream can be right and the renderer can still emit a broken
    document. This runs the code block as it appears in the published file.
    """

It pulls the python block out of the markdown, runs it, and compares against the text block that follows. That is a different assertion from check_code_runs, and it is the one that catches a renderer that drops a line, mangles indentation, or emits the wrong expected-output block. Check the artifact you ship, not the object you meant to ship.

Running it

$ python3 run_writer.py

run 1
  lap 1: score 0.00  issues=structure(major); structure(blocker); citations(blocker); citations(blocker); code(blocker)
  lap 2: score 0.00  issues=structure(major); diagram(major); code(blocker)
  lap 3: score 0.20  issues=structure(major); diagram(major)
  lap 4: score 0.60  issues=length(major)
  lap 5: score 1.00  issues=clean
  stopped=converged published=True

Read that trajectory as a story.

Lap 1: no diagram, two unsourced numbers, and a snippet that crashes. Five issues. Lap 2: the blockers are fixed — and fixing them created the diagram issue, because the diagram the writer added has an edge to an undeclared node. The code is still blocking, now for a different reason: it runs, but its output disagrees with the documented output. Lap 3: code agrees with the document. Two majors left. Lap 4: sections and diagram fixed. Adding the second section pushed the document to 121 words against a 115-word budget, so a new major appears. Lap 5: trimmed. Clean. Published.

Nothing about that sequence was scripted. Each lap’s action is a function of the critique the evaluator produced, and the critique is a function of the document. Change the word budget and the trajectory changes.

The document it publishes, in full:

# The evaluator-optimizer loop

The evaluator-optimizer loop explains how a constrained writing workflow turns a brief into a reviewed document. The loop generates, evaluates against a rubric, and revises until it converges or runs out of laps.

## How it works

Every lap costs one model call, and the median document converges in 3 laps [F1].

## What it costs

A lap is one generate call plus the evaluators, which run locally and cost nothing. The rubric checks execute the code sample in a subprocess, so a document that claims an output the code does not produce cannot pass.

## Diagram

```mermaid
flowchart TD
    B[Brief] --> G[Generate draft]
    G --> E[Evaluate against rubric]
    E -->|blockers| R[Revise]
    R --> E
    E -->|clean or budget spent| P[Publish]
```

## Example

```python
laps = [0.62, 0.81, 0.94]
for n, score in enumerate(laps, start=1):
    print(f"lap {n}: score {score:.2f}")
print("converged" if laps[-1] >= 0.9 else "lap budget exhausted")
```

Output:

```text
lap 1: score 0.62
lap 2: score 0.81
lap 3: score 0.94
converged
```

## Sources

- **[F1]** The median document converges in 3 laps. — internal telemetry, 2026-06
- **[F2]** 9 percent of briefs hit the lap budget and go to a human. — internal telemetry, 2026-06  _(not used)_
- **[F3]** Rubric checks run locally in under two seconds. — benchmark, 2026-06  _(not used)_

## Build record

- laps: 5 (converged)
- score: 1.00
- words: 87 / 115

Three things that document does that a hand-written one usually does not.

The output block is true, because it was produced by running the code block during the last lap.

The unused facts are marked, and they are computed from the finished prose rather than from what the generator intended:

    # Which facts the finished prose actually cites — not which ones we intended to.
    used = set(CITE.findall(" ".join([doc.summary, *(b for _, b in doc.sections)])))

That distinction matters here, because the length trim in lap 5 deleted the sentence that cited F2. An intent-based marker would have kept claiming F2 was used. A prose-based one tells the truth: the brief supplied three facts and the finished document needed one.

The build record is in the artifact, so anyone reading the file knows it took five laps and cleared the rubric — provenance for the process, in the same spirit as Chapter 1’s provenance for the claims.

And the promise:

run 2 (same brief, same process)
  sha(run1)=a6a04dc20852270c
  sha(run2)=a6a04dc20852270c
  identical=True

The three ways it stops

$ python3 stop_demos.py

converged      stopped=converged    laps=5  scores: 0.00 -> 0.00 -> 0.20 -> 0.60 -> 1.00
lap-budget     stopped=lap-budget   laps=3  scores: 0.00 -> 0.00 -> 0.20
no-progress    stopped=no-progress  laps=4  scores: 0.00 -> 0.00 -> 0.00 -> 0.00

A stubborn run is held, not published:
  published=False  log=['optimize:no-progress:4 laps', 'render', 'verify:ok', 'quarantine']
  > **HELD FOR REVIEW** — this document did not clear the rubric (structure(major); structure(blocker)).

The third case uses a StubbornWriter — a subclass that fixes everything except the diagram, standing in for the model that cannot see its own bug:

class StubbornWriter(TemplateWriter):
    """Fixes everything except the diagram — the model that cannot see its own bug."""

    def _apply(self, brief, doc, issue):
        if issue.check in {"structure", "diagram"} and "diagram" in issue.detail:
            return doc
        return super()._apply(brief, doc, issue)

Six lines of subclass, and it exercises the failure mode that costs the most money in production. Test the stuck case explicitly; it is the one that quietly triples your bill.

Tests

$ python3 -m pytest test_writer.py -q
............                                                             [100%]
12 passed in 1.48s

The suite splits three ways, and the split is the architecture.

Checks are pure functions, tested with hand-built documents:

def test_code_whose_output_disagrees_with_the_document_is_a_blocker():
    doc = replace(DOC, code="print('a')", code_expect="b")
    assert "does not match" in check_code_runs(BRIEF, doc)[0].detail


def test_valid_mermaid_with_edge_labels_passes():
    doc = replace(DOC, mermaid=("flowchart TD\n  A[Start] --> B[Work]\n"
                                "  B -->|ok| C[Done]\n  B -.->|retry| A\n"))
    assert check_diagram(BRIEF, doc) == []

The loop is tested through its stopping conditions and its invariants:

def test_every_lap_weakly_improves():
    run = optimize(BRIEF, TemplateWriter(), convergence=Convergence(max_laps=8))
    penalties = [lap.critique.penalty for lap in run.laps]
    assert penalties == sorted(penalties, reverse=True)

That is a property test, and it is the most valuable assertion in the file: a revision must never make a document worse. It would have caught the diagram-regex bug immediately if it had existed at the time, because the penalty stopped falling.

The system is tested end to end for the promise it makes:

def test_same_brief_gives_byte_identical_output():
    a = Workflow(TemplateWriter(), verbose=False).run(BRIEF).artifact
    b = Workflow(TemplateWriter(), verbose=False).run(BRIEF).artifact
    assert a == b

Add a set iteration anywhere in the render path and this test fails. That is exactly what it is for.

The same thing on LangGraph

Nothing above depends on the hand-rolled engine. Porting to LangGraph 1.2 is fifty lines and no rewrite of the logic:

class WriterState(TypedDict):
    brief: Brief
    doc: Document | None
    run: Run | None
    artifact: str
    published: bool
    log: Annotated[list[str], operator.add]


def build():
    g = StateGraph(WriterState)
    g.add_node("optimize", node_optimize)
    g.add_node("render", node_render)
    g.add_node("verify", node_verify)
    g.add_node("quarantine", node_quarantine)
    g.add_edge(START, "optimize")
    g.add_edge("optimize", "render")
    g.add_edge("render", "verify")
    g.add_conditional_edges("verify", route, {END: END, "quarantine": "quarantine"})
    g.add_edge("quarantine", END)
    return g.compile()

$ python3 langgraph_port.py
log       : ['optimize:converged', 'render', 'verify:ok']
published : True
laps      : 5 converged
first line: # The evaluator-optimizer loop

Annotated[list[str], operator.add] is LangGraph’s reducer syntax — the same idea as the hand-rolled engine’s special case for log, declared on the type instead of hidden in the merge function (https://docs.langchain.com/oss/python/langgraph/use-graph-api).

Take the framework when you want what the framework adds: checkpointing so a run can resume after a crash, interrupt() so a human can approve a document mid-flight days later, streaming so a UI can show lap-by-lap progress. Do not take it for the graph. You can write the graph.

Going live

Swap TemplateWriter for a ModelWriter behind the same Protocol:

class ModelWriter:
    """The live seam: same interface, a real model behind it.

    The rubric, the loop, and the artifact checks do not change — only who writes
    the words. That is the point of keeping generation behind a Protocol.
    """

Four things to get right when you do.

temperature=0. You are not going to get bit-identical output from a model, but you should get close, and you should treat any variation as a defect to investigate rather than as weather.

Ask for JSON matching Document. Parse it, construct the frozen dataclass, and let a parse failure be a blocker issue like any other. A model that cannot produce the shape gets told so and tries again — that is a lap, and laps are the mechanism you already have.

Put the rubric in the revision prompt, and keep enforcing it in code. Sending the critique’s detail and fix_hint lines to the model is what makes revision converge in three laps instead of eight. It is not what makes the document correct. The checks are.

Budget in currency, not laps. Five laps of a long document is a real amount of money. Track tokens per lap, and lower max_laps for briefs where the marginal lap is not worth it.

What this system still gets wrong

The rubric only measures the measurable. Nothing here can tell whether the document is good. It can tell you the code runs, the numbers are sourced, and the shape is right — the floor, not the ceiling. Adding an LLM judge for prose quality is reasonable; keep it advisory, and never let a nondeterministic check gate a deterministic pipeline.

One diagram, one code block. The Document shape is deliberately rigid. Real houses need multiple figures, tables, and footnotes, and each addition costs a check.

No incremental publishing. A brief that fails goes to a human whole. Real editorial workflows want “publish the three sections that passed, hold the one that did not.”

The fingerprint check can stop early. A generator making genuine progress that happens to produce an identically-worded issue twice will be cut off. Including a document hash in the fingerprint would fix it and would also mask a generator that thrashes between two documents; pick your failure.

Determinism is only as strong as your weakest dependency. This pipeline is deterministic because nothing in it is random and nothing in it is a model. Introduce either and you are back to arguing about seeds.

What you should be able to do now

  • State the rule for choosing a workflow over an agent — data-dependent path versus data-dependent repetition count — and defend it with the four properties you lose when you hand over control.
  • Write a rubric as executable checks that return structured issues with severity and a fix hint, rather than as a paragraph in a prompt.
  • Validate generated code by executing it and comparing its real output to the output the document claims, and know why that second comparison is the valuable one.
  • Explain why a validator needs its own tests, and recognise the failure signature of a broken one: a loop that spends its whole budget and quarantines a fine document.
  • Build an evaluator-optimizer loop with three independent stopping conditions, and say which failure each one catches.
  • Separate the score you report to humans from the signal you use for control, and use critique identity rather than score deltas to detect a stuck generator.
  • Inject the generator and the rubric so the same loop can be tested with a stub writer and a two-check rubric.
  • Re-verify the rendered artifact rather than the object graph, and hold a non-converged document for review instead of publishing or discarding it.
  • Port a hand-rolled graph to LangGraph without touching the domain logic, and articulate what the framework is actually buying you.

Further reading