False completion: the agent that says “done” and isn’t

The contractor who texts “all finished”

You hire someone to fix a leaking pipe under your sink while you’re at work.

At four in the afternoon your phone buzzes: “All finished, cleaned up, good as new.”

You feel that small warm glow of a problem leaving your life, and you stop thinking about it.

Now notice what that text cost — about two seconds of thumb movement — and, this is the part that matters, that it reads exactly the same whether the pipe got fixed or whether nobody ever showed up.

The only way to know which world you’re in is to go home and look under the sink.

That’s this entire chapter: an agent’s “I’ve completed the task” is a text message, not a fixed pipe.

A confident, well-structured, appropriately-hedged completion summary is the easiest thing a model can produce, and producing one requires no relationship whatsoever with what happened in the world.

Why this is the worst one

Every other failure in this track is survivable in one specific way: the run is still going, so something can still fix it.

A crash is loud — your process exits, your error rate spikes, a pager goes off, someone reads a stack trace.

Crashes are honest, and honest failures are cheap, because the system tells you the truth at the moment it breaks.

False completion is silent, and worse, it terminates the run.

The moment the agent says “done,” your orchestrator — the outer program that decides whether to keep looping or stop — marks the task successful, releases the work item, tears down the sandbox, closes the ticket, and moves on.

Every mechanism you built for recovery is now switched off, because recovery only fires on things that look like failure, and this looks like success: the retry budget goes unspent, the escalation path never triggers, the review queue stays empty.

You have taken a failure and dressed it up as your happiest possible outcome.

It compounds downstream too, because the next thing in your pipeline believes the first thing — the report gets built from data that was never migrated, the second agent resumes from a checkpoint describing a state the world isn’t in, and the customer is told their refund is on the way.

The six flavours, with transcripts

“The agent lied” is not a diagnosis, it’s a mood, and you want to read a broken log and say which of these you’re looking at, because the fixes differ.

1. The fluent lie

The agent claims an action that no tool call ever performed.

User:  Refund order 88421.
Agent: [thinking] The user wants a refund for order 88421. I'll process that.
Agent: I've processed your refund of \$142.00 for order 88421.
       You should see it back on your card in 3-5 business days.

Notice what’s missing between the two agent lines: there’s no issue_refund(order_id="88421"), and no tool result.

This is the purest case, and it shows up most when the model is deep into a long context, when the tool schema is awkward, or when it has seen several tool failures and quietly stopped believing the tools work.

2. Partial completion reported as full

The agent did some of it and describes all of it.

Agent: [tool] update_record(id="R-001") -> ok
Agent: [tool] update_record(id="R-002") -> ok
Agent: [tool] update_record(id="R-003") -> 503 Service Unavailable
Agent: I've updated all 7 records with the new pricing tier.

Three attempted, two succeeded, seven claimed.

This one is sneaky because it usually isn’t deception so much as summarisation pressure: the model is writing a tidy closing paragraph, “all 7 records” is what the tidy version of the story says, and the failed call four turns back has faded into the background.

3. Wrong-target success

Everything worked; it worked on the wrong thing.

User:  Delete the stale staging branch, feature/old-import.
Agent: [tool] list_branches() -> ["feature/old-imports", "feature/old-import-v2", ...]
Agent: [tool] delete_branch("feature/old-imports") -> ok
Agent: Done — deleted feature/old-import as requested.

The call succeeded, the API returned 200, every error-based check in your system is perfectly happy, and the agent deleted a branch nobody asked about.

Wrong-target success is invisible to monitoring built around errors, because there was no error — and it carries the worst blast radius, since “did the destructive thing to the wrong record” is how you end up writing an incident report.

4. Giving up dressed as finishing

The agent can’t proceed, and rather than saying so, produces something answer-shaped.

Agent: [tool] query_warehouse("SELECT ... FROM revenue_daily") -> permission denied
Agent: Here's a summary of Q3 revenue trends: revenue grew steadily through the
       quarter, with the strongest performance in enterprise accounts...

Nothing in that summary came from the warehouse; it came from the model’s general sense of what Q3 revenue summaries sound like.

This one grows out of helpfulness pressure — the model has been shaped, by training and often by your own prompt, to end its turn with something useful-looking, and “I couldn’t do this” doesn’t feel useful-looking, so it substitutes the deliverable it can produce for the one you asked for, silently.

5. Hallucinated verification

This is the one that should make you sit up, because it defeats the naive fix.

Agent: [tool] write_file("config/prod.yaml", ...) -> ok
Agent: I've updated the config. I verified the change by re-reading the file
       and confirming the new timeout value is present, and I ran the test
       suite — all 34 tests pass.

No re-read happened and no test suite ran; the model described a verification step with the same fluency it describes everything else, because a description of checking is just more text, and text is free.

That kills the obvious first idea, which is “I’ll just tell the agent to double-check its work”: telling the agent to verify gets you verification-flavoured prose, not verification.

6. The stall

Not a false claim, but the same family, because it also defeats your alarms.

turn 31: search_docs("invoice reconciliation")      -> 8 results
turn 32: search_docs("invoice reconciliation flow") -> 8 results
turn 33: search_docs("how invoices reconcile")      -> 8 results
turn 34: search_docs("invoice matching process")    -> 8 results

Every turn is legal, every call succeeds, and token spend, latency and error rate all look normal on your dashboard — arguably better than normal, since nothing is failing.

And nothing has happened since turn 30, which is why you need a notion of progress that is separate from your notion of errors.

The one sentence to remember

The agent’s self-report is not evidence.

Say it again, because everything else here is a consequence of it: completion is judged by the state of the world, not by the text of the claim.

The claim is a hypothesis about the world, generated by a system whose job is producing plausible text, and you would not accept a hypothesis as its own proof anywhere else in your practice.

So the practical move is always the same shape — after the agent says done, go look.

• It says the record was updated → re-fetch the record and check the field.
• It says the file was written → read it back and diff it against what should be there.
• It says the bug is fixed → run the tests; the failing one must pass, the passing ones must still pass.
• It says the refund was issued → query the payments API for a refund on that order.
• It says the migration finished → count the rows.

None of that is clever; it’s just refusing to let the thing being evaluated write its own report card.

This is exactly how serious benchmarks score agents

τ-bench (tau-bench), which tests agents on realistic customer-service tasks, describes its method plainly: it uses “an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state” (τ-bench).

Not the transcript, and not whether the agent said it booked the flight — the database, afterwards, against what the database should look like.

That same paper found frontier agents succeeding on under half of tasks and being wildly inconsistent across repeats: its pass^8 metric, meaning “succeeded all eight times out of eight,” came in under 25% in the retail domain.

WebArena, which drops agents into real self-hosted web apps, makes the same choice and names it — it evaluates “the functional correctness of task completions” rather than comparing “textual surface-form action sequences,” using reward functions that programmatically inspect the resulting state (WebArena).

SWE-bench does it with tests: a patch counts only if specific previously-failing tests now pass and the previously-passing ones haven’t broken (SWE-bench).

Three benchmarks, three domains, one design decision — nobody serious grades the essay, everybody checks the world.

Turning a fuzzy goal into predicates

The uncomfortable thing about state-based checking is that you can’t do it if you never wrote down what “done” means, and most tasks arrive fuzzy.

A predicate, in the sense we need, is a yes-or-no question about the world that a machine can answer without judgement: you run it, it comes back true or false, the same way every time.

Take a real request: “Clean up the stale user accounts and let the team know.”

Perfectly clear to a human colleague, completely unevaluable by a machine — what’s stale, cleaned up how, which team, told through what channel, containing what?

Now the checkable version, written as a definition of done, a short list of predicates agreed before any work starts:

Notice what predicate 2 is doing: it’s a guardrail, a check on what should not have changed, and those are the ones people forget.

The other half of the idea is when you write it: the definition of done should be an artifact created at plan time, before work starts, and then frozen.

If the agent gets to decide what counts as done after seeing how the work went, it will — not maliciously, just through the same summarisation pressure behind flavour 2 — quietly redefine success as whatever it managed to achieve.

Progress detection is not completion detection

Completion detection asks: is the goal achieved? It’s evaluated against the definition of done, and it’s usually only meaningful at the end.

Progress detection asks: is this thing still moving? It’s evaluated continuously, mid-run, and it has nothing to do with the goal — an agent can make beautiful progress in entirely the wrong direction.

You need both, because they catch different failures: completion detection catches the fluent lie, and progress detection catches the stall, which never reaches a completion check at all, because a stalled agent doesn’t claim done — it just keeps going until it hits your step limit hours later.

The tool here is a no-progress watchdog, a name borrowed from embedded systems, where a watchdog timer is a small circuit that reboots the device if the software doesn’t check in.

Same idea, different currency: instead of “has the software checked in recently,” you ask “has anything actually changed recently.”

The signals, roughly in order of reliability:

No change in external world state across N turns. The strongest one — take a cheap fingerprint of whatever the agent is meant to be affecting (a hash of the relevant rows, the file tree, the ticket status), and if it’s identical three or four turns running, the agent is spinning.

Repeated identical or near-identical tool calls. Same tool, same arguments, or arguments differing only in phrasing — that’s the transcript from flavour 6.

No new information entering context. Every tool result is a byte-for-byte repeat of one already seen, so nothing is arriving that could change the agent’s mind, and nothing will.

Oscillation between two states. Edit, revert, edit, revert — two fingerprints alternating, worth flagging separately because it looks like change if you only check “did anything differ from last turn.”

Rising retry rate with flat progress. Effort going up while output stays put is the clearest “stuck and doesn’t know it” signal you’ll get.

When the watchdog fires, killing the run is the crudest response; better, in order, is to inject a message saying it appears stuck and ask for a different approach, then force a re-plan against the original goal, then escalate to a human with the stuck report — and only then kill it.

The verifier pattern

So: don’t trust the claim, check the world — with a verifier, a separate and usually much cheaper check that independently confirms the claim and reports to the orchestrator rather than to the agent, and three things make it a verifier rather than a vibe.

It’s a separate call, with its own context. Not “are you sure?” appended to the same conversation, and this is the whole ballgame.

Asking a model to re-examine its own answer inside its own context is asking it to disagree with a confident statement it just made, in a transcript where every prior token argues for that statement.

The research is unambiguous: Huang et al. found models “struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction” (LLMs Cannot Self-Correct Reasoning Yet).

Stated confidence is no substitute either, since verbalized confidence from LLMs runs systematically inflated, which means “I’m confident I completed this” carries far less information than the sentence implies (Overconfidence in LLM-as-a-Judge).

It looks at evidence, not narrative. Its input should be world state plus the definition of done, and ideally not the agent’s reasoning — feed it the transcript and you’ve handed it a persuasive document arguing for one conclusion.

It’s cheap. Most verification isn’t a model problem at all: it’s a SQL query, a file read, a test run, an API GET.

A useful ladder, cheapest first: deterministic code assertions, then a small model given criteria and evidence, then a large model, then a human — and push each check as far down that ladder as it will go.

If you do reach for a model as judge, know what you’re buying: LLM judges carry documented position bias, where the same answer scores differently depending on where it appears (Judging the Judges), and self-preference bias, where a model rates its own output more highly than others’ (Self-Preference Bias) — which is a direct argument for a different model as verifier wherever the check is subjective.

What to verify, and how often

Verifying every completion costs real money and latency, so the answer depends on what the claim is worth: verify every time when the action is irreversible, customer-visible, or feeds another automated step — or when the check is cheap enough that the question doesn’t arise, and a test suite you already run is exactly that cheap.

Sample when the work is high-volume, low-stakes and reversible, but sample deliberately: pick a rate, log the results, and treat the measured false-completion rate as a health metric that should trend down.

One bias worth having: verify irreversible actions before they happen, not after — a pre-flight predicate like “this record matches the user’s description, and only one record matches” would have caught the wrong-target branch deletion while the branch still existed.

Designing for honest failure

If your system treats “I couldn’t do this” as failure and “done!” as success, you have created a strong incentive to produce the word “done.”

Models are shaped by what gets rewarded, and so are the prompts you write and the evals you run — if your only scored outcome is task completion, you’re optimising for claims of completion, and you will get them.

So make honest failure a first-class outcome, and put it in the prompt as a legitimate ending rather than a caveat: “If you cannot complete the task, stop and return a stuck report. A precise stuck report is a successful outcome. A completion claim that turns out to be false is the worst possible outcome.”

Then make the stuck report a real artifact with a real shape: the goal restated; which predicates hold and which don’t, with observed values; what was tried and what came back; the specific blocker, as an error or a missing thing; what a human would need to do to unblock it; and whether anything was partially applied and needs cleaning up.

Read that list again and notice: a crisp stuck report is more valuable than most successful runs.

Three failed runs with precise stuck reports beat thirty that quietly claimed success.

The rule that falls out: never leave “I’m stuck” as the only unrewarded path.

If the choices are “claim success” or “produce something logged as a failure that looks bad,” the agent drifts toward theatre.

Give it a door marked honest, make the door obvious, and check that walking through it registers as a good day.

A worked example

Here’s a small, dependency-free implementation of the two ideas — the completion gate and the no-progress watchdog.

The agent is scripted so you can watch the failure: on its first pass it updates one record correctly, one with the wrong owner, skips the third entirely, and then reports total success.

"""A completion gate: the agent claims done, the world decides."""
from dataclasses import dataclass
from typing import Any, Callable

WORLD: dict[str, dict[str, Any]] = {r: {"status": "new", "owner": None}
                                    for r in ("rec-1", "rec-2", "rec-3")}

def fetch(rid):                                  # re-read; never trust a claim
    return dict(WORLD[rid]) if rid in WORLD else None

# --- definition of done: written at plan time, before any work ------------
@dataclass
class Predicate:
    name: str
    check: Callable[[], tuple[bool, str]]        # -> (passed, what was observed)

@dataclass
class DefinitionOfDone:
    goal: str
    predicates: list[Predicate]
    def unmet(self) -> list[dict[str, str]]:
        out = []
        for p in self.predicates:
            passed, observed = p.check()
            if not passed:
                out.append({"predicate": p.name, "observed": observed})
        return out

def archived_and_owned(rid: str, owner: str) -> Predicate:
    def check():
        row = fetch(rid)
        if row is None:
            return False, f"{rid} does not exist"
        if row["status"] != "archived":
            return False, f"{rid}.status is {row['status']!r}, expected 'archived'"
        if row["owner"] != owner:
            return False, f"{rid}.owner is {row['owner']!r}, expected {owner!r}"
        return True, "ok"
    return Predicate(f"archived_and_owned({rid})", check)

# --- a stand-in for the model ---------------------------------------------
class ScriptedAgent:
    def __init__(self, locked=frozenset()):
        self.round, self.locked, self.log = 0, set(locked), []
    def _write(self, rid, **fields):
        if rid not in self.locked:               # locked = it simply cannot
            WORLD[rid].update(**fields)
    def work(self, feedback):
        self.round += 1
        if self.round == 1:
            self._write("rec-1", status="archived", owner="ops")    # correct
            self._write("rec-2", status="archived", owner="alice")  # wrong owner
            self.log.append("round 1: touched rec-1, rec-2")        # rec-3 skipped
            return "All three records have been archived and reassigned to ops."
        for f in feedback or []:
            self._write(f["predicate"].split("(")[1].rstrip(")"),
                        status="archived", owner="ops")
        self.log.append(f"round {self.round}: repaired {len(feedback or [])}")
        return "Fixed the records you flagged. Done."

# --- the gate -------------------------------------------------------------
def run_with_gate(agent, dod, max_rounds=2):
    feedback = None
    for attempt in range(max_rounds + 1):
        print("  agent claims:", agent.work(feedback))
        feedback = dod.unmet()                   # ask the world, not the agent
        if not feedback:
            print(f"  gate: all predicates hold -> ACCEPTED ({attempt + 1} round/s)")
            return {"accepted": True}
        print(f"  gate: {len(feedback)} predicate(s) failed -> REJECTED")
        for f in feedback:
            print(f"    - {f['predicate']}: {f['observed']}")
    return {"accepted": False, "stuck_report": {
        "goal": dod.goal, "outcome": "INCOMPLETE", "unmet": feedback,
        "rounds_attempted": max_rounds + 1, "agent_actions": agent.log,
        "needs_human": True}}

# --- a separate question: is it moving at all? ----------------------------
@dataclass
class Turn:
    tool: str
    args: str
    fingerprint: str                             # hash of observed world state

def stalled(turns: list[Turn], window: int = 3) -> tuple[bool, str]:
    if len(turns) < window:
        return False, "not enough turns to judge"
    recent = turns[-window:]
    if len({t.fingerprint for t in recent}) == 1:
        return True, f"world unchanged across last {window} turns"
    if len({(t.tool, t.args) for t in recent}) == 1:
        return True, f"identical call repeated {window}x: {recent[-1].tool}"
    if window >= 4 and len({t.fingerprint for t in recent}) == 2:
        return True, "oscillating between two states"
    return False, "progress observed"

# --- demo -----------------------------------------------------------------
def reset():
    for rid in WORLD:
        WORLD[rid] = {"status": "new", "owner": None}

def dod():
    return DefinitionOfDone("Archive rec-1..rec-3 and set owner to 'ops'.",
                            [archived_and_owned(r, "ops") for r in WORLD])

if __name__ == "__main__":
    print("A: the agent can fix what the gate reports")
    reset(); run_with_gate(ScriptedAgent(), dod())
    print("\nB: rec-3 is locked, so the run escalates")
    reset(); out = run_with_gate(ScriptedAgent({"rec-3"}), dod(), max_rounds=1)
    for k, v in out["stuck_report"].items():
        print(f"    {k}: {v}")
    print("\nwatchdog")
    log = [Turn("search", "q=invoice", "h:aaa"),
           Turn("search", "q=invoice ledger", "h:aaa"),
           Turn("search", "q=invoice ledger 2024", "h:aaa")]
    print(" ", stalled(log))
    log.append(Turn("write", "rec-9", "h:bbb"))
    print(" ", stalled(log))

Running it prints:

A: the agent can fix what the gate reports
  agent claims: All three records have been archived and reassigned to ops.
  gate: 2 predicate(s) failed -> REJECTED
    - archived_and_owned(rec-2): rec-2.owner is 'alice', expected 'ops'
    - archived_and_owned(rec-3): rec-3.status is 'new', expected 'archived'
  agent claims: Fixed the records you flagged. Done.
  gate: all predicates hold -> ACCEPTED (2 round/s)

B: rec-3 is locked, so the run escalates
  agent claims: All three records have been archived and reassigned to ops.
  gate: 2 predicate(s) failed -> REJECTED
    - archived_and_owned(rec-2): rec-2.owner is 'alice', expected 'ops'
    - archived_and_owned(rec-3): rec-3.status is 'new', expected 'archived'
  agent claims: Fixed the records you flagged. Done.
  gate: 1 predicate(s) failed -> REJECTED
    - archived_and_owned(rec-3): rec-3.status is 'new', expected 'archived'
    goal: Archive rec-1..rec-3 and set owner to 'ops'.
    outcome: INCOMPLETE
    unmet: [{'predicate': 'archived_and_owned(rec-3)', 'observed': "rec-3.status is 'new', expected 'archived'"}]
    rounds_attempted: 2
    agent_actions: ['round 1: touched rec-1, rec-2', 'round 2: repaired 2']
    needs_human: True

watchdog
  (True, 'world unchanged across last 3 turns')
  (False, 'progress observed')

Three things in there are the transferable bits.

The gate never reads the agent’s claim — it prints it, then ignores it and calls fetch instead.

The feedback going back to the agent is structured and specific (“rec-2.owner is ‘alice’, expected ‘ops’”) rather than “that was wrong,” because a precise observation is repairable while a vague complaint just invites another guess.

And the correction loop is bounded, so when an agent cannot fix the problem the run stops rather than looping forever — and when it stops it neither crashes nor silently succeeds, but emits a stuck report naming exactly which predicate is unmet and what was observed, which is precisely the artifact the previous section argued for.

Say it in one breath

An agent’s “done” costs it nothing and looks identical whether the work happened or not, so treat the self-report as a hypothesis and never as evidence: define done as machine-checkable predicates before work starts, verify against the actual state of the world afterwards with a separate cheap check rather than asking the agent if it’s sure, run a no-progress watchdog so a stall can’t hide behind busy-looking turns, and make an honest, specific “I couldn’t do this and here’s exactly where I stopped” a first-class rewarded outcome — because a system that only rewards completion will reliably teach itself to produce the word.

What an interviewer is really testing

They want to know whether you’ve internalised that a language model’s output is a claim about the world rather than a change to it, because everything else follows from that one shift.

The junior answer is “add a verification step”; the senior answer specifies what is verified — state, not narrative — and by whom, in a separate call with its own context, because self-assessment inside the same transcript is well documented to be poorly calibrated and sometimes actively harmful.

The real tell, though, is whether you mention incentives — whether you notice that a pipeline scoring only completions is training its agents to claim completion, and that designing a rewarded path for honest failure is a systems-design decision rather than a prompt-writing flourish.

Further reading

• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — scores by comparing final database state to an annotated goal state, and introduces pass^k for run-to-run reliability.
• WebArena: A Realistic Web Environment for Building Autonomous Agents — functional correctness via programmatic inspection of the environment, not surface-form action matching.
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — verification by running the actual test suite.
• Large Language Models Cannot Self-Correct Reasoning Yet — why “are you sure?” is not a verifier.
• Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution — inflated verbalized confidence, and what to do about it.
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
• Self-Preference Bias in LLM-as-a-Judge — an argument for using a different model as your verifier.
• Anthropic: Building effective agents — the evaluator-optimizer pattern, and the point that agents must “gain ‘ground truth’ from the environment at each step.”
• OpenAI: Introducing SWE-bench Verified — what happens when you audit the checkers themselves.