CI/CD when your tests are nondeterministic

Every CI pipeline you have ever built answers a yes/no question. Did the tests pass.

Your agent pipeline answers a different question, and pretending otherwise is where teams get hurt:

Is this version’s behaviour acceptably close to, or better than, the version currently serving customers — given that both are samples from a noisy distribution?

That is a statistical question, and it has statistical failure modes. Gate too tight and the pipeline is red three days a week, someone adds continue-on-error: true, and you have a decorative gate. Gate too loose and a real regression walks through.

This chapter builds the pipeline properly. A funnel that catches cheap problems cheaply, an eval gate that is trustworthy enough that nobody wants to disable it, one immutable artifact that gets promoted rather than rebuilt, and a complete GitHub Actions workflow you can copy into a repository today.

The funnel

The organising idea is old and still correct: catch errors as early and as cheaply as possible. DevOps calls it shifting left. For agents it matters more than usual, because your expensive checks are genuinely expensive — an eval run makes hundreds of model calls and costs real money, and you do not want to spend it discovering that someone left a syntax error in.

Three phases, in increasing cost and decreasing frequency.

Phase 1 — pre-merge, on every pull request. Fast, cheap, and blocking. Lint, type check, unit tests, dependency and secret scanning, and a fast subset of the eval suite. Target: under ten minutes and under a dollar. This is the gatekeeper for your main branch, and its job is to keep it clean.

Phase 2 — post-merge, into staging. The build happens once here and produces the artifact everything downstream uses. Deploy to a staging environment that resembles production, then run the checks that need a running system: integration tests against real dependencies, the full eval suite, load and latency checks, and adversarial or red-team cases. This is also where humans inside the company use it before anyone outside does.

Phase 3 — gated promotion to production. No rebuild. The exact artifact validated in staging is promoted, with a human approval in front of it and the rollout strategy from the next chapter behind it.

The reason to be strict about “no rebuild” is not purity. It is that a rebuild can pull a different transitive dependency, a different base image layer, or a different model default, and then the thing you tested is not the thing you shipped.

Saying it out loud. The organising idea is the old one — catch errors as early and as cheaply as you can — and it matters more here because your expensive checks are genuinely expensive. An eval run is hundreds of model calls and real money, and you don’t want to spend it finding out someone left a syntax error in. So three phases: fast cheap blocking checks on every pull request, aiming for under ten minutes and under a dollar; a single build post-merge into staging where the slow stuff runs against a live system; and then gated promotion of that exact artifact to production. The rule I’d defend hardest is no rebuild between staging and production — not for purity, but because a rebuild can pull a different transitive dependency or base layer, and then the thing you tested isn’t the thing you shipped.

What runs where, exactly

Sorting checks into the right phase is most of the design work. Here is the split that holds up.

Every pull request:

Post-merge to staging:

Nightly, on main:

That nightly row deserves emphasis, because it is the agent-specific one. A traditional service does not change when nobody commits. Yours does, whenever the vendor updates the model behind an alias. The nightly job is how you find out on a Tuesday morning rather than from a customer.

Saying it out loud. Sorting checks into the right phase is most of the design work. Pull requests get lint, types, tool unit tests, prompt and schema linting, secret and dependency scans, and a ten-to-fifteen-case eval smoke test. Staging gets the container build, integration tests against real MCP servers, the full fifty-to-two-hundred-case suite, the adversarial suite, and a cost-per-task number. Then there’s a nightly job on main, and that’s the agent-specific one worth calling out: a normal service doesn’t change when nobody commits, but yours does every time the vendor updates the model behind an alias. So you run the suite nightly against both the live alias and a pinned version — the pinned run isolates “did we change” from “did they change.” That job is how you find out on a Tuesday morning instead of from a customer.

Versioning the whole agent as one thing

Your agent’s behaviour is a function of five inputs. Code, prompts, tool schemas, model identifier, and configuration. A version number that covers only the first one is not a version number.

Put all five into a single manifest file, committed, and make it the thing you version.

# agent.lock.yaml — generated by CI, committed, and the unit of rollback
version: 1.4.0
git_sha: a1b2c3d4
model:
  id: claude-sonnet-4-5-20250929   # pinned, not an alias
  max_tokens: 1024
  temperature: 0.0
prompts:
  system: prompts/support_system.md
  system_sha256: 9f2c...e41
tools:
  - name: find_order
    schema_sha256: 3ab9...77c
    effect: read
  - name: issue_refund
    schema_sha256: c410...0d2
    effect: write
    confirm: true
eval:
  dataset: evals/golden_v7.jsonl
  dataset_sha256: 771e...b03
  baseline_pass_rate: 0.85

Three properties make this worth the effort.

Rollback becomes one atomic operation. You are not rolling back a deploy, you are rolling back a manifest, and the prompt goes back with the code.

Eval comparisons become meaningful. Two runs are comparable when their manifests differ in exactly one field. If you cannot tell whether last week’s score drop came from your prompt edit or a model alias moving underneath you, the score is decoration.

Incidents become reconstructable. “What was serving at 03:14” has an answer, and the answer includes the prompt text.

Note the pinned model ID rather than an alias. Aliases are convenient and they are exactly what makes an agent change without a commit. Pin in the manifest, and let the nightly job be the thing that tests the alias so you choose when to move.

Saying it out loud. Behaviour is a function of five things — code, prompts, tool schemas, model ID, and config — so a version number covering only the code isn’t a version number. You put all five in one committed manifest and version that. It buys you three things. Rollback becomes atomic, because you’re rolling back a manifest and the prompt goes back with the code. Eval comparisons become meaningful, because two runs are only comparable if their manifests differ in exactly one field. And incidents become reconstructable — “what was serving at 03:14” has an answer that includes the prompt text. One detail people miss: pin the model ID rather than using an alias, because the alias is precisely the thing that changes your agent without a commit. Let the nightly job test the alias so you choose when to move.

Statistical gates that survive contact with reality

Now the part that is genuinely different from normal CI.

Your eval suite produces a pass rate. Run the same code twice and you get two different pass rates, because the model samples. So “the pass rate went down” is not, by itself, information.

Saying it out loud. This is the part that’s genuinely different from normal CI: run the same code twice and you get two different pass rates, so “the pass rate went down” isn’t information on its own. Three moves make it into a gate. Know your suite’s resolution — forty cases at 85 percent can’t see an eleven-point move. Compare paired rather than independent, because only the cases that flipped carry any signal. And gate on a layered set of conditions rather than one test: a floor, a significance test, a tolerance band, a coverage check, and a cost ceiling. The failure mode you’re designing against is a gate that either never fires, which makes it decoration, or fires on noise, which makes people turn it off.

First: understand how noisy your number is

For a suite of \( n \) independent binary cases with true pass probability \( p \), the standard error of the observed rate is

\[ \mathrm{SE} = \sqrt{\frac{p(1-p)}{n}} \]

Put numbers in it. With \( n = 40 \) and \( p = 0.85 \), \( \mathrm{SE} \approx 0.056 \) — about 5.6 percentage points. A rough 95% interval is two standard errors wide in each direction, so your forty-case suite cannot distinguish 85% from 74% or from 96%.

That is the single most useful fact in this chapter. Before you argue about thresholds, compute your suite’s resolution. If it cannot see the regression you care about, no threshold setting will help and the fix is more cases, not more arguing.

Two corollaries.

Setting temperature to 0 for evaluation reduces but does not eliminate variance — providers do not guarantee determinism, and tool-use branching amplifies whatever variance remains. Take it anyway; it is free.

Running each case \( k \) times and averaging reduces the noise by \( \sqrt{k} \), at \( k \) times the cost. For a small suite this is often a better spend than adding mediocre cases.

Saying it out loud. Before you argue about thresholds, work out what your suite can actually see. For n binary cases at pass rate p, the standard error is the square root of p times one minus p over n — so with forty cases at 85 percent, that’s about 5.6 points, and a rough 95 percent interval is two of those either way. Which means your forty-case suite cannot distinguish 85 percent from 74 percent or from 96 percent. That’s the single most useful fact in the chapter: if the suite can’t resolve the regression you care about, no threshold setting saves you and the fix is more cases, not more arguing. Temperature zero shaves the variance and you should take it because it’s free, but providers don’t guarantee determinism and tool branching amplifies whatever’s left. Running each case k times cuts noise by root k at k times the cost, which on a small suite is often a better spend than adding mediocre cases.

Second: compare paired, not independent

Both runs execute the same cases. That is a paired design, and treating it as two independent samples throws away most of your statistical power.

What you want to look at is the discordant cases: how many the baseline passed and the candidate failed (call it \( b \)), and how many went the other way (\( c \)). Cases that both got right, or both got wrong, tell you nothing about the difference. McNemar’s test formalises this — under the null hypothesis of no change, each discordant case is a coin flip, so the two-sided exact p-value is the binomial tail

\[ p = 2 \sum_{i=0}^{\min(b,c)} \binom{b+c}{i} 2^{-(b+c)} \]

capped at 1.

You do not need to love statistics to use this. You need to know that four new failures and one new pass out of forty cases is not strong evidence of a regression, and the p-value tells you so.

Saying it out loud. Both runs execute the same cases, so it’s a paired design, and treating it as two independent samples throws away most of your statistical power. All the information is in the discordant cases — the ones the baseline passed and the candidate failed, and the ones that went the other way. Cases both got right, or both got wrong, tell you nothing about the difference. McNemar’s test formalises that: under no change, every discordant case is a coin flip, so you’re just reading a binomial tail. You don’t have to love statistics to use it. You need to know that four new failures and one new pass out of forty cases is not strong evidence of a regression, and the p-value is what tells you so before you spend a day bisecting.

Third: gate on several things, not one

A single significance test is a bad gate on its own, because with a small suite it will almost never fire, which makes it useless, and a big real regression can sit under the threshold.

Use a layered gate:

1. An absolute floor. Below some pass rate you do not ship, full stop, regardless of what the baseline did. This protects you when the baseline is also bad.
2. A significance test on the paired comparison. Catches the statistically real regression.
3. A tolerance band. Any drop larger than some margin blocks, significant or not. Small suites hide real drops; this is the backstop.
4. A coverage check. If the candidate did not run every case the baseline ran, block. Otherwise “delete the failing case” becomes a way to make CI green.
5. A cost ceiling. A version that is 1% better and 3x more expensive is not obviously an improvement, and you want the pipeline to make you look at that.

Here is a gate script that implements exactly that. It takes two JSONL files — one row per case, {"case_id": ..., "passed": bool, "cost_usd": ...} — and exits non-zero when the candidate should be blocked.

#!/usr/bin/env python3
"""Compare a candidate eval run against the production baseline: pass or block."""
import argparse, json, math, sys
from pathlib import Path


def load(path: Path) -> dict[str, dict]:
    rows = [json.loads(l) for l in path.read_text().splitlines() if l.strip()]
    return {r["case_id"]: r for r in rows}


def mcnemar_exact(b: int, c: int) -> float:
    """Two-sided exact p-value for paired binary outcomes.
    b = baseline passed, candidate failed.   c = baseline failed, candidate passed."""
    n = b + c
    if n == 0:
        return 1.0
    k = min(b, c)
    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--baseline", type=Path, required=True)
    ap.add_argument("--candidate", type=Path, required=True)
    ap.add_argument("--min-pass-rate", type=float, default=0.80)
    ap.add_argument("--max-regression-pp", type=float, default=5.0)
    ap.add_argument("--alpha", type=float, default=0.05)
    ap.add_argument("--max-cost-usd", type=float, default=None)
    args = ap.parse_args()

    base, cand = load(args.baseline), load(args.candidate)
    shared = sorted(set(base) & set(cand))
    missing = sorted(set(base) - set(cand))
    if not shared:
        print("BLOCK: no overlapping cases between baseline and candidate")
        return 1

    n = len(shared)
    b = sum(1 for k in shared if base[k]["passed"] and not cand[k]["passed"])
    c = sum(1 for k in shared if not base[k]["passed"] and cand[k]["passed"])
    base_rate = sum(base[k]["passed"] for k in shared) / n
    cand_rate = sum(cand[k]["passed"] for k in shared) / n
    delta_pp = (cand_rate - base_rate) * 100
    p = mcnemar_exact(b, c)
    cost = sum(cand[k].get("cost_usd", 0.0) for k in shared)

    print(f"cases compared      {n}"
          + (f"  ({len(missing)} baseline cases missing)" if missing else ""))
    print(f"baseline pass rate  {base_rate:6.1%}")
    print(f"candidate pass rate {cand_rate:6.1%}   ({delta_pp:+.1f} pp)")
    print(f"newly failing       {b}    newly passing  {c}")
    print(f"paired p-value      {p:.3f}")
    print(f"candidate cost      ${cost:.2f}")
    print()

    blocks = []
    if cand_rate < args.min_pass_rate:
        blocks.append(f"pass rate {cand_rate:.1%} below floor {args.min_pass_rate:.0%}")
    if delta_pp < 0 and p < args.alpha:
        blocks.append(f"significant regression ({delta_pp:+.1f} pp, p={p:.3f})")
    if delta_pp < -args.max_regression_pp:
        blocks.append(f"regression {delta_pp:+.1f} pp exceeds tolerance "
                      f"-{args.max_regression_pp:.1f} pp")
    if missing:
        blocks.append(f"candidate did not run {len(missing)} baseline case(s): "
                      f"{', '.join(missing[:5])}")
    if args.max_cost_usd is not None and cost > args.max_cost_usd:
        blocks.append(f"cost ${cost:.2f} over budget ${args.max_cost_usd:.2f}")

    if blocks:
        for reason in blocks:
            print(f"BLOCK: {reason}")
        return 1
    print("PASS: candidate cleared the gate")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A regression that should block:

$ python gate.py --baseline baseline.jsonl --candidate candidate.jsonl
cases compared      40
baseline pass rate   85.0%
candidate pass rate  77.5%   (-7.5 pp)
newly failing       4    newly passing  1
paired p-value      0.375
candidate cost      $0.96

BLOCK: pass rate 77.5% below floor 80%
BLOCK: regression -7.5 pp exceeds tolerance -5.0 pp
exit=1

Read that carefully, because it is the whole argument for a layered gate. The p-value is 0.375 — statistically, four new failures against one new pass is well within coin-flip territory, and a significance test alone would have waved this through. The floor and the tolerance band caught it.

An improvement that should ship:

$ python gate.py --baseline baseline.jsonl --candidate candidate_good.jsonl
cases compared      40
baseline pass rate   85.0%
candidate pass rate  87.5%   (+2.5 pp)
newly failing       1    newly passing  2
paired p-value      1.000
candidate cost      $0.88

PASS: candidate cleared the gate
exit=0

And the coverage check earning its place:

$ python gate.py --baseline baseline.jsonl --candidate candidate_short.jsonl
cases compared      38  (2 baseline cases missing)
baseline pass rate   84.2%
candidate pass rate  86.8%   (+2.6 pp)
newly failing       1    newly passing  2
paired p-value      1.000
candidate cost      $0.84

BLOCK: candidate did not run 2 baseline case(s): case_038, case_039
exit=1

The numbers improved and the gate still blocked, because two cases quietly stopped running. That is the check that catches the accidental and the deliberate version of the same mistake.

Saying it out loud. A single significance test makes a bad gate, because on a small suite it almost never fires — and a real regression can hide under the threshold. So I’d layer five checks. An absolute floor you never ship below, which protects you when the baseline is also bad. A significance test on the paired comparison. A tolerance band that blocks any drop over some margin whether or not it’s significant, as the backstop for small suites. A coverage check, because otherwise “delete the failing case” becomes a way to make CI green. And a cost ceiling, because a version that’s one percent better and three times more expensive is not obviously an improvement, and you want the pipeline to force that conversation.

Handling genuinely flaky infrastructure

Statistical noise in model output is one thing. A tool whose backend times out twice a week is another, and conflating them will make you distrust your own gate.

Separate them at the source. Classify each case failure as quality (the agent did the wrong thing) or infrastructure (a dependency was unreachable, a rate limit fired, the run timed out). Only quality failures count against the gate. Infrastructure failures get retried once and, if they persist, fail the job with a different message — because a flaky dependency is a real problem, it is just not a reason to block a prompt change.

The rule to hold: a check that is allowed to be flaky is a check that will be ignored. Make it either meaningful or absent.

Saying it out loud. Statistical noise in model output and a backend that times out twice a week are different problems, and conflating them will make you distrust your own gate. So classify every case failure at the source as either quality — the agent did the wrong thing — or infrastructure, meaning a dependency was unreachable, a rate limit fired, or the run timed out. Only quality failures count against the gate. Infrastructure failures get one retry and then fail the job with a different message, because a flaky dependency is a real problem, it’s just not a reason to block a prompt change. The rule I’d hold to is that a check allowed to be flaky is a check that will be ignored — make it meaningful or remove it.

Artifact promotion

Build once. Tag it with the manifest version and the git SHA. Promote the same digest through environments.

build → image sha256:9f4c...   (staging)
                ↓ same digest
             sha256:9f4c...    (production)

Two rules make this real.

Reference images by digest, not by tag, in production. Tags are mutable. myagent:1.4.0 can point at a different image tomorrow; myagent@sha256:9f4c... cannot.

Record the promotion. Which digest, which manifest, which eval report, who approved, when. GitHub environments give you the approval step and the audit record for free; use them rather than building your own.

The serving-side mechanics of getting that digest onto infrastructure — registries, image signing, Kubernetes rollouts, autoscaling — are covered in depth in the sibling llm-serving-inference-guide. This chapter stops at “the pipeline hands off a signed digest and a manifest.”

Saying it out loud. Build once, tag it with the manifest version and the git SHA, and promote the same digest through environments. Two rules make that real. Reference images by digest and not by tag in production, because tags are mutable — myagent:1.4.0 can point at a different image tomorrow and a sha256 digest cannot. And record the promotion: which digest, which manifest, which eval report, who approved, when. Most CI platforms give you the approval step and the audit trail for free, so use theirs rather than building your own. The whole point is that six months later, “what exactly was running” is a lookup and not an archaeology project.

The complete workflow

Here is the whole thing as a GitHub Actions workflow. It is long because it is real; every job in it does something you need.

Action versions are current as of mid-2026: actions/checkout@v7, actions/setup-python@v6, actions/upload-artifact@v7, and google-github-actions/auth@v3 for keyless authentication to Google Cloud via Workload Identity Federation.

# .github/workflows/agent.yml
name: agent

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "0 6 * * *"        # nightly drift check, 06:00 UTC
  workflow_dispatch:

concurrency:
  group: agent-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read

env:
  PYTHON_VERSION: "3.11"
  IMAGE: europe-west1-docker.pkg.dev/${{ vars.GCP_PROJECT }}/agents/support-agent

jobs:
  # ---------------------------------------------------------------- phase 1
  static:
    name: lint, types, unit tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v6
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
      - run: pip install -r requirements-dev.txt
      - run: ruff check .
      - run: ruff format --check .
      - run: mypy agent/
      - name: prompt and tool-schema lint
        run: python -m tools.lint_manifest agent.lock.yaml
      - run: pytest tests/unit -q --junitxml=unit.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: unit-results
          path: unit.xml

  supply_chain:
    name: dependency and secret scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v6
        with: { python-version: "3.11" }
      - run: pip install pip-audit
      - run: pip-audit -r requirements.txt --strict
      - name: secret scan
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  eval_fast:
    name: eval gate (fast subset)
    runs-on: ubuntu-latest
    needs: [static]
    if: github.event_name == 'pull_request'
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v6
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip
      - run: pip install -r requirements.txt
      - name: run candidate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python -m evals.run \
            --dataset evals/golden_v7.jsonl --subset fast \
            --manifest agent.lock.yaml \
            --out candidate.jsonl
      - name: fetch production baseline
        run: |
          gh release download baseline --pattern 'baseline-fast.jsonl' --output baseline.jsonl
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - name: gate
        id: gate
        run: |
          python evals/gate.py \
            --baseline baseline.jsonl --candidate candidate.jsonl \
            --min-pass-rate 0.80 --max-regression-pp 8 --max-cost-usd 2.00 \
            | tee gate.txt
      - name: comment the report on the PR
        if: always()
        uses: actions/github-script@v8
        with:
          script: |
            const fs = require('fs');
            const body = "### Agent eval gate\n```\n"
              + fs.readFileSync('gate.txt','utf8') + "\n```";
            await github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner, repo: context.repo.repo, body });
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-fast
          path: |
            candidate.jsonl
            gate.txt

  # ---------------------------------------------------------------- phase 2
  build:
    name: build the promotable artifact
    runs-on: ubuntu-latest
    needs: [static, supply_chain]
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    permissions:
      contents: read
      id-token: write
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v7
      - uses: google-github-actions/auth@v3
        with:
          project_id: ${{ vars.GCP_PROJECT }}
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
      - uses: google-github-actions/setup-gcloud@v3
      - run: gcloud auth configure-docker europe-west1-docker.pkg.dev --quiet
      - name: stamp the manifest with this commit
        run: |
          python -m tools.stamp_manifest agent.lock.yaml \
            --git-sha ${{ github.sha }}
      - id: push
        run: |
          docker build \
            --build-arg VERSION=$(yq '.version' agent.lock.yaml) \
            --build-arg GIT_SHA=${{ github.sha }} \
            -t $IMAGE:${{ github.sha }} .
          docker push $IMAGE:${{ github.sha }}
          echo "digest=$(docker inspect --format='{{index .RepoDigests 0}}' \
            $IMAGE:${{ github.sha }})" >> "$GITHUB_OUTPUT"

  staging:
    name: deploy to staging and validate
    runs-on: ubuntu-latest
    needs: [build]
    environment: staging
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v7
      - uses: google-github-actions/auth@v3
        with:
          project_id: ${{ vars.GCP_PROJECT }}
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
      - uses: google-github-actions/setup-gcloud@v3
      - name: deploy
        run: |
          gcloud run deploy support-agent-staging \
            --image ${{ needs.build.outputs.digest }} \
            --region europe-west1 --quiet
      - name: smoke test
        run: |
          URL=$(gcloud run services describe support-agent-staging \
            --region europe-west1 --format='value(status.url)')
          python smoke_test.py "$URL" "$(yq '.version' agent.lock.yaml)"
      - name: full eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python -m evals.run --dataset evals/golden_v7.jsonl \
            --manifest agent.lock.yaml --out candidate-full.jsonl
          gh release download baseline --pattern 'baseline-full.jsonl' \
            --output baseline.jsonl
          python evals/gate.py --baseline baseline.jsonl \
            --candidate candidate-full.jsonl \
            --min-pass-rate 0.82 --max-regression-pp 4 --alpha 0.05 | tee gate-full.txt
      - name: adversarial suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python -m evals.run --dataset evals/adversarial_v3.jsonl --fail-on-any
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-full
          path: |
            candidate-full.jsonl
            gate-full.txt

  # ---------------------------------------------------------------- phase 3
  production:
    name: promote to production
    runs-on: ubuntu-latest
    needs: [build, staging]
    environment: production          # required reviewers configured here
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v7
      - uses: google-github-actions/auth@v3
        with:
          project_id: ${{ vars.GCP_PROJECT }}
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
      - uses: google-github-actions/setup-gcloud@v3
      - name: deploy the validated digest with no traffic
        run: |
          gcloud run deploy support-agent \
            --image ${{ needs.build.outputs.digest }} \
            --region europe-west1 --no-traffic --tag candidate --quiet
      - name: smoke test the candidate revision
        run: |
          URL=$(gcloud run services describe support-agent --region europe-west1 \
            --format='value(status.traffic[0].url)')
          python smoke_test.py "https://candidate---$(echo $URL | cut -d/ -f3)"
      - name: start the canary at 5 percent
        run: |
          gcloud run services update-traffic support-agent \
            --region europe-west1 --to-tags candidate=5 --quiet

The nightly drift job is the same shape as eval_fast with two changes: if: github.event_name == 'schedule', and the eval invoked with --model-override claude-sonnet-4-5 so it runs against the moving alias rather than the pinned version in the manifest. When that job goes red and nobody committed, the vendor moved and you have a decision to make.

Four details in the workflow are worth naming, because they are the ones people leave out.

permissions: is set narrowly at the top and widened per job. The default token permissions on a repository are broader than a CI job needs, and a compromised action inherits whatever you grant.

Cloud authentication is keyless. google-github-actions/auth@v3 with a Workload Identity Federation provider means there is no long-lived service account key in your repository secrets to leak. id-token: write is what lets the runner mint the OIDC token; without it the action cannot work.

The gate report is posted as a pull request comment. This is the small thing that makes the gate cultural rather than adversarial. A reviewer who can see which cases changed will engage with the number; a reviewer who sees only a red X will ask you to rerun it.

The production job deploys with --no-traffic and a tag. The revision exists, is smoke-tested by name, and receives 5% of traffic only after it passes. That is the handoff into the next chapter.

Saying it out loud. A couple of details in the assembled pipeline are worth pointing at. The gate report gets posted as a pull request comment, and that’s the small thing that makes the gate cultural rather than adversarial — a reviewer who can see which cases flipped will engage with the number, whereas a reviewer who sees only a red X will just ask you to rerun it. And the production job deploys with no traffic and a tag, so the revision exists and gets smoke-tested by name before it receives its first five percent. Those two choices are what stop the pipeline from being something people route around.

Making the gate trustworthy

A gate people trust is a gate that stays on. Four habits get you there.

Publish the baseline as an artifact, not a number in a config file. The baseline is the full per-case result of whatever is currently in production, uploaded when you promote. Then a comparison is always against reality, and updating the baseline is a side effect of shipping rather than a manual step someone forgets.

Show the diff, not the delta. “Pass rate 82%” tells a developer nothing actionable. “These three cases went from pass to fail, here are their traces” tells them what to fix. Your gate output should link to traces.

Let a human override, loudly. Sometimes the eval set is wrong and the change is right. A documented override — a label on the PR, a required justification, an entry in the audit log — is far healthier than the alternative, which is that someone edits the threshold and nobody notices.

Re-baseline deliberately, and review it. When you accept a new baseline you are redefining acceptable. That is a decision, and it belongs in a pull request with the same scrutiny as a code change.

The agentic-ai-evaluation-guide sibling repository has a chapter on automated evaluation that goes considerably deeper on harness design, judge calibration, and metric selection. Read it alongside this one; this chapter is the plumbing, that one is the measurement.

Saying it out loud. A gate people trust is a gate that stays on, and four habits get you there. Publish the baseline as a full per-case artifact uploaded when you promote, not a number in a config file, so comparisons are always against reality and updating the baseline is a side effect of shipping rather than a chore someone forgets. Show the diff, not the delta — “pass rate 82 percent” is useless, “these three cases went pass to fail, here are the traces” is actionable. Let a human override loudly, with a label and a written justification, because the alternative is that somebody quietly edits the threshold and nobody notices. And re-baseline deliberately through a pull request, because accepting a new baseline is redefining what acceptable means, and that deserves the same scrutiny as a code change.

What you should be able to do now

• Split your checks into pre-merge, post-merge staging, and gated promotion, and justify why each check sits where it does on cost and feedback speed.
• Explain why an agent needs a nightly eval run even when nobody has committed anything, and set one up against a model alias rather than a pinned version.
• Write a manifest that versions code, prompts, tool schemas, model ID, and eval dataset together, and explain what breaks when any one of them is versioned separately.
• Compute your eval suite’s statistical resolution from \( n \) and \( p \), and say honestly whether it can detect the regression size you care about.
• Build a layered gate — absolute floor, paired significance test, tolerance band, coverage check, cost ceiling — and explain why a significance test alone is not enough on a small suite.
• Separate quality failures from infrastructure failures in your eval results, and explain why a gate that is allowed to be flaky will be ignored.
• Promote a single immutable image digest from staging to production with a human approval, rather than rebuilding per environment.

Further reading

• GitHub Actions workflow syntax reference: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax-for-github-actions
• GitHub Actions deployment environments and required reviewers — the approval gate used above: https://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/manage-environments
• google-github-actions/auth — keyless authentication to Google Cloud via Workload Identity Federation: https://github.com/google-github-actions/auth
• Google Cloud Agent Starter Pack CI/CD templates — pre-merge checks, staging, and gated production deploy: https://github.com/GoogleCloudPlatform/agent-starter-pack
• McNemar’s test, the paired comparison used by the gate script: https://en.wikipedia.org/wiki/McNemar%27s_test
• pip-audit, for the dependency scanning job: https://github.com/pypa/pip-audit
• Sibling repository agentic-ai-evaluation-guide, automated-evaluation chapter — harness design, judges, and metric selection.
• Sibling repository llm-serving-inference-guide — registries, image signing, and the serving-side mechanics downstream of this pipeline.