Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Mini-project 10: containerize and deploy your agent

You have an agent. It has been a Python object this whole book.

By the end of this chapter it is a service: a container that starts in under a second, runs as a non-root user, reports its own health, reads its secrets at runtime, serves HTTP, and proves after every deploy that the thing running is the thing you built.

Everything below runs. Every output block in this chapter is real terminal output from the code as printed.

The files:

Dockerfile              # multi-stage, non-root, healthcheck
docker-compose.yml      # local stack: agent + redis
requirements.txt
agent.lock.yaml         # the version manifest from Chapter 2
smoke_test.py           # the post-deploy proof
prompts/
  support_system.md
secrets/
  api_key.txt           # gitignored, local only
app/
  __init__.py
  agent.py              # from Part 1, unchanged
  config.py             # all configuration, one place
  server.py             # the FastAPI wrapper

Note what app/agent.py is: the ReAct agent from Part 1, copied in with no changes. That is deliberate and it is the test of whether your agent was designed well. If wrapping it in a web server required rewriting it, the agent had a dependency on being run from a terminal.

Step 1: configuration in one place

The first thing to get right, because everything else depends on it.

Three rules:

Configuration comes from the environment, so the same image runs in every environment. It is validated at startup, so a misconfigured deploy fails immediately and loudly rather than at 3 a.m. on an unusual code path. Secrets are never defaulted to a real value and never logged.

"""All configuration in one place: read from the environment, validated at import."""
from __future__ import annotations

from functools import lru_cache
from pathlib import Path

from pydantic import Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="AGENT_", extra="ignore")

    # identity of this build — stamped by CI, reported by /healthz, attached to every trace
    version: str = "dev"
    git_sha: str = "unknown"

    # behaviour
    model: str = "claude-sonnet-4-5-20250929"
    max_steps: int = Field(default=6, ge=1, le=32)
    request_timeout_s: float = Field(default=30.0, gt=0)
    log_level: str = "info"

    # runtime
    port: int = 8080
    offline: bool = True          # scripted client: no network, no key needed

    # secrets — never a real default, never logged, preferably read from a file
    api_key: str = ""
    api_key_file: str | None = None

    @model_validator(mode="after")
    def _load_secret_file(self) -> "Settings":
        if not self.api_key and self.api_key_file:
            path = Path(self.api_key_file)
            if path.is_file():
                object.__setattr__(self, "api_key", path.read_text().strip())
        return self

    def redacted(self) -> dict:
        d = self.model_dump()
        d["api_key"] = f"set ({len(self.api_key)} chars)" if self.api_key else "unset"
        return d


@lru_cache
def settings() -> Settings:
    return Settings()

Two things there are worth more than they look.

api_key_file. Every serious secret delivery mechanism — Docker secrets, Kubernetes secret volumes, Cloud Run secret mounts — writes a file rather than setting an environment variable, because environment variables leak into crash dumps, child processes, and /proc. Supporting both means the same image works with a .env locally and a mounted secret in production.

redacted(). There will come a day when you want to log the running configuration to debug a deploy. Having exactly one method that is safe to log means you will not, on that day, print your API key into a log aggregator that six teams can read.

$ AGENT_API_KEY_FILE=secrets/api_key.txt python -c \
    "from app.config import Settings; print(Settings().redacted())"
{'version': 'dev', 'git_sha': 'unknown', 'model': 'claude-sonnet-4-5-20250929',
 'max_steps': 6, 'request_timeout_s': 30.0, 'log_level': 'info', 'port': 8080,
 'offline': True, 'api_key': 'set (14 chars)', 'api_key_file': 'secrets/api_key.txt'}

Step 2: the FastAPI wrapper

Four endpoints, and the health/readiness split is not pedantry.

"""FastAPI wrapper around the agent: health, readiness, and one work endpoint."""
from __future__ import annotations

import logging
import time
import uuid
from contextlib import asynccontextmanager

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, PlainTextResponse
from pydantic import BaseModel, Field

from .agent import Agent, Block, MockClient, Reply, registry
from .config import settings

logging.basicConfig(
    level=settings().log_level.upper(),
    format='{"ts":"%(asctime)s","level":"%(levelname)s","logger":"%(name)s","msg":"%(message)s"}',
)
log = logging.getLogger("agent")

STATE = {"ready": False, "started_at": 0.0}
COUNTERS = {"requests": 0, "failures": 0, "steps": 0}


def build_client():
    cfg = settings()
    if cfg.offline:
        return MockClient([
            Reply([Block("tool_use", id="t1", name="find_order",
                         input={"order_id": "12345"})], "tool_use"),
            Reply([Block("text", text="Order 12345 (Solaris headphones) is out for "
                                      "delivery and arrives today by 8pm.")], "end_turn"),
        ])
    from .agent import AnthropicClient
    return AnthropicClient(cfg.model)


@asynccontextmanager
async def lifespan(app: FastAPI):
    cfg = settings()
    STATE["started_at"] = time.time()
    if not cfg.offline and not cfg.api_key:
        raise RuntimeError("AGENT_API_KEY is required when AGENT_OFFLINE=false")
    log.info("starting version=%s sha=%s offline=%s", cfg.version, cfg.git_sha, cfg.offline)
    _ = registry.specs()          # cheap warm-up: tool schemas resolve
    STATE["ready"] = True
    yield
    STATE["ready"] = False
    log.info("draining")


app = FastAPI(title="solaris-support-agent", lifespan=lifespan)


class ChatIn(BaseModel):
    message: str = Field(min_length=1, max_length=4000)
    session_id: str | None = None


class ChatOut(BaseModel):
    reply: str
    request_id: str
    version: str
    latency_ms: int


@app.middleware("http")
async def request_id(request: Request, call_next):
    rid = request.headers.get("x-request-id") or uuid.uuid4().hex[:12]
    request.state.request_id = rid
    response = await call_next(request)
    response.headers["x-request-id"] = rid
    response.headers["x-agent-version"] = settings().version
    return response


@app.get("/healthz")
def healthz():
    """Liveness: is the process alive? Must never touch a dependency."""
    return {"status": "ok", "version": settings().version, "sha": settings().git_sha}


@app.get("/readyz")
def readyz():
    """Readiness: should this instance receive traffic?"""
    if not STATE["ready"]:
        return JSONResponse({"status": "starting"}, status_code=503)
    return {"status": "ready", "uptime_s": round(time.time() - STATE["started_at"], 1)}


@app.get("/metrics", response_class=PlainTextResponse)
def metrics():
    return "\n".join(f"agent_{k} {v}" for k, v in COUNTERS.items()) + "\n"


@app.post("/v1/chat", response_model=ChatOut)
def chat(body: ChatIn, request: Request):
    cfg = settings()
    started = time.perf_counter()
    COUNTERS["requests"] += 1
    agent = Agent(build_client(), registry, max_steps=cfg.max_steps, verbose=False)
    try:
        reply = agent.run(body.message)
    except Exception as exc:                              # noqa: BLE001
        COUNTERS["failures"] += 1
        log.exception("run failed rid=%s", request.state.request_id)
        raise HTTPException(status_code=500, detail="agent run failed") from exc
    return ChatOut(
        reply=reply,
        request_id=request.state.request_id,
        version=cfg.version,
        latency_ms=int((time.perf_counter() - started) * 1000),
    )

Why liveness and readiness are different endpoints

This is the detail people collapse, and collapsing it causes a specific and infuriating outage.

Liveness (/healthz) answers: is this process alive? If it fails, the orchestrator kills and restarts the container. So it must never check a dependency. If liveness checks your database and the database has a bad ten minutes, every instance fails liveness, every instance gets restarted, and you have converted a degraded dependency into a full outage plus a restart storm.

Readiness (/readyz) answers: should this instance get traffic right now? If it fails, the load balancer takes the instance out of rotation and leaves it running. That is the right response to “still warming up” or “my one dependency is unreachable” — step out, stay alive, come back.

The rule: liveness is about the process, readiness is about the traffic.

Also notice /healthz returns the version and git SHA. That single detail is what makes the smoke test in Step 5 able to prove that the thing serving traffic is the thing you just built, which is the question a deploy is actually trying to answer.

The request ID middleware is the third small thing worth keeping. It accepts an inbound x-request-id if there is one — so a trace ID from an upstream service survives the hop — and echoes it on the response, so a user reporting a problem can give you an identifier that finds the trace.

Step 3: the Dockerfile

Multi-stage, non-root, healthcheck, and reproducible.

# syntax=docker/dockerfile:1

# ---------- stage 1: build the dependency tree ----------
FROM python:3.11-slim AS builder

ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_NO_CACHE_DIR=1

WORKDIR /build
COPY requirements.txt .
RUN python -m venv /opt/venv && \
    /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

# ---------- stage 2: the runtime image ----------
FROM python:3.11-slim AS runtime

ARG VERSION=dev
ARG GIT_SHA=unknown

ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    AGENT_VERSION=${VERSION} \
    AGENT_GIT_SHA=${GIT_SHA} \
    PORT=8080

LABEL org.opencontainers.image.version="${VERSION}" \
      org.opencontainers.image.revision="${GIT_SHA}" \
      org.opencontainers.image.title="solaris-support-agent"

RUN adduser --system --no-create-home --uid 10001 agent

COPY --from=builder /opt/venv /opt/venv
WORKDIR /srv
COPY --chown=agent:nogroup app/ ./app/
COPY --chown=agent:nogroup prompts/ ./prompts/
COPY --chown=agent:nogroup agent.lock.yaml ./

USER 10001

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD python -c "import urllib.request,os,sys; \
sys.exit(0 if urllib.request.urlopen(f\"http://127.0.0.1:{os.environ['PORT']}/healthz\", timeout=2).status==200 else 1)"

CMD exec uvicorn app.server:app --host 0.0.0.0 --port ${PORT} --workers 1 --no-server-header

Each decision, briefly, because these are the ones that get cargo-culted wrong.

Two stages. pip and its build machinery stay in the builder. The runtime image ships the virtualenv and nothing else, which is smaller and has a smaller attack surface.

--no-create-home, --system, numeric USER 10001. Non-root is the baseline. The numeric UID matters because Kubernetes’ runAsNonRoot check inspects the numeric user, and a USER agent by name can fail it depending on how the image is inspected.

PYTHONUNBUFFERED=1. Without it, Python buffers stdout, your logs arrive in blocks, and during an incident the last few seconds before a crash are gone.

ENV PORT=8080 and --port ${PORT}. Serverless runtimes inject PORT and expect you to honour it. Hardcoding 8080 works locally and fails on Cloud Run.

Copy the app last. Dependencies change rarely and code changes constantly. This ordering means an application-only change reuses the cached dependency layer, which is the difference between a 15-second rebuild and a two-minute one.

exec in the CMD. Without it, uvicorn runs as a child of a shell, the shell is PID 1, and SIGTERM on shutdown goes to the shell instead of your app. Your graceful drain never runs, and in-flight requests get killed. This one bites people for months before they find it.

--workers 1. Scale by adding containers, not processes inside a container. One worker per container gives you accurate per-instance metrics, clean autoscaling signals, and a memory limit that means something.

The healthcheck uses Python, not curl. A slim image does not have curl, and installing it just for a healthcheck adds a package and a CVE surface for no reason.

Step 4: docker-compose for the local stack

Compose is how you run the real image locally with its real dependencies before anything reaches a cloud.

services:
  agent:
    build:
      context: .
      args:
        VERSION: ${VERSION:-dev}
        GIT_SHA: ${GIT_SHA:-local}
    image: solaris-support-agent:${VERSION:-dev}
    ports:
      - "8080:8080"
    environment:
      AGENT_OFFLINE: "true"
      AGENT_MAX_STEPS: "6"
      AGENT_LOG_LEVEL: "info"
    secrets:
      - agent_api_key
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/healthz', timeout=2).status==200 else 1)"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
    deploy:
      resources:
        limits:
          memory: 512M

  redis:
    image: redis:7-alpine
    command: ["redis-server", "--save", "", "--appendonly", "no"]
    ports:
      - "6379:6379"

secrets:
  agent_api_key:
    file: ./secrets/api_key.txt

The hardening block — read_only, cap_drop: ALL, no-new-privileges — is here rather than only in production on purpose. If your app writes somewhere it should not, you want to find that on your laptop, not during a rollout. tmpfs: /tmp is the escape hatch for the handful of libraries that genuinely need scratch space.

Validate before you build:

$ docker compose config
name: deploy
services:
  agent:
    build:
      context: /home/claude/scratch/deploy
      dockerfile: Dockerfile
      args: {GIT_SHA: local, VERSION: dev}
    cap_drop: [ALL]
    deploy: {resources: {limits: {memory: "536870912"}}}
    ...
    read_only: true
    secrets:
      - source: agent_api_key
        target: /run/secrets/agent_api_key
    security_opt:
      - no-new-privileges:true

Note where the secret lands: /run/secrets/agent_api_key. Set AGENT_API_KEY_FILE=/run/secrets/agent_api_key and the config module from Step 1 picks it up with no code change.

Then:

VERSION=1.4.0 GIT_SHA=$(git rev-parse --short HEAD) docker compose up --build
docker compose ps          # STATUS should read "healthy", not just "running"

healthy is the bit to look for. A container that is running tells you the process started. A container that is healthy tells you it can serve.

Step 5: run it and prove it

Here is the service actually running, outside a container, so you can see the whole loop before adding Docker to the equation.

$ AGENT_VERSION=1.4.0 AGENT_GIT_SHA=a1b2c3d \
    uvicorn app.server:app --host 127.0.0.1 --port 8080

{"ts":"2026-08-06 20:38:28,414","level":"INFO","logger":"agent",
 "msg":"starting version=1.4.0 sha=a1b2c3d offline=True"}

$ curl -s localhost:8080/healthz
{"status":"ok","version":"1.4.0","sha":"a1b2c3d"}

$ curl -s localhost:8080/readyz
{"status":"ready","uptime_s":3.6}

$ curl -si -X POST localhost:8080/v1/chat \
    -H 'content-type: application/json' \
    -d '{"message":"where is order 12345?"}'
HTTP/1.1 200 OK
content-type: application/json
x-request-id: 20074a36c131
x-agent-version: 1.4.0

{"reply":"Order 12345 (Solaris headphones) is out for delivery and arrives today by 8pm.",
 "request_id":"20074a36c131","version":"1.4.0","latency_ms":0}

The agent ran its loop, called find_order, and answered — through HTTP, with a request ID, stamped with the version that produced it.

The smoke test

The most important file in this chapter, and the one most projects do not have.

A smoke test is not an integration test. It runs after a deploy, against the deployed URL, and it answers one question: is the thing now serving traffic the thing I intended to deploy, and does it work?

#!/usr/bin/env python3
"""Post-deploy smoke test. Exit non-zero and the rollout stops."""
from __future__ import annotations

import sys, time
import httpx

BASE = sys.argv[1].rstrip("/")
EXPECT_VERSION = sys.argv[2] if len(sys.argv) > 2 else None
failures: list[str] = []


def check(name: str, fn) -> None:
    t0 = time.perf_counter()
    try:
        fn()
        print(f"  PASS  {name}  ({(time.perf_counter()-t0)*1000:.0f} ms)")
    except AssertionError as exc:
        failures.append(name)
        print(f"  FAIL  {name}: {exc}")


with httpx.Client(timeout=20.0) as http:
    def live():
        r = http.get(f"{BASE}/healthz")
        assert r.status_code == 200, r.status_code
        if EXPECT_VERSION:
            got = r.json()["version"]
            assert got == EXPECT_VERSION, f"serving {got}, expected {EXPECT_VERSION}"

    def ready():
        r = http.get(f"{BASE}/readyz")
        assert r.status_code == 200, f"{r.status_code} {r.text}"

    def happy_path():
        r = http.post(f"{BASE}/v1/chat", json={"message": "where is order 12345?"})
        assert r.status_code == 200, f"{r.status_code} {r.text}"
        body = r.json()
        assert "12345" in body["reply"], f"no order id in reply: {body['reply']!r}"
        assert r.headers.get("x-request-id"), "no request id echoed"

    def rejects_garbage():
        r = http.post(f"{BASE}/v1/chat", json={"message": ""})
        assert r.status_code == 422, f"empty message should 422, got {r.status_code}"

    check("liveness", live)
    check("readiness", ready)
    check("happy path returns grounded answer", happy_path)
    check("input validation rejects empty message", rejects_garbage)

print()
if failures:
    print(f"SMOKE TEST FAILED: {len(failures)} check(s): {', '.join(failures)}")
    sys.exit(1)
print("SMOKE TEST PASSED")

Against the running service:

$ python smoke_test.py http://localhost:8080 1.4.0
  PASS  liveness  (4 ms)
  PASS  readiness  (1 ms)
  PASS  happy path returns grounded answer  (2 ms)
  PASS  input validation rejects empty message  (1 ms)

SMOKE TEST PASSED

And now the check that earns the whole file — asking for a version that is not deployed:

$ python smoke_test.py http://localhost:8080 1.5.0
  FAIL  liveness: serving 1.4.0, expected 1.5.0
  PASS  readiness  (2 ms)
  PASS  happy path returns grounded answer  (2 ms)
  PASS  input validation rejects empty message  (1 ms)

SMOKE TEST FAILED: 1 check(s): liveness
exit=1

Everything works. The service is healthy, it answers correctly, it validates input. And it is the wrong build, which happens more often than anyone admits — a failed push, a stale tag, traffic still pinned to the previous revision, a deploy that silently no-opped.

Without the version assertion, that deploy reports success and you spend the afternoon wondering why your fix did not take effect.

Three properties of a good smoke test:

It runs against the deployed URL, not a test client, so it exercises the load balancer, TLS, and routing. It asserts the version, so “deploy succeeded” means what you think it means. It is fast and non-destructive, because it runs on every deploy including the rollback, and it must not create real side effects — a read-only happy path, never issue_refund.

Step 6: deploy it

Cloud Run is the shortest path from this container to a URL, and its revision model matches Chapter 3 exactly. The same shapes exist on AWS App Runner, Azure Container Apps, and Fly.io; only the nouns change.

Put the secret in a manager first, never in an environment variable in your deploy command:

gcloud secrets create agent-api-key --replication-policy=automatic
printf '%s' "$ANTHROPIC_API_KEY" | gcloud secrets versions add agent-api-key --data-file=-

Then deploy the image, mounting the secret as a file:

REGION=europe-west1
IMAGE=europe-west1-docker.pkg.dev/$PROJECT/agents/support-agent:1.4.0

gcloud run deploy support-agent \
  --image "$IMAGE" \
  --region "$REGION" \
  --no-allow-unauthenticated \
  --set-env-vars AGENT_VERSION=1.4.0,AGENT_GIT_SHA=$(git rev-parse --short HEAD),AGENT_OFFLINE=false \
  --set-env-vars AGENT_API_KEY_FILE=/run/secrets/api_key \
  --set-secrets /run/secrets/api_key=agent-api-key:latest \
  --cpu 1 --memory 512Mi \
  --concurrency 20 \
  --min-instances 1 --max-instances 50 \
  --timeout 120s \
  --no-traffic --tag candidate

The flags that matter for an agent specifically:

--concurrency 20, not the default 80. An agent request holds an in-flight model call for seconds and uses meaningful memory for its context. Packing 80 of those onto one instance produces memory pressure and tail latency. Tune it from measurements; start low.

--timeout 120s must exceed your longest expected trajectory, or the platform will cut off a run mid-loop and you will misdiagnose it as a model failure.

--min-instances 1 if cold starts hurt. A Python container with a virtualenv takes a second or two to come up, on top of whatever your first model call costs.

--no-traffic --tag candidate deploys the revision without giving it any users, and gives it a stable URL for the smoke test. That is Chapter 3’s shadow-then-canary flow, and it is why the deploy command and the traffic command are separate.

Cloud Run runs an HTTP startup probe against your container by default, and you can point it at /readyz explicitly — which is the right target, because readiness is exactly “should this receive traffic.”

Then the sequence that makes a deploy safe:

# 1. smoke-test the candidate revision by its tag URL, with zero users on it
#    (`--tag candidate` gives the revision a stable https://candidate---<service>... URL)
CANDIDATE=$(gcloud run services describe support-agent --region "$REGION" \
  --format=json | jq -r '.status.traffic[] | select(.tag=="candidate") | .url')
python smoke_test.py "$CANDIDATE" 1.4.0

# 2. only then, start the canary
gcloud run services update-traffic support-agent --region "$REGION" --to-tags candidate=5

# 3. watch (Chapter 3's gates), then expand
gcloud run services update-traffic support-agent --region "$REGION" --to-tags candidate=25
gcloud run services update-traffic support-agent --region "$REGION" --to-latest

# 4. and the command you must have run at least once on purpose
gcloud run services update-traffic support-agent --region "$REGION" \
  --to-revisions support-agent-00042-abc=100

Step 4 is the rollback. Run it deliberately, today, on a service that does not matter, so that the first time you run it in anger you are not also reading the documentation.

What is still missing

An honest list, because this chapter shipped a service, not a finished production system.

Authentication on the endpoint. --no-allow-unauthenticated means IAM guards it; a public product needs real user auth, and the principal it produces is what feeds Chapter 4’s authorization layer. Persistent sessions. The compose file starts Redis and the app does not use it. Wire in the session store from Part 3, and the service becomes horizontally scalable for real. Streaming. A ten-second silent wait is a bad experience; server-sent events fix the perception even when they do not fix the latency. Tracing. OpenTelemetry export to a backend, per Chapter 5. The authorization layer. Chapter 4’s Authorizer between the model’s tool request and your function. Rate limits and budgets. Per user and per tenant, enforced in code.

Each of those is a chapter you have already read. The container is where they all get to run.

What you should be able to do now

  • Write a configuration module that reads from the environment, validates at startup, supports file-based secrets, and has exactly one method that is safe to log.
  • Explain the difference between liveness and readiness, and describe the outage you cause by having liveness check a dependency.
  • Write a multi-stage, non-root Dockerfile with a numeric UID, an honoured PORT, correct layer ordering, exec in the CMD, and a healthcheck that does not need curl installed.
  • Say why --workers 1 and one process per container is the right default for an agent, and why exec in the CMD determines whether graceful shutdown works at all.
  • Run the real image locally under compose with read-only root, dropped capabilities, and a mounted secret, and check for healthy rather than running.
  • Write a post-deploy smoke test that asserts the served version, and explain the failure it catches that every other test misses.
  • Deploy to a serverless runtime with the secret mounted from a manager, tune concurrency and timeout for agent workloads, deploy with no traffic behind a tag, smoke-test the candidate, and roll back with one command.

Further reading