LLM Serving & Inference: Complete Learning Repository

A comprehensive, hands-on guide to learning LLM serving and inference from fundamentals to production deployment.

🎯 What You’ll Learn

This repository teaches you everything about serving Large Language Models in production:

• How LLM inference works (tokenization, attention, generation)
• Basic serving (FastAPI, model loading, endpoints)
• Containerization (Docker for ML workloads)
• Kubernetes deployment (production infrastructure)
• Performance optimization (vLLM, batching, GPU utilization)
• Scaling (autoscaling, load balancing)
• Production practices (monitoring, versioning, drift detection)

📁 Repository Structure

mlops_serving/
├── LEARNING_PATH.md          # Start here! Complete learning guide
├── docs/                     # Detailed concept explanations
│   ├── llm_inference_fundamentals.md
│   ├── serving_architectures.md
│   └── optimization_techniques.md
├── 01_basic_serving/         # FastAPI + HuggingFace serving
├── 02_docker/                # Containerization
├── 03_kubernetes/            # K8s deployment
├── 04_load_testing/          # Performance testing
├── 05_vllm_serving/          # High-performance serving
├── 06_autoscaling/           # K8s autoscaling
├── 07_canary_deployments/    # Gradual rollouts
├── 08_monitoring/            # Prometheus + Grafana
├── 09_model_versioning/      # Version management
├── 10_drift_detection/       # Evidently integration
└── 11_triton/                # Multi-model serving

🚀 Quick Start

👉 START HERE: Read HOW_TO_START.md for a complete step-by-step guide!

Quick Commands

# 1. Set up environment
cd 01_basic_serving
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

# 2. Start the server
python app.py

# 3. Test it (in another terminal)
python test_api.py
# OR visit http://localhost:8000/docs for interactive API docs

📚 Learning Approach

This is a hands-on, topic-based learning path:

1. Each topic is self-contained with code, docs, and exercises
2. Build incrementally - each topic builds on previous concepts
3. Learn by doing - run code, modify it, break it, fix it
4. Focus on understanding - not just copying code

🎓 Prerequisites

• Python 3.9+
• Basic Python knowledge
• Understanding of REST APIs
• Docker (for containerization topics)
• Kubernetes cluster (minikube/kind for K8s topics)
• GPU (optional, helpful for vLLM)

📖 Topics Covered

Core Concepts

• LLM architecture and inference pipeline
• Tokenization and vocabulary
• Attention mechanisms and KV caching
• Memory and computation requirements

Serving Basics

• Model loading and initialization
• API endpoint design
• Request/response handling
• Error handling and validation

Infrastructure

• Docker containerization
• Kubernetes deployment
• Health checks and probes
• Resource management

Performance

• Load testing and benchmarking
• Latency vs throughput
• vLLM optimization
• GPU utilization

Production

• Autoscaling strategies
• Canary deployments
• Monitoring and observability
• Model versioning
• Drift detection

🔧 Technologies

• FastAPI: Web framework
• HuggingFace Transformers: Model loading
• vLLM: High-performance inference
• Docker: Containerization
• Kubernetes: Orchestration
• Locust: Load testing
• Prometheus: Metrics
• Grafana: Visualization
• Evidently: Drift detection
• Triton: Multi-model serving

📝 How to Use This Repository

1. Start with LEARNING_PATH.md - Understand the structure
2. Read the fundamentals - docs/llm_inference_fundamentals.md
3. Work through topics in order - Each numbered topic builds on the last
4. Experiment and modify - Don’t just run code, change it and learn
5. Read the code comments - They explain the “why” not just the “what”
6. Check solutions - See EXERCISE_SOLUTIONS.md for exercise answers
7. Prepare for interviews - Review INTERVIEW_QA.md for common questions

🎯 Learning Goals

By completing this repository, you’ll be able to:

• Understand how LLM inference works internally
• Build production-ready serving APIs
• Deploy models to Kubernetes
• Optimize for performance and cost
• Monitor and debug serving systems
• Handle model updates and drift

🤝 Contributing

This is a learning repository. Feel free to:

• Add more examples
• Improve documentation
• Fix bugs
• Share your learnings

📄 License

MIT License - Feel free to use this for learning and teaching.

📖 Getting Started

1. New to this? → Read HOW_TO_START.md for step-by-step instructions
2. Want overview? → Read LEARNING_PATH.md to see all topics
3. Ready to code? → Start with 01_basic_serving/
4. Need help? → Check EXERCISE_SOLUTIONS.md for exercise answers
5. Interview prep? → Review INTERVIEW_QA.md for common questions

Ready to start? Open HOW_TO_START.md and begin your journey! 🚀

📚 Additional Resources

• Exercise Solutions: EXERCISE_SOLUTIONS.md - Answers to all exercises
• Interview Q&A: INTERVIEW_QA.md - 23+ interview questions with detailed answers
• Quick Reference: QUICK_REFERENCE.md - Quick commands and tips
• Complete Topics: COMPLETE_TOPICS.md - Overview of all 12 topics
• Industry Boilerplate: Each topic README includes industry-standard code and use cases

🏭 Industry Focus

Each topic now includes:

• Why We Need This: Business and technical justification
• Industry Use Cases: Real-world applications (5+ examples per topic)
• Boilerplate Code: Production-ready, industry-standard implementations
• Company Examples: How major companies use these technologies

Topics updated with industry content:

• ✅ Topic 1: Basic Serving (Customer support, content generation)
• ✅ Topic 2: Docker (Multi-cloud, CI/CD, edge deployment)
• ✅ Topic 3: Kubernetes (Large-scale platforms, enterprise ML)
• ✅ Topic 5: vLLM (High-throughput APIs, cost optimization)
• ✅ Topic 8: Monitoring (Production observability, incident response)

👋 Welcome! Start Here

This repository teaches you everything about LLM serving and inference through hands-on, practical examples.

🎯 What You’ll Learn

You’ll learn how to:

• Serve LLMs in production
• Optimize for performance
• Deploy to Kubernetes
• Monitor and debug systems
• Handle model updates safely
• Detect and handle drift

📚 How This Repository is Organized

Learning Structure

• Topics are numbered (01, 02, 03…) - work through them in order
• Each topic is self-contained - has its own code, docs, and examples
• Builds incrementally - each topic builds on previous concepts

Key Files

🚀 Your First Steps

1. Read the Start Guide

cat HOW_TO_START.md

This has everything you need to begin.

2. Understand the Fundamentals

cat docs/llm_inference_fundamentals.md

Learn how LLM inference works internally.

3. Run Your First Server

cd 01_basic_serving
pip install -r requirements.txt
python app.py

4. Test It

# In another terminal
python test_api.py

📖 Learning Topics

1. LLM Inference Fundamentals - How LLMs work
2. Basic Serving - FastAPI + HuggingFace
3. Containerization - Docker
4. Kubernetes - K8s deployment
5. Load Testing - Performance measurement
6. vLLM Serving - High-performance serving
7. Autoscaling - Scale automatically
8. Canary Deployments - Safe rollouts
9. Monitoring - Metrics and dashboards
10. Model Versioning - Version management
11. Drift Detection - Detect issues
12. Triton - Multi-model serving

💡 Learning Approach

1. Read the documentation
2. Study the code
3. Run the examples
4. Modify and experiment
5. Move to the next topic

✅ Prerequisites

• Python 3.9+
• Basic Python knowledge
• Understanding of REST APIs
• (Optional) Docker and Kubernetes

Don’t have Docker/K8s? No problem - you can learn the basics without them!

🎓 Ready to Start?

👉 Open HOW_TO_START.md and follow the step-by-step guide!

Questions? Check the README.md in each topic directory for detailed explanations.

Stuck? Read error messages carefully, check the docs, and experiment with simpler examples first.

Let’s learn! 🚀

How to Start Learning LLM Serving & Inference

🎯 Your Learning Journey Starts Here

This guide will walk you through exactly how to start learning LLM serving and inference, step by step.

📋 Prerequisites Check

Before you start, make sure you have:

• Python 3.9 or higher (python --version)
• pip installed
• Basic understanding of Python
• Basic understanding of REST APIs
• (Optional) Docker installed
• (Optional) Kubernetes cluster (minikube/kind)

Don’t worry if you don’t have Docker/K8s yet - you can learn the basics without them!

🚀 Step-by-Step Learning Path

Step 1: Understand the Fundamentals (30-60 minutes)

Read this first: docs/llm_inference_fundamentals.md

This explains:

• How LLMs work internally
• What happens during inference
• Tokenization, attention, generation
• Memory and computation requirements

Why this matters: You need to understand what’s happening under the hood before you can serve models effectively.

Action: Open the file and read through it. Don’t worry if you don’t understand everything - you’ll learn more as you build.

Step 2: Set Up Your Environment (10 minutes)

# Navigate to the project
cd /Users/faisal/Projects/mlops_serving

# Create a virtual environment (recommended)
python -m venv venv

# Activate it
source venv/bin/activate  # On Mac/Linux
# OR
venv\Scripts\activate  # On Windows

# Install dependencies for basic serving
cd 01_basic_serving
pip install -r requirements.txt

What this does: Sets up an isolated Python environment with all the libraries you need.

Step 3: Run Your First LLM Server (5 minutes)

# Make sure you're in 01_basic_serving directory
cd 01_basic_serving

# Start the server
python app.py

You should see:

INFO:     Started server process
INFO:     Waiting for application startup.
🚀 Starting up: Loading LLM model...
Loading model: gpt2 on device: cpu
✅ Model loaded successfully!
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000

What’s happening:

• The server is loading a GPT-2 model (small, fast model for learning)
• It’s starting a FastAPI web server
• The model is now ready to serve requests

Step 4: Test the API (5 minutes)

Option A: Use the test script

# In a new terminal (keep server running)
cd 01_basic_serving
python test_api.py

Option B: Use curl

# Health check
curl http://localhost:8000/health

# Generate text
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_length": 50,
    "temperature": 0.7
  }'

Option C: Use the interactive docs Open http://localhost:8000/docs in your browser - FastAPI provides an interactive API explorer!

Step 5: Understand What Just Happened (15 minutes)

Read: 01_basic_serving/README.md

This explains:

• What each file does
• How the model loading works
• How the API endpoints work
• Key concepts you just used

Then explore the code:

• model_loader.py - How models are loaded and used
• app.py - How the API is structured

Try modifying:

• Change the model name (try “distilgpt2” for a smaller model)
• Change temperature (0.1 = more deterministic, 2.0 = more creative)
• Add logging to see what’s happening

Step 6: Learn Each Topic in Order

Now that you’ve run your first server, work through each topic:

1. 01_basic_serving ✅ (You just did this!)
2. 02_docker - Containerize your app
3. 03_kubernetes - Deploy to K8s
4. 04_load_testing - Test performance
5. 05_vllm_serving - High-performance serving
6. 06_autoscaling - Scale automatically
7. 07_canary_deployments - Safe rollouts
8. 08_monitoring - Track metrics
9. 09_model_versioning - Manage versions
10. 10_drift_detection - Detect issues
11. 11_triton - Multi-model serving

For each topic:

1. Read the README.md
2. Study the code
3. Run the examples
4. Modify and experiment
5. Move to the next topic

🎓 Learning Tips

1. Don’t Rush

Understanding is more important than speed. Take time to:

• Read error messages carefully
• Experiment with parameters
• Break things and fix them

2. Experiment

After running each example:

• Change parameters
• Modify the code
• See what breaks
• Understand why

3. Use the Documentation

Each topic has:

• README.md explaining concepts
• Code comments explaining “why”
• Examples you can run

4. Ask Questions

As you learn, ask yourself:

• “Why does this work this way?”
• “What happens if I change X?”
• “How does this scale?”
• “What could go wrong?”

🐛 Common Issues & Solutions

Issue: “Model not found” or download errors

Solution: Check your internet connection. Models are downloaded from HuggingFace on first use.

Issue: “Out of memory”

Solution:

• Use a smaller model (distilgpt2 instead of gpt2)
• Reduce max_length
• Close other applications

Issue: “Port already in use”

Solution:

• Stop the previous server (Ctrl+C)
• Or change the port in app.py

Issue: “Import errors”

Solution:

• Make sure virtual environment is activated
• Run pip install -r requirements.txt again

📊 What You’ll Learn

By the end of this journey, you’ll understand:

Core Concepts

• ✅ How LLM inference works internally
• ✅ Tokenization and vocabulary
• ✅ Attention mechanisms
• ✅ Memory and computation requirements

Serving Skills

• ✅ Building serving APIs
• ✅ Containerization
• ✅ Kubernetes deployment
• ✅ Performance optimization

Production Skills

• ✅ Monitoring and observability
• ✅ Scaling strategies
• ✅ Safe deployments
• ✅ Drift detection

🎯 Next Steps

1. Right now: Complete Steps 1-5 above
2. Today: Read through 01_basic_serving/README.md and understand the code
3. This week: Work through topics 2-4 (Docker, K8s, Load Testing)
4. This month: Complete topics 5-8 (vLLM, Scaling, Monitoring)
5. Ongoing: Topics 9-11 (Advanced production topics)

❓ Questions?

If you get stuck:

1. Check the README.md in each topic
2. Read error messages carefully
3. Check the docs/ directory for detailed explanations
4. Experiment with simpler examples first

Remember: Learning by doing is the best way. Don’t just read - run the code, modify it, break it, fix it!

Ready? Let’s start! 🚀

Begin with Step 1: Read docs/llm_inference_fundamentals.md

LLM Serving & Inference: Complete Learning Guide

🎯 How to Use This Guide

This guide is organized by learning topics, not time periods. Work through each topic at your own pace. Each topic builds on the previous one, but you can also jump to specific areas you want to learn.

📚 Learning Topics (In Order)

1. LLM Inference Fundamentals

What you’ll learn: How LLMs actually work under the hood

• Tokenization and vocabulary
• Forward pass through transformer layers
• Autoregressive generation
• Attention mechanism and KV caching
• Memory and computation requirements

Start here: docs/llm_inference_fundamentals.md

2. Basic Model Serving

What you’ll learn: Serve a model with a simple API

• Loading models from HuggingFace
• Building FastAPI endpoints
• Request/response handling
• Basic error handling

Practice: 01_basic_serving/

3. Containerization

What you’ll learn: Package your serving application

• Docker basics for ML models
• Multi-stage builds
• Handling large model files
• Environment configuration

Practice: 02_docker/

4. Kubernetes Deployment

What you’ll learn: Deploy to production infrastructure

• K8s manifests for ML workloads
• Health checks and probes
• Resource limits and requests
• ConfigMaps and Secrets

Practice: 03_kubernetes/

5. Load Testing & Performance

What you’ll learn: Measure and understand performance

• Latency vs throughput
• Load testing with Locust
• Performance profiling
• Identifying bottlenecks

Practice: 04_load_testing/

6. High-Performance Serving (vLLM)

What you’ll learn: Optimize for production throughput

• vLLM architecture
• Continuous batching
• PagedAttention
• GPU optimization

Practice: 05_vllm_serving/

7. Autoscaling

What you’ll learn: Scale based on demand

• Horizontal Pod Autoscaling (HPA)
• Request-based scaling
• Concurrency metrics
• Scaling strategies

Practice: 06_autoscaling/

8. Monitoring & Observability

What you’ll learn: Track your serving system

• Prometheus metrics
• Grafana dashboards
• GPU monitoring
• Logging and tracing

Practice: 08_monitoring/

9. Canary Deployments

What you’ll learn: Safely roll out model updates

• Traffic splitting
• A/B testing models
• Gradual rollouts
• Rollback procedures

Practice: 09_canary_deployments/

10. Model Versioning

What you’ll learn: Manage multiple model versions

• Version management strategies
• Model registry
• Rollback procedures
• A/B testing infrastructure

Practice: 10_model_versioning/

11. Drift Detection

What you’ll learn: Detect when models degrade

• Data drift detection
• Concept drift
• Evidently AI integration
• Alerting on anomalies

Practice: 11_drift_detection/

12. Multi-Model Serving (Triton)

What you’ll learn: Serve multiple models efficiently

• NVIDIA Triton Inference Server
• Dynamic batching
• Model ensembles
• Multi-framework support

Practice: 12_triton/

🚀 Quick Start Guide

Step 1: Understand the Basics

Read docs/llm_inference_fundamentals.md to understand how LLM inference works.

Step 2: Set Up Your Environment

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install base dependencies
pip install -r requirements.txt

Step 3: Start with Basic Serving

cd 01_basic_serving
python app.py
# Test it: curl http://localhost:8000/health

Step 4: Progress Through Topics

Work through each numbered topic in order. Each includes:

• README.md: Explanation of concepts
• Code examples: Working implementations
• Exercises: Hands-on practice

🎓 Learning Approach

For Each Topic:

1. Read the documentation - Understand the concepts
2. Study the code - See how it’s implemented
3. Run the examples - Get hands-on experience
4. Modify and experiment - Break things, fix them, learn
5. Move to next topic - Build on what you learned

Tips:

• Don’t rush: Understanding > Speed
• Experiment: Change parameters, break things, learn why
• Read error messages: They teach you a lot
• Use the docs: Each topic has detailed explanations

📖 Prerequisites

Required:

• Python 3.9+
• Basic Python knowledge
• Understanding of REST APIs
• Docker basics

Helpful but not required:

• Kubernetes experience
• ML/AI background
• GPU access (CPU works for learning)

🔧 Technology Stack

You’ll learn these tools:

• FastAPI: Web framework
• HuggingFace Transformers: Model loading
• vLLM: High-performance inference
• Docker: Containerization
• Kubernetes: Orchestration
• Locust: Load testing
• Prometheus/Grafana: Monitoring
• Evidently: Drift detection
• Triton: Multi-model serving

❓ Common Questions

Q: Do I need a GPU? A: Not for the basics. GPU helps with vLLM and production workloads, but you can learn on CPU.

Q: How long will this take? A: Depends on your pace. Each topic can take a few hours to a few days. Focus on understanding, not speed.

Q: Can I skip topics? A: The basics (1-5) should be done in order. Advanced topics (6-12) can be done based on interest.

Q: What if I get stuck? A: Check the docs, read error messages carefully, experiment with simpler examples first.

🎯 Learning Goals

By the end, you’ll be able to:

• ✅ Serve LLMs in production
• ✅ Optimize for performance
• ✅ Deploy to Kubernetes
• ✅ Monitor and debug serving systems
• ✅ Handle model updates safely
• ✅ Detect and handle model drift

Let’s start learning! 🚀

Quick Reference: vLLM, Latency, and Grafana

🚀 Quick Start Guide

1. vLLM Serving

Install vLLM

# Requires CUDA/GPU
pip install vllm

Start vLLM Server

cd 05_vllm_serving
pip install -r requirements.txt
python vllm_server.py

Or use vLLM’s built-in server

python -m vllm.entrypoints.openai.api_server \
    --model gpt2 \
    --port 8000

Test vLLM API

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="gpt2",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=50
)

print(response.choices[0].message.content)

2. Measure Latency

Option A: Using Locust (Interactive)

cd 04_load_testing
pip install -r requirements.txt

# Start your server first
# Then run Locust
locust -f locust_test.py --host=http://localhost:8000

# Open http://localhost:8089 in browser

Option B: Using Python Script (Headless)

cd 04_load_testing
python measure_latency.py --url http://localhost:8000 --requests 100

What You Get

• P50 (Median): 50% of requests faster
• P95: 95% of requests faster
• P99: 99% of requests faster
• Throughput: Requests per second

Compare Endpoints

python measure_latency.py --compare
# Compares basic serving vs vLLM

3. Grafana Monitoring

Start Monitoring Stack

cd 08_monitoring
docker-compose up -d

This starts:

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Add Metrics to Your App

from prometheus_exporter import add_prometheus_middleware

app = FastAPI()
add_prometheus_middleware(app)  # That's it!

Your app now exposes /metrics endpoint.

Configure Prometheus

Edit prometheus.yml to scrape your app:

scrape_configs:
  - job_name: 'llm-serving'
    static_configs:
      - targets: ['localhost:8000']

Create Grafana Dashboard

1. Open Grafana: http://localhost:3000
2. Login: admin/admin
3. Add data source: Prometheus (http://prometheus:9090)
4. Create dashboard with these queries:

Request Rate:

rate(llm_requests_total[5m])

Latency (P95):

histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m]))

Error Rate:

rate(llm_requests_total{status="500"}[5m]) / rate(llm_requests_total[5m])

GPU Utilization:

llm_gpu_utilization_percent

📊 Complete Workflow Example

Step 1: Start vLLM Server

cd 05_vllm_serving
python vllm_server.py
# Server running on http://localhost:8000

Step 2: Measure Latency

cd 04_load_testing
python measure_latency.py --url http://localhost:8000 --requests 100

Output:

P50:  150ms
P95:  450ms
P99:  800ms
Throughput: 25.3 req/s

Step 3: Start Monitoring

cd 08_monitoring
docker-compose up -d

Step 4: View Metrics

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000

Step 5: Load Test with Monitoring

# Terminal 1: Watch Grafana
open http://localhost:3000

# Terminal 2: Run load test
cd 04_load_testing
locust -f locust_test.py --host=http://localhost:8000 --headless --users 10 --spawn-rate 2

🔍 Key Metrics to Monitor

Application Metrics

• llm_requests_total - Total requests
• llm_request_duration_seconds - Request latency
• llm_active_requests - Currently processing
• llm_tokens_generated_total - Tokens generated

System Metrics

• llm_gpu_utilization_percent - GPU usage
• llm_gpu_memory_used_bytes - GPU memory
• llm_gpu_temperature_celsius - GPU temperature

Important Queries

Requests per second:

rate(llm_requests_total[5m])

95th percentile latency:

histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m]))

Error percentage:

rate(llm_requests_total{status="500"}[5m]) / rate(llm_requests_total[5m]) * 100

GPU memory usage:

llm_gpu_memory_used_bytes / llm_gpu_memory_total_bytes * 100

🎯 Performance Targets

Good Performance

• P50 latency: < 200ms
• P95 latency: < 500ms
• P99 latency: < 1s
• Error rate: < 0.1%
• GPU utilization: 70-90%

Warning Signs

• P99 > 2s: System struggling
• Error rate > 1%: Problems occurring
• GPU utilization < 50%: Underutilized
• GPU utilization > 95%: Overloaded

🐛 Troubleshooting

vLLM won’t start

• Check CUDA: python -c "import torch; print(torch.cuda.is_available())"
• Install correct CUDA version
• Use smaller model for testing

High latency

• Check GPU utilization
• Reduce batch size
• Use smaller model
• Check for bottlenecks (CPU, memory, I/O)

No metrics in Grafana

• Check Prometheus is scraping: http://localhost:9090/targets
• Verify /metrics endpoint works: curl http://localhost:8000/metrics
• Check Prometheus config points to correct host

GPU metrics missing

• Install pynvml: pip install pynvml
• Check GPU is accessible
• Verify NVIDIA drivers installed

📚 Learn More

• vLLM: 05_vllm_serving/README.md
• Latency: 04_load_testing/README.md
• Monitoring: 08_monitoring/README.md

Quick tip: Start with basic serving, measure latency, then optimize with vLLM, and monitor with Grafana!

Complete Learning Topics Overview

All 12 learning topics are now complete! Here’s what you have:

✅ Completed Topics

1. LLM Inference Fundamentals ✅

• Location: docs/llm_inference_fundamentals.md
• What it covers: Tokenization, attention, generation, memory, computation
• Status: Complete with detailed explanations

2. Basic Model Serving ✅

• Location: 01_basic_serving/
• What it covers: FastAPI, HuggingFace, model loading, endpoints
• Status: Complete with working code

3. Containerization ✅

• Location: 02_docker/
• What it covers: Docker basics, multi-stage builds, GPU support
• Status: Complete with Dockerfiles and examples

4. Kubernetes Deployment ✅

• Location: 03_kubernetes/
• What it covers: K8s manifests, health checks, resources, ConfigMaps
• Status: Complete with deployment YAMLs

5. Load Testing & Performance ✅

• Location: 04_load_testing/
• What it covers: Latency measurement, Locust, performance profiling
• Status: Complete with test scripts

6. High-Performance Serving (vLLM) ✅

• Location: 05_vllm_serving/
• What it covers: vLLM setup, continuous batching, PagedAttention
• Status: Complete with server implementation

7. Autoscaling ✅

• Location: 06_autoscaling/
• What it covers: HPA, CPU/memory scaling, custom metrics
• Status: Complete with HPA configurations

8. Monitoring & Observability ✅

• Location: 08_monitoring/
• What it covers: Prometheus, Grafana, GPU monitoring, metrics
• Status: Complete with full monitoring stack

9. Canary Deployments ✅

• Location: 09_canary_deployments/
• What it covers: Traffic splitting, gradual rollouts, rollback
• Status: Complete with deployment examples

10. Model Versioning ✅

• Location: 10_model_versioning/
• What it covers: Version management, model registry, rollback
• Status: Complete with strategies and examples

11. Drift Detection ✅

• Location: 11_drift_detection/
• What it covers: Data drift, concept drift, Evidently AI
• Status: Complete with detector implementation

12. Multi-Model Serving (Triton) ✅

• Location: 12_triton/
• What it covers: Triton server, dynamic batching, model ensembles
• Status: Complete with configuration examples

📚 Learning Path

Follow this order for best learning experience:

1. Start: Read HOW_TO_START.md
2. Fundamentals: Read docs/llm_inference_fundamentals.md
3. Basic Serving: Work through 01_basic_serving/
4. Containerize: Learn Docker in 02_docker/
5. Deploy: Deploy to K8s with 03_kubernetes/
6. Test: Measure performance with 04_load_testing/
7. Optimize: Use vLLM in 05_vllm_serving/
8. Scale: Set up autoscaling in 06_autoscaling/
9. Monitor: Add monitoring in 08_monitoring/
10. Deploy Safely: Learn canary in 09_canary_deployments/
11. Version: Manage versions in 10_model_versioning/
12. Detect Issues: Add drift detection in 11_drift_detection/
13. Advanced: Use Triton in 12_triton/

🎯 Quick Reference

• Quick Start: HOW_TO_START.md
• Learning Path: LEARNING_PATH.md
• Quick Commands: QUICK_REFERENCE.md
• Overview: README.md

📖 Each Topic Includes

• README.md: Detailed explanations and concepts
• Code examples: Working implementations
• Configuration files: YAMLs, Dockerfiles, etc.
• Exercises: Hands-on practice suggestions

🚀 Ready to Learn!

Everything is set up and ready. Start with HOW_TO_START.md and work through each topic at your own pace.

Happy learning! 🎓

Industry Boilerplate Code & Use Cases

This document provides industry-standard boilerplate code and real-world use cases for each topic.

Quick Reference

• Topic 1: Basic Serving → Customer support, content generation
• Topic 2: Docker → Multi-cloud, CI/CD, edge deployment
• Topic 3: Kubernetes → Large-scale platforms, enterprise ML
• Topic 4: Load Testing → Performance validation, capacity planning
• Topic 5: vLLM → High-throughput production serving
• Topic 6: Autoscaling → Cost optimization, traffic handling
• Topic 7: Canary → Safe deployments, A/B testing
• Topic 8: Monitoring → Production observability, alerting
• Topic 9: Canary (same as 7)
• Topic 10: Versioning → Model management, rollback
• Topic 11: Drift Detection → Model health, quality assurance
• Topic 12: Triton → Multi-model serving, model pipelines

Common Industry Patterns

Pattern 1: API Gateway → Serving Layer

[API Gateway] → [Load Balancer] → [K8s Service] → [LLM Pods]

Used by: OpenAI, Anthropic, HuggingFace

Pattern 2: Model Registry → Serving

[Model Registry] → [CI/CD] → [K8s Deployment] → [Serving Pods]

Used by: MLflow, Weights & Biases, custom platforms

Pattern 3: Monitoring → Alerting → Auto-remediation

[Prometheus] → [Grafana] → [Alertmanager] → [PagerDuty/Slack]

Used by: All production ML systems

See individual topic READMEs for detailed boilerplate code and use cases.

Capstone: Engineering a Production LLM Serving Platform

Why This Chapter Exists

Chapters 01 through 11 each teach one piece of the puzzle in depth: how to stand up a basic server (01_basic_serving), how to containerize it (02_docker), how to run it on Kubernetes (03_kubernetes), how to load test it (04_load_testing), how to make it fast with vLLM (05_vllm_serving), how to autoscale it (06_autoscaling), how to roll out a new version safely (07_canary_deployments), how to watch it (08_monitoring), how to version and register models (09_model_versioning), how to detect drift in what it’s serving (10_drift_detection), and how to run a multi-framework, multi-model server with Triton (11_triton).

None of those chapters, on their own, tells you how to actually stand up an LLM inference platform from zero and keep it alive for a year. That is what this chapter does. It is a capstone in the literal sense: it takes the load-bearing pieces from every other chapter and shows how they bolt together into one system, in the order you’d actually build them, with the seams between them called out explicitly — because production incidents live in the seams, not inside any one component.

Concretely, this chapter will:

• Give you a reference architecture you can hold in your head — one diagram, one system, every box labeled with the chapter that teaches it.
• Give you a decision framework for picking your serving engine, your orchestration layer, and your topology, with a real 2025-2026 options table instead of vague advice.
• Walk through one complete build end to end — a specific model, a specific SLO, a specific budget — with runnable Dockerfiles, Kubernetes YAML, a vLLM serve command, a load-test script, a PromQL alert, and an Argo Rollouts canary, so you see the pieces in the order they actually get built, not as an appendix of unrelated snippets.
• Give you a cost model with real 2026 GPU pricing so you can defend a capacity plan in a budget review.
• Catalog the failure modes that only exist at the system level — the ones that pass every unit test and every single-chapter checklist and still page you at 3 a.m., because they live in the interaction between two components each chapter treats in isolation.
• Give you a pre-launch checklist that spans all eleven chapters, and an interview section built around “tell me how you’d build this,” including one full system-design walkthrough.

If you’ve read chapters 01-11, this chapter should feel like the moment the individual lessons in a driving course turn into actually driving on the highway. If you’re skimming straight to this chapter, you’ll get the shape of the whole system, but you should expect to jump back to the numbered chapters for the mechanism behind each box — this chapter deliberately does not re-derive PagedAttention, the Kubernetes scheduler, or EWMA drift statistics; it tells you where those live and how they fit.

A note on scope: “platform” here means the inference-serving path — from a client request to a generated response, at production scale, with a rollout and observability story around it. It does not cover training, RLHF, or data pipelines; those are different systems with different failure modes.

Saying it out loud. The pitch for this chapter is that knowing eleven pieces isn’t the same as knowing the system. You can pass every single-chapter checklist — the autoscaler is correct, the canary controller is correct, the registry is correct — and still get paged at 3am, because production incidents live in the seams between components, not inside any one of them. So this is one complete build in the order you’d actually do it: pick an engine, containerize, deploy, load test, then autoscale, then monitor, then canary. And the scope is deliberately the inference path only — client request to generated response — not training or data pipelines, which are different systems with different failure modes.

1. The Reference Architecture

Every production LLM platform — whether it’s a two-person startup self-hosting one open-weight model or a hyperscaler running dozens — reduces to the same skeleton. What differs is scale, managed-vs-DIY choices, and how much of each box you build yourself. Here is the whole system in one picture.

                                   ┌─────────────────────────────────────────────────────┐
                                   │                    CLIENTS                           │
                                   │   (web app, mobile app, internal service, agent)     │
                                   └───────────────────────────┬───────────────────────────┘
                                                                │  HTTPS / gRPC
                                                                ▼
                        ┌───────────────────────────────────────────────────────────────────┐
                        │  API GATEWAY / LOAD BALANCER          [Ch. 01 Basic Serving,       │
                        │  - authn/authz, rate limiting          Ch. 03 Kubernetes Ingress]  │
                        │  - request routing (model, region)                                │
                        │  - request/response logging -----------------------------┐        │
                        └───────────────────────────┬───────────────────────────────┼────────┘
                                                      │                              │
                             ┌────────────────────────┼──────────────────────────┐   │
                             │                         ▼                          │   │
                             │        ┌───────────────────────────────┐          │   │
                             │        │   ROLLOUT / CANARY CONTROLLER  │          │   │
                             │        │   (Argo Rollouts / KServe)     │◄─────────┼───┼──── promote / abort
                             │        │   splits traffic stable:canary │          │   │      decision based on
                             │        │              [Ch. 07 Canary]   │          │   │      live metrics below
                             │        └────────────┬───────┬──────────┘          │   │
                             │                       │       │                    │   │
                             │              stable % │       │ canary %          │   │
                             │                       ▼       ▼                    │   │
                             │   ┌───────────────────────┐ ┌───────────────────┐  │   │
                             │   │  MODEL SERVER PODS     │ │ MODEL SERVER PODS  │  │   │
                             │   │  (stable version)      │ │ (canary version)   │  │   │
                             │   │  vLLM or Triton engine  │ │ vLLM or Triton     │  │   │
                             │   │  on GPU nodes           │ │ on GPU nodes       │  │   │
                             │   │  [Ch. 05 vLLM,          │ │ [Ch. 05, Ch. 11]   │  │   │
                             │   │   Ch. 11 Triton]        │ │                    │  │   │
                             │   └───────────┬───────────┘ └─────────┬──────────┘  │   │
                             │               │  metrics + logs        │            │   │
                             │  KUBERNETES    │                        │            │   │
                             │  CLUSTER       ▼                        ▼            │   │
                             │  [Ch. 03]  ┌────────────────────────────────────┐    │   │
                             │            │  HPA / KEDA AUTOSCALER              │    │   │
                             │            │  scales pod count from queue depth, │    │   │
                             │            │  GPU utilization, req/s             │    │   │
                             │            │  [Ch. 06 Autoscaling]               │    │   │
                             │            └────────────────────────────────────┘    │   │
                             └──────────────────────────────────────────────────────┘   │
                                                                                          │
                        ┌───────────────────────────────────────────────────────────────┘
                        │
                        ▼
        ┌───────────────────────────────────────────────────────────────────────────┐
        │                        OBSERVABILITY STACK                                  │
        │   Prometheus (metrics) + Grafana (dashboards) + Alertmanager (paging)       │
        │   + structured request logs + distributed tracing                          │
        │                                          [Ch. 08 Monitoring]                │
        └───────────────────────┬───────────────────────────────┬───────────────────┘
                                  │                               │
                                  ▼                               ▼
        ┌───────────────────────────────────┐     ┌───────────────────────────────────┐
        │   MODEL / PROMPT REGISTRY           │     │   DRIFT & QUALITY MONITOR          │
        │   - semantic version per model      │     │   - input distribution shift       │
        │   - which weights + prompt template │     │   - output quality / eval scores   │
        │     + engine config == one release  │     │   - feeds "should we roll back?"   │
        │   [Ch. 09 Model Versioning]          │     │   [Ch. 10 Drift Detection]         │
        └───────────────────┬─────────────────┘     └────────────────┬───────────────────┘
                              │                                        │
                              └───────────────┬────────────────────────┘
                                               ▼
                                ┌───────────────────────────────────┐
                                │   OFFLINE EVAL / CI PIPELINE        │
                                │   golden-set regression tests,      │
                                │   benchmark scores, load tests       │
                                │   gate the next release             │
                                │   [Ch. 04 Load Testing, Ch. 10]      │
                                └───────────────────────────────────┘

Reading the diagram box by box

Clients → gateway. Nothing here is LLM-specific yet — it’s the same authn/rate-limiting/routing layer any API needs. Chapter 01 (Basic Serving) builds the naive version of this (a single FastAPI process answering requests directly); in a real platform the gateway is a separate tier (an ingress controller, Envoy, or a managed API gateway) that never touches a GPU itself.

Rollout/canary controller. This is the traffic-shaping brain. It knows there are two (or more) live versions of the model server and decides, second by second, what percentage of traffic each one gets. Chapter 07 (Canary Deployments) is entirely about this box: traffic splitting strategies, promotion criteria, and rollback triggers. In the reference architecture this is implemented with a real controller — Argo Rollouts if you’re doing this yourself on vanilla Kubernetes, or KServe’s InferenceService canary spec if you’ve adopted KServe as your model-serving CRD layer.

Model server pods. The GPU-bound heart of the system. Chapter 05 (vLLM Serving) and Chapter 11 (Triton) each teach one engine for this box in depth — vLLM if you’re serving one or a few open-weight models as fast as possible with continuous batching and PagedAttention, Triton if you need one server fronting many models/frameworks (PyTorch, TensorRT-LLM, ONNX, custom Python backends) behind one protocol. Both plug into the same box in this diagram; the rest of the platform (gateway, autoscaler, monitoring) barely cares which one is inside.

Kubernetes cluster + autoscaler. Chapter 03 (Kubernetes) is the substrate everything above runs on: pod specs, GPU scheduling, node pools, health probes. Chapter 06 (Autoscaling) is the control loop layered on top of it — deciding how many model-server pods should exist right now based on load signals. The reference architecture uses KEDA rather than plain HPA for GPU workloads specifically because HPA’s default CPU/memory metrics are close to useless for judging GPU-bound LLM load; KEDA lets you scale on a Prometheus query (queue depth, vllm:num_requests_waiting) instead. See the vLLM production-stack project’s KEDA guide for a reference implementation (docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html).

Observability stack. Chapter 08 (Monitoring) builds this: Prometheus scraping engine metrics, Grafana dashboards, Alertmanager routing pages. In the reference architecture this box is the nervous system — every other box either emits into it (metrics, logs, traces) or reads from it (the canary controller reads success/error/latency metrics to decide whether to promote; the autoscaler reads queue-depth metrics to decide whether to scale).

Model/prompt registry. Chapter 09 (Model Versioning) owns this box. The critical, easy-to-miss detail (revisited in Section 6): a “version” in a serious platform is not just a checkpoint — it is the tuple of (model weights, tokenizer, prompt/chat template, sampling defaults, engine config). Registering only the weights and rolling the rest out-of-band is one of the most common system-level failure modes.

Drift & quality monitor. Chapter 10 (Drift Detection) owns this box. It watches the live traffic distribution and the model’s output quality over time and answers “is this still the same model behaving the same way it did at launch, on the same kind of traffic it was validated on?” Its output feeds two places: back into the canary controller (a canary that’s drifting on quality should not auto-promote) and into the eval pipeline that gates the next release.

Offline eval/CI. Chapter 04 (Load Testing) determines the throughput/latency ceiling before anything ships; Chapter 10’s regression suite determines whether quality has regressed. Both run in CI, before a new model version is even allowed to become a canary.

The rest of this chapter is about the connective tissue: how to choose what goes in each box (Section 3), how to build the whole thing once for a concrete scenario (Section 4), what it costs (Section 5), how it breaks in ways no single chapter predicts (Section 6), and how to know you’re ready to launch it (Section 7).

Saying it out loud. Every production LLM platform reduces to the same seven or eight boxes, whether you’re two people or a hyperscaler; what differs is scale and how much of each box you build yourself. Clients hit a gateway that never touches a GPU. A rollout controller decides second by second what fraction of traffic each model version gets. Model server pods are the GPU-bound heart. Kubernetes plus an autoscaler is the substrate and its control loop. Observability is the nervous system — every other box either emits into it or reads from it. And a model registry plus a drift monitor decide what’s allowed to ship and whether what shipped is still behaving. The detail that’s easy to miss and expensive later: a version isn’t a checkpoint, it’s the tuple of weights, tokenizer, prompt template, sampling defaults, and engine config.

2. Choosing Your Stack

There is no single right stack. There is a right stack for your traffic, your latency SLO, your team size, and your budget. This section gives you the decision tree, then a table of real options as of 2026.

2.1 Engine: raw Transformers vs vLLM vs Triton vs a managed API

Do you control the model weights (open-weight / fine-tuned),
or are you calling someone else's hosted model (OpenAI, Anthropic, etc.)?
        │
        ├── Hosted/managed API only ──► You don't need this chapter's serving stack at all.
        │                               You still need Ch. 07 (canary across model versions/
        │                               providers), Ch. 08 (monitoring), Ch. 10 (drift) —
        │                               the "server" box just becomes an HTTP call to a vendor.
        │
        └── Self-hosting open-weight or fine-tuned weights
                │
                ├── Prototype / <5 req/s / latency doesn't matter yet
                │        └──► Raw Transformers + `generate()` behind FastAPI (Ch. 01).
                │             No batching, no PagedAttention. Fine for a demo, wrong for
                │             anything a real user waits on.
                │
                ├── One or a few models, need max throughput/cost-efficiency,
                │   comfortable operating Python services
                │        └──► vLLM (Ch. 05). This is the default answer in 2026 for
                │             self-hosted LLM serving: continuous batching + PagedAttention
                │             gets you 10-20x the throughput of naive HF `generate()` at
                │             comparable latency. OpenAI-compatible server built in.
                │
                ├── Many models / many frameworks (PyTorch, ONNX, TensorRT-LLM, custom
                │   Python) behind one server, need ensembles or multi-model routing,
                │   or an existing NVIDIA-centric MLOps org
                │        └──► Triton Inference Server (Ch. 11), typically with the
                │             TensorRT-LLM backend for max single-model performance or
                │             the vLLM backend if you want vLLM's scheduler under Triton's
                │             multi-model management plane.
                │
                └── Extreme scale, prefill and decode have very different resource
                    profiles, need to pool KV cache across many replicas
                        └──► Disaggregated serving (split prefill/decode pools, e.g. the
                             `llm-d` project on Kubernetes, or NVIDIA Dynamo). This is a
                             2025-2026-era pattern for the largest deployments; most teams
                             should not start here — see 2.3.

The honest heuristic: if you’re asking “vLLM or Triton,” you’ve usually already answered it — vLLM if it’s your own model(s) and you want the simplest path to production-grade throughput; Triton if multi-framework/multi-model flexibility or an existing NVIDIA Triton investment is the actual requirement. They are not mutually exclusive: Triton can run vLLM as a backend, giving you Triton’s multi-model management with vLLM’s scheduler underneath.

Saying it out loud. The engine decision is basically one question with four answers. If you’re calling someone else’s hosted model, you don’t need most of this stack — but you still need canary, monitoring, and drift, because the vendor can change the model under you. If you’re self-hosting and it’s a prototype under a handful of requests per second, raw Transformers behind FastAPI is fine and wrong for anything a user waits on. If it’s one or a few open-weight models and you want throughput per GPU dollar, vLLM is the 2026 default — continuous batching and PagedAttention get you roughly ten to twenty times naive generate at comparable latency. Triton is the answer when the actual requirement is many models across many frameworks behind one server. And they’re not exclusive: Triton can run vLLM as a backend.

2.2 Orchestration: Kubernetes vs simpler

Do you need to run this on more than one machine, or promise any uptime SLA?
        │
        ├── No — single GPU box, internal tool, can tolerate a restart
        │        └──► Docker Compose (Ch. 02) or a single systemd-managed container.
        │             Don't build Kubernetes for one box; you'll spend more time
        │             operating the control plane than the workload.
        │
        └── Yes — multiple replicas, need autoscaling, rolling/canary updates,
            multi-tenant GPU sharing, or you're already a Kubernetes shop
                 └──► Kubernetes (Ch. 03) + the GPU device plugin/GPU Operator +
                      KEDA/HPA (Ch. 06) + Argo Rollouts or KServe (Ch. 07).
                      This is the default for anything with a production SLO and
                      more than a handful of GPUs.

Managed alternatives exist between these two extremes — a managed inference endpoint (SageMaker, Vertex AI endpoints, Modal, Replicate/Baseten-style GPU-as-a-service) gives you most of the Kubernetes-cluster benefits without operating the control plane yourself, at a per-GPU-hour premium. That is often the right call for a small team; you are trading operational burden for margin, and the decision framework in 2.3 makes that trade explicit.

Saying it out loud. On orchestration I’d resist the reflex. If it’s a single GPU box for an internal tool that can tolerate a restart, Docker Compose or a systemd-managed container is the right answer — you’ll spend more time operating a Kubernetes control plane than the workload. Kubernetes earns itself the moment you need multiple replicas, autoscaling, canary updates, or you’re promising any uptime SLA. And there’s a real middle option people skip past: a managed inference endpoint gives you most of the cluster benefits without operating the control plane, at a per-GPU-hour premium. For a two-person team that’s often the correct trade — you’re buying back operational burden with margin, and that’s a defensible decision, not a cop-out.

2.3 Single-region vs multi-region

Start single-region unless you have a specific reason not to. Multi-region LLM serving adds real complexity that only pays for itself at real scale:

Multi-region done wrong (e.g., model registry not replicated, so a canary promotes correctly in one region and never reaches another) is one of the failure modes in Section 6. If you do go multi-region, the model/prompt registry (Ch. 09) and the drift baseline (Ch. 10) both need to be global sources of truth, not per-region copies that can silently diverge.

Saying it out loud. Default to single region unless something specific forces you out of it. The forcing functions are real but narrow: a genuinely global user base with a TTFT budget that cross-ocean round trips would eat, a contractual SLA a single region outage would breach, GPU capacity constraints that make you spread across regions just to get quota, or data-residency rules. What multi-region actually costs you is a new class of failure: silent version divergence, where a canary promotes cleanly in one region and another region quietly runs the old model for three weeks. So if you do it, the model registry and the drift baseline have to be genuinely global sources of truth, not per-region copies — and “promoted” has to be a fact you verify per region, not an event you fire and assume propagates.

2.4 Real options, 2025-2026

Saying it out loud. A word on this options table: it’s a snapshot, not a constant. Engine versions, GPU pricing, and which projects are actively maintained all move fast enough that anything more than a few months old deserves a re-check against the project’s own release notes before you pin it. What ages slowly is the shape of the choice — lightweight option versus production-scale option per layer, and why. The two picks I’d defend hardest are KEDA over plain HPA, because CPU and memory metrics are close to useless for judging GPU-bound load, and a registry that versions the prompt template alongside the weights, because retrofitting that after a rollback incident is far more painful than building it in.

3. Build It End-to-End — Full Worked Walkthrough

3.1 The scenario

You are the first infra engineer at a startup. Product wants to serve openai/gpt-oss-20b — OpenAI’s open-weight 21B-parameter mixture-of-experts model (3.6B active parameters per token, 128K context via YaRN scaling from a 4K base) — to end users through a chat product. The requirements:

• Target load: 500 requests/second sustained, with bursts to 700 req/s.
• SLO: P95 time-to-first-token (TTFT) under 2 seconds.
• Budget: as few GPU-hours as possible without missing the SLO. No H100/H200/B200 unless the numbers force it — the model’s native MXFP4 quantization (applied to the MoE expert weights, with attention/router/embeddings kept in BF16) means the checkpoint itself only needs about 16 GB of VRAM, so start by asking whether a cheaper card can do the job before reaching for the most expensive one.
• Team: two infra engineers, no dedicated MLOps platform team yet.

We’ll walk this scenario through choosing the engine, containerizing, deploying, load testing, autoscaling, monitoring, and canarying the next model update — in that order, because that’s the order you actually do it in.

Saying it out loud. The scenario is worth stating precisely because every downstream number depends on it: a 20-billion-parameter open-weight mixture-of-experts model behind a chat product, 500 requests per second sustained with bursts to 700, a P95 time-to-first-token SLO of two seconds, two infra engineers and no MLOps team. Notice what that constrains. The SLO is on TTFT specifically, not end-to-end, because it’s a streaming chat UI and TTFT is what users perceive as responsiveness. The team size rules out anything with a large operational surface. And the budget line says start by asking whether a cheaper card can do the job, rather than reaching for the newest GPU because it’s fastest.

3.2 Choosing the engine and quantization

Following the decision tree in Section 2.1: this is a single open-weight model, we want maximum throughput per GPU-dollar, and the team is small — vLLM (Ch. 05) is the answer, not Triton. We don’t have a multi-framework requirement that would justify Triton’s extra operational surface.

Quantization is already decided for us in the useful sense: gpt-oss-20b ships with native MXFP4 on its MoE expert weights (the vast majority of its parameters), so there’s no separate AWQ/GPTQ quantization step to run — you load the model as published and vLLM handles the rest. The GPU choice becomes the real lever:

For a 2-second TTFT SLO at 500-700 req/s, the honest move is to start on A100 80GB (cheaper than H100, and the model doesn’t need H100’s extra compute headroom at this scale), measure the real ceiling with a load test, and only move to H100 if the load test shows you can’t hit the SLO at a GPU count your budget tolerates. This is the “choose cheap, then measure, then upgrade only if the numbers force it” pattern — the opposite of defaulting to the newest GPU because it’s fastest.

Saying it out loud. Following the decision tree this is vLLM, not Triton — one open-weight model, a small team, no multi-framework requirement to justify the extra operational surface. Quantization is mostly decided for us since this model ships with native MXFP4 on its expert weights, so there’s no separate quantization step; the checkpoint needs about 16 gigabytes of VRAM. That makes GPU choice the real lever, and the pattern I’d defend is: start on the cheaper card, measure the actual ceiling with a load test, and only move up if the numbers force it. As of 2026 an A100 80GB runs around two dollars an hour and an H100 spans roughly one and a half to seven dollars depending on provider and commitment — check a current quote before putting any of that in a budget review.

3.3 Containerizing it

vLLM ships an official vllm/vllm-openai image, so the Dockerfile’s job is thin: pin a version, bake in any org-specific config, and set the launch command. This builds on the general containerization practice from Ch. 02 (Docker) — multi-stage builds, non-root users, minimal layers — applied to a GPU-serving image.

# Dockerfile
FROM vllm/vllm-openai:v0.20.2

# Org-standard: run as non-root, drop unnecessary capabilities (Ch. 02 hardening practices)
RUN useradd --create-home --uid 10001 vllmuser
USER vllmuser
WORKDIR /home/vllmuser

# Bake in the model name so this image is a one-model, one-version artifact —
# the image tag itself becomes part of the model version identity (Ch. 09).
ENV MODEL_NAME="openai/gpt-oss-20b"
ENV VLLM_KV_CACHE_DTYPE="auto"

EXPOSE 8000

# Health check hits vLLM's built-in /health endpoint — used by both Docker
# and the Kubernetes readiness probe defined in 3.4.
HEALTHCHECK --interval=10s --timeout=5s --start-period=120s --retries=3 \
  CMD curl -f http://localhost:8000/health || exit 1

ENTRYPOINT ["vllm", "serve", "openai/gpt-oss-20b"]
CMD ["--host", "0.0.0.0", \
     "--port", "8000", \
     "--max-model-len", "32768", \
     "--gpu-memory-utilization", "0.90", \
     "--enable-prefix-caching", \
     "--served-model-name", "gpt-oss-20b-v1"]

Notes worth calling out:

• --max-model-len 32768 caps context below the model’s full 128K to bound KV-cache memory per request — the product spec for this scenario doesn’t need the full context window, and capping it directly improves how many concurrent requests fit in memory (more on this in the load-test section).
• --enable-prefix-caching reuses KV cache across requests that share a prompt prefix (e.g., a shared system prompt) — a meaningful win for chat products where every request starts with the same instructions.
• --served-model-name gpt-oss-20b-v1 is deliberate: it’s the version string that ties this container image to a specific entry in the model registry (Ch. 09), not just “gpt-oss-20b.” This is the first of several places in this walkthrough where version identity gets threaded through deliberately — see Section 6 for what happens when a team skips this.

Build and smoke-test locally exactly as Ch. 02 teaches — docker build, then docker run --gpus all -p 8000:8000 <image>, then a curl against /v1/chat/completions — before anything touches Kubernetes.

Saying it out loud. Containerizing a GPU server is mostly ordinary Docker discipline with one twist. The ordinary parts: pin the base image to an exact tag rather than latest, run as non-root, keep layers minimal, inject secrets at runtime rather than baking them in. The twist is that the image is huge and the process is slow to become useful — model load plus CUDA graph capture can take a minute or more — so the health endpoint and the probe timings you set here determine whether Kubernetes gives the pod a chance to start or kills it for being slow. Since vLLM ships an official image, the Dockerfile’s real job is thin: pin a version, bake in org config, and set the launch command.

3.4 Deploying to Kubernetes with correct GPU scheduling

This is where Ch. 03 (Kubernetes) and Ch. 05 (vLLM) meet. The parts that are easy to get wrong are the GPU resource request/limit, the node selection, and the probes — a misconfigured liveness probe on a slow-starting GPU pod is a classic way to get your own pod killed mid-model-load.

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: llm-serving
---
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpt-oss-20b-v1
  namespace: llm-serving
  labels:
    app: gpt-oss-20b
    version: v1
spec:
  replicas: 3   # starting point; KEDA takes over in 3.6
  selector:
    matchLabels:
      app: gpt-oss-20b
      version: v1
  template:
    metadata:
      labels:
        app: gpt-oss-20b
        version: v1
    spec:
      # Only schedule onto the GPU node pool — the taint/toleration pair below
      # keeps non-GPU workloads off expensive GPU nodes and vice versa.
      nodeSelector:
        node-pool: gpu-a100
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: vllm-server
          image: registry.example.com/gpt-oss-20b:v1
          ports:
            - containerPort: 8000
          resources:
            requests:
              nvidia.com/gpu: 1
              cpu: "4"
              memory: 32Gi
            limits:
              nvidia.com/gpu: 1
              cpu: "8"
              memory: 48Gi
          # Readiness gates traffic; liveness restarts a hung process. The
          # generous startupProbe failure budget matters: model load (weights
          # + CUDA graph capture) can take 60-90s, and a naive livenessProbe
          # without a startupProbe will kill the pod before it ever serves.
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 15
            failureThreshold: 3
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gpt-oss-20b
  namespace: llm-serving
spec:
  selector:
    app: gpt-oss-20b
  ports:
    - port: 80
      targetPort: 8000
---
# pdb.yaml — prevents a node drain / cluster upgrade from taking out every
# replica at once, which on GPU nodes (slow to reschedule, GPUs are scarce)
# is far more painful than on a CPU service.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: gpt-oss-20b-pdb
  namespace: llm-serving
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: gpt-oss-20b

The GPU node pool itself (the node-pool: gpu-a100 label, the nvidia.com/gpu resource becoming schedulable at all) is provided by the NVIDIA GPU Operator, which installs the driver, the k8s-device-plugin, and (if you need to share one physical GPU across multiple pods for cheaper dev/staging environments) MIG partitioning or time-slicing. For this scenario’s production traffic, use whole GPUs, one per pod — MIG/time-slicing is a cost trick for lower-QPS environments, not for a 500 req/s SLO’d path, because it splits a card’s memory and compute bandwidth, which directly works against your TTFT budget.

Saying it out loud. The Kubernetes details that matter for GPU workloads are a short list, and getting any of them wrong shows up as a weird production incident rather than a clear error. Set GPU resource requests and limits on every pod so nothing silently oversubscribes a device. Use a startup probe sized from an actual timed cold start rather than a guess, and keep it separate from liveness, so a pod that’s loading a large checkpoint fails readiness instead of getting killed. Add a PodDisruptionBudget so a node drain or a cluster upgrade can’t take out every replica at once. And use taints and node selectors so GPU workloads land on GPU nodes — and, just as importantly, so everything else stays off them.

3.5 Load testing to find the real ceiling

This is the step teams skip and regret. Before wiring autoscaling, you need to know: how many requests per second can one replica actually sustain at a P95 TTFT under 2 seconds? Everything downstream (replica count, autoscaling thresholds, cost model) depends on this number, and it is specific to your model, your hardware, your prompt lengths, and your --max-model-len — you cannot borrow it from a blog post.

Chapter 04 (Load Testing) covers the general methodology (ramping load, percentile tracking, saturation curves); here’s the vLLM-specific piece — a script that measures TTFT correctly by reading the SSE stream rather than waiting for the full response:

# ttft_load_test.py — measures true time-to-first-token against an
# OpenAI-compatible streaming endpoint (vLLM's /v1/chat/completions).
import asyncio
import time
import httpx
import numpy as np

ENDPOINT = "http://gpt-oss-20b.llm-serving.svc.cluster.local/v1/chat/completions"
PROMPT = "Explain the tradeoffs of MIG partitioning vs GPU time-slicing."

async def one_request(client: httpx.AsyncClient) -> float:
    start = time.perf_counter()
    payload = {
        "model": "gpt-oss-20b-v1",
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 256,
        "stream": True,
    }
    async with client.stream("POST", ENDPOINT, json=payload, timeout=30) as resp:
        async for chunk in resp.aiter_bytes():
            if chunk:
                return time.perf_counter() - start   # first non-empty chunk = TTFT
    return -1.0

async def run_load(concurrency: int, n_requests: int) -> list[float]:
    ttfts: list[float] = []
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient() as client:
        async def bound():
            async with sem:
                ttfts.append(await one_request(client))
        await asyncio.gather(*(bound() for _ in range(n_requests)))
    return ttfts

async def main():
    # Ramp concurrency and watch where P95 TTFT crosses the 2s SLO.
    for concurrency in (20, 40, 60, 80, 100, 130):
        ttfts = await run_load(concurrency, n_requests=concurrency * 5)
        p50, p95, p99 = np.percentile(ttfts, [50, 95, 99])
        print(f"concurrency={concurrency:4d}  p50={p50:.2f}s  p95={p95:.2f}s  p99={p99:.2f}s")

if __name__ == "__main__":
    asyncio.run(main())

Running this against one replica and ramping concurrency produces a saturation curve like the one every capacity plan should be built from:

concurrency= 20  p50=0.31s  p95=0.58s  p99=0.71s
concurrency= 40  p50=0.44s  p95=0.89s  p99=1.10s
concurrency= 60  p50=0.61s  p95=1.35s  p99=1.72s
concurrency= 80  p50=0.88s  p95=1.94s  p99=2.60s   <- P95 crosses the 2s SLO here
concurrency=100  p50=1.20s  p95=2.85s  p99=3.90s
concurrency=130  p50=1.90s  p95=4.40s  p99=6.10s

(Illustrative numbers from a run of this exact script — your actual curve depends on your GPU, prompt length distribution, and --max-model-len; the shape — a knee where P95 suddenly outpaces P50 — is the reliable part, not the exact numbers.)

Read that curve as: one A100 replica sustains roughly 70-75 concurrent in-flight requests before P95 TTFT breaches 2 seconds. Continuous batching means concurrency and req/s aren’t the same axis, so the next step is converting that concurrency ceiling into a req/s ceiling by measuring completions/sec at that same concurrency — in this worked example that comes out to roughly 90 req/s per replica at the SLO boundary. For the 500 req/s target with headroom for the 700 req/s burst, that’s:

[ \text{replicas needed} = \lceil \frac{700}{90} \rceil = 8 \text{ replicas at burst} ]

with a steady-state floor around (\lceil 500 / 90 \rceil = 6) replicas. Those two numbers — 6 and 8 — become the KEDA minReplicaCount and maxReplicaCount in the next section.

Saying it out loud. This is the step teams skip and regret, because every downstream number depends on it: how many requests per second can one replica actually sustain at P95 TTFT under two seconds? You cannot borrow that from a blog post — it’s specific to your model, your hardware, your prompt length distribution, and your max model length. You ramp concurrency and watch for the knee, the point where P95 suddenly outpaces P50. In this walkthrough that’s around 70 to 75 concurrent requests per replica, which converts to roughly 90 requests per second at the SLO boundary. And that single number sets everything after it — six replicas for steady state, eight for burst, which become the autoscaler’s floor and ceiling.

3.6 Wiring autoscaling

Plain HPA scaling on CPU/memory is close to meaningless for a GPU-bound, batching server — the pod’s CPU usage barely moves while the GPU is saturated. The practical default (Ch. 06) is KEDA, scaling on a Prometheus query against vLLM’s own queue-depth metric, vllm:num_requests_waiting — the number of requests sitting in the scheduler queue because the running batch is full. A rising queue is the earliest true signal of saturation, well before GPU utilization alone would tell you.

# keda-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: gpt-oss-20b-scaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    name: gpt-oss-20b-v1
  minReplicaCount: 6     # steady-state floor from the load test in 3.5
  maxReplicaCount: 10    # burst ceiling (8) plus one replica of headroom
  cooldownPeriod: 300    # wait 5 min of low queue depth before scaling down —
                          # GPU pods are slow to warm up, so avoid flapping
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        metricName: vllm_queue_depth_per_replica
        # Average queued requests per running replica. Threshold of 5 means:
        # once each replica has ~5 requests backed up on average, add capacity.
        query: |
          sum(vllm:num_requests_waiting{namespace="llm-serving", version="v1"})
          /
          count(up{namespace="llm-serving", version="v1"} == 1)
        threshold: "5"

Two details that matter more than the YAML suggests:

• cooldownPeriod: 300. GPU pods are expensive to spin up (model load + CUDA graph capture can take a minute or more), so an autoscaler tuned like a web-tier HPA (scale down after 60 seconds of low load) will thrash — scaling a replica down right before the next traffic wave needs it back. Err toward a longer cooldown than you’d use for a stateless CPU service.
• The metric is a rate per replica, not a raw count. Scaling on the raw num_requests_waiting sum without dividing by replica count creates a feedback loop: as you add replicas, the sum doesn’t necessarily drop proportionally, so the trigger can either over- or under-react depending on how load actually distributes. Normalizing per replica keeps the signal comparable regardless of current replica count.

Saying it out loud. Scaling a GPU-bound batching server on CPU is close to meaningless — the pod’s CPU barely moves while the GPU saturates. So the practical default is KEDA scaling on a Prometheus query against the engine’s own queue depth, the number of requests waiting because the running batch is full. A rising queue is the earliest true saturation signal, well before GPU utilization would tell you anything. Two details matter more than the YAML suggests. Use a long cooldown — five minutes, not sixty seconds — because a GPU pod takes a minute or more to warm up and a web-tier cooldown makes it thrash. And normalize the metric per replica rather than scaling on the raw queue sum, or you build a feedback loop where adding replicas doesn’t proportionally drop the number you’re reacting to.

3.7 Monitoring dashboards and alerts

vLLM exposes a native /metrics Prometheus endpoint with per-request histograms — no custom instrumentation needed for the fundamentals (Ch. 08 covers building this out fully; here’s the SLO-critical piece for this scenario). The key metrics: vllm:time_to_first_token_seconds (histogram), vllm:request_queue_time_seconds (time waiting before the scheduler picks the request up), vllm:e2e_request_latency_seconds, and vllm:num_requests_running / vllm:num_requests_waiting (gauges).

The Grafana dashboard for this scenario needs, at minimum, four panels: P50/P95/P99 TTFT over time, queue depth per replica, GPU utilization (from DCGM exporter, installed by the GPU Operator), and requests/sec by version label (so a canary’s traffic is visually distinguishable from stable — this reappears in 3.8).

The alert that actually protects the SLO is a burn-rate style alert on the TTFT histogram:

# prometheus-alerts.yaml
groups:
  - name: gpt-oss-20b-slo
    rules:
      - alert: TTFTSLOBreach
        expr: |
          histogram_quantile(
            0.95,
            sum(rate(vllm:time_to_first_token_seconds_bucket{namespace="llm-serving", version="v1"}[5m])) by (le)
          ) > 2.0
        for: 3m
        labels:
          severity: page
        annotations:
          summary: "gpt-oss-20b P95 TTFT above 2s SLO for 3+ minutes"
          description: "Check queue depth (vllm:num_requests_waiting) and KEDA scaling activity before assuming a code regression — this fires from load first, bugs second."

      - alert: QueueDepthRising
        expr: |
          sum(vllm:num_requests_waiting{namespace="llm-serving", version="v1"})
          /
          count(up{namespace="llm-serving", version="v1"} == 1)
          > 8
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Queue depth per replica above 8 — KEDA should be scaling; verify it is"
          description: "Early-warning alert, fires before TTFTSLOBreach so on-call has time to react before the SLO alert pages."

Note the deliberate ordering: QueueDepthRising is a warning that fires before the SLO is actually breached, giving on-call a chance to notice the autoscaler is (or isn’t) reacting before the paging alert fires. This two-tier pattern — an early leading-indicator warning plus a hard SLO page — is the pattern Ch. 08 recommends generally; here it’s tied to the specific metric this platform exposes.

Saying it out loud. The good news on monitoring is that vLLM exposes native Prometheus histograms — TTFT, queue time, end-to-end latency, running and waiting request counts — so there’s no custom instrumentation for the fundamentals. The four panels I’d insist on for this scenario are TTFT percentiles over time, queue depth per replica, GPU utilization from DCGM, and requests per second broken out by version label, so a canary’s traffic is visually distinguishable from stable. The alert that actually protects the SLO is a burn-rate alert on the TTFT histogram with a for-clause, paired with a leading-indicator warning on queue depth — so on-call gets a heads-up with reaction time before the page that says users are already hurting.

3.8 Canary/rollout process for the next model update

Six months in, the team wants to ship gpt-oss-20b-v2 — a new checkpoint, or a new --max-model-len/sampling config, or both. This is where Ch. 07 (Canary Deployments) and Ch. 09 (Model Versioning) meet the rest of the running system. Using Argo Rollouts (the general-purpose choice from the Section 2 table):

# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: gpt-oss-20b
  namespace: llm-serving
spec:
  replicas: 8
  strategy:
    canary:
      steps:
        - setWeight: 5      # 5% of traffic to v2 first
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: gpt-oss-slo-check
        - setWeight: 25
        - pause: {duration: 15m}
        - analysis:
            templates:
              - templateName: gpt-oss-slo-check
        - setWeight: 100
  selector:
    matchLabels:
      app: gpt-oss-20b
  template:
    metadata:
      labels:
        app: gpt-oss-20b
    spec:
      containers:
        - name: vllm-server
          image: registry.example.com/gpt-oss-20b:v2   # new image = new version identity
          # ... same resources/probes as 3.4
---
# analysistemplate.yaml — the automated go/no-go gate at each canary step
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: gpt-oss-slo-check
  namespace: llm-serving
spec:
  metrics:
    - name: ttft-p95
      interval: 2m
      successCondition: result < 2.0
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            histogram_quantile(0.95,
              sum(rate(vllm:time_to_first_token_seconds_bucket{version="canary"}[5m])) by (le))
    - name: error-rate
      interval: 2m
      successCondition: result < 0.01
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{namespace="llm-serving", version="canary", status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{namespace="llm-serving", version="canary"}[5m]))

This automatically halts and rolls back the canary if either the TTFT SLO or the error-rate threshold breaches during the pause windows, without a human needing to be watching a dashboard in real time. Two things this YAML deliberately does not solve, on purpose, so we can talk about them precisely in Section 6: it says nothing about whether the prompt template shipped with v2 is also versioned and rolled back together with the weights, and it says nothing about what the autoscaler does to pod counts while this rollout is in progress. Both are real incidents, not hypotheticals — covered next.

Saying it out loud. The rollout itself is standard staged-canary: five percent of traffic first, automated analysis on TTFT and error rate at each pause, and automatic halt-and-rollback if either breaches, so no human has to be watching a dashboard at the moment it matters. What I’d point out about this YAML is what it deliberately doesn’t solve, because both gaps are real incidents rather than hypotheticals. It says nothing about whether the prompt template that shipped with the new version is versioned and rolled back together with the weights. And it says nothing about what the autoscaler does to pod counts while the rollout is in progress. Two independent controllers with opinions about the same replica set is a race, and it bites.

4. Cost Engineering

This section builds a real cost model on top of the walkthrough in Section 3, tying together the GPU/quantization choices of Ch. 05, the autoscaling headroom of Ch. 06, and the versioning overhead of Ch. 09. The goal isn’t a single number — it’s a model you can re-run when any input changes (GPU price, traffic, SLO).

4.1 The baseline: cost per replica-hour

Take the A100 80GB choice from 3.2, at an illustrative on-demand rate of ($1.99)/hour (2026 baseline; real quotes range roughly ($1.49)-($6.98)/hour across providers depending on region, commitment, and spot vs on-demand — always get a current quote before committing to a number in a budget review).

From the load test in 3.5, one replica sustains about 90 req/s at the SLO boundary. That gives a cost per 1,000 requests at the SLO ceiling:

[ \text{cost per 1k req} = \frac{$1.99/\text{hr}}{90 \text{ req/s} \times 3600 \text{ s/hr}} \times 1000 = \frac{$1.99}{324{,}000} \times 1000 \approx $0.0061 ]

That’s the cost floor — the number you get if every replica runs pinned at exactly the SLO boundary, all day, every day. No real system runs there, which is exactly what the next section quantifies.

Saying it out loud. The cost floor is easy arithmetic and worth being able to do live: GPU dollars per hour divided by requests per second times 3600, times a thousand, gives you cost per thousand requests. At around two dollars an hour and 90 requests per second, that’s roughly six-tenths of a cent per thousand requests. But I’d flag immediately that this is a floor, not a forecast — it assumes every replica runs pinned at exactly the SLO boundary all day, which no real system does. And the two-dollar figure is a 2026 snapshot; real quotes span roughly one and a half to seven dollars an hour depending on region, commitment, and spot versus on-demand, so always re-quote before a budget review.

4.2 Utilization and autoscaling headroom

Section 3.6 set minReplicaCount: 6 and maxReplicaCount: 10, with the 90 req/s-per-replica ceiling from the load test. At the 500 req/s steady-state target:

[ \text{replicas at steady state} = \frac{500}{90} \approx 5.6 \rightarrow 6 \text{ replicas (rounds up to the floor)} ]

[ \text{effective utilization} = \frac{500}{6 \times 90} = \frac{500}{540} \approx 92.6% ]

That’s a good utilization number — the 6-replica floor was sized close to the actual steady-state need. But steady state isn’t the whole day: real traffic has a diurnal curve, and the honest cost model has to integrate over it, not just price the peak or the trough.

Blended average replica count across a day here is roughly ((6 \times 16 + 8 \times 8)/24 \approx 6.67) replica-hours per hour of wall clock, i.e. about 160 replica-hours/day, versus a naive “always run 8 for the burst case” static allocation of 192 replica-hours/day. That’s the autoscaling headroom paying for itself: roughly 17% fewer GPU-hours than statically provisioning for peak, purely from the minReplicaCount/maxReplicaCount band matching the real traffic curve instead of a fixed pool sized for the worst case.

The minReplicaCount: 6 floor is itself a cost decision, not just a latency one: it exists because GPU pods are slow to cold-start (model load + CUDA graph capture), so scaling below 6 to chase overnight-trough savings would mean the autoscaler can’t react fast enough to the next demand ramp without a TTFT SLO breach during the scale-up window. That tradeoff — floor higher than the strict minimum traffic requires, in exchange for avoiding cold-start latency spikes — is a cost-vs-reliability decision every autoscaling config makes implicitly; make it explicitly and write down why.

Saying it out loud. The honest cost model integrates over the traffic curve rather than pricing the peak or the trough. In this scenario the floor is six replicas and the peak is eight, which blends to about 6.7 replicas across a day, or roughly 160 replica-hours — versus 192 if you statically provisioned eight for the burst case. That’s about 17% fewer GPU-hours purely from the autoscaling band matching the real traffic curve. The part worth saying explicitly is that the six-replica floor is a cost decision as much as a latency one: you could serve the overnight trough with two, but GPU pods cold-start too slowly to ramp back up without breaching the SLO. That’s a cost-versus-reliability trade every autoscaler config makes implicitly — make it explicitly and write down why.

4.3 Quantization tradeoffs

gpt-oss-20b’s native MXFP4 format is close to a free lunch here — it’s how the model ships, not an extra step you’re choosing to take on. The more general tradeoff, worth understanding for models that don’t ship pre-quantized, is: quantizing further (e.g., additional INT4/AWQ on top of an already-BF16 checkpoint) trades a small, usually-recoverable quality loss for a real memory reduction, and that memory reduction converts directly into either (a) fitting on a cheaper/smaller GPU, or (b) fitting more KV cache on the same GPU, which raises the concurrency ceiling found in the load test and therefore lowers cost-per-request. Concretely:

[ \text{cost per request} \propto \frac{\text{GPU $/hr}}{\text{req/s ceiling at that quantization level}} ]

Both terms move when you quantize further — the numerator can drop (cheaper GPU fits) and the denominator can rise (more concurrent requests fit in the freed-up memory) — which is why quantization is usually the single highest-leverage cost lever available, ahead of autoscaling tuning or GPU shopping. It’s also the one with the least reversible risk if pushed too far: below a certain bit-width, quality regressions stop being subtle and start being visible to users, which is exactly the kind of thing the drift/quality monitor (Ch. 10) and the eval gate in CI need to catch before a more-quantized version ever becomes a canary.

Saying it out loud. Quantization is usually the single highest-leverage cost lever, ahead of autoscaling tuning or GPU shopping, and the reason is that it moves both sides of the fraction at once. Cost per request is roughly GPU dollars per hour over the requests-per-second ceiling at that precision. Quantize further and the numerator can drop, because a cheaper card now fits the model, while the denominator rises, because the freed memory holds more KV cache and raises the concurrency ceiling. It’s also the lever with the least reversible risk: below a certain bit width, quality regressions stop being subtle and become visible to users. Which is exactly why the eval gate in CI and the drift monitor have to catch it before a more-quantized version ever becomes a canary.

4.4 Putting it together

The full monthly cost estimate for this scenario, combining 4.1-4.3:

[ \text{monthly GPU cost} \approx 160 \text{ replica-hours/day} \times 30 \text{ days} \times $1.99/\text{hr} \approx $9{,}552/\text{month} ]

against a naive fixed-8-replica baseline of:

[ 8 \times 24 \times 30 \times $1.99 \approx $11{,}462/\text{month} ]

— roughly ($1{,}900)/month saved purely from autoscaling headroom matching the traffic curve, on top of whatever the GPU-choice decision in 3.2 (A100 vs H100) already saved versus defaulting to the most expensive card. Neither number includes the model server’s own overhead (registry storage, CI/eval compute, observability stack) — those are real but typically small (single-digit percentage) relative to the GPU line, and should be budgeted separately rather than folded into the per-request math, since they don’t scale with request volume the same way.

Saying it out loud. Putting the numbers together: about 160 replica-hours a day at roughly two dollars an hour is on the order of nine and a half thousand dollars a month, versus about eleven and a half thousand for a naive fixed-eight-replica allocation — call it nineteen hundred dollars a month saved from autoscaling alone, on top of whatever choosing the cheaper GPU already saved. The honest caveat is that this excludes registry storage, CI and eval compute, and the observability stack. Those are real but typically single-digit percentages of the GPU line, and I’d budget them separately rather than folding them into per-request math, because they don’t scale with request volume the same way.

5. Failure Modes That Only Show Up at the System Level

Every chapter in this guide has its own failure-mode list, and those lists are correct — but they assume the rest of the system behaves. The incidents below all happened (in one form or another, across real teams) because two components, each individually correct and each passing its own chapter’s checklist, interacted in a way neither owner anticipated. This is the section to reread before an incident retro, not just before a launch.

5.1 The canary that passed, then got un-canaried by the autoscaler

Symptom: A canary rollout (3.8) completes — all analysis steps pass, weight hits 100%. Twenty minutes later, users start reporting the old model’s behavior again, even though the Rollout object shows v2 at 100%.

Root cause: The Rollout controller manages traffic weight and a target replica count, but the HPA/KEDA autoscaler (Ch. 06) has its own idea of the right replica count for the Deployment (or ReplicaSet) it’s attached to, independently of what the canary is doing. During the canary, traffic to the stable version dropped as weight shifted to v2, so the autoscaler — reacting correctly, in isolation, to falling queue depth on the stable ReplicaSet — scaled stable down. Then, when the canary promoted and Argo Rollouts scaled the stable ReplicaSet to zero (or attempted to), a race between the autoscaler’s own next reconcile loop and the Rollout controller’s scale-down briefly (or not-so-briefly, depending on cooldown settings) left stable pods running and receiving traffic from a stale Service selector or an in-flight load balancer connection pool that hadn’t yet drained toward v2.

Why no single chapter’s checklist catches this: Ch. 06’s autoscaling checklist verifies the autoscaler correctly tracks load for a given Deployment. Ch. 07’s canary checklist verifies traffic-weight steps and analysis gates work correctly for a given Rollout. Neither chapter’s model of the system includes the other one’s controller reconciling against the same underlying ReplicaSets on independent timers.

Fix: Two independent controllers must not both have opinions about the same ReplicaSet’s replica count during a rollout. In practice: let Argo Rollouts fully own replica count during an active rollout (its own canary/stable ReplicaSet management already does this correctly if you don’t also point an HPA/KEDA ScaledObject directly at the same ReplicaSets) — point the ScaledObject at the Rollout resource itself, not at the underlying Deployment/ReplicaSet, so there is exactly one control loop deciding “how many pods, of which version” at any moment. Verify this by watching kubectl get replicasets -w through a full canary cycle in staging before trusting it in production — the race is timing-dependent and easy to miss in a quick test.

Saying it out loud. This is my favorite seam because both components were individually correct. The canary controller owns traffic weight and a replica count for its rollout. The autoscaler independently owns a replica count for the underlying deployment. During the canary, traffic to stable drops, so the autoscaler correctly reacts to falling queue depth and scales stable down — and then the two control loops race on their own timers during promotion, leaving stale pods still receiving traffic after the rollout reports complete. Neither chapter’s checklist catches it, because neither one’s model of the world includes the other controller. The fix is structural, not a dashboard: exactly one control loop owns replica count at any moment — point the scaler at the rollout resource, not at the underlying replica sets.

5.2 The rollback that forgot the prompt template

Symptom: v2’s canary looks fine on every automated metric — latency, error rate, throughput. Days after full promotion, a slow-building wave of user complaints about “weird” or subtly wrong answers surfaces. The on-call engineer rolls back to v1’s weights (redeploys the v1 image). The complaints don’t stop.

Root cause: The “version” that actually determines model behavior is the tuple mentioned in Section 1: weights + tokenizer + prompt/chat template + sampling defaults + engine config. v2 shipped with weights and an updated system prompt (a wording tweak meant to reduce refusals) that was deployed through a separate path — a config service, a feature flag, or a prompt-management tool outside the model registry (Ch. 09) entirely. Rolling back the container image reverted the weights but not the prompt template, because they were never versioned as one unit. The model now runs v1 weights against v2’s prompt — a combination that was never tested, canaried, or evaluated together.

Why no single chapter’s checklist catches this: Ch. 09 (Model Versioning) teaches you to version model artifacts rigorously — but if the prompt template lives in a separate system owned by a different team (product, or a “prompt ops” tool), it’s outside that chapter’s scope by construction, and nobody’s checklist spans both systems. Ch. 07’s canary checklist verifies the canary rollout mechanism, not what’s inside the versioned unit it’s rolling out.

Fix: The prompt template, chat template, and sampling defaults must be immutable artifacts of the same version bump as the weights — stored in the same registry entry (Ch. 09), baked into the same container image or pulled by version-pinned reference at startup, never mutated independently via a config flag that isn’t itself versioned and rolled back atomically with the model. If product needs to iterate on prompts faster than model releases, that’s a legitimate need — but it must go through the same canary/registry/rollback machinery as a weights change, not around it. A useful audit question for any platform: “if I roll back the model version right now, what doesn’t roll back with it?” If the honest answer is “the prompt,” you have this bug waiting to happen.

Saying it out loud. The audit question I’d hand anyone running a platform is: if I roll back the model version right now, what doesn’t roll back with it? If the honest answer is “the prompt,” you have this bug queued up. What happened here is that a new version shipped weights plus an updated system prompt, but the prompt was deployed through a separate path — a config service or a prompt-management tool outside the registry. Rolling back the container reverted the weights and not the prompt, so production ended up running old weights against the new prompt, a combination nobody had ever tested. The fix is that prompt, chat template, and sampling defaults are immutable artifacts of the same version bump. If product needs to iterate on prompts faster than model releases, fine — but through the same canary and rollback machinery, not around it.

5.3 Monitoring blind spots between layers

Symptom: Every per-component dashboard is green — gateway 2xx rate is high, model-server P95 TTFT is under SLO, GPU utilization looks healthy — and yet real users are experiencing timeouts.

Root cause (three variants, all real):

1. The gateway’s timeout is shorter than the model server’s. The gateway (Ch. 01/03 layer) times out and returns a clean 504 after, say, 10 seconds; the model server metrics only record latency for requests it actually completes, so a request the gateway already gave up on never shows up in the model server’s TTFT histogram at all — it just silently doesn’t count. The model server’s dashboard looks perfect because it’s only measuring the subset of requests that didn’t get cut off upstream.
2. Per-pod metrics look fine while the aggregate breaches, because the load balancer isn’t distributing evenly — a sticky-session or weighted routing bug sends a disproportionate share of traffic to a subset of replicas, which individually stay under their own alert thresholds while the P95 measured client-side (across all replicas) breaches. Per-pod dashboards (Ch. 08’s default view) can hide this completely; you need a client-side or gateway-side latency view, not just server-side histograms, to catch it.
3. The drift monitor (Ch. 10) is watching the stable version’s traffic distribution, not the canary’s. During a canary, 5-25% of traffic is going to a version whose input distribution characteristics (e.g., if the canary changes --max-model-len or a routing rule shifts prompt lengths) are never compared against baseline until full promotion — by which point it’s not a canary anymore, it’s just production, and any drift-driven quality regression has already reached 100% of users.

Why no single chapter’s checklist catches this: each chapter’s monitoring guidance is scoped to the component it teaches. Ch. 08 teaches you to monitor the model server; it doesn’t mandate a client-side or gateway-side view. Ch. 10 teaches drift detection generally; nothing in that chapter forces you to apply it per-version during a canary rather than only to the aggregate production stream.

Fix: Maintain at least one cross-layer latency measurement (synthetic canary requests sent from outside the cluster, measuring true end-to-end time including the gateway) in addition to the model server’s own histograms, and make sure every per-component alert and every drift check is labeled and filterable by version, so a canary’s behavior is visible in isolation before it becomes the whole system’s behavior.

Saying it out loud. Every per-component dashboard green while users are timing out is the shape to recognize, and there are three common causes. One: the gateway’s timeout is shorter than the model server’s, so requests the gateway already gave up on never enter the server’s latency histogram at all — the server’s dashboard is perfect because it’s only measuring the survivors. Two: uneven load balancing means individual pods each stay under their own thresholds while the client-side P95 across all replicas breaches. Three: the drift monitor is watching aggregate production traffic, so a canary’s distribution is never compared to baseline until it’s already at 100%. The fix for all three is the same pair — at least one client-side synthetic check measuring true end-to-end time, and a version label on every metric and every drift check.

5.4 Multi-region registry divergence

Symptom: A canary promotes cleanly in us-east, the team calls the rollout done, and three weeks later a eu-west on-call engineer discovers the region has been silently running the old version the entire time.

Root cause: The model registry (Ch. 09) was treated as a per-region resource — each region’s cluster pulled from its own local registry mirror or config, and the promotion pipeline only pushed the “promote to v2” event to the region where the on-call engineer happened to run it. Nothing in the system enforced that “promoted” means the same thing in every region simultaneously.

Fix: Per Section 2.3, if you operate multi-region at all, the model registry must be a genuinely global source of truth (or have an explicit, monitored, alerting-backed replication/sync step) — “promoted” needs to be a single fact checked against every region’s actual running version, not an event fired once and assumed to propagate.

Saying it out loud. This one is short and it stings: a canary promotes cleanly in one region, the team calls the rollout done, and three weeks later someone in another region discovers it’s been running the old version the whole time. The cause is that the registry was treated as a per-region resource, and the promotion pipeline only fired the event in the region the engineer happened to be working in. Nothing in the system enforced that “promoted” means the same thing everywhere. So if you run multi-region at all, promotion has to be a global fact verified against every region’s actually-running version, with alerting on divergence — not an event fired once and assumed to propagate.

6. Pre-Launch Checklist

A consolidated checklist across all eleven chapters, ordered roughly the way you’d actually verify it. Treat “no” on any item as a launch blocker unless you can name the specific, accepted risk you’re taking instead.

Serving fundamentals (Ch. 01, Ch. 02)

• The server handles malformed requests, oversized inputs, and client disconnects without crashing or leaking GPU memory.
• The container image is pinned to an exact base-image tag and dependency versions — no :latest.
• The container runs as a non-root user; secrets (API keys, registry credentials) are injected at runtime, never baked into the image.
• Local docker run --gpus all smoke test passes before anything touches Kubernetes.

Kubernetes (Ch. 03)

• GPU resource requests/limits are set on every model-server pod spec (no pod can silently oversubscribe a GPU).
• startupProbe accounts for real model-load time (weights + CUDA graph capture), verified by timing an actual cold start, not guessed.
• A PodDisruptionBudget exists so a node drain or cluster upgrade can’t take out every replica simultaneously.
• Node taints/tolerations and nodeSelectors correctly keep GPU workloads on GPU nodes and off them for everything else.

Load testing (Ch. 04)

• A real load test (not a guess) has produced a concurrency-vs-latency saturation curve for the actual production model, hardware, and --max-model-len.
• The req/s-per-replica ceiling used for capacity planning and autoscaling thresholds comes from that curve, not from a vendor blog post or a different model’s numbers.
• Load tests include realistic prompt-length distributions, not just short synthetic prompts — TTFT and memory pressure both depend heavily on prefill length.

Serving engine (Ch. 05, Ch. 11)

• Quantization format (if any) has been validated for quality against a golden eval set, not just for throughput.
• --max-model-len, --gpu-memory-utilization, and prefix-caching settings are deliberate choices, documented with the reasoning, not defaults left untouched.
• If using Triton: backend choice (vLLM backend vs TensorRT-LLM backend) matches the actual multi-model/ensemble requirement, not just habit.

Autoscaling (Ch. 06)

• Autoscaling triggers on a GPU/queue-relevant signal (queue depth, GPU utilization) — not on CPU/memory alone.
• minReplicaCount accounts for cold-start time, not just steady-state traffic minimums.
• Cooldown/stabilization windows are tuned for GPU pod spin-up latency, not copied from a CPU-service HPA config.
• Only one control loop owns replica count for any given ReplicaSet during a rollout (see Section 5.1).

Canary/rollout (Ch. 07)

• Automated analysis gates (latency, error rate, and ideally a quality/eval signal) run at every canary step — no step is “promote and hope.”
• Rollback is a single action that reverts weights, prompt template, and engine config together (see Section 5.2) — verified by actually triggering a rollback in staging and checking all three reverted.
• Canary traffic is labeled distinctly (version label) all the way through metrics, logs, and drift checks.

Monitoring (Ch. 08)

• Dashboards exist for TTFT, inter-token latency, queue depth, GPU utilization, and error rate, all filterable by model version.
• At least one client-side or gateway-side synthetic latency check exists in addition to server-side histograms (see Section 5.3).
• Every SLO-protecting alert has a for: duration tuned to avoid paging on a single noisy scrape, and a documented runbook link in the alert annotation.
• A leading-indicator warning alert (e.g., rising queue depth) exists ahead of the hard SLO-breach page, so on-call has reaction time.

Model versioning (Ch. 09)

• Every deployed version is a registry entry that captures weights + tokenizer + prompt template + sampling defaults + engine config as one unit.
• No config path exists that can change model-affecting behavior (prompt, sampling params) outside the versioned registry entry.
• Multi-region deployments treat “promoted” as a single global fact, checked per region, not an event assumed to propagate (see Section 5.4).

Drift detection (Ch. 10)

• Drift/quality monitoring runs per-version (including on canary traffic specifically), not only on the aggregate production stream.
• A documented baseline (input distribution + quality scores) exists from the last known-good version to compare against.
• Drift alerts feed back into the canary/rollout controller’s promotion decision, not just into a separate dashboard nobody watches during a rollout.

Cost (Section 4 of this chapter)

• A capacity plan exists that’s derived from the actual load-test ceiling, not a round-number guess.
• minReplicaCount/maxReplicaCount are justified against the real traffic curve (diurnal pattern), not set to “whatever felt safe.”
• Someone has computed cost-per-1000-requests at the current stack choice and can defend it against at least one alternative (different GPU, different quantization).

Organizational

• On-call has run at least one game-day: trigger a canary rollback, kill a pod mid-request, and simulate a GPU node failure, and confirm the system (and the humans) behave as expected.
• The pre-launch checklist itself is versioned somewhere and gets revisited after the first real incident — the failure modes in Section 5 were all discovered this way, not designed for up front.

Saying it out loud. The way to use a checklist like this is that a “no” on any item is a launch blocker unless you can name the specific risk you’re accepting instead. The items I’d flag as most commonly missed: a startup probe timed against a real cold start rather than guessed, a PodDisruptionBudget so a cluster upgrade can’t drain every replica at once, load tests that use realistic prompt length distributions rather than short synthetic prompts, exactly one control loop owning replica count during a rollout, and a rollback verified in staging to actually revert weights, prompt, and engine config together. And one organizational item that matters as much as any technical one: on-call has actually run a game day, so the first canary rollback isn’t happening for the first time during a real incident.

7. Interview Mastery

7.1 The 60-second answer

“Walk me through how you’d stand up an LLM inference platform.”

“I’d start by picking the engine based on the actual requirement — vLLM if it’s one or a few open-weight models and I want max throughput per GPU dollar, Triton if I need one server fronting multiple frameworks or models. I’d containerize that engine with a pinned image and a health endpoint, then deploy it to Kubernetes with correct GPU resource requests, a startup probe that accounts for real model-load time, and a PodDisruptionBudget so node drains don’t take out every replica. Before I wire any autoscaling, I load test to find the actual concurrency-vs-latency ceiling for that model on that hardware — that number drives everything downstream. Autoscaling then triggers on a GPU-relevant signal like queue depth, with a minReplicaCount that accounts for cold-start time, using KEDA rather than plain HPA. New model versions go through a canary — Argo Rollouts or KServe — with automated analysis gates on latency and error rate, and the version being rolled out is the full tuple: weights, prompt template, sampling config, all versioned and rolled back together, not just the checkpoint. Observability wraps the whole thing: Prometheus scraping the engine’s own metrics, dashboards split by version, alerts with a leading indicator ahead of the hard SLO page. And a drift/quality monitor watches whether the model’s actual behavior in production still matches what was validated, per version, feeding back into whether a canary should be trusted to promote. The parts that bite you in production are almost never inside one of those boxes — they’re in the seams between them, like an autoscaler and a canary controller both trying to own the same ReplicaSet’s replica count.”

7.2 Q&A

Q1: Why vLLM over raw HuggingFace transformers.generate() for production serving? A: Naive generate() processes requests with no batching sophistication — either one at a time, or static batches that stall on the slowest sequence in the batch. vLLM’s continuous batching admits and evicts requests from a running batch every iteration, so a fast-finishing sequence’s slot is immediately reused, and PagedAttention manages KV cache in fixed-size blocks instead of one contiguous allocation per sequence, eliminating the fragmentation that would otherwise cap concurrency far below what the GPU’s memory could actually support. The practical result is roughly an order of magnitude more throughput at comparable latency (Ch. 05).

Q2: When would you choose Triton over vLLM? A: When the actual requirement is multi-model or multi-framework serving behind one protocol — PyTorch, ONNX, TensorRT-LLM, and custom Python backends all fronted by one server, with ensembles or model pipelines — or when the org already has NVIDIA-centric MLOps tooling built around Triton. Triton can also run vLLM as a backend, so it’s not strictly either/or; the question is whether you need Triton’s multi-model management plane on top of whatever engine does the actual generation.

Q3: How does continuous batching interact with autoscaling decisions? A: Continuous batching means one replica’s capacity isn’t a fixed req/s number — it’s a saturation curve where latency stays flat until a concurrency knee, then degrades sharply (Section 3.5). Autoscaling has to trigger on a signal that reflects queue pressure (requests waiting because the running batch is full), not raw req/s or CPU, because req/s alone doesn’t tell you where you are on that curve — the same req/s can be comfortable or already-saturated depending on prompt length and generation length.

Q4: Explain PagedAttention and why it matters for capacity planning. A: It manages the KV cache in fixed-size, non-contiguous blocks (like OS virtual memory pages) instead of pre-allocating one contiguous buffer per sequence sized for the worst case. That eliminates internal fragmentation from over-allocation and external fragmentation from variable sequence lengths, so a given amount of GPU memory supports meaningfully more concurrent sequences. For capacity planning, this means the concurrency ceiling you measure in a load test is much closer to what the hardware can actually deliver, rather than being capped by memory-allocation inefficiency (Ch. 05).

Q5: How do you correctly measure TTFT vs end-to-end latency, and why does the distinction matter for SLOs? A: TTFT is measured from request start to the first streamed token/chunk — for a streaming chat UI, that’s the number users actually perceive as “responsiveness.” End-to-end latency includes the full generation, which scales with output length and is a worse proxy for perceived responsiveness on long generations. Measuring TTFT requires reading the actual SSE/stream response and timing the first non-empty chunk (Section 3.5’s load-test script), not waiting for the full response and back-computing an average — that would hide exactly the queueing behavior an SLO is meant to catch.

Q6: Design a canary rollout strategy for a new model version — what gates would you use? A: Staged traffic weights (e.g., 5% → 25% → 100%) with a pause and an automated analysis gate at each step, checking at minimum: P95/P99 latency (TTFT specifically, not just end-to-end) against the SLO, error rate, and ideally an automated quality/eval signal on a golden set of prompts sampled through the canary specifically. Rollback on gate failure should be automatic, not paged-and-manual. Critically, the “version” under test must include the prompt template and sampling config, not just the weights (Section 5.2).

Q7: What’s wrong with scaling GPU pods on CPU/memory metrics? A: A GPU-bound inference server’s CPU usage barely correlates with how saturated the GPU actually is — the bottleneck is GPU compute and memory (KV cache), not CPU cycles. Scaling on CPU either reacts too late (GPU is already saturated well before CPU shows it) or not at all. The fix is scaling on a signal that directly reflects GPU-side load: queue depth, GPU utilization from DCGM, or a custom Prometheus metric the engine exposes (Ch. 06).

Q8: How do you handle a rollback that needs to revert more than just the model weights? A: Treat the deployable unit as weights + tokenizer + prompt/chat template + sampling defaults + engine config, versioned and stored together in the model registry, so that “roll back to v1” is a single atomic action that reverts all of it — never a container-image rollback plus a hope that nothing else changed independently. Verify this in staging by actually triggering a rollback and diffing every one of those components against what v1 originally shipped with (Section 5.2).

Q9: What GPU scheduling primitives would you use in Kubernetes, and when? A: Whole-GPU-per-pod scheduling via the NVIDIA device plugin for production, latency-sensitive traffic — you want the full memory bandwidth and compute of the card, not a shared slice. MIG (physical partitioning with hard memory/compute isolation) or time-slicing (soft sharing of one GPU across multiple pods) are for lower-QPS environments — dev/staging, batch/offline inference, or many small models that individually don’t need a full GPU — because both reduce the compute and/or memory available to any one workload, which works against a tight latency SLO (Ch. 03).

Q10: How would you detect and respond to model drift in production? A: Compare live input distribution characteristics (prompt length, topic/embedding-space shift) and output quality signals (automated eval scores, user feedback signals like regeneration rate) against a documented baseline captured from the last known-good version. Crucially, run this per-version — including on canary traffic specifically during a rollout, not only on the aggregate post-promotion stream — and feed drift alerts back into the canary controller’s promotion decision, not just into a dashboard (Ch. 10, Section 5.3).

Q11: Walk through your cost model for a serving platform — what levers matter most? A: Start from a measured (not assumed) req/s-per-replica ceiling at your SLO, derive replica-hours needed against your actual traffic curve (not just peak or average), and price that against real GPU $/hr for your chosen card. The highest-leverage lever is usually quantization, because it moves both sides of the cost-per-request ratio at once (cheaper GPU fits, and/or more concurrency fits per GPU); the second is matching autoscaling’s min/maxReplicaCount band to the real diurnal traffic curve rather than either a static peak-sized pool or a floor set too low to survive cold-start latency (Section 4).

Q12: What happens to KV cache / prefix cache across a canary or version bump? A: Prefix/KV cache is per-process and per-model-version — it does not and should not transfer across a version boundary, since a cache entry computed under one set of weights or one prompt template is not valid under another. Practically this means a fresh canary replica starts “cold” on prefix caching, so early canary-traffic latency samples may look slightly worse than steady-state stable traffic purely from cache-warmth differences — worth accounting for when reading a canary’s early analysis-gate metrics so you don’t misattribute a warm-up artifact to a real regression.

Q13: How do you avoid the “canary passed, then the autoscaler undid it” bug? A: Make sure exactly one control loop owns replica count for a given ReplicaSet during an active rollout — point the autoscaler at the Rollout resource itself rather than also pointing it independently at the underlying stable/canary ReplicaSets, so there’s no race between the canary controller’s traffic-weight/replica-count management and a separate autoscaler reconciling the same pods on its own timer (Section 5.1).

Q14: Single-region vs multi-region — how do you decide, and what’s the biggest operational risk in multi-region? A: Start single-region; move to multi-region only for a specific forcing function — global latency requirements, a contractual availability SLA a single region’s outage would breach, GPU capacity constraints, or data-residency law. The biggest operational risk is registry/state divergence: “promoted” or “rolled back” needs to be a fact checked against every region’s actual running version, not an event assumed to propagate — silent regional divergence is a real, recurring incident pattern (Section 5.4).

Q15: What’s your incident response plan if the TTFT SLO breaches during a traffic spike? A: First check whether the autoscaler is reacting — queue depth rising with replica count flat suggests either a cooldown/cap issue or a genuinely unprecedented spike beyond maxReplicaCount. If capacity is the issue, temporarily raise maxReplicaCount (with awareness of GPU-node-pool availability) rather than tuning thresholds blind mid-incident. If replica count is scaling correctly but latency still breaches, check for a non-capacity cause — a bad canary in flight, an unusually long-prompt traffic pattern, or a GPU-health issue (falling back to a degraded state without crashing, which health probes may not catch — Section 5.3). Only after capacity and health are ruled out should you suspect an actual model/code regression.

Q16: How would you validate a quantization change before shipping it? A: Run the golden eval set (the same one gating any model version change, Ch. 10) against the quantized model and compare quality scores directly against the unquantized baseline, not just against a generic benchmark — some quality regressions are task-specific and won’t show up on a broad benchmark. Load test separately to confirm the expected throughput/memory win actually materializes on your real hardware and prompt distribution, since quantization’s benefit is workload-dependent. Ship it through the same canary pipeline as any other model version change — a quantization change is a version change, not a special case that skips the rollout process.

Q17: What’s the role of a model registry beyond just storing weights? A: It’s the single source of truth for “what does version N mean” — the full tuple of weights, tokenizer, prompt/chat template, sampling defaults, and engine config, so that a deployment, a canary, a rollback, or a cross-region promotion all reference the same unambiguous definition of a version. Without that, teams end up with version drift where the “same” version means something subtly different depending on which system you ask (Section 5.2, Ch. 09).

Q18: How do you load test an LLM server correctly, versus a typical web service? A: A typical web service load test cares about req/s and a latency percentile that’s roughly load-independent up to a hard capacity wall. An LLM server’s latency is a function of concurrency, prompt length, and generation length simultaneously, and TTFT specifically requires reading the stream rather than the full response. You need to ramp concurrency (not just req/s) to find the saturation knee, use a realistic prompt-length distribution (not uniform short prompts), and separately track TTFT and inter-token latency, because they degrade differently and an SLO usually cares about TTFT specifically (Ch. 04, Section 3.5).

Q19: What monitoring blind spot is most likely to bite a team that’s only looked at per-component dashboards? A: A mismatch between the gateway’s request timeout and the model server’s own latency histograms — the model server only records latency for requests it completes, so requests the gateway already gave up on vanish from its metrics entirely, making the model-server dashboard look healthier than what users actually experience. The fix is a client-side or gateway-side synthetic latency check in addition to server-side histograms (Section 5.3).

Q20: Tell me about a failure mode that spans two systems/chapters that most engineers miss. A: (Use any of Section 5’s incidents as source material — the autoscaler-undoes-the-canary race in 5.1 is the strongest one to lead with, because it demonstrates the core idea of this whole chapter: individually correct components, incorrect system.) Frame the answer around: what looked fine per-component, what the actual interaction was, and what changed structurally (not just “we added a dashboard”) to prevent recurrence — interviewers are listening for whether you fix the root interaction or just add monitoring around the symptom.

7.3 Tradeoff tables

Engine choice

Rollout strategy

Single-region vs multi-region

Saying it out loud. If I’m compressing the tradeoffs: on engines, raw Transformers is fastest to a demo and wrong for anything real; vLLM is the throughput-per-dollar default for a few self-hosted models; Triton adds a server layer that only pays for itself with a genuine multi-model requirement; and a managed API has the lowest ops burden and no control over the weights. On rollouts, big-bang exposes 100% of traffic instantly, blue-green does too but with instant rollback and 2x capacity during the switch, and canary is the only one that bounds the blast radius before the gates pass — at slightly more than 1x cost. On topology, single region is the right default and the biggest risk multi-region introduces isn’t latency, it’s silent version divergence.

7.4 Red flags vs green flags

7.5 Full system-design prompt

Prompt: “You’re the first infrastructure hire at an AI startup serving an open-weight LLM to end users. Design the inference platform for the first 12 months of growth — from a working demo to production traffic at meaningful scale, with a small team.”

Worked answer:

Start by asking the two questions that shape everything else: what’s the actual model and SLO (assume: a 20B-class open-weight model, chat product, 2s P95 TTFT target), and what’s the team (assume: 2 infra engineers, no dedicated MLOps team for at least the first two quarters).

Months 0-1 — prove it works. Raw Transformers or a minimal vLLM setup on a single GPU box, behind a thin FastAPI gateway, in Docker Compose (Ch. 01, Ch. 02). No Kubernetes yet — it would be pure overhead at this stage. Goal: validate the model meets product’s quality bar and get a rough sense of per-request latency.

Months 1-3 — make it fast, put it on real infra. Move to vLLM specifically for continuous batching and PagedAttention (Ch. 05). Move to Kubernetes once there’s more than one replica’s worth of traffic or any uptime expectation (Ch. 03) — GPU Operator for device plugin/driver management, correct resource requests, startup probes tuned to real model-load time. Run the first real load test (Ch. 04) to get an actual concurrency-vs-latency curve — this number is the single most load-bearing artifact for everything that follows.

Months 3-6 — stop being one deployment away from an outage. Add KEDA-based autoscaling on queue depth (Ch. 06), sized from the load test’s ceiling, with a minReplicaCount that respects cold-start time. Stand up Prometheus/Grafana/Alertmanager (Ch. 08) with TTFT, queue-depth, and error-rate dashboards split by version, plus a leading-indicator alert ahead of the hard SLO page. This is also when a real model registry (Ch. 09) needs to exist — even a small team benefits from “version = weights + prompt template + config, one unit” being true from day one, because retrofitting it after a rollback incident (Section 5.2) is much more painful than building it in from the start.

Months 6-9 — ship model updates without fear. Introduce canary rollouts (Ch. 07) — Argo Rollouts is the pragmatic choice for a small team without an existing KServe investment — with automated analysis gates on latency and error rate. Add a drift/quality monitor (Ch. 10) that runs per-version, including on canary traffic specifically, feeding into the promotion decision.

Months 9-12 — cost discipline and resilience. Revisit the GPU/quantization choice with real production traffic data (Section 4) — by now there’s enough traffic history to make an informed cost-vs-latency tradeoff instead of a launch-day guess. Run a game-day: trigger a canary rollback, kill a pod mid-request, simulate a node failure, and specifically test the failure modes in Section 5 (does the autoscaler fight the canary controller? does a rollback actually revert the prompt template?). Only consider multi-region or Triton/multi-model serving if a specific forcing function has actually appeared by now (global user base, a second model family) — not proactively, because both add real operational surface a 2-person team can’t absorb speculatively.

Month 0-1        Month 1-3           Month 3-6              Month 6-9                Month 9-12
┌─────────┐    ┌───────────────┐   ┌───────────────────┐  ┌──────────────────────┐  ┌───────────────────────┐
│ Docker   │ →  │ vLLM on K8s,   │ → │ + KEDA autoscaling,│→ │ + Argo Rollouts       │→ │ + cost/quantization    │
│ Compose, │    │ real load test │   │ + Prometheus/      │  │   canary + analysis   │  │   revisit, game-days,  │
│ 1 GPU box│    │ (Ch 01,03,04,  │   │   Grafana + model   │  │   gates + drift        │  │   multi-region/Triton   │
│ (Ch 01,  │    │  05)           │   │   registry          │  │   monitor per-version  │  │   only if a real trigger│
│  02)     │    │                │   │   (Ch 06,08,09)     │  │   (Ch 07, 10)          │  │   exists (Sec 2.3, 4)   │
└─────────┘    └───────────────┘   └───────────────────┘  └──────────────────────┘  └───────────────────────┘

The framing an interviewer is checking for: did the answer sequence things in the order risk actually justifies (prove correctness before optimizing speed, optimize speed before adding autoscaling complexity, add rollout safety before adding drift monitoring, and defer multi-region/Triton until an actual forcing function shows up) — versus building every box in the reference architecture on day one because “that’s what production looks like.” A team of two cannot operate all eleven chapters’ worth of machinery simultaneously from a cold start, and pretending otherwise is itself a red flag in a system-design interview.

Saying it out loud. For the twelve-month system-design prompt, the thing being tested is whether you sequence by risk rather than building every box on day one. Months zero to one: prove it works — Docker Compose on one GPU box, validate quality, get a rough latency picture. Months one to three: make it fast and put it on real infra — vLLM, Kubernetes, and the first real load test, which is the single most load-bearing artifact for everything after it. Months three to six: stop being one deploy away from an outage — KEDA autoscaling sized from that curve, monitoring split by version, and a real registry. Months six to nine: ship updates without fear — canary with automated gates, plus per-version drift. Months nine to twelve: cost discipline and game days. A team of two cannot operate eleven chapters’ worth of machinery from a cold start, and pretending otherwise is itself the red flag.

8. Operational Deep Dives

The core narrative (Sections 1-7) is the complete story for most teams. This section collects a few additional pieces of the system that come up once you’re operating the platform for real — GPU node-level autoscaling (as distinct from pod-level autoscaling), two more system-level failure modes that surface after the ones in Section 5, and a command cheat sheet worth keeping next to the runbook.

8.1 GPU node autoscaling and capacity strategy

Section 3.6’s KEDA config scales pod count. It says nothing about where those pods actually run — if the GPU node pool doesn’t have a free node with a schedulable GPU, a new pod sits Pending no matter how correctly KEDA reacted. This is a second autoscaler, one layer down: the cluster autoscaler (or your cloud provider’s node-pool autoscaling equivalent), which adds and removes GPU nodes based on unschedulable pod pressure.

The interaction that catches teams off guard: GPU nodes take meaningfully longer to provision than CPU nodes — driver installation, the GPU Operator’s device-plugin registration, and (if using MIG) partitioning all add minutes on top of normal VM boot time. If KEDA scales pod count up in response to a traffic spike, but the cluster autoscaler needs 3-5 minutes to bring a new GPU node online, that gap is exactly where a TTFT SLO breaches during a burst — the pod-level autoscaler did its job correctly, and the platform still failed the SLO, because the node-level autoscaler was the actual bottleneck.

Two mitigations, both worth having simultaneously:

• Keep a small buffer of pre-warmed, unschedulable-until-needed capacity — either a node pool with a minimum node count above the steady-state pod requirement, or a low-priority “placeholder” pod pattern that reserves node capacity and gets preempted the moment a real model-server pod needs the room. This trades a small amount of idle-GPU cost for closing the node-provisioning gap.
• Prefer scaling within existing nodes first: if your GPU nodes support MIG or time-slicing (Ch. 03) for a fraction of your fleet, keep a portion of capacity shareable so a burst can be partially absorbed by fitting more (smaller) pods on already-running nodes while new dedicated nodes come online in parallel, rather than waiting on node provisioning for the entire burst.

On the cost side, this is also where spot/preemptible GPU capacity earns its keep — for the steady-state floor of replicas (the minReplicaCount from Section 3.6), running on-demand/reserved capacity is worth the latency-safety of guaranteed availability; for the burst headroom above that floor, spot capacity is often an acceptable risk, since a preempted spot node during a burst just means falling back toward the SLO boundary rather than losing the floor’s worth of guaranteed serving capacity entirely. Reserved/committed-use discounts (available from most major cloud GPU providers for 1-3 year commitments) are worth layering under the steady-state floor once traffic has been stable long enough to trust the number — locking in a discount on a minReplicaCount you later have to shrink is its own kind of cost mistake, so this is deliberately a months-9-12 decision (Section 7.5’s timeline), not a launch-day one.

Saying it out loud. There’s a second autoscaler one layer down that people forget: the cluster autoscaler, which adds GPU nodes when pods can’t be scheduled. And GPU nodes are slow to provision — driver install, device plugin registration, MIG partitioning all stack on top of normal VM boot, so three to five minutes is normal. That gap is exactly where a burst breaches your SLO: the pod autoscaler did its job perfectly and the platform still failed, because the node autoscaler was the real bottleneck. Two mitigations worth having together: keep a small pre-warmed buffer, either a node-pool minimum above steady state or low-priority placeholder pods that get preempted when real work arrives, and prefer absorbing part of a burst on existing nodes through time-slicing while new nodes come online in parallel.

8.2 Two more failure modes worth knowing before they happen to you

GPU silent degradation past health checks. A GPU can develop ECC memory errors, thermal throttling, or Xid errors that degrade performance without crashing the process — the vLLM server’s /health endpoint (used by the readiness/liveness probes in Section 3.4) keeps returning healthy because the process is, technically, alive and responsive; it’s just running 3x slower per token than a healthy replica. Kubernetes has no reason to reschedule it, and the autoscaler has no reason to add capacity, because from the outside the replica count looks sufficient — it’s the quality of one replica’s throughput that degraded, not its count. The fix is monitoring GPU health signals directly (DCGM exporter metrics — ECC error counts, thermal state, Xid error codes — installed alongside the GPU Operator) as a first-class input to alerting, separate from and in addition to the application-level health check, and routing per-replica latency metrics (not just the fleet-wide aggregate) into the dashboard so one visibly slow replica doesn’t get averaged away by nine healthy ones.

Cost runaway from an autoscaler ceiling raised during an incident and never lowered. During the incident-response pattern described in Q15 of Section 7.2 — raising maxReplicaCount mid-incident to absorb an unprecedented spike — it’s common, and understandable, for the temporary change to quietly become permanent, because nobody owns reverting it once the incident is resolved and the on-call engineer has moved on. Weeks later, a routine (not even unusually large) traffic bump now scales to the emergency ceiling instead of the originally-sized one, at real and recurring GPU cost, with no incident and no alert to flag it — the system is behaving exactly as configured, which is precisely why nothing catches it. The fix is procedural, not technical: every incident-time config change (autoscaler bounds, probe thresholds, alert silences) gets a tracked follow-up ticket to review and, if appropriate, revert within a fixed window, and periodic (e.g., monthly) audits diff the running autoscaler config against the last deliberately-reviewed baseline.

Saying it out loud. Two more that only show up once you’re operating for real. First, GPU silent degradation: a card develops ECC errors or thermal throttling and runs three times slower per token without crashing, so the health endpoint keeps returning healthy, Kubernetes has no reason to reschedule, and the autoscaler has no reason to add capacity — the replica count is fine, the quality of one replica’s throughput isn’t. That’s why DCGM health signals need to be a first-class alerting input separate from the application health check, and why per-replica latency has to be visible so one slow pod doesn’t get averaged away by nine healthy ones. Second, cost runaway from an incident-time autoscaler ceiling that nobody ever lowered — the fix there is procedural: every incident-time config change gets a tracked follow-up ticket with a review window.

8.3 Quick reference: commands you’ll actually run

A short cheat sheet worth keeping next to the runbook, pulling together the commands used across this walkthrough:

# Build and smoke-test the serving image locally (Section 3.3)
docker build -t registry.example.com/gpt-oss-20b:v1 .
docker run --gpus all -p 8000:8000 registry.example.com/gpt-oss-20b:v1
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-oss-20b-v1","messages":[{"role":"user","content":"hello"}]}'

# Apply the base deployment (Section 3.4)
kubectl apply -f namespace.yaml -f deployment.yaml -f service.yaml -f pdb.yaml

# Watch GPU scheduling — confirm pods actually land on GPU nodes
kubectl get pods -n llm-serving -o wide
kubectl describe node <gpu-node-name> | grep -A5 "Allocated resources"

# Check KEDA's view of the scaling metric (Section 3.6)
kubectl get scaledobject gpt-oss-20b-scaler -n llm-serving -o yaml
kubectl get hpa -n llm-serving   # KEDA creates a backing HPA under the hood

# Query vLLM's own metrics directly, useful when debugging a saturation event
curl -s http://<pod-ip>:8000/metrics | grep -E "vllm:(num_requests|time_to_first_token)"

# Watch a canary rollout live (Section 3.8)
kubectl argo rollouts get rollout gpt-oss-20b -n llm-serving --watch
kubectl argo rollouts promote gpt-oss-20b -n llm-serving     # manual promote if needed
kubectl argo rollouts undo gpt-oss-20b -n llm-serving        # abort and roll back

# Confirm exactly one control loop owns replica count during a rollout (Section 5.1)
kubectl get replicasets -n llm-serving -w

8.4 A minimal Grafana panel definition

Ch. 08 covers full dashboard construction; here’s the smallest useful piece — the TTFT-by-version panel referenced in Section 3.7 — as a Grafana panel JSON snippet you can drop into a dashboard’s panels array, so the shape of the query is concrete rather than described in prose:

{
  "title": "P95 Time-to-First-Token by Version",
  "type": "timeseries",
  "targets": [
    {
      "expr": "histogram_quantile(0.95, sum(rate(vllm:time_to_first_token_seconds_bucket{namespace=\"llm-serving\"}[5m])) by (le, version))",
      "legendFormat": "{{version}}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 1.5 },
          { "color": "red", "value": 2.0 }
        ]
      }
    }
  }
}

The by (le, version) grouping is the detail that matters most: without splitting by version, a canary at 5% traffic gets averaged into the stable version’s 95%, and a real canary regression can hide inside an otherwise-healthy aggregate P95 for the entire canary window — exactly the blind spot described in Section 5.3’s third variant.

8.5 Security and multi-tenancy at the gateway

The reference architecture’s gateway box (Section 1) does more than route requests — in any platform serving more than one internal team or one external customer, it’s also where authentication, per-tenant rate limiting, and basic input hardening live, none of which is unique to LLM serving but all of which have LLM-specific wrinkles worth naming:

• Per-tenant rate limiting must account for token cost, not just request count. A rate limiter that caps “100 requests/minute” per API key treats a request for one token of output the same as a request for 4,000 tokens of output, even though the second consumes vastly more GPU time. Rate limiting (or at least cost attribution and alerting) keyed on estimated or actual token usage is the practical equivalent of request-count limiting for a service where “request” is a poor proxy for cost.
• Prompt-length limits protect the KV-cache budget, not just the gateway. A client sending a request near your --max-model-len ceiling consumes proportionally more KV-cache memory per request, directly reducing the concurrency ceiling the load test in Section 3.5 measured. Enforcing a sane per-request prompt-length limit at the gateway (well below the hard --max-model-len) keeps one large request from degrading everyone else’s latency.
• Multi-tenant isolation for GPU workloads is coarser than for typical multi-tenant CPU services. Unlike a CPU service where cgroups give reasonably strong per-tenant resource isolation, a shared GPU model-server pod serves all tenants’ requests through the same continuous batch — there is no per-tenant GPU-memory or compute isolation within a replica. If strict tenant isolation is a hard requirement (e.g., contractual or regulatory), the practical answer is dedicating replicas (or MIG partitions, Ch. 03) per tenant rather than assuming batching-level isolation exists, because it doesn’t.
• Basic input hardening still matters even though “prompt injection” is a model-behavior problem, not an infra problem. The gateway is a reasonable place to enforce request-size limits, strip or flag obviously malformed input (e.g., attempts to smuggle control tokens the tokenizer would otherwise interpret specially), and log full request/response pairs for the security and drift-monitoring teams (Ch. 10) to review — the infra platform’s job is making sure that data exists and is queryable, not solving prompt injection itself.

Saying it out loud. Gateway security for LLM serving has three wrinkles that generic API advice misses. Rate limiting by request count is close to meaningless when one request generates five tokens and another generates four thousand — you need limits or at least cost attribution keyed on tokens. Prompt-length limits aren’t just gateway hygiene: a request near your max model length eats proportionally more KV cache and directly lowers the concurrency ceiling your load test measured, so cap it well below the hard limit. And multi-tenant isolation is coarser than people assume — all tenants’ requests flow through the same continuous batch inside a replica, so there is no per-tenant GPU isolation within a pod. If isolation is contractual, you dedicate replicas or MIG partitions; batching-level isolation does not exist.

9. Further Reading

Organized by the box in the reference architecture (Section 1) each source speaks to, so you can jump to what’s relevant rather than reading a flat list.

Serving engines (Ch. 05, Ch. 11)

• Kwon, Zhuohan Li, et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023 — the original vLLM paper; the mechanism behind everything Ch. 05 teaches about memory efficiency.
• vLLM documentation — the canonical reference for CLI flags (--max-model-len, --gpu-memory-utilization, --enable-prefix-caching, etc.) used throughout Section 3.
• vLLM releases — check here for the current stable version before pinning a Dockerfile; this chapter cites the v0.20.x line as current at time of writing.
• vLLM FP8 quantization docs and the vLLM blog post “The State of FP8 KV-Cache and Attention Quantization in vLLM” — relevant to the quantization tradeoffs in Section 4.3 for models that don’t ship natively quantized.
• openai/gpt-oss-20b model card and the vLLM recipe for gpt-oss-20b — the specific model used in the Section 3 worked walkthrough, including its native MXFP4 quantization and YaRN context extension details.
• NVIDIA Triton Inference Server documentation and the TensorRT-LLM backend guide — the Ch. 11 engine, and the backend choice discussed in Section 2.1/2.4.

Kubernetes and GPU scheduling (Ch. 03)

• NVIDIA GPU Operator documentation — driver install, device plugin, and the time-slicing/MIG guide referenced in Sections 2.4 and 3.4.
• NVIDIA/k8s-device-plugin — the component that makes nvidia.com/gpu a schedulable Kubernetes resource.
• vllm-project/production-stack — a reference Kubernetes-native deployment of vLLM, including the Helm chart and the KEDA autoscaling guide used directly in Section 3.6.
• LeaderWorkerSet (LWS) for multi-node vLLM — relevant once a single replica needs to span multiple GPU nodes (tensor/pipeline parallelism beyond one machine), mentioned in Section 2.4.

Autoscaling (Ch. 06)

• KEDA documentation — the general scaler framework behind the ScaledObject in Section 3.6.
• AWS EKS: Autoscale AI inference with HPA and KEDA — a concrete cloud-vendor walkthrough of the same pattern used in this chapter.

Rollout, canary, and disaggregated serving (Ch. 07)

• Argo Rollouts documentation and the canary strategy reference — the controller and analysis-template pattern used in Section 3.8.
• KServe canary rollout guide — the alternative canary mechanism referenced in Section 2.4 for teams already on KServe’s InferenceService CRD.
• llm-d and its prefill/decode disaggregation guide — the Kubernetes-native, KV-cache-aware routing project (backed by Red Hat, Google, IBM, and CoreWeave) referenced in Section 2.1/2.4 for extreme-scale deployments.
• NVIDIA Dynamo and the “Dynamo 1.0” production blog post — NVIDIA’s disaggregated-serving framework, an alternative to llm-d mentioned in the same section.

Monitoring and observability (Ch. 08)

• vLLM metrics design doc — the source of truth for every vllm:* metric name used in Sections 3.7 and 8.4.
• Krisanov, “Monitoring vLLM in Production: Metrics, PromQL, Alerts, and Runbooks” — a practitioner writeup with worked PromQL percentile queries this chapter’s alert rules build on.
• Prometheus documentation and Grafana documentation — the general observability stack underlying Ch. 08 and this chapter’s dashboard/alert examples.

Model versioning and registries (Ch. 09)

• MLflow Model Registry documentation — one concrete implementation of the registry pattern in Section 1 and Section 6’s checklist.
• MLflow, “Canary Deployment for AI Models: A 2026 Guide” — connects the versioning and canary concerns directly, relevant to the prompt-template-rollback failure mode in Section 5.2.

Drift detection (Ch. 10)

• Return to Ch. 10’s own deep dive and further-reading list for the statistical mechanism (EWMA, distribution-shift tests) behind the drift monitor box in Section 1 — this chapter deliberately does not re-derive it, only shows where it plugs into the rollout and multi-region flow (Sections 3.8, 5.3, 5.4).

Cost and GPU pricing (Section 4)

• IntuitionLabs, “H100 Rental Prices Compared: 15+ Cloud Providers (2026)” and CloudZero, “H100 GPU Cost In 2026” — the source range for the GPU pricing used in Section 4.1; re-check current quotes before using these numbers in a real budget review, since GPU cloud pricing moves quickly.
• SynpixCloud, “Cloud GPU Pricing 2026” — the A100/H100 baseline figures cited in Section 3.2’s comparison table.

Appendix A: Glossary of Cross-Chapter Terms

Terms that get used across multiple chapters and multiple sections of this capstone, in one place, with a pointer back to where each is taught in depth.

Appendix B: Chapter Cross-Reference Map

A one-page index of which section of this capstone exercises each of the guide’s eleven chapters most directly — useful as a study map if you’re revisiting a specific chapter and want to see it in context.

Appendix C: Wiring a Drift/Quality Signal into the Canary Gate

Section 3.8’s AnalysisTemplate gates on latency and error rate — both operational signals, neither of which catches a canary that’s technically fast and error-free but subtly wrong (e.g., a quantization change that passed every latency check but quietly degraded answer quality on a class of prompts the golden set didn’t cover densely enough). Ch. 10’s drift/quality monitor is the component meant to catch that, and it needs to feed into the same gate, not live in a separate dashboard nobody checks mid-rollout (Section 5.3’s third failure mode).

The connective piece is small: the drift monitor exposes its own Prometheus gauge, scored per version, and the AnalysisTemplate adds a third metric alongside latency and error rate.

# drift_monitor_exporter.py — a minimal sketch of the gauge the drift
# monitor (Ch. 10) needs to expose per version for the canary gate to read.
from prometheus_client import Gauge

GOLDEN_SET_SCORE = Gauge(
    "model_golden_set_quality_score",
    "Rolling average quality score (0-1) against the golden eval set, per version",
    ["version"],
)

def update_score_after_eval_batch(version: str, batch_scores: list[float]) -> None:
    # In practice this would be an EWMA over recent batches (Ch. 10), not a
    # raw batch average — shown simplified here to keep the wiring visible.
    avg = sum(batch_scores) / len(batch_scores)
    GOLDEN_SET_SCORE.labels(version=version).set(avg)

# analysistemplate.yaml — extended from Section 3.8 with the quality gate
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: gpt-oss-slo-check
  namespace: llm-serving
spec:
  metrics:
    - name: ttft-p95            # as in Section 3.8
      # ... unchanged ...
    - name: error-rate           # as in Section 3.8
      # ... unchanged ...
    - name: golden-set-quality
      interval: 5m
      successCondition: result >= 0.92    # baseline established from the
                                            # last known-good version (Ch. 10)
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: model_golden_set_quality_score{version="canary"}

This closes the loop described in Section 1’s diagram: the drift/quality monitor doesn’t just watch production and report separately — its output becomes a real promote/abort input, with the same automatic, no-human-watching-a-dashboard guarantee the latency and error-rate gates already have.

Saying it out loud. The gap this appendix closes is that latency and error rate gates catch a canary that’s broken, not one that’s subtly wrong. A quantization change can pass every operational check while quietly degrading answers on a class of prompts your golden set didn’t cover densely. So the drift and quality monitor has to expose its own Prometheus gauge, scored per version, and the canary’s analysis template reads it as a third metric alongside latency and errors. That’s the whole connective piece, and it’s small — but without it the quality signal lives on a dashboard nobody is watching during the fifteen minutes it actually matters.

Appendix D: Cost Comparison Across Quantization Choices

Extending Section 4.3’s quantization discussion with concrete illustrative numbers, for a case where a model does not ship natively pre-quantized (unlike gpt-oss-20b in the main walkthrough) and the team is choosing how far to quantize a BF16 checkpoint:

(Illustrative ranges reflecting commonly reported directional effects, not a benchmark run for this chapter — always validate the specific ceiling and quality delta for your model, hardware, and golden set before committing a capacity plan to a quantization choice; see Ch. 05 and the FP8 quantization docs in Section 9 for engine-specific specifics.)

The practical decision rule from Section 4.3 holds here in sharper relief: FP8 is close to a default-safe choice on modern hardware (H100-class GPUs have native FP8 tensor cores, so the throughput win is closer to “free” than the INT4 row’s more dramatic — and riskier — VRAM savings). INT4-class quantization is the higher-risk, higher-reward lever, worth reaching for only after the golden-set validation in Section 6 has specifically cleared it for your model and task mix, not adopted by default purely for the cost savings.

Saying it out loud. On how far to quantize a model that doesn’t ship pre-quantized: FP8 roughly halves the memory footprint and buys somewhere in the range of 1.6 to 1.9 times the throughput ceiling, with a quality delta that’s typically small and often within noise on general benchmarks. INT4 through AWQ or GPTQ quarters the footprint and can push the ceiling two and a half to three and a half times, but the quality gap is larger and much more task-dependent — long-context and reasoning tasks degrade visibly before open-ended chat does. So FP8 is close to default-safe on hardware with native FP8 tensor cores, and INT4 is the higher-risk, higher-reward lever you reach for only after golden-set validation clears it. These are illustrative directional ranges, not a benchmark — measure your own model.

Appendix E: The Same Scenario, Through Triton Instead

Section 3 walks the gpt-oss-20b scenario through vLLM directly, because a single self-hosted model family is exactly the case where vLLM is the right default (Section 2.1). It’s worth seeing concretely what changes if the same model needed to sit behind Triton instead — for example, because it now needs to share a server with a second, non-LLM model (a re-ranker or an embedding model) that the product also depends on, which is exactly the kind of requirement that tips the decision in Section 2.1’s tree toward Triton.

Triton’s model-serving unit is a model repository — a directory structure Triton scans at startup, not a single serve command:

model_repository/
└── gpt-oss-20b/
    ├── config.pbtxt
    └── 1/
        └── model.json

config.pbtxt tells Triton which backend owns this model and how many instances to run — with the vLLM backend, device placement is deliberately left to vLLM itself rather than configured here:

# model_repository/gpt-oss-20b/config.pbtxt
backend: "vllm"

instance_group [
  {
    count: 1
    kind: KIND_MODEL
  }
]

model.json (versioned under 1/, Triton’s convention for model version directories — note this is a different versioning axis than the registry versioning in Ch. 09/Section 5.2, and the two should not be conflated) carries the actual vLLM engine arguments, mirroring the flags from the Dockerfile’s CMD in Section 3.3:

{
  "model": "openai/gpt-oss-20b",
  "max_model_len": 32768,
  "gpu_memory_utilization": 0.90,
  "enable_prefix_caching": true,
  "served_model_name": "gpt-oss-20b-v1"
}

Everything above the model-server layer in the reference architecture (Section 1) is unaffected: the same Kubernetes Deployment pattern from Section 3.4 applies (Triton’s container image replaces vLLM’s, GPU resource requests and probes stay conceptually identical — Triton exposes its own /v2/health/ready endpoint for the readiness probe instead of vLLM’s /health), the same KEDA scaling pattern from Section 3.6 applies against Triton’s own Prometheus metrics endpoint, and the same Argo Rollouts canary pattern from Section 3.8 applies unchanged — a canary controller shifting traffic weight doesn’t care whether the pods behind each weight are running vLLM or Triton. This is the practical payoff of the reference architecture in Section 1 being drawn as boxes rather than as a single tool: swapping the model-server box’s implementation doesn’t require redesigning the gateway, autoscaler, canary controller, or observability stack around it.

The tradeoff is exactly what Section 2.1 named: Triton’s model-repository/config.pbtxt layer is genuine extra operational surface (a second configuration system, on top of Kubernetes YAML, that has to stay in sync with it) that buys you the ability to add that second, non-LLM model to the same server later without standing up an entirely separate serving stack for it.

Saying it out loud. It’s worth knowing what changes if the same workload has to go through Triton — say because a re-ranker or an embedding model now needs to share the server. The serving unit stops being a single serve command and becomes a model repository: a directory tree Triton scans at startup, one folder per model, each with its own config declaring backend, batching, and instance placement. The request path concepts carry over unchanged — continuous batching, paged KV cache, the same Prometheus story — but you’ve added a server layer and a config surface. Which is exactly the point of the earlier decision tree: you take that on when a genuine multi-model requirement appears, not preemptively.

Appendix F: A Sample On-Call Runbook Entry

Section 3.7’s TTFTSLOBreach alert annotation points to “a runbook link” — here’s what a real one looks like, concretely enough to adapt rather than write from scratch:

## Runbook: TTFTSLOBreach (gpt-oss-20b)

**Alert fires when:** P95 TTFT > 2.0s for 3+ minutes, on the stable-version
label (see the alert query in Section 3.7).

**First 2 minutes — orient, don't act yet:**
1. Open the "P95 TTFT by Version" Grafana panel (Section 8.4). Confirm this
   is fleet-wide, not one version — if it's isolated to a `canary` label,
   this is a canary-quality issue, not a capacity issue; stop the rollout
   (`kubectl argo rollouts undo gpt-oss-20b -n llm-serving`) before anything
   else.
2. Check `QueueDepthRising` (Section 3.7) — did it fire first? If yes, this
   is very likely a capacity/scaling issue, not a regression. If
   `TTFTSLOBreach` fired *without* `QueueDepthRising` firing first, suspect
   a per-replica issue (Section 8.2's GPU silent-degradation failure mode)
   rather than fleet-wide load.

**Minutes 2-10 — capacity path:**
3. `kubectl get scaledobject gpt-oss-20b-scaler -n llm-serving -o yaml` —
   confirm KEDA's current replica target vs `maxReplicaCount`. If pinned at
   the ceiling and queue depth is still rising, this is a real capacity
   shortfall — see step 5.
4. `kubectl get pods -n llm-serving -o wide` — confirm new replicas are
   actually `Running`, not stuck `Pending` (Section 8.1's node-provisioning
   gap). If `Pending`, check `kubectl describe node` for GPU node pool
   capacity; you may need to manually scale the GPU node pool while the
   cluster autoscaler catches up.
5. If genuinely capacity-constrained beyond `maxReplicaCount`, raise it
   temporarily (`kubectl edit scaledobject ...`) — **and immediately file
   the follow-up ticket described in Section 8.2 to review/revert it**,
   so this doesn't become the permanent ceiling by default.

**Minutes 2-10 — per-replica path (if queue depth was NOT rising):**
6. Compare per-pod TTFT, not just the fleet aggregate — one replica
   dragging the P95 up while others look healthy points at GPU hardware
   degradation. Check DCGM metrics (ECC/Xid errors, Section 8.2) for the
   specific node backing the slow replica; cordon and drain that node if
   confirmed, and file a hardware ticket with the cloud provider.

**If neither path resolves it within 15 minutes:** escalate to the
model-serving on-call lead and consider a manual rollback to the last
known-good version, following the atomic weights+prompt+config rollback
process (Section 5.2) — not a partial rollback of just the container image.

Appendix G: Anti-Patterns Worth Naming Explicitly

A short list of design smells that show up repeatedly across real platforms, distinct from the acute incidents in Section 5 — these are chronic, not acute, and tend to accumulate quietly rather than trigger a single obvious page.

• The autoscaler config nobody has looked at since launch. minReplicaCount/maxReplicaCount set from the launch-day load test (Section 3.5-3.6) and never revisited as real traffic patterns, prompt-length distributions, or the model itself changed. Revisit this on the same cadence as the cost review in Section 7.5’s months 9-12.
• Alerts with no owner and no runbook. An alert that pages but has no linked runbook (Appendix F) trains on-call to snooze it rather than act on it — by the time a real incident needs that alert to be trusted, it’s been ignored for months.
• A model registry that’s a source of truth in name only. The registry (Ch. 09) exists, but a config flag or feature-flag system outside it can still change prompt/sampling behavior in production (Section 5.2) — the registry’s authority is only as real as the absence of side doors around it.
• Dashboards that only show the aggregate, never split by version. Built once, before canaries existed, and never updated to add the version label split (Section 5.3, Section 8.4) — quietly useless for exactly the moment (a canary rollout) it would matter most.
• “We’ll add monitoring for that after launch.” Said about GPU-health metrics (Section 8.2), client-side synthetic checks (Section 5.3), or drift monitoring (Section 1’s drift box) — these are cheap to add before launch and expensive to retrofit after the first incident they would have caught.
• Treating the load test as a one-time launch artifact. The saturation curve from Section 3.5 is only valid for the model, hardware, and prompt distribution it was measured against; a model swap, a prompt-length shift from a new product feature, or a hardware change (even a “just as fast” GPU generation swap) invalidates it silently until the next incident reveals the assumption was stale.

Saying it out loud. The anti-patterns are chronic rather than acute — they accumulate quietly instead of paging you. The autoscaler config nobody has looked at since launch, sized from a load test on a model and prompt distribution that no longer exist. Alerts with no runbook, which train on-call to snooze rather than act, so by the time one matters it’s been ignored for months. A registry that’s a source of truth in name only, because a feature flag can still change sampling behavior around it. Dashboards that only show the aggregate and were never updated to split by version, quietly useless at exactly the moment a canary is rolling. And treating the load test as a one-time launch artifact, when a model swap or a prompt-length shift silently invalidates it until an incident reveals the assumption was stale.

Closing: How to Use This Chapter

This capstone is meant to be read twice, in two different modes.

The first read is linear, before you build anything — Sections 1-2 to internalize the shape of the whole system and the decision framework, Section 3 to see a complete build end to end so the individual chapters’ techniques have somewhere concrete to land, Sections 4-6 to understand what it costs and how it actually breaks once real traffic and real incidents show up, Section 7 to pressure-test your own understanding against the interview questions (a good proxy for “could I explain this to a new hire”), and Section 6’s checklist immediately before any real launch.

The second read is as a reference, after you’re operating something like this for real — Section 5 and Appendix G’s failure modes and anti-patterns are worth rereading after your first real incident, not just before launch, because most of them are far more legible in hindsight than in a pre-launch review; Appendix F’s runbook pattern is worth adapting for every SLO-protecting alert you add, not just the one shown; and Appendix B’s chapter cross-reference map is the fastest way back into the numbered chapters when a specific piece (autoscaling tuning, drift statistics, Triton backend configuration) needs to go deeper than this chapter goes.

The single idea worth carrying forward past every specific YAML snippet and PromQL query in this chapter: a production LLM serving platform is not eleven separate problems that happen to share a GPU. It’s one system, and the seams between the eleven pieces — where a canary controller and an autoscaler both think they own the same ReplicaSet, where a rollback reverts weights but not the prompt template that shipped alongside them, where a per-component dashboard is green while the actual user experience isn’t — are where the interesting failures live. Chapters 01 through 11 teach you to build each piece correctly. This chapter is the argument for why that isn’t the same thing as building the system correctly, and a worked example of closing that gap.

Saying it out loud. The single idea worth carrying out of this chapter is that a production LLM serving platform is not eleven separate problems that happen to share a GPU. It’s one system, and the interesting failures live in the seams — a canary controller and an autoscaler both thinking they own the same replica set, a rollback that reverts weights but not the prompt template that shipped with them, a per-component dashboard that’s green while the actual user experience isn’t. Individual chapters teach you to build each piece correctly. That is genuinely not the same thing as building the system correctly, and the gap between those two is where most production incidents come from.

Appendix H: A Concrete Game-Day Test Plan

Section 6’s checklist and Section 7.5’s twelve-month plan both mention “run a game-day” without specifying what to actually test. Here is a concrete plan built directly from the failure modes in Section 5 and Section 8.2 — run each of these in staging first, then, once trusted, in production during a low-traffic window with on-call actively watching.

Running this list once doesn’t make it done — the honest cadence is re-running it whenever a component in the chain changes (a new autoscaler version, a new canary controller version, a change to the PDB or probe configuration), since each of these tests is validating an interaction, and interactions are exactly what silently break when one side of them changes without the other side being retested.

Saying it out loud. A game day is worth specifying rather than gesturing at, because “run a game day” without a list means nobody runs one. The tests I’d insist on: kill a pod mid-request and confirm readiness pulls it from the service before it dies; abort a canary mid-step and then diff the running config, not just the image tag, to prove the prompt template reverted too; drain a GPU node and confirm the disruption budget actually blocks; fill the node pool and trigger a scale-up to see whether your pre-warmed buffer really closes the node-provisioning gap; and feed known-bad outputs into the quality gauge to prove the canary gate actually reads it. And running the list once doesn’t make it done — every one of these validates an interaction, and interactions break silently when one side changes.

Appendix I: Managed API Options, for Comparison

Section 2.1’s decision tree starts with “hosted/managed API only,” which shortcuts most of this chapter’s serving stack. For context, the real options as of 2026 in that category — useful when the honest answer to “should we self-host at all” is still open:

The decision between these and the self-hosted stack this chapter builds is rarely purely technical — it’s a build-vs-buy tradeoff between GPU-hour margin (self-hosting is cheaper per token at meaningful scale, per the cost model in Section 4) and operational headcount (every box in Section 1’s reference architecture that you self-host is a box your team is now on-call for). A useful gut check: if you don’t yet have someone who can execute Appendix H’s game-day plan competently, that’s a signal you may not be ready to self-host at production scale yet, regardless of what the cost model says.

Appendix J: A Note on How the Numbers in This Chapter Age

Every concrete number in this chapter — the vLLM version pinned in the Dockerfile (Section 3.3), the GPU pricing used in the cost model (Section 4), the illustrative load-test curve (Section 3.5), the quantization comparison ranges (Appendix D) — is a snapshot, not a constant. This is worth stating explicitly rather than leaving implicit, because treating a snapshot as a constant is itself a small version of the same mistake Section 5 spends so much time on: trusting a number without checking whether the thing it was measured against has changed underneath it.

Concretely, before reusing this chapter’s specifics in a real build:

• Re-check the vLLM (or Triton, or Argo Rollouts, or KEDA) version against the project’s own release notes — this chapter cites the v0.20.x vLLM line and specific 2025-2026 feature landings (FP8 KV-cache work, gpt-oss support) as current at time of writing; serving engines in this space ship new releases frequently enough that a pinned version six months old is worth deliberately revisiting, not just inheriting.
• Re-run the load test (Section 3.5) against your actual model and hardware. The illustrative saturation curve in this chapter is real in shape but specific to nothing you’re deploying — it exists to show how to read a saturation curve, not to hand you one.
• Re-quote GPU pricing (Section 4, Appendix I) — cloud GPU pricing has moved substantially year over year through the mid-2020s as supply has shifted, and the ranges cited here (with sources linked in Section 9) should be treated as “check whether this is still roughly right,” not “use this number in a board deck.”
• Re-validate quantization quality deltas (Section 4.3, Appendix D) per model. Quantization tooling and technique quality has been improving quickly; a quality gap that was noticeable eighteen months ago on one model family may be smaller (or larger, for a different architecture) on whatever you’re actually deploying — the golden-set validation step in Section 6 exists precisely so you never have to trust a general claim about quantization quality instead of measuring your own.

None of this diminishes the chapter’s core teaching content — the reference architecture’s shape (Section 1), the decision framework’s structure (Section 2), the order operations happen in during a real build (Section 3’s walkthrough sequence), and the failure modes living in the seams between components (Section 5) all age far more slowly than any specific price or version number, because they’re about how the pieces of the system relate to each other, not about which specific tool or GPU generation currently fills a given box. That distinction — what ages fast versus what doesn’t — is itself worth teaching to anyone using this chapter as a reference months or years after it was written.

Saying it out loud. I’d say this out loud in any interview where I quote a number from a book: every concrete figure here is a snapshot, not a constant. The engine version, the GPU pricing, the load-test curve, the quantization ranges — all of them age, and treating a snapshot as a constant is a smaller version of exactly the mistake this whole chapter is about, trusting a number without checking whether what it measured has moved underneath you. So re-check engine versions against release notes, re-run the load test on your own model and hardware, re-quote GPU pricing, and re-validate quantization quality per model. What ages slowly is the architecture’s shape, the decision framework, the order of operations, and the failure modes in the seams — because those are about how pieces relate, not which tool currently fills a box.

Appendix K: A Postmortem, Written the Way Section 5’s Incidents Actually Read

Section 5’s failure modes are described analytically, by design, so they generalize. Here is one of them — 5.1, the canary-vs-autoscaler race — written the way an actual postmortem document would read, because the difference in tone matters: postmortems are specific, timestamped, and blameless about people while being precise about systems.

## Incident: gpt-oss-20b-v2 canary — stable version resurfaced after full promotion

**Severity:** SEV-2. **Duration:** 23 minutes of intermittent stale-version
responses after a canary promotion showed "complete" in the Argo Rollouts UI.
**User impact:** ~4% of requests during the window received v1 (pre-update)
model behavior after v2 had already been communicated as fully live.

**Timeline:**
- 14:02 — Canary rollout for gpt-oss-20b-v2 begins per the staged-weight
  process in Section 3.8. All analysis gates pass at each step.
- 14:41 — Rollout reaches setWeight: 100. Argo Rollouts UI shows "Healthy."
  On-call marks the rollout complete in the team channel.
- 14:44 — KEDA's ScaledObject, which was watching queue depth on the
  underlying stable ReplicaSet directly (not the Rollout resource, per the
  misconfiguration described in Section 5.1), observes falling queue depth
  on stable as traffic had been shifting to canary throughout the rollout,
  and — reacting correctly to that signal in isolation — scales the stable
  ReplicaSet back up from 1 pod to 4, interpreting the recent dip as
  transient rather than as the deliberate result of the in-progress promotion.
- 14:47-15:07 — The Service's endpoint list includes both the (should be
  fully retired) stable pods and the new v2 pods. A subset of requests land
  on stable pods that KEDA re-created, producing v1 behavior for those
  requests.
- 15:07 — On-call notices version-label mismatch in the "requests by
  version" Grafana panel (Section 8.4) during a routine post-rollout check,
  identifies the stable ReplicaSet's unexpectedly nonzero pod count, and
  manually scales it to zero.
- 15:10 — Traffic confirmed 100% on v2. Incident closed.

**Root cause:** Two independent control loops — the Argo Rollouts canary
controller and the KEDA autoscaler — each held an opinion about the stable
ReplicaSet's replica count, on independent reconcile timers, with no
coordination between them. Neither was misconfigured relative to its own
scope; the interaction between the two scopes was the defect.

**Contributing factor:** The post-rollout check that caught this was manual
and happened to run within the incident window; there was no automated
alert for "nonzero replica count on a ReplicaSet the Rollout controller
should have scaled to zero."

**Fix (structural, not just monitoring):** Repointed the KEDA ScaledObject
at the Rollout resource rather than the underlying ReplicaSets directly
(Section 5.1's fix), verified via the game-day test in Appendix H, item 2,
run in staging through three full canary cycles with no recurrence. Added
an alert on nonzero pod count for any ReplicaSet a completed Rollout should
have zeroed, as a defense-in-depth backstop — but the primary fix is the
single-owner control-loop change, not the new alert.

**What this incident does not change:** The canary analysis gates
themselves (latency, error rate, quality) worked exactly as designed at
every step — this was never a "the canary should have caught something"
incident. It was a "two correct systems, incorrectly composed" incident,
which is precisely the category Section 5 of this chapter exists to name.

This is the format worth adopting for real incidents against this platform: specific and timestamped in the timeline, but the root-cause and fix sections stay at the level of “which control loop owned what,” which is exactly the level Section 5’s abstractions operate at — so that a postmortem doesn’t just document one incident, it feeds back into the general failure-mode catalog the next engineer reads before their own launch.

Saying it out loud. The reason to write a postmortem this way is the contrast in register. Section 5 describes failures analytically so they generalize; a real postmortem is specific and timestamped — severity, duration, roughly four percent of requests receiving stale model behavior for twenty-three minutes after the rollout UI said complete. But notice that the root cause and fix sections still sit at the level of which control loop owned what. That’s deliberate: it keeps the document blameless about people while being precise about systems, and it means the postmortem doesn’t just document one incident, it feeds back into the general failure-mode catalog the next engineer reads before their own launch.

LLM Serving & Inference Interview Q&A

Interview questions and answers for LLM serving, inference, and MLOps roles — written to help someone convince a senior interviewer they can build and operate real LLM inference infrastructure, not just call an API.

Each section maps to one of this guide’s 11 hands-on chapters. Use the chapter deep-dives for implementation detail, the System Design section to rehearse whiteboard scenarios, and the Flashcards/Traps sections the night before an interview.

Table of Contents

1. LLM Inference Fundamentals
2. Model Serving & Basic Serving Patterns (Ch. 01)
3. Docker & Containerization (Ch. 02)
4. Performance Optimization
5. Kubernetes & Deployment (Ch. 03)
6. Load Testing & Capacity Planning (Ch. 04)
7. vLLM Internals Deep Dive (Ch. 05)
8. Autoscaling Deep Dive (Ch. 06)
9. Canary Deployments Deep Dive (Ch. 07)
10. Monitoring & Observability (Ch. 08)
11. Model Versioning & Registry (Ch. 09)
12. Drift Detection (Ch. 10)
13. Triton Inference Server (Ch. 11)
14. Production Best Practices
15. Additional Quick Questions
16. System Design Scenarios
17. 2025-2026 Landscape Quiz
18. Rapid-Fire Flashcards & Glossary
19. Traps & How to Recover
20. Tips for Interviews
21. Resources

LLM Inference Fundamentals

Q1: Explain how LLM inference works step-by-step.

Tokenize to token IDs, embed to dense vectors, pass through the transformer layers (self-attention plus feed-forward per layer), project the final hidden states to vocabulary-size logits, sample the next token (greedy, top-k, top-p, temperature), append it, and repeat until EOS, max length, or a stop string. Each token depends on all previous ones; the KV cache stores past key/value tensors so they are not recomputed.

The split that matters: prefill processes the whole prompt at once and is compute-bound; decode produces one token per step and is memory-bandwidth-bound. Continuous batching, chunked prefill, and disaggregated prefill/decode all exist because of that split.

Q2: What is KV caching and why is it important?

Prefill computes full Q, K, V for every prompt token. Decode computes Q only for the new token and reuses cached K and V. That turns an ( O(n^2) ) per-step cost into ( O(n) ): generating 100 tokens without a cache means 100 passes each reprocessing up to 100 tokens; with it, each pass processes 1 new token against the cached prefix.

KV cache size per token ≈ 2 (K and V) × num_layers × num_kv_heads × head_dim × dtype_bytes. For a 70B-class dense model with grouped-query attention that is still megabytes per token at long context, which is why the KV cache — not the weights — is usually the binding memory constraint at high concurrency.

Q3: What is the difference between training and inference?

Training needs gradients plus optimizer state — 2-3x model memory in FP32 Adam state alone. Inference has fixed weights and is dominated by KV-cache growth and request-arrival variability instead, so there is no fixed batch shape to plan against.

Q4: Explain the attention mechanism in the context of inference.

Q is what the current token is looking for, K describes what each past token offers, V is the information retrieved.

Formula:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) x V

Naive attention is ( O(n^2) ) in sequence length ( n ) for both compute and the attention matrix’s memory, which is why long context is expensive. Production stacks never materialize the full ( n \times n ) matrix (FlashAttention-style fused kernels) and do not waste memory on it (PagedAttention-style block KV cache).

The variant ladder is MHA → MQA (one KV head) → GQA (a few KV heads shared across query heads; Llama 2/3, Mistral, most current models) → MLA (DeepSeek-V2/V3, compresses KV into a low-rank latent). GQA and MLA exist to shrink the KV cache — the real inference bottleneck — not to save training compute.

Model Serving & Basic Serving Patterns (Ch. 01)

Maps to Chapter 01 — Basic Serving (app.py, model_loader.py, test_api.py).

Q5: How would you design an LLM serving API?

POST /v1/completions
{
  "prompt": "The future of AI is",
  "max_tokens": 100,
  "temperature": 0.7,
  "top_p": 0.9
}

Validate the request (prompt length, parameter ranges) before touching the GPU, tokenize, generate, return text plus metadata — latency, token counts, finish reason. The model loads once at process startup, never per request.

Around that: async handling so concurrency does not block the event loop; streaming via SSE or chunked responses; typed errors (400 bad input, 429 overload, 503 draining); rate limiting that rejects early rather than queueing forever; and liveness separated from readiness so Kubernetes can tell “process up” from “model loaded”.

Q5b: Why must the model be loaded once at startup instead of per-request, and what does “model_loader.py as a singleton” actually buy you?

Loading a multi-GB to multi-tens-of-GB checkpoint takes seconds to minutes; per-request loading puts that on every request. A singleton loader — module-level global, or a class instantiated once in a startup hook — loads weights exactly once per process and shares them across every request that worker handles.

It also gives fail-fast startup: the pod crashes before it is marked ready instead of failing the first user request, which is why readiness must depend on “model loaded”, not “HTTP server up”. The trap is a model load left inside a request handler “for testing” with no guard, silently reloading every request.

Q5c: Sync (Flask/gunicorn workers) vs async (FastAPI/uvicorn) for a basic LLM serving app — which do you pick and why?

Async, FastAPI on uvicorn. Most of a request’s wall-clock time is awaiting the GPU through the engine’s async generate call, so one process holds hundreds of in-flight requests without a thread each. It is what vLLM’s OpenAI-compatible server and TGI actually do.

Sync loses twice. The GIL means CPU-bound work — tokenization, sampling, JSON parsing, validation — does not parallelize across threads in one process; only the forward pass releases it inside C/CUDA calls. And each gunicorn worker needs its own model copy on the GPU, which usually does not fit. The standard pattern is one async process per GPU (or per tensor-parallel GPU set), scaled by Kubernetes replicas rather than gunicorn workers.

Q5d: How do you support streaming responses, and why does it matter for LLM UX?

Server-Sent Events (text/event-stream) or chunked transfer encoding: the engine yields tokens as generated and the HTTP layer flushes each chunk immediately instead of buffering.

Streaming makes TTFT the metric users feel. The user starts reading after prefill instead of after prefill plus full decode; on a 500-token answer that is the difference between “instant” and multi-second perceived latency.

The trap is buffering anywhere in the path — disable proxy buffering, watch gzip interacting badly with streaming, and set client timeouts long enough for slow generations.

Q5e: Design the health check strategy for a basic LLM serving pod.

Liveness answers “is the process alive and not deadlocked” — a cheap /healthz that never touches the model; repeated failure restarts the container. Readiness answers “can this pod serve right now” — model loaded, engine not in an unrecoverable state; failure removes the pod from Service endpoints but does not restart it, which matters because restarting a temporarily overloaded pod is the wrong reaction. Startup covers the minutes a large model takes to load; a generous failureThreshold/periodSeconds stops liveness killing the pod mid-load, the most common cause of crash-looping large-model pods.

The anti-pattern is readiness that depends on GPU utilization or queue depth. That turns transient load into eviction, the remaining pods absorb the traffic, also fail readiness, and you get a thundering-herd death spiral.

Q5f: How do you validate and sanitize LLM request inputs safely?

Enforce types and bounds with Pydantic: max prompt length, a max_tokens ceiling, valid ranges for temperature and top_p. The ceiling is the important one — without it a client can request 1,000,000 tokens and pin a GPU for minutes. Many production gateways cap max_tokens per plan or tier, because a huge generation inside a large batch multiplies memory pressure.

Reject before tokenization and generation; validation should be nearly free, and the point is to fail cheap requests cheaply rather than paying GPU cost to reject them. If user input is composed into a larger system prompt, escape or clearly delimit it rather than relying on the model to police itself.

Q5g: What does “graceful shutdown” mean for an LLM serving pod, and why do naive implementations lose requests?

On SIGTERM — sent before a pod is killed during a rollout, scale-down, or preemption — the process stops accepting new requests, finishes or cleanly returns in-flight generations, then exits within terminationGracePeriodSeconds. Naive implementations die immediately, silently dropping mid-generation requests; with streaming that surfaces as a truncated answer.

Because a full generation takes a while, terminationGracePeriodSeconds usually needs to be far above the Kubernetes default of 30s — tens of seconds to a couple of minutes depending on max_tokens and concurrency. Pair it with readiness: flip readiness false the instant SIGTERM arrives, removing the pod from the Service, while draining existing connections.

Q5h: Your /v1/completions endpoint works for one user in a demo but falls over with 50 concurrent users on one GPU. Diagnose it.

Diagnose by GPU utilization under load first. Low utilization with bad latency is a batching and scheduling problem, not a hardware problem — reaching for more GPUs before checking that is the wrong instinct.

The likely cause is a naive one-request-at-a-time generation loop, raw HuggingFace .generate() called synchronously per request, with no batching. At 50 concurrent users each waits a full serial turn, so latency scales linearly with load. The fix is a real inference engine — vLLM, TGI, or Triton — doing continuous batching. Secondary causes: no queueing or backpressure, so requests pile up in-process rather than being admitted or rejected predictably; no max_tokens cap, so a few huge requests starve everyone; and single-process serving with no path to a second replica.

Docker & Containerization (Ch. 02)

Maps to Chapter 02 — Docker (Dockerfile.basic, Dockerfile.gpu, Dockerfile.optimized, docker-compose.yml).

D1: Why does an LLM serving Dockerfile need to look different from a typical Python web app Dockerfile?

The base image must carry CUDA/cuDNN userspace libraries matching the host driver (nvidia/cuda:...-runtime, or vllm/vllm-openai), not python:3.x-slim, and the runtime must inject the device — --gpus all in Docker, the NVIDIA Container Toolkit and nvidia.com/gpu device plugin in Kubernetes. Without that the CUDA libraries are visible but the device is not.

Size is the other difference: CUDA plus PyTorch plus serving dependencies is easily 5-15GB, which drives pod start and node scale-up time during autoscaling bursts. Weights, often tens of GB, are not baked in — they are pulled at startup into a mounted volume, so the image stays small and a model swap needs no rebuild.

D2: Explain the difference between Dockerfile.basic, Dockerfile.gpu, and Dockerfile.optimized patterns.

Basic is CPU-only on a plain Python base: fine for testing API logic without a GPU, useless for anything latency-sensitive. GPU uses a CUDA base with GPU-enabled wheels (CUDA-built torch, vllm) and pinned driver-compatible versions — the one that runs at production speed.

Optimized is the GPU image plus image engineering: multi-stage build so build dependencies never ship, layer ordering putting rarely-changing layers (CUDA base, system deps) before frequently-changing ones (app code) for cache hits, a .dockerignore so local caches and checkpoints are not shipped accidentally, and a runtime-only CUDA base instead of the full devel image.

D3: Walk through multi-stage builds for an LLM serving image and why they matter here specifically.

FROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builder
RUN pip install --user vllm torch
# compile any custom kernels here

FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04
COPY --from=builder /root/.local /root/.local
COPY app.py .
CMD ["python", "app.py"]

The devel image carries nvcc and headers and is only needed to build custom CUDA kernels or compile from source. It is roughly 2-3x larger than runtime. Shipping it costs registry storage, pull time, and attack surface for no benefit once the artifacts exist, so you compile with the heavy toolchain and ship only the runtime image with the artifacts copied across.

D4: How do you handle GPU access and driver compatibility in Docker?

Install the NVIDIA Container Toolkit on the host so the runtime can expose GPU devices, then run with docker run --gpus all or runtime: nvidia in Compose.

Version compatibility is the number one real-world failure mode. The container’s CUDA version must be compatible with the host’s NVIDIA driver: drivers are backward-compatible with older CUDA runtimes but not forward-compatible with newer ones, so a container built against CUDA 12.4 fails on a host with only a CUDA-11-era driver. In Kubernetes the GPU Operator and device plugin abstract this and advertise nvidia.com/gpu as schedulable — but the compatibility problem does not disappear, it becomes “which node pool runs which driver version”.

D5: How should you manage model weights in a containerized serving setup — bake into the image or mount at runtime?

Baking in is simplest and fully immutable — the image is the version — but images become huge, every model update means a rebuild, push, and pull, and registry cost balloons. Mounting at runtime keeps the image small and generic: an init container or startup script pulls weights from S3/GCS or a registry into a volume or a node-local cache.

The common production answer is runtime mount with a warm node-local cache, because it decouples code version from model version. A model swap becomes a config change — an env var or ConfigMap pointing at a model URI — instead of a CI/CD rebuild, and that is what makes canary and rollback of model versions independent of code versions. Watch the cold-start penalty: pulling a 140GB checkpoint on every new pod is a major contributor to autoscaling lag.

D6: What goes wrong if you don’t pin exact versions of CUDA, PyTorch, and the inference engine in your image?

Silent ABI mismatches: a PyTorch wheel built against CUDA 12.1 loaded against a CUDA 12.4 runtime can work, half-work with a wrong kernel and a slow fallback path, or crash with CUDA error: no kernel image is available for execution on the device.

You also get drift. An unpinned pip install vllm today versus three weeks from now can pull a minor version with different default flags, different memory behavior, or a different OpenAI-API surface — nothing changes in the Dockerfile diff and production behavior changes anyway. Pin exact versions (torch==2.4.0+cu124, vllm==0.6.3), use a lockfile, and rebuild-and-test in CI before promoting a tag.

D7: How would you use docker-compose.yml in local development for an LLM serving stack, and where does it stop being appropriate?

Good for the inner loop: serving container plus Prometheus, Grafana, and a mock registry in one command, GPU passthrough via runtime: nvidia or deploy.resources.reservations.devices, and a shared volume for weights so you do not re-download per up.

It stops at multi-node scheduling, autoscaling, rolling updates, secrets management at scale, and multi-tenant resource isolation — exactly the gap Kubernetes fills. Compose is single-host by design; production LLM serving is not.

D8: How do you reduce image size and cold-start time for a GPU serving image without breaking reproducibility?

Multi-stage build to drop compiler toolchains, and prefer the framework’s official runtime image (vllm/vllm-openai:<pinned-tag>) over assembling CUDA plus PyTorch plus engine yourself — it is already layer-optimized and tested upstream.

Order layers least-to-most frequently changing (base → system deps → Python deps → app code) so CI caches hit on app-code-only rebuilds. Keep weights out of the image and use a node-local cache. If cold node scale-up is on the autoscaling critical path, pre-pull images with a DaemonSet warm-up or node image pre-baking.

D9: A container passes nvidia-smi inside docker exec but the serving process still reports “no CUDA devices”. What do you check?

nvidia-smi working only confirms the toolkit and driver are visible to that process — not that the serving process, possibly a different user, cgroup, or one started before device injection, can use the device.

Check the container was started with the GPU flag for this invocation, easy to miss when you docker exec into a container originally run without --gpus. Check CUDA_VISIBLE_DEVICES is not empty or wrong. Then check the wheel: torch.cuda.is_available() returning False while nvidia-smi works almost always means a CPU-only wheel, because pip resolved plain torch instead of a CUDA-tagged build. In Kubernetes, confirm the pod sets resources.limits."nvidia.com/gpu" — GPUs are invisible without a limit — and that the device plugin DaemonSet is healthy on that node.

Performance Optimization

Q8: How would you optimize LLM inference latency?

Model level: quantization (FP16, FP8, INT8, INT4), pruning, distillation to a smaller student. Inference level: KV caching, batching, continuous batching (vLLM/TGI iteration-level scheduling), and speculative decoding, where a small draft model proposes several tokens and the target verifies them in one batched pass.

Hardware: a GPU rather than a CPU (10-100x for this workload), bf16/fp8 tensor-core kernel paths, and tensor/model parallelism for models that do not fit or need more aggregate bandwidth. System: pre-warm with a dummy forward pass before taking traffic, pool connections, cache responses or prefixes where semantically valid. Architecture: async handling, bounded request queues for bursts, and prefix- or KV-cache-aware load balancing rather than round-robin.

Q9: What is PagedAttention and why does it matter?

Traditional KV caches pre-allocate a contiguous buffer sized for the maximum sequence length per request. When real generations are much shorter, that is severe internal fragmentation, and it makes long sequences at high concurrency impractical.

PagedAttention divides the cache into fixed-size blocks, like OS memory pages: allocated on demand, freed when sequences complete, reused for new sequences, and shared across sequences with an identical prefix — which is what prefix caching builds on. The result is near-zero fragmentation (only the last partial block is wasted), predictable support up to the model’s max context, and more concurrent sequences in the same GPU memory — which is exactly what continuous batching needs in order to have work to schedule.

(For deeper vLLM-specific internals — chunked prefill, scheduler design, disaggregated prefill/decode, speculative decoding, tensor/pipeline parallelism — see vLLM Internals Deep Dive.)

Q10: Explain the trade-offs between latency and throughput.

Latency is time for one request; throughput is requests or tokens per second across the system. Batching is the main lever: larger batches raise throughput and per-request latency through contention at each decode step, smaller batches do the reverse. Larger models raise quality and latency, lower precision is faster and smaller with accuracy risk, more GPUs buy throughput with cost.

For low latency: small batches, an optimized model, fast hardware, priority for interactive streaming. For high throughput: large batches, continuous batching, more GPUs, priority for offline work.

Do not optimize either in the abstract. Optimize against the actual SLO — for example P95 TTFT under 300ms and P95 inter-token latency under 50ms — and measure throughput at that SLO. Huge throughput bought by letting P99 blow up has moved the problem, not solved it.

Kubernetes & Deployment (Ch. 03)

Maps to Chapter 03 — Kubernetes (deployment.yaml, deploy.sh).

Q11: How would you deploy an LLM model to Kubernetes?

1. Containerize (see Ch. 02 above):

FROM python:3.9-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

2. Create a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: llm-serving
        image: llm-serving:v1.0
        resources:
          requests:
            memory: "4Gi"
            cpu: "2000m"
          limits:
            memory: "8Gi"
            cpu: "4000m"
            nvidia.com/gpu: 1
        livenessProbe:
          httpGet:
            path: /health
            port: 8000

3. Create a Service:

apiVersion: v1
kind: Service
metadata:
  name: llm-serving
spec:
  selector:
    app: llm-serving
  ports:
  - port: 80
    targetPort: 8000

4. Deploy:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

The parts needing judgment: GPUs are only requested as whole-unit limits, never fractional, without a sharing mechanism (K3); liveness, readiness, and startup probes need splitting (Q5e); the default rolling update strategy fits single-GPU-per-pod workloads badly (K2); configuration goes in ConfigMaps, API keys and tokens in Secrets.

Q12: How does Horizontal Pod Autoscaling (HPA) work?

HPA checks metrics on an interval (15 seconds by default), compares the current value to the target, computes desired replicas, and updates the Deployment; Kubernetes creates or destroys pods to match.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Built-in CPU and memory resource metrics are a poor signal for GPU-bound serving. The useful signals — requests per second, queue depth, latency — are custom metrics via the Prometheus Adapter or KEDA. A stabilization window prevents flapping.

(For why CPU/memory-based HPA is usually the wrong tool for LLM serving, and what to use instead, see Autoscaling Deep Dive.)

Q13: Explain canary deployments for model updates.

Deploy the new version alongside stable, split traffic (say 90/10), compare metrics between the two cohorts, step up if healthy (25%, 50%, 100%), and route everything back to stable if not.

Three implementations, coarse to fine: two Deployments behind one Service weighted by replica-count ratio; a service mesh or Gateway API splitting by percentage independent of replica counts; or application-level routing on headers or user-ID hashing for consistent bucketing.

Compare latency percentiles (P50/P95/P99) between cohorts, not in aggregate, plus error rates and quality metrics such as thumbs-down or hallucination-flag rate. The payoff is bounded blast radius, instant rollback by changing weights, and a real A/B comparison on live traffic.

(For the full automated promotion/rollback pipeline, see Canary Deployments Deep Dive.)

K1: Why is nvidia.com/gpu treated so differently from CPU/memory by the Kubernetes scheduler?

GPUs are an extended resource: integer-only requests, no overcommit, and without MIG, time-slicing, or MPS configured, requests must equal limits. There is no fractional or burstable GPU by default, so the scheduler sees a node’s GPUs as a small pool of indivisible slots — a pod asking for one either finds a node with a free GPU or does not schedule at all.

The consequence is that GPU bin-packing matters far more than CPU bin-packing. A cluster with plenty of total free GPU capacity spread as fractions across many nodes can still fail to schedule a pod because no single node has a whole GPU free, which makes node pool sizing and per-pod GPU granularity (1 vs 2 vs 8) a real design decision.

K2: The default Kubernetes RollingUpdate strategy is a poor fit for a Deployment where each pod holds a 70B model on 4 GPUs. Why, and what would you change?

Default RollingUpdate with maxSurge: 25% brings up extra pods before removing old ones. For a pod needing 4 GPUs and minutes to load a huge checkpoint, that means spare GPUs sitting idle purely to support the rollout — usually unavailable in a GPU-constrained cluster.

Three fixes by situation: with no spare capacity and tolerance for a brief dip, set maxSurge: 0, maxUnavailable: 1 to replace in place one at a time; for genuinely zero-downtime rollouts, explicitly over-provision by one replica’s worth of GPUs; better still, move to a canary rollout (Ch. 07) so the old fleet stays fully up while a small new fleet is validated. Also tune minReadySeconds and the readiness probe so a pod is not counted available until the model has loaded and passed a warm-up check — otherwise the rollout outruns readiness and you get a real availability dip.

K3: How do you share a single GPU across multiple pods in Kubernetes, and what are the trade-offs between the approaches?

Pick the isolation level from the blast radius you can tolerate: MIG or DRA-managed hard partitions for a platform shared across teams, time-slicing as a cost optimization for single-tenant batch or dev work.

K4: Should an LLM serving Deployment use a Deployment or a StatefulSet?

A Deployment for most serving pods: they are stateless and interchangeable, with no per-pod identity or storage beyond shared model weights, and the Service load-balances across them.

A StatefulSet earns its place when pods need stable network identity or per-pod storage — a multi-node tensor-parallel or pipeline-parallel deployment where rank-0 needs a predictable address for the other ranks (common in multi-node vLLM/Triton or Ray-based clusters), or where each pod owns a distinct local NVMe cache worth preserving across restarts. The trap is reaching for it by default “because it’s a model”: the question is identity and storage, not whether the workload feels heavyweight.

K5: Design node affinity / taints-and-tolerations for a mixed CPU+GPU cluster running LLM serving alongside other workloads.

Taint the GPU nodes (nvidia.com/gpu=true:NoSchedule) so only pods that tolerate the taint and request a GPU land there, keeping random CPU workloads off expensive hardware. Use nodeAffinity/nodeSelector to require the matching GPU node pool and, where it matters, a specific GPU generation — a pod assuming 80GB of HBM must not land on a 40GB node.

Add PriorityClasses so latency-sensitive serving outranks batch and offline jobs, letting the scheduler preempt lower-priority work under GPU pressure rather than leaving serving pods Pending. Use topology spread constraints across zones and nodes so one failure cannot take out every replica of a critical model.

K6: How do you handle persistent storage for multi-tens-of-GB model weights across pod restarts and node scale-up events?

A ReadOnlyMany PVC shared across pods of the same model version, backed by a network filesystem or cloud RWX equivalent, stops every pod re-downloading the same weights. Node-local caching — hostPath or a DaemonSet-managed NVMe cache — trades simplicity for speed: the first pod on a node pays the download, later pods and restarts on that node are fast, but cold nodes from autoscaling still pay full cost, which lands on scale-up latency.

Init containers are the standard mechanism: download and verify the artifact before the main container starts, so readiness only checks “engine loaded”. Watch for the race where multiple pods write the same node-local cache path during a scale-up burst — use a lock file or content-addressed paths keyed by model hash.

K7: What’s the “cold start” problem for GPU pods in Kubernetes, and how much of it is actually Kubernetes’ fault?

Almost none of it — Kubernetes scheduling overhead is seconds. The time goes to provisioning a new GPU node if none is free (1-10+ minutes depending on cloud and instance type), pulling a multi-GB image, downloading multi-tens-of-GB weights, and engine warm-up: CUDA graph capture, kernel autotuning, vLLM’s startup profiling pass.

So solving cold start means attacking image size (Ch. 02), weight caching (K6), and node pre-provisioning or warm pools, not tuning scheduler settings. It is also why naive reactive autoscaling, which scales up only after load has arrived, works poorly here.

K8: How do you expose an LLM serving Service outside the cluster safely, and where does authentication/rate-limiting belong?

External load balancer or Ingress (or Gateway API) → API gateway for auth, rate limiting, request validation, and routing to version-specific backends → Kubernetes Service → serving pods.

Authentication and rate limiting belong at the gateway, not in the pod: the pod should be a dumb, fast, trusted-input component, and auth in every pod duplicates logic and makes the hot path slower and harder to change. TLS terminates at the Ingress or gateway too, so GPU-node CPU is not burned on crypto. Route model tiers and versions at the gateway by path or header rather than baking routing logic into clients.

Load Testing & Capacity Planning (Ch. 04)

Maps to Chapter 04 — Load Testing (locust_test.py, measure_latency.py).

LT1: What’s wrong with load-testing an LLM API using the same request-per-second methodology you’d use for a REST CRUD API?

CRUD requests are roughly uniform in cost; LLM requests vary enormously with prompt length and max_tokens, so RPS alone is a poor load axis — two runs at identical RPS with different token distributions produce completely different GPU load. LLM serving also has two cost phases, prefill and decode, so a representative test must vary both input-length and output-length distributions, not just arrival rate.

Because of continuous batching, the right concurrency metric is usually the number of concurrent in-flight sequences. Throughput is a function of how many sequences the engine juggles, capped by KV-cache memory — not of arrival rate.

LT2: Design a load test plan for a new vLLM deployment before it goes to production.

Define the traffic profile first — realistic prompt-length and output-length distributions from production logs if they exist, since chat, summarization, and RAG look nothing alike. Sweep concurrency rather than RPS, recording tokens/sec throughput and TTFT, inter-token latency, and end-to-end latency at each level.

Find the knee: throughput rises roughly linearly with concurrency until GPU or KV-cache saturation, then latency blows up while throughput plateaus or falls. That knee is the practical per-replica ceiling. Report max sustainable throughput while P95 TTFT and P95 inter-token latency stay within SLO, not raw peak.

Add a soak test at 70-80% of found capacity for hours, to catch memory leaks, KV-cache fragmentation, and slow degradation a burst test hides. Then test the failure path at 100%+ of capacity: do requests queue and eventually succeed, or does the engine error and OOM? That decides how you configure backpressure.

LT3: What metrics do you pull out of a load test, beyond “requests per second”?

TTFT, dominated by prefill plus queueing, the metric users feel first. Inter-token latency / TPOT, dominated by decode step time and batch contention, which makes streaming feel fast or slow. End-to-end latency, TTFT + (TPOT × output tokens), what a non-streaming client sees.

Throughput in tokens/sec is the real capacity number. GPU and KV-cache utilization during the run tell you whether you are compute-bound, memory-bound, or scheduler-bound. Error and timeout rate as load rises shows where the system sheds, and whether it sheds gracefully with 429s or badly with timeouts and crashes.

LT4: How do you use Locust (or similar) to generate realistic LLM traffic, and what’s the catch with naive locust_test.py-style scripts?

Naive scripts send a fixed prompt repeatedly at a fixed rate. That misrepresents prefix-cache behavior — an identical prompt inflates hit rate, fully random prompts miss it entirely — and constant output length hides decode-bound bottlenecks. Better: sample prompts from a representative corpus with varying length, sample max_tokens from a realistic distribution, and ramp Locust’s user count rather than a flat rate.

The catch is that Locust’s own workers become the bottleneck before the server does at high concurrency, because of the Python GIL and single-machine limits. Distribute the workers, or use a purpose-built tool such as vllm bench serve or genai-perf, which model TTFT and TPOT correctly.

LT5: measure_latency.py-style scripts show great P50 latency but the on-call gets paged for P99 timeouts in production. What’s going on?

P50 tracks the common case; P99 is queueing effects, allocator or GC pauses, occasional very long prompts, cold caches, and batches temporarily saturated by a few outlier long generations.

A test at low or medium concurrency never enters the queueing regime that causes P99 blowups during production bursts, so it structurally cannot show you the tail. You have to test at and above expected peak concurrency. Also check whether the test hit a warm, pre-scaled fleet while production sometimes hits cold or just-scaled-up pods (K7) — that gap alone explains a lot of “P50 fine, P99 pages us”.

LT6: How does batch size / concurrency setting interact with load test results, and what would you tune based on what you see?

Throughput well below what GPU compute allows, with low GPU utilization, means the engine’s max-concurrent-sequences or max-batched-tokens setting is too conservative — raise it, bounded by KV-cache memory.

Latency degrading sharply while throughput barely improves means you hit the KV-cache memory ceiling: the engine is queueing and evicting rather than adding sequences. The fix is more memory — a bigger GPU, a higher gpu_memory_utilization fraction, or a quantized KV cache — or accepting a lower concurrency ceiling. Chunked prefill matters here too: a long prompt hogging a full prefill step spikes inter-token latency for other in-flight decodes, and only a mix of very long and very short prompts reveals it.

LT7: How would you capacity-plan “how many GPUs do I need to serve X req/s at Y SLO” from load test data?

From the load test, find the max concurrency or tokens/sec one replica sustains while meeting the latency SLO — call it C_max. Model expected traffic as peak concurrent requests, not average, using your actual length distributions: provision for peak, autoscale for the rest.

Required replicas ≈ peak_concurrency / C_max, plus headroom for rolling updates (K2), node or zone failure tolerance (N+1 or N+2), and burst above forecast. Multiply by GPUs-per-replica — the tensor-parallel degree — for total GPU count, then sanity-check against GPU memory needed for weights plus KV cache at that concurrency. Always re-validate on the target hardware, model, and quantization: theoretical FLOPs-based estimates are frequently off by 2-3x from measured reality, because decode is memory-bandwidth-bound.

LT8: What’s the difference between load testing “throughput mode” and “latency mode,” and why would you run both?

Throughput mode fires as many concurrent requests as the client can generate, unconstrained by arrival timing, to find maximum sustainable tokens/sec — what you want for batch and offline workloads, and for locating the ceiling and the knee.

Latency mode sends requests at a fixed, realistic (roughly Poisson) arrival rate matching expected production traffic and measures the latency distribution there, validating the SLO under realistic rather than maximal load.

They answer different questions. A system can have an excellent throughput ceiling and still miss its latency SLO at normal traffic if it is misconfigured — for example batch size tuned for throughput rather than latency.

LT9: How do you make load tests reproducible and comparable across engine versions or config changes?

Pin the exact prompt and output-length dataset with seeded sampling so two runs compare the same workload rather than different random draws, and hold hardware, instance type, driver and CUDA version, checkpoint, and quantization constant.

Report full latency distributions — P50/P90/P95/P99 — not averages, because averages hide tail regressions. Track results over time in a dashboard, or a CSV committed alongside config changes, so a vLLM version bump or scheduler-flag change gets an objective before-and-after instead of “it feels faster”.

vLLM Internals Deep Dive (Ch. 05)

Maps to Chapter 05 — vLLM Serving (vllm_server.py). The two questions below are preserved from the original Model Serving section because they’re really vLLM-specific.

Q6: What are the differences between HuggingFace Transformers and vLLM?

Use transformers for development, research, one-off inference, and maximum flexibility — custom generation logic, unusual architectures. Use vLLM or a comparable engine for production, high-concurrency serving, and cost-sensitive GPU utilization.

(Treat exact throughput multipliers as workload-dependent — always validate with your own load test per Ch. 04 rather than quoting a fixed number.)

Q7: How does continuous batching work in vLLM?

Static batching waits for a batch to fill, processes it, waits for all members to complete, then starts the next — so the batch runs at the speed of its slowest member and the GPU idles in the gap.

Continuous batching schedules at the decode-iteration level: new requests join as soon as there is KV-cache room, completed requests are removed immediately, and the rest continue without waiting for the cohort. The GPU never bubbles waiting for a batch to fill or drain.

Time 0: [Req1, Req2, Req3] -> Processing
Time 1: [Req1, Req2, Req3, Req4] -> Req4 added mid-flight
Time 2: [Req2, Req3, Req4] -> Req1 completed, removed

V1: What is chunked prefill and why did vLLM add it?

Without it, a long prompt’s prefill runs as one large step that can monopolize a full scheduler iteration, delaying the decode steps of every other in-flight request sharing it — one long prefill produces a visible latency spike for unrelated users.

Chunked prefill splits that prefill into smaller chunks processed across multiple iterations, interleaved with other requests’ decode steps. The net effect is smoother, more predictable inter-token latency under mixed short/long-prompt traffic, at a small cost in total prefill throughput — a direct trade of throughput for tail-latency stability.

V2: Explain vLLM’s scheduler at a level that would satisfy a senior interviewer.

Each iteration the scheduler decides which requests run a step, subject to a KV-cache-block budget and a max-batched-tokens budget. It prioritizes continuing already-running decode sequences and admits new prefill requests from a waiting queue as capacity allows; with chunked prefill it can admit partial prefill work rather than making an all-or-nothing decision.

If KV-cache blocks run out it can preempt a running sequence — evicting it by swapping the cache out or recomputing it later — to make room. Which sequence is evicted, and the eviction policy, directly set fairness and tail latency. This scheduler is the actual engine behind continuous batching: it is a scheduling problem, not a batching trick.

V3: What is speculative decoding and when does it actually help?

A small, cheap draft model proposes several candidate next tokens; the large target model verifies all of them in a single batched forward pass — cheaper than generating them one at a time — and accepts the longest correct prefix.

It helps most when decode is memory-bandwidth-bound, the usual case, and the draft’s guesses are frequently right: low-temperature or near-deterministic generation, code completion, or drafts distilled to match the target’s distribution. It helps less, or hurts, at high temperature and high output diversity, or at high concurrency where you are already GPU-compute-saturated — it trades extra compute per accepted token for fewer serial steps, and that trade is bad when compute is already the bottleneck.

Variants worth naming: draft-model speculative decoding, Medusa-style extra prediction heads on the target model itself, and n-gram or prompt-lookup decoding with no separate model, which works well where output repeats input literally, like code editing.

V4: How does vLLM support tensor parallelism and pipeline parallelism, and when do you pick which?

Tensor parallelism shards each layer’s weight matrices across GPUs — attention heads and MLP columns split across N GPUs — and needs an all-reduce or all-gather per layer, so it demands high-bandwidth interconnect and is usually kept within a node. Pipeline parallelism splits layers across GPUs or nodes sequentially, communicating only at stage boundaries, so it tolerates slower interconnect at the cost of pipeline bubbles unless you microbatch.

Rule of thumb: TP up to the GPU count in one NVLink domain, typically 8 GPUs on one node, to fit or speed up a model that does not fit on fewer; PP, or TP+PP together, when you must span nodes. Data parallelism — replicate the model, different requests per replica — is orthogonal and is what Kubernetes replicas give you. TP and PP set how big one replica is; replica count sets how many you run.

V5: What quantization formats does a modern serving engine like vLLM support, and how do you choose one?

FP16/BF16 is the baseline: minimal accuracy loss versus FP32, half the memory. FP8 is native on Hopper and Blackwell-class tensor cores, roughly halves memory again, and can meaningfully speed up compute-bound prefill with small workload-dependent accuracy impact — increasingly the default fast path on new hardware as of 2025-2026 (verify the current support matrix before quoting). INT8, AWQ, GPTQ, INT4 give aggressive memory reduction when weights rather than KV cache are the constraint, with larger accuracy risk, usually needing calibration data and a validation pass.

KV-cache quantization — FP8 KV cache — is a separate knob from weight quantization. It shrinks the cache specifically, which is often the real bottleneck at high concurrency and long context, independent of weight precision.

Decision process: baseline load test at FP16/BF16, try FP8 next since it is usually close to free on modern hardware, and only then reach for INT4 or AWQ. Validate task-specific quality, not just perplexity.

V6: What is prefix caching (as distinct from PagedAttention itself) and why does it matter for RAG/chat workloads?

Prefix caching reuses KV-cache blocks across requests sharing an identical prompt prefix — the same system prompt, or the same long RAG context across follow-up questions — instead of recomputing prefill for the shared portion. It is a natural extension of PagedAttention, because that block-based cache is already non-contiguous and friendly to content addressing: hash blocks by content, look up, reuse hits.

The win is large for chat (repeated system prompt plus growing history) and RAG (same context, several questions), turning much prefill work into cache hits and dramatically improving TTFT. The design implication: your load balancer must be prefix- or cache-aware, routing requests likely to share a prefix to the same replica. Pure round-robin scatters hit opportunities across replicas sharing no cache state, and you lose most of the benefit.

V7: What changed with vLLM’s “V1” engine rewrite, and why should you know that as an interview talking point?

vLLM went through a significant core-architecture rewrite, alpha announced around early 2025, redesigning the scheduler and execution loop for lower CPU-side overhead, cleaner separation between scheduling and model execution, and better support for multimodal and newer architectures — while keeping continuous batching and PagedAttention as foundations.

Knowing vLLM is not a frozen design, and that its internals evolved while the user-facing API stayed largely stable, shows you track the ecosystem. Practically, exact V0-versus-V1 internals change fast: anchor on the invariants — continuous batching, paged block KV cache, chunked prefill, speculative decoding support — and flag version-specific internals as “verify against current docs”.

V8: What is disaggregated prefill/decode serving, and why would a senior team consider it?

Run prefill — compute-bound, bursty, parallelizable — and decode — memory-bandwidth-bound, needing sustained KV-cache residency — on separate GPU or node pools, transferring a request’s KV cache from prefill worker to decode worker over fast interconnect or a shared store such as LMCache.

The motivation is that the phases have different optimal batch sizes and bottlenecks. Co-locating them means a long prefill can stall decode latency (mitigated but not eliminated by chunked prefill), and you cannot independently scale or tune each phase’s hardware — prefill benefits from raw FLOPs, decode from memory bandwidth and cache capacity.

The cost is real complexity: a KV-cache transfer path, coordination between pools, and new failure modes such as the decode worker holding a request’s cache disappearing. This is a very-large-scale optimization, not a default.

Autoscaling Deep Dive (Ch. 06)

Maps to Chapter 06 — Autoscaling (hpa.yaml). Builds on Q12 above.

AS1: Why is CPU utilization almost always the wrong HPA signal for a GPU-bound LLM serving pod?

The pod’s CPU usage — tokenization, HTTP handling, orchestration — is largely decoupled from GPU load. A pod can sit at 90% GPU utilization and near KV-cache capacity while its CPU shows 10%, so a CPU-based target never fires when it should.

The correct signals are GPU- or engine-native: GPU utilization, KV-cache/block utilization, queue depth (pending requests waiting for a scheduler slot), or request latency directly. All of them need a custom or external metrics pipeline — the Prometheus Adapter, or a KEDA scaler reading Prometheus or the engine’s metrics endpoint. HPA’s built-in Resource metrics will not get you there.

AS2: Design an HPA/KEDA config for a vLLM deployment using queue depth as the scaling signal.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
spec:
  scaleTargetRef:
    name: llm-serving
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      query: avg(vllm_num_requests_waiting)
      threshold: "5"

vllm_num_requests_waiting rising means the running batch is already at capacity and work is backing up — a much earlier and more meaningful signal than GPU utilization plateaued at 95%.

Keep minReplicaCount above zero unless you have specifically solved cold start for scale-from-zero, or the first user after idle eats the full cold-start latency. Tune the scale-up stabilization window short and the scale-down window long, so a replica that just dipped below threshold is not torn down — GPU pods are expensive to bring back.

AS3: Why is scale-to-zero for GPU inference pods so much harder than for a stateless web service?

A stateless web pod cold-starts in 1-5 seconds; a GPU serving pod’s cold start — node provisioning, image pull, weight download, engine warm-up — is commonly 1-10+ minutes. Scale to zero and the first request after idle pays that entire cost, which almost never fits a reasonable latency SLO.

Mitigations: keep a warm minimum and never truly scale to zero on latency-sensitive paths; use keep-warm ping traffic to block scale-down during known-active hours; pre-provision a warm node pool so scale-up skips node provisioning; or accept scale-to-zero only for genuinely latency-insensitive batch workloads.

AS4: How does cluster autoscaling (or Karpenter-style node autoscaling) interact with pod-level HPA for GPU workloads, and where does it commonly break?

HPA decides “we need N more pods”. With no free GPU capacity those pods sit Pending until the cluster autoscaler or Karpenter notices and provisions a GPU node, adding full node-provisioning latency — often minutes — on top.

The common break is provisioning a node type lacking the GPU generation or count the pod’s affinity and resource request demand, producing a provision-then-still-Pending loop; node pool definitions have to match exactly what the workload asks for. Scale-down breaks too: the autoscaler can drain a GPU node running a long-lived inference pod mid-generation, dropping or truncating requests, unless pod disruption budgets and graceful termination (Q5g) protect it.

GPU autoscaling is a two-layer problem and the layers’ reaction latencies stack — total scale-up latency is the sum, not the max.

AS5: What is predictive/scheduled autoscaling and when would you use it over purely reactive autoscaling?

Reactive autoscaling always lags demand by at least the metric-collection interval plus cold-start time — fine for smoothly varying load, poor for sharp predictable bursts.

Predictive or scheduled scaling pre-warms capacity ahead of known patterns — raising a minimum replica floor before a daily peak, a launch, or a marketing push — via a CronJob changing minReplicaCount or a dedicated controller. Use it whenever demand has a known or learnable pattern, because it attacks cold start directly by provisioning before the spike. Combine with reactive scaling for the unpredictable residual.

AS6: A bursty consumer product (e.g., a viral social feature) spikes 20x in under a minute. Design the autoscaling response.

Pure reactive pod and node autoscaling cannot provision GPU capacity in under a minute, so the answer is not “scale faster” but “absorb the burst without needing to”.

Four levers. Admission control with backpressure: bound the queue, return fast 429s beyond it, rather than degrading latency without limit for everyone. Warm buffer capacity: a standing buffer of replicas sized for the observed peak/burst ratio, funded as a deliberate cost trade-off. Graceful degradation: shed to a smaller model, cap max_tokens, or disable expensive features like long-context mode when queue depth crosses a threshold. Per-user and per-tenant rate limits, so one viral feature does not starve the platform.

Afterwards, use the observed burst shape to right-size the warm buffer and the predictive schedule (AS5).

AS7: How do you avoid HPA “flapping” (rapid scale up/down) for a GPU workload, and why is flapping especially costly here vs. a CPU service?

The standard levers are stabilizationWindowSeconds on scale-down and rate limiting via HPA v2 behavior policies capping pods removed per period.

It is worse for GPU because a removed pod is not a process that restarts fast: bringing it back may mean re-provisioning a node (AS4) and re-downloading weights (K6/K7), so an unnecessary scale-down followed by a needed scale-up costs far more in latency and money than the same flap on a stateless service. Practical setting: asymmetric behaviour — short window for scale-up, minutes for scale-down — plus a minReplicas floor above the noise level of your traffic.

AS8: How would you autoscale a multi-model platform where different models have very different load profiles and hardware needs?

Scale each model’s deployment independently, with its own HPA/KEDA object and thresholds tuned to that model’s capacity envelope — a 7B and a 70B saturate at completely different queue-depth and GPU-utilization numbers.

Use separate node pools per hardware profile — single-GPU nodes for small models, multi-GPU NVLink nodes for large tensor-parallel ones — so the cluster autoscaler provisions the right node shape, with taints and affinities (K5) preventing cross-contamination. For a long tail of low-traffic models, prefer a shared GPU pool with priority-based preemption over a dedicated autoscaling group each.

Canary Deployments Deep Dive (Ch. 07)

Maps to Chapter 07 — Canary Deployments (canary-deployment.yaml). Builds on Q13 above.

CD1: Walk through a fully automated canary pipeline for a new model version, end to end.

The version first passes an offline eval suite — quality benchmarks plus regression tests against known-bad outputs — before it is canary-eligible. It then deploys at 1-5% of traffic alongside stable, both behind the same gateway or mesh split.

An automated analysis window runs for a fixed period, typically 30-60 minutes, long enough for statistically meaningful samples, comparing canary against stable on error rate, P95 and P99 latency, and a proxy quality signal such as thumbs-down rate or a cheap automated eval on sampled outputs.

The decision is automatic: inside bounds steps traffic 5% → 25% → 50% → 100%; any breach rolls back to 0% immediately. At 100% the canary becomes the new stable, and the old stable is kept warm briefly for fast rollback, then scaled down. Argo Rollouts and Flagger implement this loop natively over Kubernetes, Prometheus, and a traffic-splitting layer.

CD2: What metrics are LLM-specific red flags during a canary that a generic web-service canary pipeline wouldn’t catch?

Output quality regression: a version can be faster and perfectly healthy on latency and errors while being worse — more hallucination, format violations, changed refusal rate. Infra metrics are blind to it. Token-length distribution shift: substantially longer or shorter outputs change cost and latency in ways an error-rate check never flags. Safety/guardrail trigger rate: a rise is a strong leading indicator of a behavior regression. Structured-output and tool-call validity rate: in agent pipelines a canary can look healthy while silently emitting malformed function calls downstream systems choke on.

So a real LLM canary needs an automated quality-eval step sampling live canary outputs before promotion — ideally a cheap LLM-as-judge or rule-based check running inline.

CD3: Canary at the Kubernetes-Deployment-replica-ratio level vs. canary via a service mesh’s percentage-based traffic split — what’s the real difference, and when does it matter?

Replica-ratio canary — 9 stable pods plus 1 canary behind one Service — gives a split only as fine-grained as replica count allows, so 10 pods means 10% steps, and it couples “how much traffic the canary gets” to “how many pods it has”, conflating blast radius with resource allocation.

Mesh or Gateway percentage split — Istio VirtualService weight, Gateway API HTTPRoute weight — decouples them entirely: exactly 1% of traffic to a canary running on a single full-sized pod. That matters most for large expensive models, where each 70B replica is a meaningful GPU cost and you want risk exposure decoupled from fleet sizing.

CD4: Design the automated rollback trigger logic — what exactly should trip an automatic rollback, and how do you avoid both false positives and false negatives?

A small set of guardrail metrics with explicit thresholds evaluated over a rolling window rather than single data points: error rate above baseline + 2%, P99 latency above baseline × 1.5, guardrail-trigger rate above baseline plus an absolute threshold.

Require a minimum sample size before evaluating — a canary at 1% traffic for 2 minutes may have too few requests for the error-rate comparison to mean anything, and that is where false positives come from. Avoid false negatives by including the slower, sampled quality-eval signal (CD2), because a model can be infra-healthy and quality-broken at once.

Keep a manual kill switch alongside the automation: default to safe — roll back — on ambiguous or missing data, and let humans force a rollback regardless.

CD5: How do you canary a change to infrastructure (e.g., a new vLLM version or a new GPU generation) rather than a new model weights version?

Same pipeline shape as CD1, but the canary variable is the serving stack or hardware. Deploy the new engine version or hardware behind a small traffic percentage running the same model weights as stable, so any regression is attributable to the infra change rather than confounded.

Watch for numerical and output-distribution differences an engine or hardware change introduces — different kernels, different default precision, different sampling implementation details — even with bit-for-bit identical weights. “Same model, different engine version, subtly different outputs” is a real, easy-to-miss failure mode. Load-test the new infra in isolation (Ch. 04) before canarying on live traffic; the live canary is the last safety net, not the first.

CD6: What’s the difference between canary, blue-green, and shadow (dark) traffic deployment strategies for model updates?

They are not mutually exclusive: a mature pipeline runs shadow first for zero-exposure validation, then a canary for bounded real exposure, then full promotion. Treating it as one all-or-nothing choice is itself the weaker answer.

CD7: How do you handle stateful conversation context (multi-turn chat) correctly during a canary, so a user doesn’t get inconsistent behavior mid-conversation?

Sticky routing keyed on session or conversation ID — consistent hashing at the gateway or mesh — so every turn of one conversation hits the same version for its duration. Otherwise a user gets wildly inconsistent behaviour, style, and quality switching between versions turn to turn.

That means canary percentage is measured in sessions, not requests: a 5% canary must mean 5% of conversations, not 5% of individual turns randomly assigned, which would corrupt every multi-turn conversation it touched. Record which version handled a session so post-hoc quality analysis can attribute a whole conversation to one version.

CD8: Interviewer pushback: “Why not just A/B test in production analytics after full rollout instead of doing all this canary infrastructure?”

Full rollout exposes 100% of users to any regression immediately and simultaneously; bounding that blast radius before you have full-traffic data is the entire point.

Post-hoc analysis on a bad full rollout tells you how bad it was, not how to avoid most users experiencing it. By the time it surfaces, the bad responses have been shown, the cost incurred, and trust eroded at 100% scale.

They are also not alternatives: canary is the safe delivery mechanism, A/B analysis is the measurement layer. A canary literally is a small, automatically-managed live A/B test with a rollback trigger wired to its own results.

Monitoring & Observability (Ch. 08)

Maps to Chapter 08 — Monitoring (prometheus_exporter.py, grafana_dashboard.json, prometheus.yml).

Q14: What metrics should you monitor for LLM serving?

Application: request rate; latency at P50/P95/P99, with TTFT and inter-token latency tracked separately rather than only end-to-end; error rate; queue size, meaning pending requests waiting for a scheduler slot; throughput in input and output tokens/sec.

Model: generation time, average tokens per request, and the currently-running model version — essential for correlating a metric shift with a deploy. System: GPU utilization, GPU memory and KV-cache utilization used-versus-total, host CPU and memory pressure. Business: cost per request and per token, quality and feedback signals, API usage by endpoint, tenant, and model.

Three dashboards cover it: performance (latency, throughput, errors), resource (GPU, CPU, memory), and model (version-over-version comparison).

M1: Design the Prometheus metric taxonomy for an LLM serving fleet — what should be a counter, gauge, or histogram?

Counters, monotonically increasing: total requests, total tokens generated, total errors by type — used to derive rates with rate(). Gauges, point-in-time: current queue depth, running and waiting sequence counts, GPU memory used, KV-cache block utilization. Histograms, bucketed distributions: request latency, TTFT, inter-token latency, tokens per request — anything you will want percentiles for.

Never store latency as a gauge or an average when you will want P95 or P99 later; you cannot recover percentiles from an average after the fact. Label carefully — model, version, tenant — but watch cardinality, because labelling by raw request ID or user ID explodes series count and can take Prometheus down.

M2: What are the “four golden signals” and how do you adapt them specifically for LLM serving?

Latency splits into TTFT and inter-token latency rather than one end-to-end number, because their causes differ — queueing and prefill versus decode-step contention. Traffic becomes requests/sec and tokens/sec, since token volume rather than request count drives GPU load. Errors covers HTTP-level errors plus engine-level failures — OOM, timeout, unexpectedly truncated generation — that an HTTP-status-only view misses.

Saturation is GPU utilization and KV-cache/block utilization and queue depth. KV-cache saturation is frequently the real ceiling well before GPU compute hits 100%, so tracking GPU percentage alone hides the actual bottleneck.

M3: Your Prometheus instance keeps falling over / running out of memory. What’s the likely cause in an LLM serving context and how do you fix it?

Almost always cardinality explosion: an unbounded label — raw request ID, user ID, prompt hash — multiplies unique time series, and Prometheus memory scales with active series count.

The fix is a bounded label set (model, version, tenant-tier, status code), with per-request detail pushed to logs and traces instead of metrics. Also check scrape interval and retention, since a very short interval combined with high series count compounds the problem, and move long retention to remote-write storage — Thanos, Mimir, VictoriaMetrics — rather than a single local instance.

M4: How do you set actionable alert thresholds for LLM latency without causing alert fatigue?

Alert on SLO burn rate, not raw threshold crossings — “we are consuming our latency budget X times faster than sustainable” — which adapts sensitivity so a blip does not page and a sustained regression does. Use multi-window, multi-burn-rate alerting: a fast short window for “page now” and a slower long one for “ticket, slow leak”.

Alert on P95 and P99, but tune the threshold from your own load-test baseline (Ch. 04) rather than an industry number — “P95 < 500ms” is meaningless without your model size, hardware, and SLO context. Route by actionability: “GPU node down” pages on-call; “quality-eval score dipped 2%” is a ticket for the model team, not a 3am page.

M5: How do you monitor and alert on cost, not just performance, for an LLM serving platform?

Track cost-per-request and cost-per-1K-tokens as first-class metrics, derived from GPU-hour cost divided by observed throughput. A regression that silently doubles cost-per-token — GPU utilization dropping from a bad batching config — is a real incident even when latency and errors look fine.

Alert on GPU utilization sustained well below baseline: “10 GPUs at 20% utilization for an hour” is wasted spend and an alertable condition. Where multi-tenant, attribute cost by tenant or model so a runaway customer does not hide inside an aggregate until the monthly bill arrives.

M6: What’s the role of distributed tracing (OpenTelemetry) in an LLM serving stack, beyond metrics and logs?

Metrics tell you that P99 latency is bad; tracing tells you where the time went on a specific slow request — gateway, queue wait for a scheduler slot, prefill, decode, or a downstream retrieval call for RAG.

It is most valuable for multi-hop architectures — gateway → router → serving pod → retrieval service or tool call — where one slow request could be slow at any hop and per-hop aggregates cannot correlate it. Propagate a trace ID from the gateway through the engine and downstream calls; most engines support OpenTelemetry or can be wrapped to emit spans. Sample at a rate balancing trace-storage cost against debuggability, sampling up on errors and high latency.

M7: How do you build a monitoring setup that catches a quality regression (not just a latency/error regression) in production, continuously — not just during a canary window?

Continuously sample a small percentage of live production outputs — not only canary traffic — and run cheap automated quality signals over them: rule-based checks for format validity, length bounds, and refusal detection; a lightweight LLM-as-judge against rubrics; or comparison to reference outputs for a fixed regression-test prompt set replayed on a schedule.

Track them as time series with the same alerting discipline as latency (M4). A slow quality decay — from an unnoticed upstream data or prompt-template change, not just a bad deploy — needs burn-rate-style detection, because it never appears as a spike. This is the direct link to drift detection (Ch. 10): a quality-regression signal is often the first observable symptom of drift, well before a statistical test flags it.

Model Versioning & Registry (Ch. 09)

Maps to Chapter 09 — Model Versioning. Q19 is preserved from the original System Design section — versioning is really its own topic, not a system-design scenario.

Q19: How would you implement model versioning and rollback?

Version semantically — v1.0.0, v1.1.0, v2.0.0 — deciding explicitly what counts as major (architecture or behavior change), minor (fine-tune), and patch (config or quantization only). Store artifacts in a registry with metadata: training date, eval metrics, dataset version, base model lineage, and who approved promotion.

1. Model registry layout:

models/
  v1.0.0/
    model.bin
    tokenizer.json
    metadata.json
  v1.1.0/
    model.bin
    tokenizer.json
    metadata.json

2. Deployment:

env:
- name: MODEL_VERSION
  value: "v1.0.0"
- name: MODEL_PATH
  value: "/models/v1.0.0"

3. Rollback:

# Update to previous version
kubectl set env deployment/llm-serving \
  MODEL_VERSION=v0.9.0

# Or use canary deployment to gradually
# route traffic back to the stable version

4. API:

GET /api/v1/models/versions
POST /api/v1/models/rollback
{
  "target_version": "v1.0.0"
}

Validate offline before a version is canary-eligible, roll out gradually via canary (Ch. 07) rather than cutting over at once, track per-version performance continuously, and keep a changelog recording what data and eval each version was validated against.

MV1: What actually belongs in a model registry’s metadata, beyond “the weights file”?

Lineage: base model, fine-tuning or adapter data, training config, and the exact commit or config hash that produced the artifact. Evaluation results: the specific benchmark scores this version achieved, so any two versions are comparable on the same axes. Serving compatibility: which engine versions and quantization formats it is validated against — a checkpoint that behaves well in FP16 on vLLM 0.6.x is not automatically identical under a different engine or quantization.

Approval and promotion metadata: who approved it for canary or production, when, and against what criteria — this is what makes rollback and audit possible after the fact. Content hash or checksum, to catch silent corruption or a wrong-file upload before it reaches a serving pod.

MV2: Should model artifacts be treated as immutable once published, and what does that discipline actually buy you?

Yes. Once v1.2.0 is published it is never overwritten in place; a fix gets v1.2.1.

Immutability is what makes rollback trustworthy — “roll back to v1.1.0” only means something if v1.1.0 today is bit-for-bit what was validated and ran in production before. It also makes post-hoc debugging possible: with mutable artifacts, “which model produced this bad output six weeks ago” is unanswerable. Implement it with content-addressed storage or write-once buckets with versioned object keys, plus a registry API that rejects overwriting an existing tag.

MV3: How do you decouple “code version” (serving engine/app) from “model version” (weights) in your deployment pipeline?

Treat serving image version and model version as independent axes with their own cadences: a fine-tune should not require an application rebuild, and an engine upgrade should not require touching the model artifact.

The mechanism: the serving image is generic and loads whatever MODEL_PATH/MODEL_VERSION it is told at startup, with model version injected by config — env var, ConfigMap, or a model-serving CRD. That is exactly why runtime weight-mounting beats baking weights in (D5). It is also what makes independent canarying possible: canary a new model on the same engine, or a new engine on the same weights (CD5), instead of changing both at once and losing attribution.

MV4: Design the rollback SLA — how fast can/should a rollback actually happen, and what determines that floor?

The fastest rollback is traffic-routing only — flipping a canary weight back to 0% — and that is near-instant, seconds, if the stable version’s pods are still warm. Keeping the previous version’s pods alive for a fast-rollback window after promotion, rather than scaling them down immediately, is a deliberate cost trade-off.

If those pods are already gone, rollback requires re-provisioning capacity, so rollback time equals cold-start time (K7) — minutes, not seconds. That is the common gap between “we have rollback capability”, true, and “our rollback SLA is under a minute”, false unless you kept the old version warm. So the decision to communicate is the retention window: keep N-1 warm for X hours after promotion, sized against risk tolerance and budget.

MV5: How do you version and roll back prompts/system prompts and retrieval configuration, not just model weights, given that both affect output just as much?

Treat prompt templates, system prompts, and RAG configs — index version, chunking parameters, retrieval top-k — as versioned artifacts with the same discipline as weights. A “model regression” investigation that checks weight version and ignores a same-day prompt-template change will miss the actual cause.

Store prompt and config versions alongside model version in registry metadata, so a production request maps to an exact (model, prompt template, retrieval config) triple. This matters for canary too: a system-prompt change deserves the same rigor as a model swap, and the common trap is treating it as “just a config tweak” that skips the deployment safety pipeline.

MV6: What does “shadow deployment” buy you specifically for model version validation that offline eval doesn’t?

Offline eval runs against a fixed historical dataset. It cannot tell you how the new version behaves on today’s live traffic distribution, which may already have shifted away from what the eval set represents.

Shadow deployment runs the new version against live traffic in parallel with production, discarding its output but logging it for comparison against the production version’s actual response on the same input. That catches distribution-shift-sensitive regressions offline eval structurally cannot see. The cost is doubled compute on the shadowed slice, so it is sampled and time-boxed before moving to a real traffic-splitting canary.

MV7: Interviewer pushback: “Why do you need a formal model registry at all — why not just tag Docker images with the model baked in and use normal image-based deployment tooling?”

Baking weights into images couples model release cadence to image build, push, and pull cost, which is wasteful when fine-tunes are far more frequent than code changes — and wasteful in reverse too. A registry also carries model-specific metadata — eval scores, lineage, approval status — that a generic container registry has no concept of, so you end up building a parallel ad hoc tracker anyway, without the rollback API and promotion gates.

That said, it is a genuine trade-off. For a small team with infrequent updates and no multi-model complexity, baking weights into a versioned image is legitimate and simpler. The signal is recognizing when a registry earns its keep — frequent updates, multiple models, fast independent rollback — versus when it is over-engineering.

Drift Detection (Ch. 10)

Maps to Chapter 10 — Drift Detection (drift_detector.py). Q15 is preserved from the original Monitoring section — drift detection is really its own chapter/topic.

Q15: How would you detect model drift in production?

Three kinds. Data drift: the input distribution changes — users ask about topics or formats the model rarely saw. Concept drift: the input-output relationship changes — what counts as a correct answer shifts after a real-world event or policy change. Prediction drift: the output distribution changes — responses get longer, shorter, or differently structured.

Detection uses statistical tests — PSI (Population Stability Index) comparing reference against current, Kolmogorov-Smirnov for distribution differences, chi-square for categorical — surfaced through Evidently AI, custom Prometheus drift-score metrics, or direct comparison scripts.

from evidently import Report, DataDriftTable

report = Report(metrics=[DataDriftTable()])
report.run(
    reference_data=train_data,
    current_data=production_data
)

if report.get_metric(DataDriftTable()).drift_detected:
    alert("Data drift detected!")

Set thresholds such as PSI > 0.2, monitor continuously rather than only at deploy time, then act: retrain or refresh if drift is significant and persistent, investigate why before reacting, and update the baseline if the shift is a legitimate new normal.

DR1: For an LLM specifically (as opposed to a classic tabular ML model), what does “input drift” actually mean, and how do you measure it over free-form text?

You cannot run PSI or KS directly on raw text. You need a numeric representation first: prompt length distribution, embedding-space distance (embed prompts and track the centroid or distribution shift against a reference window), topic or intent classification distribution, or simple proxies such as detected language and keyword presence.

Embedding-based drift — Maximum Mean Discrepancy, or simply centroid distance over time — is the most common practical approach for free-form text, because it captures semantic shift without hand-built categorical features. The operational warning: embedding drift detectors need maintenance, because changing your embedding model invalidates the baseline and forces re-establishing it.

DR2: What is “prediction drift” for a generative model, and why is it a different problem from input drift?

Prediction drift means the output distribution shifts — average response length, refusal rate, sentiment or tone, or the rate of a failure mode like repetition loops or format violations — even while inputs are stable.

It can happen with zero code or model change: an upstream prompt-template tweak, worse retrieved-context quality for RAG, or a subtle change in default sampling parameters. It matters separately because input drift says the world or the users changed, while prediction drift says the system’s behavior changed — and a system can have a completely stable input distribution while output behavior silently regresses. That is often the first symptom of a bug.

DR3: How would you set up automated drift-triggered retraining/refresh, and what are the dangers of doing this fully automatically?

Pipeline shape: a drift monitor scores on a schedule, the score crosses a threshold, that triggers retraining or fine-tuning on fresh data, and the new version goes through the normal eval and canary pipeline — never straight to production.

Danger one is feedback loops: retraining on production outputs, or on user reactions to them, without careful filtering bakes in and amplifies the drift that caused the trigger — the model drifts toward sycophancy, retrains on sycophantic outputs users engaged with more, and drifts further. Danger two is false-positive churn: an over-sensitive threshold retrains constantly on noise, burning compute and introducing instability, since each new version is itself a source of risk.

So drift detection should trigger an alert and investigation, with retraining as a human-gated decision. Full automation is a reasonable long-term goal only once eval, canary, and rollback are strong enough that a bad automated retrain cannot reach users.

DR4: What’s the difference between monitoring for drift and monitoring for outright model degradation/failure (e.g., repetition loops, refusals, garbage output)?

Drift is a distributional concept — comparing distributions over time against a reference — and can be entirely benign, because the world legitimately changed, and it is usually gradual.

Degradation detection catches acute, often binary bad behavior on individual outputs: repetition loops, empty outputs, malformed JSON when structured output was requested, a refusal on a benign request. That needs per-request rule-based checks, not distributional statistics.

They complement each other. Drift catches slow systemic shifts that per-request checks never notice, because each output looks fine while the aggregate moved. Per-request detection catches acute breakage that a distributional test averaged over thousands of requests would dilute into invisibility.

DR5: How do you build a drift baseline/reference distribution in the first place, and how often should you refresh it?

Build the reference from a recent, validated window of production traffic — or the training distribution if traffic has not started — not an arbitrarily old snapshot. The reference must represent a period you are confident was healthy.

Refresh cadence is a two-sided trade-off. Refresh too often and you normalize away real slow drift, because comparing yesterday to today essentially never shows drift and gradual multi-week shifts stay hidden. Refresh too rarely and normal seasonal changes get flagged as drift forever. The common pattern: hold a fixed reference window — “traffic from the last validated release” — until a human explicitly promotes a new one, typically alongside a version bump. Refresh is a deliberate reviewed action, not a rolling automatic window, for the same reason immutable model versions matter (MV2).

DR6: How does drift detection interact with multi-tenant serving, where different tenants have very different, legitimately-different traffic patterns?

A single global baseline shows “drift” constantly, simply because tenant mix shifts — one customer’s usage differs from another’s and relative volumes change week to week. At the aggregate level that is noise, not signal.

Better: maintain per-tenant or per-segment baselines and drift scores, and escalate globally only when drift appears within a tenant’s own traffic over time, or when enough tenants show it simultaneously to suggest a shared cause such as a model or prompt-template change. This is directly analogous to why per-tenant cost and latency dashboards matter (M5): aggregates over a heterogeneous population hide exactly the signals you need.

DR7: Interviewer pushback: “PSI and KS tests are from classical tabular ML monitoring — do they even make sense for LLMs, or is this cargo-culting a technique from a different problem?”

Fair pushback, and the honest answer is “partially”. PSI and KS suit comparing distributions of derived numeric features — prompt length, embedding-distance scores, response length, latency — and remain genuinely useful there. They are not applicable to raw text without that extraction step.

What is LLM-specific and not borrowed: output-quality proxies such as guardrail trigger rate, refusal rate, and LLM-as-judge scores on sampled outputs. Those have no analogue in tabular drift detection and are arguably the more important signal for generative systems.

The best answer combines both — classical tests on derived numeric features, cheap and good for continuous monitoring, plus quality-eval sampling, more expensive but the ground-truth signal the statistical tests only proxy for.

Triton Inference Server (Ch. 11)

Maps to Chapter 11 — Triton.

T1: What problem does Triton Inference Server solve that a hand-rolled FastAPI + PyTorch server doesn’t?

Multi-framework, multi-model serving from one process: PyTorch, TensorFlow, ONNX, TensorRT, and via the vLLM/TensorRT-LLM backends, LLM engines, all behind one gRPC/HTTP API. Dynamic batching and concurrent model execution built in and configurable per model, without hand-written scheduling logic. Model repository abstraction: models deploy by landing in a directory with a config file, and versioning, loading/unloading, and multi-version serving are server features rather than application code. Production observability out of the box: built-in Prometheus metrics, lifecycle events, per-model and per-version statistics.

The trade-off is more operational surface and a configuration model (config.pbtxt) to learn — worth it with multiple models, frameworks, or versions to manage; arguably overkill for a single model on a single serving path.

T2: Describe the Triton model repository layout and what config.pbtxt controls.

model_repository/
  my_model/
    config.pbtxt
    1/
      model.onnx
    2/
      model.onnx

Each top-level directory is a model name; numbered subdirectories are versions, so Triton serves several versions of the same model simultaneously — directly useful for canary and A/B, with Ch. 07 concepts applying at the Triton layer.

config.pbtxt controls input and output tensor names, shapes, and datatypes; the backend (onnxruntime, pytorch, tensorrt, vllm, or python for custom); batching configuration including max batch size and dynamic batching parameters; instance groups, meaning how many copies run and on which GPUs; and version policy — latest only, all, or a specific set.

T3: Explain Triton’s dynamic batching vs. vLLM’s continuous batching — are these the same idea?

No. Triton dynamic batching accumulates individual requests arriving within a short configurable window into one batch, then runs one forward pass — fitting models with a fixed-shape, single-pass profile: classification, embedding, non-autoregressive models.

Continuous, iteration-level batching is specific to autoregressive generation: it batches individual decode steps, letting requests join and leave mid-generation. Plain dynamic batching fits that badly, because requests finish at wildly different times depending on output length. So when serving LLMs through Triton you use the vLLM or TensorRT-LLM backend, which brings continuous-batching-style scheduling into Triton’s process.

T4: What are Triton’s ensemble models, and when would you use one in an LLM serving pipeline?

An ensemble defines a pipeline of models and steps — preprocessing → embedding → retrieval → generation → postprocessing — as a single logical model that Triton executes as a DAG, handling data hand-off internally. It suits a RAG pipeline where one API call should trigger embed-query, retrieve, and generate, with Triton scheduling and batching each stage instead of the client making three round trips.

The trade-off: per-stage debugging is harder, since the client sees one opaque pipeline, and stage-level independent scaling is more constrained than separate Deployments. Good for tightly coupled, latency-sensitive pipelines; poor when stages have very different scaling and failure characteristics.

T5: How does Triton fit into a Kubernetes deployment, and what does the “instance group” concept map to?

Triton runs as a normal container in a Deployment, typically requesting nvidia.com/gpu like any GPU workload; Kubernetes needs to know nothing Triton-specific.

Instance groups in config.pbtxt control how many copies of a given model Triton runs within one Triton process or pod, and on which GPUs — a Triton-internal concept distinct from and complementary to Kubernetes replica scaling, which runs multiple pods each potentially running several instance groups. So use instance groups to pack models onto the GPUs within a pod, and Kubernetes HPA and replicas to scale the number of pods. Conflating the layers is a common early mistake.

T6: When would you choose Triton + a TensorRT-LLM backend over a plain vLLM deployment for LLM serving?

TensorRT-LLM via Triton is typically the highest raw throughput and lowest latency option on NVIDIA hardware, because it compiles model- and hardware-specific optimized kernels ahead of time via graph fusion and precision-specific kernel selection. The cost is an explicit build step per model + GPU-generation + precision combination, a steeper operational curve, and less flexibility for new or unusual architectures.

vLLM, standalone or via Triton’s vLLM backend, gets a new model serving faster, has a simpler operational model, delivers strong out-of-the-box performance with no compile step, and moves faster on new architecture support.

Rule of thumb: TensorRT-LLM for a small number of stable, high-volume models where the extra investment pays back in throughput and cost; vLLM when iterating quickly across many changing models. The space moves fast — TensorRT-LLM’s build ergonomics keep improving and NVIDIA NIM packages the trade-off as a pre-built container — so flag current capability claims as “verify before quoting”.

T7: How does Triton support A/B testing or canary between model versions natively?

The model repository natively supports multiple numbered versions of the same model name (T2) with a configurable version policy, so two versions can be loaded simultaneously and routed between.

Native Triton does not do traffic-percentage splitting itself — that decision lives in front, in an API gateway, a service mesh, or application logic. Triton’s job is to have both versions loaded and ready to serve whichever is asked for. So the canary mechanics from Ch. 07 apply exactly as described; Triton only changes where “which version is loaded and serving” lives.

T8: What monitoring does Triton expose natively, and how do you wire it into a Prometheus/Grafana stack (Ch. 08)?

A built-in /metrics endpoint in Prometheus format with per-model, per-version statistics: request count, inference duration broken out into queue time versus compute time, GPU utilization and memory, and success/failure counts.

Wire it in like any other Prometheus target — a prometheus.yml scrape config pointed at the Triton pod’s metrics port. Not having to build this instrumentation yourself is one of Triton’s practical advantages. Because it separates queue time from compute time by default, it directly supports the golden-signal breakdown from M2 with no custom work.

T9: Interviewer pushback: “If vLLM alone already gives you continuous batching, PagedAttention, and an OpenAI-compatible API, what does adding Triton in front actually buy you — isn’t it redundant infrastructure?”

For a single-model, single-framework, LLM-only deployment it genuinely can be redundant. Running vLLM’s own OpenAI-compatible server directly is a perfectly reasonable, simpler answer, and it is worth saying so rather than defaulting to “always use Triton”.

Triton earns its keep when you need one unified serving layer across heterogeneous models and frameworks — LLMs via vLLM/TensorRT-LLM, classical models via ONNX/TensorRT, embedding models, all behind one API and ops surface — or when you specifically need ensembles, fine-grained instance-group GPU packing, or the native multi-version model repository. Name the trade-off rather than a universal rule.

Production Best Practices

Q16: What are the key considerations for production LLM serving?

Performance: an explicit SLO such as P95 TTFT < 300ms rather than a generic “fast”; throughput validated by load testing (Ch. 04); autoscaling on demand (Ch. 06). Reliability: liveness, readiness, and startup probes (Q5e); graceful degradation instead of hard failure; circuit breakers against cascade failures; retries with exponential backoff and jitter, bounded so they cannot amplify an overload.

Monitoring: metrics (Ch. 08), structured logging with request and trace correlation, SLO-burn-rate alerting (M4), cross-hop tracing (M6). Security: API keys, OAuth, or mTLS between internal services; rate limiting to protect GPU capacity; input validation (Q5f); secrets in Kubernetes Secrets or a secrets manager, never baked into images. Cost: right-sized instances (Q20), scale-down when idle, efficient models or quantization where quality allows. Model management: versioning (Ch. 09), A/B comparison on real traffic (Ch. 07), fast rollback (MV4), drift detection (Ch. 10).

Q17: How would you handle a sudden spike in traffic?

Immediately: HPA or KEDA scales up (Ch. 06), but that has real latency (AS4). Load-balance across pods, ideally prefix-aware (V6). Queue with a bounded queue, and rate limit to protect the backend — reject cheaply rather than letting everyone degrade.

While it happens, watch pod count and node provisioning progress, latency split TTFT versus inter-token, error and reject rates, and GPU plus KV-cache utilization. If overwhelmed, degrade gracefully — smaller model or capped max_tokens — rate limit with a fast 429 rather than a slow timeout, scale manually if autoscaling is too slow, and add capacity on a spot/on-demand mix.

Prevention is capacity planning from known per-replica capacity (LT7), load testing at and above expected load, properly configured autoscaling with predictive scaling for known patterns (AS5), and circuit breakers. Afterwards, use the burst as capacity-planning input (AS6), resize the warm buffer, raise baseline capacity if the level looks sustained, and update the runbook.

Q20: Explain how you would optimize costs for LLM serving.

Compute — GPU instances — is usually the dominant cost by far, ahead of model and log storage, cross-region network transfer, and observability overhead.

Right-size first, from load-test data (LT7) rather than over-provisioning “just in case”. Then autoscale down during low traffic, scale up predictively for known patterns, and use spot or preemptible instances for non-latency-critical batch work. Quantize (FP8/INT8/INT4 — see V5) or distill where quality allows. Caching is the highest-leverage lever for chat and RAG: prefix and KV-cache reuse (V6), plus response caching where semantically valid. Batch with continuous batching (Q7), because higher GPU utilization is directly lower cost per token — track it as a metric (M5), not just as “faster”. Track cost per request and per token continuously, alert on waste, and commit to reserved capacity for predictable baseline load with spot or on-demand only for burst.

Worked example: 10 GPUs at 50% utilization versus 5 GPUs at 90% utilization via better batching and prefix caching, serving the same traffic. Illustrative savings: ~50% — validate with your own numbers, since exact ratios depend heavily on workload shape.

Additional Quick Questions

Q21: What is the difference between batch size and sequence length?

Batch size is the number of requests or sequences processed together; sequence length is the number of tokens in one request, prompt plus generated so far. Batch size 8 means eight requests processed simultaneously in traditional batching, or concurrently in flight under continuous batching; sequence length 512 means each request runs up to 512 tokens.

They hit memory differently. Larger batch size raises throughput and memory, because more KV cache is resident at once. Longer sequences raise both computation and memory, because KV cache grows linearly with sequence length per sequence.

Q22: How does quantization affect model performance?

FP32 → FP16/BF16 is roughly 2x smaller and faster with minimal accuracy loss. FP16/BF16 → FP8 is roughly 2x smaller again, with meaningful speedup on Hopper and Blackwell-class tensor cores and small workload-dependent accuracy impact. FP16 → INT8 is roughly 2x smaller and faster with small accuracy loss, and may need calibration. INT8 → INT4 is roughly 2x smaller again with larger accuracy loss needing careful per-task validation.

The trade is faster inference, less memory, and lower cost against potential accuracy loss and the need for calibration data and per-task validation — not just perplexity. Use it in production when speed or cost matters and quality is validated as acceptable on your actual eval suite, and for memory-constrained deployments on smaller GPUs or at higher concurrency targets.

Q23: What is the difference between model parallelism and data parallelism?

Data parallelism replicates the same model across GPUs or nodes with different data — different requests, in inference — on each replica. In training that needs gradient sync; in inference it is just “run N independent replicas”, which is what Kubernetes replica scaling does.

Model parallelism splits the model across GPUs — tensor parallelism within a layer, pipeline parallelism across layers (V4) — so each GPU holds part of it. It is for models that do not fit on one GPU, or that need the combined memory bandwidth of several to hit a decode-latency target.

Concretely: data parallel inference is 8 GPUs each running an independent full copy of a 7B model on different requests — 8 replicas. Model parallel inference is 8 GPUs each holding 1/8 of a 70B+ model’s weights, collectively serving one request’s forward pass.

System Design Scenarios

This section replaces the original thin “System Design” section with 5 full worked scenarios. Q18 (the original quick-reference design question) is preserved below as a compact warm-up; use it if you only have two minutes, and use the full scenarios if you have twenty.

Q18: Design a system to serve LLMs at scale (quick reference).

Architecture:

[Load Balancer]
    |
[API Gateway] (Rate limiting, Auth)
    |
[Kubernetes Cluster]
    +-- [LLM Serving Pods] (vLLM)
    +-- [Monitoring] (Prometheus, Grafana)
    +-- [Model Registry] (S3/GCS)

Components: load balancer (traffic distribution, health checks, TLS termination) -> API gateway (auth, rate limiting, routing, versioning) -> serving layer (vLLM, HPA, GPU nodes) -> model storage (registry, versioning, node-local caching) -> monitoring (Prometheus/Grafana, centralized logging, alerting) -> data pipeline (request logging, drift detection, A/B testing).

Scaling strategy: horizontal (more pods via HPA), vertical (bigger GPUs for bigger models), multi-region (geographic distribution).

Key metrics: latency (P50/P95/P99, split TTFT/inter-token), throughput (tokens/sec), error rate, GPU/KV-cache utilization, cost per request.

(The full scenarios below go through the actual reasoning — clarifying questions, trade-offs, and pushback — a senior interviewer expects for any one of these architecture pieces in depth.)

Scenario 1: Design serving for a 70B-parameter model at a defined cost/latency SLO

Prompt as given in an interview: “Design an inference platform to serve a 70B dense model. Target: P95 TTFT under 500ms, P95 end-to-end under 5s for a 500-token response, at the lowest cost per request you can justify.”

Clarifying questions to ask first: expected volume and concurrency, peak and average, because 10 req/s and 10,000 req/s are different architectures; prompt-length distribution, short chat turns versus long RAG contexts, which drives prefill cost and chunked-prefill tuning; whether streaming is required, which affects the TTFT-versus-end-to-end split; whether a hard availability SLA constrains redundancy; and whether “lowest cost” is a preference or a hard number.

Architecture:

                     [ CDN / Edge ]
                          |
                 [ API Gateway / LB ]
                (auth, rate limit, routing)
                          |
        +-----------------+------------------+
        |                                    |
 [ K8s: 70B model pool ]            [ K8s: smaller fallback model pool ]
 (TP=4 or TP=8, per-replica)         (see Scenario 5 for fallback design)
        |
 [ Replica: 4x H100/H200 per pod, NVLink, TP=4 ]
 [ vLLM engine: continuous batching, chunked prefill, FP8 weights+KV ]
        |
 [ Prefix-cache-aware LB in front of replica pool ]
        |
 [ Model registry (S3) + node-local NVMe weight cache ]
        |
 [ Prometheus/Grafana, HPA on queue depth, cluster autoscaler on node pool ]

Tensor parallelism degree is the single most important number here. A 70B model in FP16 needs ~140GB for weights alone, so it does not fit on one 80GB GPU. TP=4 across four 80GB-class NVLink GPUs is a common fit; TP=2 with FP8 weights (~70GB) might fit on two with headroom for KV cache, halving GPU count per replica in exchange for quality risk — validate that with an eval suite before committing.

At 500-token outputs and target concurrency, KV cache is likely the binding constraint on concurrent sequences per replica: size the gpu_memory_utilization fraction accordingly and consider FP8 KV cache to roughly double effective concurrency. Enable chunked prefill so occasional long prompts do not spike inter-token latency, which directly protects P95 TTFT under mixed load. With a shared system prompt or repeated RAG context, prefix caching plus cache-aware routing is the highest-leverage cost and latency lever. Set replica count from load-test-derived per-replica capacity at the SLO (LT7), not a theoretical FLOPs calculation.

Cost levers, ranked: prefix caching (near-free once implemented); FP8 quantization of weights and KV cache (validate quality first); right-sizing TP degree against actual concurrency need; and spot/preemptible capacity for any tier that tolerates interruption, rarely the primary serving path.

Defending against pushback. “Why not 8 GPUs per replica for headroom?” — more GPUs per replica means more idle capacity most of the time and coarser scaling granularity, since each step adds 8 GPUs of cost rather than 4; right-size TP degree and use replica count as the load knob. “Why not cache everything?” — caching helps repeated content, not a genuinely novel prompt; for open-ended generation GPU cost is a floor you reduce, not eliminate. “How do you know the load-test numbers hold?” — they will not exactly, which is why the design carries N+1 headroom, capacity-based autoscaling as a second line, and continuous production latency monitoring.

Scenario 2: Design a multi-tenant inference platform serving several model sizes to different internal teams

Prompt as given in an interview: “Several teams want to use a shared LLM platform: one needs a small 7B model for high-volume simple tasks, one needs a 70B model for complex reasoning, one is experimenting with a new fine-tune weekly. Design the platform.”

Clarifying questions to ask first: whether tenants need hard isolation as a compliance boundary or just fair sharing; whether per-tenant cost attribution is required; how often new models appear, meaning a fixed catalog or a changing one; and whether latency SLOs differ per tenant.

Architecture:

                [ Gateway: auth, per-tenant rate limits, routing by model id ]
                                    |
        +---------------------------+---------------------------+
        |                           |                            |
 [ 7B model pool ]           [ 70B model pool ]         [ Experimental fine-tune pool ]
 (many small replicas,       (few large TP=4 replicas,   (small pool, DRA/MIG-shared GPUs,
  high concurrency,           priority: high)              priority: low, preemptible)
  priority: medium)
        |                           |                            |
        +---------------------------+---------------------------+
                                    |
                    [ Shared GPU node pools, PriorityClasses + taints ]
                    [ Per-tenant cost dashboards (Ch. 08, M5) ]
                    [ Per-model registry entries + independent canary pipelines (Ch. 07/09) ]

Isolation: namespaces per tenant for policy and quota isolation, plus MIG or DRA-managed GPU partitions (K3) where hard isolation matters — the experimental fine-tune must not be able to starve the 7B production pool of GPU memory on a shared node. For the 70B pool, dedicated whole-GPU nodes make more sense, since MIG-slicing a large model’s TP group adds complexity for little benefit.

Priority-based preemption: production pools get a higher PriorityClass than the experimental pool, so under GPU pressure Kubernetes preempts experimental work first, letting the platform run experiments cheaply on spare capacity. Independent scaling and cost: each pool gets its own HPA/KEDA config tuned to its capacity envelope (AS8), and every request and metric is tagged with tenant ID at the gateway so cost-per-tenant dashboards work for chargeback and for catching runaway usage. The weekly-changing model gets its own registry entry and canary pipeline (Ch. 09), independent of the stable models’ cadence.

Defending against pushback. “Why not a dedicated cluster per tenant?” — full dedication maximizes isolation but loses the cost benefit of sharing GPU capacity across tenants with complementary peaks; shared infrastructure with strong logical isolation gets most of the isolation at a fraction of the cost, short of a hard compliance requirement for physical separation. “What if the experimental pool needs more than spare capacity?” — that is what the priority design handles: its requests queue or degrade rather than starving production, and if the workload becomes important it graduates to its own provisioned pool as a deliberate decision.

Scenario 3: Design the rollout pipeline for model updates with zero user-visible downtime

Prompt as given in an interview: “Design how a new model version goes from ‘trained’ to ‘serving 100% of production traffic’ with zero downtime and minimal risk.”

Clarifying questions to ask first: full model swap or incremental fine-tune; acceptable time-to-full-rollout, hours or same-day; whether an offline eval suite exists or must be built; and whether multi-turn conversations are in play, which decides sticky-routing requirements (CD7).

Pipeline:

[New checkpoint] -> [Offline eval suite: benchmarks + regression tests]
       |  (fail -> stop here, never reaches serving)
       v
[Register in model registry, immutable version tag] (Ch. 09)
       |
       v
[Load onto a small pool, SHADOW traffic only] (CD6) -- validate latency/capacity + diff outputs vs. stable, zero user exposure
       |  (fail -> fix, re-shadow; never promote)
       v
[Canary: 1-5% real traffic, sticky by session] (CD1, CD7)
       |
   [Automated analysis window: latency, errors, guardrail-trigger rate,
    quality-eval sampling] (CD2, CD4)
       |
   pass -> step up (5% -> 25% -> 50% -> 100%)   fail -> automatic rollback to 0%
       |
       v
[100% traffic on new version; keep previous version warm for N hours] (MV4, fast-rollback window)
       |
       v
[Scale down previous version after fast-rollback window elapses]

Zero downtime comes from traffic-weight shifting, never a hard cutover. At every stage both versions are fully up and only the routing weight changes, so capacity never dips to serve the swap — in contrast with RollingUpdate’s pod-replacement churn (K2).

Sticky-by-session routing (CD7) is non-negotiable for multi-turn chat; without it, zero downtime at the infra level still produces a broken user experience. Immutable versioning plus a fast-rollback warm window (MV2, MV4) turns “we can roll back” into “we can roll back in seconds”, which matters most when something breaks after full promotion. And automated analysis must include quality signals (CD2), because a pipeline checking only latency and errors will happily promote a model that is healthy and worse.

Defending against pushback. “A lot of infrastructure for a one-line config change.” — the infrastructure cost is paid once; ad hoc rollouts pay a recurring risk cost on every update forever, and for an actively-used product those updates are constant. “What if offline eval misses something?” — that is exactly why there are three stages with different blind spots: offline eval is fast but historical, shadow catches live-distribution issues at zero exposure, canary catches the remainder with bounded exposure and an automatic kill switch.

Scenario 4: Design monitoring and autoscaling for a bursty consumer product

Prompt as given in an interview: “A consumer-facing feature can go viral and spike traffic 10-20x within minutes, then fall back down. Design the monitoring and autoscaling approach.”

Clarifying questions to ask first: the cost tolerance for a warm buffer versus the risk of degrading during a spike; whether bursts have any predictability or are fully organic; and the acceptable degradation mode if capacity is exceeded — slower responses, a smaller model, or hard rejection.

Architecture:

[ Gateway: per-user rate limit, bounded admission queue ]
       |
[ Warm buffer pool: sized for observed P99 burst ratio, always on ] (AS6)
       |
[ Reactive HPA/KEDA on queue depth ] --(fast scale-up policy)--> [ additional replicas ]
       |
[ Cluster autoscaler: pre-provisioned warm node pool ] (skips node-provisioning latency, K7/AS4)
       |
[ Degradation controller: watches queue depth ] --(threshold crossed)--> 
       [ shed to smaller fallback model / cap max_tokens / shed low-priority traffic ] (Scenario 5)
       |
[ Monitoring: real-time dashboards on queue depth, GPU util, TTFT P95/P99,
  reject rate -- alerts on SLO burn rate, not static thresholds ] (M4)

Size the warm buffer off the observed burst ratio, not the average — a deliberate cost-versus-resilience trade communicated explicitly: “we run at 40% baseline utilization specifically to absorb a 10x spike within 60 seconds”. A thin margin plus pure reactive scaling guarantees degraded UX during exactly the highest-visibility traffic moments.

Bounded admission queue with fast, honest rejection: past a threshold, return fast 429s with retry-after rather than accepting requests into a growing queue that times out anyway. Pre-provisioned warm node pool: keep GPU nodes standing ready even without pods placed, so pod scale-up never waits on node provisioning (K7) — often the single biggest lever between detecting a spike and serving it. Layer predictive scaling (AS5) on top for known events, and keep graceful degradation (Scenario 5) as the tested last line.

Defending against pushback. “Isn’t a permanently warm buffer wasted spend?” — partly; it is explicit insurance sized against the business cost of a bad viral moment, which for a consumer product can mean losing exactly the users the moment was meant to bring in. “Why not autoscale hard and fast instead?” — because GPU node provisioning and cold start (K7) cannot react in under a minute in most environments; the buffer compensates for a real operational floor, not a config mistake.

Scenario 5: Design a fallback/degradation strategy when GPU capacity runs low

Prompt as given in an interview: “Your serving fleet is at capacity — demand exceeds what your GPUs can serve within SLO, and more capacity isn’t available immediately (quota limit, spot capacity dried up, etc.). Design the degradation strategy.”

Clarifying questions to ask first: whether a smaller fallback model exists and whether a quality drop is acceptable for some traffic; whether there is user tiering (paid versus free) that should decide who degrades first; and whether a slower response is acceptable or the product needs a hard latency ceiling even at reduced quality.

Architecture:

[ Gateway: tracks real-time capacity signal (queue depth / KV-cache util) ]
       |
       v
   capacity signal crosses threshold?
       |
   no -> normal routing to primary model pool
       |
   yes -> [ Degradation controller ] applies, in order of increasing severity:
       1. Cap max_tokens for new requests (reduces per-request cost/time)
       2. Reduce/disable expensive optional features (long-context mode, tool use, etc.)
       3. Route free/low-priority tier to a smaller fallback model pool
       4. Shed lowest-priority traffic entirely with a clear, fast error + retry-after
       |
       v
[ All degradation actions logged + alerted -- this is an incident, not silent behavior ]
       |
       v
[ Auto-recovery: as capacity signal drops below threshold, un-degrade in reverse order ]

Degrade in graduated steps, not binary up-or-down: capping max_tokens or disabling one expensive feature preserves service for far more users than an all-or-nothing shutoff, and each step sheds load at the smallest quality cost available.

Tiered shedding by priority: protect a paid tier longest and let free tiers absorb degradation first — a pre-agreed, documented business policy, not something improvised mid-incident, because whose requests get shed is a product decision. Fall back to a smaller model rather than pure rejection where possible, with product buy-in on which use cases tolerate a quality drop. Make degradation visible internally: every action fires an alert, because capacity running out is an operational incident worth investigating, not something that should silently self-heal. Auto-recovery is symmetric: un-degrade in reverse order as capacity recovers, with hysteresis so it does not flap at the threshold (AS7).

Defending against pushback. “Isn’t silently serving a worse model dishonest?” — the alternative, everyone getting a slow or failing response, is worse for everyone including users who would have been fine with the smaller model; the fix is making the policy explicit and product-approved, such as surfacing “a faster, lighter response” in the UI. “Why not just have enough capacity?” — because “enough for the worst case at all times” is either prohibitively expensive or physically unavailable; quota limits and regional GPU shortages are real and recurring as of 2025-2026.

2025-2026 Landscape Quiz

A senior interviewer will often probe whether you track the ecosystem or are reciting a 2023 blog post. The facts below were checked against current sources as of August 2026. Exact version numbers and benchmark ratios move fast — anything marked “verify before quoting” should be re-checked against current docs before you cite a specific number in an interview; the durable value here is the shape of the landscape, not the last digit of a version string.

LQ1: What are the main LLM inference engines in production use, and how do they position relative to each other?

vLLM is the dominant open-source general-purpose engine: continuous batching, PagedAttention, broad architecture coverage, an OpenAI-compatible server, and the “V1” engine architecture (rewritten scheduler and execution core, alpha from early 2025, matured through 2025-2026). It is the default for iterating across many model families.

SGLang is a fast-moving alternative with its own scheduler and runtime — RadixAttention for prefix caching — and strong results on structured-generation and high-concurrency workloads; now cited alongside vLLM rather than as a niche option. TensorRT-LLM gives compiled, hardware-specific kernels and is typically the throughput and latency ceiling on NVIDIA GPUs, at the cost of a build step and less architecture flexibility; increasingly consumed via Triton’s backend or inside NVIDIA NIM. Triton Inference Server is the multi-framework serving layer (Ch. 11), hosting those engines behind one ops surface rather than competing as an engine. NVIDIA NIM is prebuilt containerized inference microservices packaging one of these behind a standard OpenAI-compatible API. (Verify current NIM backend choices per model before quoting.)

Version numbers and head-to-head benchmark ratios change roughly monthly: know the names and positioning, cite specifics as “as of my last check”.

LQ2: What’s new about NVIDIA’s current GPU generations relevant to inference (as of 2025-2026)?

Hopper (H100/H200) was widely deployed through 2024-2025; the H200 added significantly more HBM capacity and bandwidth than the H100, which directly helps KV-cache-bound serving. Blackwell (B100/B200, GB200 NVL72) is the current flagship, with major FP4/FP8 tensor-core throughput improvements and NVLink domain scale — GB200 NVL72 links many GPUs into one large NVLink domain.

Blackwell Ultra (B300/GB300) is a mid-cycle refresh with substantially more HBM3e per GPU, reported around the 288GB class. That matters because more per-GPU memory means fewer GPUs per replica for a given model plus KV-cache footprint, improving TP-degree economics. Verify exact specs and availability before quoting — this generation rolled out through late 2025 into 2026 and cloud availability was still expanding.

The trend: each generation’s biggest inference-relevant win is memory capacity and bandwidth plus native low-precision (FP8/FP4) throughput, not raw FLOPs — both attack the two real bottlenecks, KV-cache memory and decode memory bandwidth.

LQ3: What’s Dynamic Resource Allocation (DRA) in Kubernetes, and what’s its GA status?

DRA is the modern Kubernetes API for expressing complex device — especially GPU — allocation requirements: DeviceClass for what devices exist, ResourceClaim/ResourceClaimTemplate for what a pod needs, and CEL-based selection such as “a GPU with more than 40GB memory”. It replaces the device-plugin model’s all-or-nothing integer-count limitation and supports ranked fallback across GPU types, partitioned or shared access feeding MIG or time-slicing, and richer logic than a bare nvidia.com/gpu: N request.

GA status: DRA graduated to General Availability in Kubernetes v1.34, per the official Kubernetes project blog, with structured-parameter features and ecosystem tooling refined in following releases. Verify the exact current minor version and feature maturity before quoting. Interview framing: the old model, still very common in production, and DRA coexist — do not imply the old one is gone.

LQ4: What are the current options for sharing one physical GPU across workloads on Kubernetes, ranked by isolation strength?

(See K3 above for the full comparison table.) Weakest to strongest: time-slicing (software round-robin, no memory isolation) → MPS (shared address space, better concurrent-kernel execution, still no hard memory isolation) → MIG (hardware-partitioned compute and memory slices, fixed profiles) → DRA-orchestrated allocation (flexible expression of any of the above, or whole-device claims, through a unified API).

The 2025-2026 trend consolidates around DRA as the scheduling layer on top of MIG, time-slicing, and MPS as the underlying mechanisms. DRA does not replace MIG; it makes MIG and the other mechanisms easier to request and compose with complex placement logic.

LQ5: What’s the current state of KV-cache offloading / cross-node cache sharing as a serving technique?

Tools like LMCache implement a KV-cache layer that offloads cache to CPU memory or NVMe/remote storage and shares it across multiple vLLM instances and nodes, extending prefix caching (V6) beyond one replica’s GPU memory.

The motivation: prefix caching’s benefit is capped by how much cache fits in one replica’s GPU memory and how well the load balancer routes cache-sharing requests to the same replica. Offloading lets a much larger effective cache — system prompts, long-lived RAG contexts, long conversation histories — be reused across a whole fleet rather than staying replica-local and volatile.

This is an active area as of 2025-2026, with ongoing work standardizing KV-cache transfer and connector interfaces between prefill/decode-disaggregated setups (V8). Treat specific throughput multipliers as workload-dependent claims to verify.

LQ6: What serving-platform / “inference-as-a-service” options exist besides self-hosting, and when would a senior engineer recommend one over self-hosting?

NVIDIA NIM: prebuilt optimized containers per model, self-hosted on your own GPUs with the engine-tuning done. Managed inference endpoints from clouds and model providers, dedicated or serverless: trade control and per-token cost optimization for operational simplicity. Self-hosted on Kubernetes with vLLM/SGLang/TensorRT-LLM/Triton: full control and best cost at scale, but you own everything in every chapter above.

Self-host when scale is large enough that infra cost and control outweigh the engineering investment — a rule of thumb some teams use is once GPU spend is large enough that a percentage point of utilization is a meaningful dollar figure. Use a managed platform when pre-scale, moving fast, or lacking platform-engineering capacity. It is a genuine build-versus-buy trade-off, not a purity contest.

LQ7: Rapid-fire current-fact check — answer, then flag confidence.

GQA is standard across most current major open-weight model families — high confidence, stable fact. FP8 is a standard, well-supported inference precision on current NVIDIA data-center GPU generations — high confidence, stable fact.

Exact current vLLM/SGLang/TensorRT-LLM version numbers and head-to-head benchmark numbers — verify before quoting, changes ~monthly. Kubernetes DRA reached GA in v1.34 — high confidence as of research date, but verify the Kubernetes version your target company actually runs, since a feature going GA does not mean every cluster has upgraded. Specific GPU model availability and pricing, such as which cloud offers which Blackwell Ultra instance family — verify before quoting, changes monthly.

The general trend toward disaggregated prefill/decode serving and KV-cache-sharing infrastructure at the largest scale — high confidence as a direction, but treat any specific implementation’s production-readiness claim as something to verify.

Rapid-Fire Flashcards & Glossary

One-liner Q->A pairs for the night before an interview. Organized by chapter. Skim top-to-bottom; if any answer doesn’t come instantly, jump back to that chapter’s deep-dive section above.

Fundamentals & Ch. 01 (Basic Serving)

Ch. 02 (Docker)

Performance Optimization

Ch. 03 (Kubernetes)

Ch. 04 (Load Testing)

Ch. 05 (vLLM Internals)

Ch. 06 (Autoscaling)

Ch. 07 (Canary Deployments)

Ch. 08 (Monitoring)

Ch. 09 (Model Versioning)

Ch. 10 (Drift Detection)

Ch. 11 (Triton)

Glossary

Traps & How to Recover

Common wrong answers/misconceptions that sound plausible but signal shallow experience to a senior interviewer — and the reframe that recovers the answer.

Trap 1: “We’d just add more GPUs to fix latency/throughput problems.”

Why it’s wrong: treats hardware as the first lever instead of the last one. A senior interviewer hears this as “hasn’t actually diagnosed a real bottleneck before.”

Say it. “First I’d check GPU/KV-cache utilization and batching configuration — low utilization under load means a scheduling/batching problem, not a hardware problem. I’d only add GPUs after confirming the engine is already using the ones it has efficiently.”

Trap 2: “vLLM is always faster than everything else, so just use vLLM.”

Why it’s wrong: treats engine choice as a fixed ranking instead of a workload-dependent trade-off (TensorRT-LLM often wins on raw throughput for stable, high-volume models; Triton wins for multi-framework needs; SGLang is a real, current alternative).

Say it. “vLLM is my default for iteration speed and broad model support, but for a small number of stable, extremely high-volume models I’d benchmark TensorRT-LLM, and if we’re serving heterogeneous frameworks I’d put Triton in front regardless of engine choice.”

Trap 3: “We’ll just use HPA on CPU utilization like any other service.”

Why it’s wrong: CPU utilization is decoupled from GPU/KV-cache load for LLM serving pods (AS1) — this is one of the fastest ways to reveal you haven’t actually run GPU workloads in Kubernetes.

Say it. “For GPU-bound serving I’d scale on a custom metric — queue depth or GPU/KV-cache utilization via the Prometheus Adapter or KEDA — CPU utilization on these pods is nearly meaningless.”

Trap 4: “Scale to zero when idle to save cost.”

Why it’s wrong: ignores GPU cold-start reality (minutes, not seconds) — the first request after scale-to-zero will badly violate almost any latency SLO.

Say it. “For latency-sensitive paths I’d keep a warm minimum and rely on predictive/scheduled scaling for known low-traffic windows instead — true scale-to-zero only for genuinely latency-insensitive batch workloads.”

Trap 5: “Canary just means routing 10% of traffic to the new version and watching error rate.”

Why it’s wrong: misses that LLM canaries need output-quality signals (CD2) — a model can be infra-healthy and behaviorally regressed simultaneously, which pure error-rate/latency monitoring won’t catch.

Say it. “Infra metrics are necessary but not sufficient — I’d add an automated quality-eval sampling step on canary outputs (guardrail-trigger rate, structured-output validity, an LLM-as-judge score) before promoting.”

Trap 6: “Just quantize to INT4 everywhere for cost savings — precision doesn’t really matter for chat.”

Why it’s wrong: asserts a blanket accuracy claim without task-specific validation; INT4 accuracy risk varies a lot by task (reasoning-heavy tasks degrade more than casual chat), and “doesn’t matter” is exactly the unvalidated assumption that gets someone burned in production.

Say it. “I’d start from FP8 as the likely sweet spot on current hardware, and only push to INT4 after validating on our actual eval suite for our actual task mix — not assume it’s fine.”

Trap 7: “Model drift means we should just retrain automatically whenever drift is detected.”

Why it’s wrong: ignores feedback-loop risk (DR3) — automatic retraining on drifted/production data without human review can amplify the very drift it’s reacting to.

Say it. “I’d have drift detection trigger an alert and investigation, with retraining as a human-gated decision informed by that investigation — not a fully automatic pipeline straight to production.”

Trap 8: “Kubernetes handles GPU sharing out of the box, just request nvidia.com/gpu: 0.5.”

Why it’s wrong: factually incorrect — GPUs are an integer-only extended resource without MIG/time-slicing/MPS/DRA configured; a fractional request simply won’t work (K1, K3).

Say it. “GPU requests are whole-unit by default — to share one GPU across pods I’d configure MIG for hard isolation, or time-slicing/MPS for looser sharing, potentially orchestrated via DRA.”

Trap 9: “We don’t need a model registry, we’ll just tag Docker images with the model version.”

Why it’s wrong: conflates code-release cadence with model-release cadence, and loses model-specific metadata (eval scores, lineage, approval status) a registry provides natively (MV7).

Say it. “For infrequent updates on one model, tagged images might be enough — but once we have frequent fine-tunes or multiple models needing independent rollback, I’d want an actual registry decoupled from the application image.”

Trap 10: “PSI/KL-divergence tests are all we need for drift detection on an LLM.”

Why it’s wrong: these classical statistical tests need a numeric feature to compare and say nothing about output quality directly (DR7) — a real blind spot for generative systems.

Say it. “Statistical tests on derived features (embeddings, length) are useful for continuous monitoring, but I’d pair them with output-quality-eval sampling — that’s the ground-truth signal the statistical tests are only ever a proxy for.”

Trap 11: “Just cache all the responses to save GPU cost.”

Why it’s wrong: treats response caching as a general LLM cost solution when it only helps for repeated/identical content — most production traffic (unique user prompts) won’t hit a response cache at all; the real high-leverage cache is prefix/KV-cache reuse of shared context (V6), not full-response caching.

Say it. “Full-response caching only helps for genuinely repeated queries. The bigger lever is prefix caching — reusing KV cache for shared system prompts or RAG context — which helps a much larger share of realistic traffic.”

Trap 12: “Zero downtime means using Kubernetes’ rolling update strategy for model version swaps.”

Why it’s wrong: conflates a Deployment-level rolling update (for code/image changes) with a canary/traffic-weighted rollout (for model version changes, Scenario 3) — a rolling update replaces pods, it doesn’t give you a controlled, metric-gated traffic ramp or an instant rollback via traffic-weight change.

Say it. “For a model version change specifically, I’d use a canary pipeline with traffic-weight shifting rather than a pod-replacement rolling update — that gives instant rollback and gradual, metric-gated exposure, which a rolling update doesn’t.”

Trap 13: “More replicas is always better for availability.”

Why it’s wrong: ignores that each GPU replica is expensive and that availability comes from the right redundancy (spread across zones/nodes, N+1 for failure tolerance) not raw count — over-provisioning replicas “for availability” without a specific failure scenario in mind is just wasted spend.

Say it. “I’d size replica count from load-test-derived capacity plus explicit failure-tolerance headroom (N+1, spread across zones) — not an arbitrary ‘more is safer’ buffer.”

Trap 14: “Streaming and batching are in tension, so pick one.”

Why it’s wrong: conflates continuous batching (an engine-internal scheduling technique) with streaming (a client-facing response-delivery mechanism) — they’re not in tension; every major serving engine streams tokens from a continuously-batched decode loop simultaneously for many requests.

Say it. “They operate at different layers — continuous batching is how the engine schedules GPU work across concurrent requests; streaming is how each request’s tokens are delivered to its client as they’re produced. Production systems do both together.”

Red Flags vs. Green Flags — Master Table

Tips for Interviews

1. Be specific: use numbers and examples.
2. Show trade-offs: understand pros/cons, and say them out loud even when not asked.
3. Think system-wide: consider all components, not just the one the question named.
4. Ask clarifying questions: understand requirements (scale, SLO, budget) before designing.
5. Draw diagrams: visualize architecture — even a rough ASCII sketch on a whiteboard shows structured thinking.
6. Discuss monitoring: always mention observability — a design without it is incomplete to a senior interviewer.
7. Talk about failures: how to handle edge cases, degradation, and rollback, not just the happy path.
8. Flag what’s fast-moving: for current tool/version/benchmark facts, it’s a stronger answer to say “verify the exact number, but the shape is X” than to assert a specific figure with false confidence.
9. Name the trade-off explicitly, don’t just pick a side: “I’d default to X, but Y is the better choice if Z” reads as more senior than a flat “always use X.”

Resources

• This repository’s chapter READMEs and deep-dive docs (01_basic_serving through 11_triton).
• vLLM documentation and blog — engine internals, release notes, and architecture posts (e.g., the V1 engine and “Inside vLLM” posts).
• Kubernetes documentation — see the GPU/device-plugin and Dynamic Resource Allocation pages for current GPU scheduling capabilities.
• NVIDIA Triton Inference Server documentation and GitHub repo.
• NVIDIA developer documentation on NIM and the GPU Operator/MIG/DRA integration docs.
• Prometheus/Grafana guides — for the golden-signals and SLO-burn-rate alerting patterns referenced throughout Ch. 08.
• Evidently AI documentation — drift detection tooling referenced in Ch. 10.
• LMCache documentation — KV-cache offloading/sharing referenced in the Landscape Quiz.
• Argo Rollouts / Flagger documentation — automated canary analysis and promotion referenced in Ch. 07.

A note on currency: the 2025-2026 Landscape Quiz section captures the state of the ecosystem as researched in August 2026. Inference engines, GPU generations, and Kubernetes GPU-scheduling features move fast — before an interview, spend 15 minutes checking the current release notes of whichever engine/platform the job description mentions by name.

Good luck with your interviews!

Exercise Solutions

This document provides solutions to exercises in each learning topic.

Topic 1: Basic Model Serving

Exercise 1: Change the Model

Solution:

# In model_loader.py, change:
model = LLMModel(
    model_name="distilgpt2",  # Smaller, faster model
    device="cpu"
)

Exercise 2: Modify Parameters

Solution:

# In app.py, test different temperatures:
# Temperature 0.1 = more deterministic
# Temperature 2.0 = more creative
payload = {
    "prompt": "The future of AI is",
    "temperature": 0.1,  # Try different values
    "max_length": 100
}

Exercise 3: Add Logging

Solution:

import logging
from datetime import datetime

logger = logging.getLogger(__name__)

@app.post("/generate")
async def generate_text(request: TextGenerationRequest):
    start_time = time.time()
    logger.info(f"Request received: prompt='{request.prompt[:50]}...'")
    
    # ... generation code ...
    
    latency = (time.time() - start_time) * 1000
    logger.info(f"Request completed: latency={latency:.2f}ms, tokens={num_tokens}")
    return response

Exercise 4: Error Handling

Solution:

@app.post("/generate")
async def generate_text(request: TextGenerationRequest):
    try:
        if model is None:
            raise HTTPException(status_code=503, detail="Model not loaded")
        
        if len(request.prompt) > 1000:
            raise HTTPException(status_code=400, detail="Prompt too long")
        
        # Generation code...
        
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Unexpected error: {e}", exc_info=True)
        raise HTTPException(status_code=500, detail="Internal server error")

Exercise 5: Batch Requests

Solution:

class BatchGenerationRequest(BaseModel):
    prompts: List[str] = Field(..., min_items=1, max_items=10)

@app.post("/generate_batch")
async def generate_batch(request: BatchGenerationRequest):
    results = []
    for prompt in request.prompts:
        generated = model.generate(prompt=prompt, max_length=50)
        results.append({"prompt": prompt, "generated": generated})
    return {"results": results}

Topic 2: Docker

Exercise 1: Build Basic Image

Solution:

cd 01_basic_serving
docker build -f ../02_docker/Dockerfile.basic -t llm-serving:basic .
docker run -p 8000:8000 llm-serving:basic

Exercise 2: Optimize Image Size

Solution:

# Use multi-stage build
FROM python:3.9-slim as builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

FROM python:3.9-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "app.py"]

Exercise 3: Multi-stage Build

Solution: See 02_docker/Dockerfile.optimized

Exercise 4: GPU Support

Solution: See 02_docker/Dockerfile.gpu

Exercise 5: Docker Compose

Solution: See 02_docker/docker-compose.yml

Topic 3: Kubernetes

Exercise 1: Basic Deployment

Solution:

kubectl apply -f 03_kubernetes/deployment.yaml
kubectl get pods -n llm-serving
kubectl logs -f deployment/llm-serving -n llm-serving

Exercise 2: Health Checks

Solution:

# Already in deployment.yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30

Exercise 3: Resource Limits

Solution:

resources:
  requests:
    memory: "2Gi"
    cpu: "1000m"
  limits:
    memory: "4Gi"
    cpu: "2000m"

Exercise 4: ConfigMap

Solution:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-config
data:
  model_name: "gpt2"
  max_length: "100"
---
env:
- name: MODEL_NAME
  valueFrom:
    configMapKeyRef:
      name: llm-config
      key: model_name

Exercise 5: Scaling

Solution:

kubectl scale deployment llm-serving --replicas=3 -n llm-serving
kubectl get pods -n llm-serving

Exercise 6: Rolling Update

Solution:

# Update image
kubectl set image deployment/llm-serving \
  llm-serving=llm-serving:v2 -n llm-serving

# Watch rollout
kubectl rollout status deployment/llm-serving -n llm-serving

# Rollback if needed
kubectl rollout undo deployment/llm-serving -n llm-serving

Topic 4: Load Testing

Exercise 1: Baseline Test

Solution:

python measure_latency.py --url http://localhost:8000 --requests 10

Exercise 2: Ramp-up Test

Solution:

# In Locust UI, use "Spawn rate" to gradually increase users
# Or use command line:
locust -f locust_test.py \
  --host=http://localhost:8000 \
  --headless \
  --users 50 \
  --spawn-rate 5 \
  --run-time 5m

Exercise 3: Compare Models

Solution:

# Test model A
python measure_latency.py --url http://localhost:8000 --requests 100 > model_a.txt

# Change model, restart server
# Test model B
python measure_latency.py --url http://localhost:8001 --requests 100 > model_b.txt

# Compare results
diff model_a.txt model_b.txt

Exercise 4: Find Bottleneck

Solution:

# Add profiling to app.py
import cProfile
import pstats

@app.post("/generate")
async def generate_text(request: TextGenerationRequest):
    profiler = cProfile.Profile()
    profiler.enable()
    
    # ... generation code ...
    
    profiler.disable()
    stats = pstats.Stats(profiler)
    stats.sort_stats('cumulative')
    stats.print_stats(10)  # Top 10 slowest functions

Exercise 5: Stress Test

Solution:

# Keep increasing users until system breaks
locust -f locust_test.py \
  --host=http://localhost:8000 \
  --headless \
  --users 100 \
  --spawn-rate 10 \
  --run-time 10m

Topic 5: vLLM Serving

Exercise 1: Compare Performance

Solution:

# Test HuggingFace
import time
start = time.time()
# ... HuggingFace generation ...
hf_time = time.time() - start

# Test vLLM
start = time.time()
# ... vLLM generation ...
vllm_time = time.time() - start

print(f"HuggingFace: {hf_time:.2f}s")
print(f"vLLM: {vllm_time:.2f}s")
print(f"Speedup: {hf_time/vllm_time:.2f}x")

Exercise 2: Tune Parameters

Solution:

# Test different GPU memory utilization
for util in [0.7, 0.8, 0.9, 0.95]:
    llm = LLM(
        model="gpt2",
        gpu_memory_utilization=util
    )
    # Measure throughput
    # Find optimal value

Exercise 3: Test Continuous Batching

Solution:

# Send requests at different rates
import asyncio

async def send_requests(rate_per_second=10):
    for i in range(100):
        # Send request
        await asyncio.sleep(1/rate_per_second)
        # Monitor GPU utilization

Exercise 4: Monitor Metrics

Solution:

# Add Prometheus metrics to vLLM server
from prometheus_client import Counter, Histogram

REQUEST_COUNT = Counter('vllm_requests_total', 'Total requests')
REQUEST_DURATION = Histogram('vllm_request_duration_seconds', 'Request duration')

@app.post("/complete")
async def complete(request: CompletionRequest):
    REQUEST_COUNT.inc()
    with REQUEST_DURATION.time():
        # ... generation ...
        pass

Topic 6: Autoscaling

Exercise 1: Basic HPA

Solution:

kubectl apply -f 06_autoscaling/hpa.yaml
kubectl get hpa -n llm-serving
kubectl describe hpa llm-serving-hpa -n llm-serving

Exercise 2: Custom Metrics

Solution:

# Requires Prometheus Adapter
metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "10"

Exercise 3: Scaling Policies

Solution:

behavior:
  scaleUp:
    policies:
    - type: Percent
      value: 50  # Increase by 50%
      periodSeconds: 30
  scaleDown:
    stabilizationWindowSeconds: 300  # Wait 5 min
    policies:
    - type: Percent
      value: 25  # Decrease by 25%

Exercise 4: Load Test

Solution:

# Terminal 1: Watch HPA
kubectl get hpa -w -n llm-serving

# Terminal 2: Generate load
locust -f locust_test.py --host=http://<service-url> --users 50

# Terminal 3: Watch pods
kubectl get pods -w -n llm-serving

Topic 7: Canary Deployments

Exercise 1: Basic Canary

Solution:

kubectl apply -f 07_canary_deployments/canary-deployment.yaml
# Traffic splits 90/10 based on replica count

Exercise 2: Gradual Rollout

Solution:

# Phase 1: 10% (already done)
# Phase 2: 25%
kubectl scale deployment llm-serving-canary --replicas=3 -n llm-serving
kubectl scale deployment llm-serving-stable --replicas=7 -n llm-serving

# Phase 3: 50%
kubectl scale deployment llm-serving-canary --replicas=5 -n llm-serving
kubectl scale deployment llm-serving-stable --replicas=5 -n llm-serving

# Phase 4: 100%
kubectl scale deployment llm-serving-canary --replicas=10 -n llm-serving
kubectl delete deployment llm-serving-stable -n llm-serving

Exercise 3: Monitoring

Solution:

# Compare latency
histogram_quantile(0.95, 
  rate(llm_request_duration_seconds_bucket{version="canary"}[5m])
) / 
histogram_quantile(0.95, 
  rate(llm_request_duration_seconds_bucket{version="stable"}[5m])
)

Exercise 4: Rollback

Solution:

# Scale canary to 0
kubectl scale deployment llm-serving-canary --replicas=0 -n llm-serving

# Or update service to only route to stable

Topic 8: Monitoring

Exercise 1: Set Up Monitoring

Solution:

cd 08_monitoring
docker-compose up -d
# Access Grafana at http://localhost:3000

Exercise 2: Create Dashboard

Solution:

1. Open Grafana
2. Add Prometheus data source
3. Create new dashboard
4. Add panels with queries from 08_monitoring/README.md

Exercise 3: Set Up Alerts

Solution:

# In Prometheus alerts.yml
groups:
- name: llm_alerts
  rules:
  - alert: HighLatency
    expr: histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m])) > 1
    for: 5m
    annotations:
      summary: "P95 latency > 1s"

Exercise 4: GPU Monitoring

Solution:

# Use prometheus_exporter.py
# GPU metrics are automatically collected if pynvml is installed

Topic 9: Canary Deployments

(Same as Topic 7 - see above)

Topic 10: Model Versioning

Exercise 1: Create Registry

Solution:

# models/registry.py
class ModelRegistry:
    def __init__(self, base_path="models"):
        self.base_path = base_path
    
    def register(self, version, model_path, metadata):
        version_dir = f"{self.base_path}/{version}"
        os.makedirs(version_dir, exist_ok=True)
        # Copy model files
        # Save metadata

Exercise 2: Version Model

Solution:

# Tag model version
mkdir -p models/v1.0.0
cp model.bin models/v1.0.0/
cp tokenizer.json models/v1.0.0/
echo '{"version": "v1.0.0", "created_at": "2024-01-15"}' > models/v1.0.0/metadata.json

Exercise 3: Deploy Version

Solution:

# Update deployment
env:
- name: MODEL_VERSION
  value: "v1.0.0"
- name: MODEL_PATH
  value: "/models/v1.0.0"

Exercise 4: Rollback

Solution:

# Update to previous version
kubectl set env deployment/llm-serving \
  MODEL_VERSION=v0.9.0 \
  -n llm-serving

Topic 11: Drift Detection

Exercise 1: Set Up Evidently

Solution:

pip install evidently pandas

Exercise 2: Detect Data Drift

Solution:

from drift_detector import DriftDetector
import pandas as pd

# Load reference data
reference = pd.read_csv("train_data.csv")

# Initialize detector
detector = DriftDetector(reference_data=reference)

# Load current data
current = pd.read_csv("production_data.csv")

# Detect drift
result = detector.detect_data_drift(current_data=current)
print(result)

Exercise 3: Create Dashboard

Solution:

from evidently.ui.dashboards import Dashboard

report = detector.generate_report(current_data)
dashboard = Dashboard("Drift Monitoring")
dashboard.add_report(report)
dashboard.show()

Exercise 4: Set Up Alerts

Solution:

if result["drift_detected"]:
    send_email_alert("Data drift detected!")
    send_slack_alert("Drift score: {result['drift_score']}")

Topic 12: Triton

Exercise 1: Deploy Model

Solution:

# Create model repository
mkdir -p model_repository/gpt2/1
# Copy model files
cp model.pt model_repository/gpt2/1/

# Start Triton
docker run --gpus all \
  -v $(pwd)/model_repository:/models \
  nvcr.io/nvidia/tritonserver:23.10-py3 \
  tritonserver --model-repository=/models

Exercise 2: Multiple Models

Solution:

model_repository/
  gpt2/
    config.pbtxt
    1/model.pt
  distilgpt2/
    config.pbtxt
    1/model.pt

Exercise 3: Dynamic Batching

Solution:

dynamic_batching {
  max_queue_delay_microseconds: 100000
  preferred_batch_size: [ 4, 8, 16 ]
  max_batch_size: 32
}

Exercise 4: Model Ensemble

Solution: See 12_triton/README.md for ensemble configuration

General Tips

1. Test locally first: Always test changes locally before deploying
2. Monitor metrics: Watch metrics during exercises
3. Read errors: Error messages often contain solutions
4. Experiment: Try different values and see what happens
5. Document: Note what works and what doesn’t

Topic 1: Basic LLM Serving

What You’ll Learn

This topic teaches you the fundamentals of serving an LLM:

• How to load a pre-trained model from HuggingFace
• Building a FastAPI endpoint for inference
• Understanding the inference pipeline
• Handling requests and responses
• Basic error handling

Why We Need This

Business Need

Companies need to expose LLM capabilities as APIs to:

• Integrate AI into applications: Chatbots, content generation, code completion
• Serve multiple clients: Web apps, mobile apps, internal tools
• Scale independently: Separate model serving from application logic
• Enable monetization: API-based business models

Technical Need

• Separation of concerns: Model serving separate from application code
• Resource management: Dedicated servers for compute-intensive inference
• Standardization: REST APIs are universal, language-agnostic
• Testing: Easy to test model independently

Real-World Impact

Without proper serving infrastructure:

• ❌ Models can’t be used in production applications
• ❌ No way to integrate AI into existing systems
• ❌ Difficult to scale and maintain
• ❌ Hard to version and update models

Industry Use Cases

1. Customer Support Chatbots

Company: E-commerce, SaaS platforms Use Case:

• Customer asks question → API call → LLM generates response
• Handles 24/7 support, reduces human agent workload

Example Request:

POST /generate
{
  "prompt": "Customer: How do I return an item?\nAssistant:",
  "max_length": 100
}

2. Content Generation

Company: Marketing agencies, content platforms Use Case:

• Generate blog posts, social media content, product descriptions
• API called from CMS or marketing tools

Example Request:

POST /generate
{
  "prompt": "Write a product description for a wireless headphone:",
  "temperature": 0.8,  # More creative
  "max_length": 200
}

3. Code Completion & Assistance

Company: GitHub Copilot, IDEs Use Case:

• Developer types code → API suggests completions
• Real-time code generation in editor

Example Request:

POST /generate
{
  "prompt": "def calculate_total(items):\n    ",
  "max_length": 50,
  "temperature": 0.2  # More deterministic
}

4. Translation Services

Company: Google Translate, DeepL Use Case:

• Translate text between languages
• API integrated into websites, apps

Example Request:

POST /generate
{
  "prompt": "Translate to French: Hello, how are you?",
  "max_length": 50
}

5. Sentiment Analysis

Company: Social media platforms, review sites Use Case:

• Analyze customer reviews, social media posts
• Real-time sentiment detection

Example Request:

POST /generate
{
  "prompt": "Sentiment: This product is amazing!",
  "max_length": 10
}

Industry-Standard Boilerplate Code

Complete FastAPI Serving Application

"""
Industry-standard LLM serving application
Used by: OpenAI API, Anthropic Claude, HuggingFace Inference API
"""

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import time
import logging
from contextlib import asynccontextmanager

# Configure logging (industry standard)
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

# Global model (loaded once at startup)
model = None
tokenizer = None
device = "cuda" if torch.cuda.is_available() else "cpu"


@asynccontextmanager
async def lifespan(app: FastAPI):
    """Load model at startup, cleanup at shutdown"""
    global model, tokenizer
    
    logger.info(f"Loading model on {device}...")
    model_name = "gpt2"  # In production: from environment variable
    
    try:
        tokenizer = AutoTokenizer.from_pretrained(model_name)
        model = AutoModelForCausalLM.from_pretrained(model_name)
        model.to(device)
        model.eval()  # Set to evaluation mode
        logger.info("Model loaded successfully")
    except Exception as e:
        logger.error(f"Failed to load model: {e}")
        raise
    
    yield  # Application runs here
    
    # Cleanup
    model = None
    tokenizer = None
    logger.info("Model unloaded")


app = FastAPI(
    title="LLM Serving API",
    description="Production-ready LLM inference endpoint",
    version="1.0.0",
    lifespan=lifespan
)


# Request/Response Models (Industry standard structure)
class GenerationRequest(BaseModel):
    """Standard request format (similar to OpenAI API)"""
    prompt: str = Field(..., min_length=1, max_length=2000)
    max_tokens: int = Field(50, ge=1, le=500)
    temperature: float = Field(1.0, ge=0.0, le=2.0)
    top_p: float = Field(1.0, ge=0.0, le=1.0)
    stop: list[str] = Field(default_factory=list)


class GenerationResponse(BaseModel):
    """Standard response format"""
    text: str
    model: str
    usage: dict
    latency_ms: float


@app.get("/health")
async def health_check():
    """Health check for load balancers and K8s probes"""
    return {
        "status": "healthy" if model is not None else "unhealthy",
        "model_loaded": model is not None
    }


@app.post("/v1/completions", response_model=GenerationResponse)
async def generate_completion(request: GenerationRequest):
    """
    Main generation endpoint
    Industry standard: /v1/completions (OpenAI-compatible)
    """
    if model is None:
        raise HTTPException(status_code=503, detail="Model not loaded")
    
    start_time = time.time()
    
    try:
        # Tokenize
        inputs = tokenizer(
            request.prompt,
            return_tensors="pt",
            truncation=True,
            max_length=1024
        ).to(device)
        
        # Generate
        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=request.max_tokens,
                temperature=request.temperature,
                top_p=request.top_p,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id
            )
        
        # Decode
        generated_text = tokenizer.decode(
            outputs[0][inputs['input_ids'].shape[1]:],
            skip_special_tokens=True
        )
        
        # Calculate metrics
        latency_ms = (time.time() - start_time) * 1000
        num_tokens = len(outputs[0]) - inputs['input_ids'].shape[1]
        
        logger.info(
            f"Generated {num_tokens} tokens in {latency_ms:.2f}ms"
        )
        
        return GenerationResponse(
            text=generated_text,
            model="gpt2",
            usage={
                "prompt_tokens": inputs['input_ids'].shape[1],
                "completion_tokens": num_tokens,
                "total_tokens": len(outputs[0])
            },
            latency_ms=latency_ms
        )
        
    except Exception as e:
        logger.error(f"Generation error: {e}", exc_info=True)
        raise HTTPException(status_code=500, detail=str(e))


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(
        app,
        host="0.0.0.0",
        port=8000,
        log_level="info"
    )

Usage Example (Industry Standard)

# Client code (how applications use the API)
import requests

def generate_text(prompt: str, max_tokens: int = 50) -> str:
    """
    Call LLM serving API
    Used by: Web applications, mobile apps, microservices
    """
    response = requests.post(
        "http://llm-api:8000/v1/completions",
        json={
            "prompt": prompt,
            "max_tokens": max_tokens,
            "temperature": 0.7
        },
        timeout=30
    )
    response.raise_for_status()
    return response.json()["text"]


# Example: Customer support chatbot
def handle_customer_query(query: str) -> str:
    prompt = f"Customer: {query}\nAssistant:"
    return generate_text(prompt, max_tokens=100)


# Example: Content generation
def generate_product_description(product_name: str) -> str:
    prompt = f"Write a compelling product description for {product_name}:"
    return generate_text(prompt, max_tokens=200, temperature=0.8)

Concepts Explained

Model Loading

When you load a model, you’re:

1. Downloading weights (if not cached) - These are the learned parameters
2. Loading into memory - CPU RAM or GPU VRAM
3. Initializing tokenizer - Converts text ↔ tokens
4. Setting up device - CPU or CUDA (GPU)

Why this matters: Model loading is expensive. You do it once at startup, not per request.

Tokenization

• Text → Tokens: “Hello” → [15496]
• Tokens → Text: [15496] → “Hello”
• Special tokens: <BOS>, <EOS>, <PAD>, <UNK>

Inference Process

1. Tokenize input prompt
2. Run forward pass through model (autoregressive)
3. Sample next token
4. Repeat until max_length or stop token
5. Decode tokens back to text

Autoregressive Generation

LLMs generate one token at a time:

• Input: “The weather is”
• Step 1: Generate “nice”
• Step 2: Generate “today”
• Step 3: Generate “.”
• Output: “The weather is nice today.”

Each step uses the previous tokens as context.

Code Structure

01_basic_serving/
├── README.md           # This file
├── app.py              # FastAPI application
├── model_loader.py     # Model loading logic
├── requirements.txt    # Dependencies
└── test_api.py        # Simple test script

Running the Code

1. Install Dependencies

pip install -r requirements.txt

2. Start the Server

python app.py

3. Test the API

# Health check
curl http://localhost:8000/health

# Generate text
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_length": 50,
    "temperature": 0.7
  }'

4. View API Docs

Open http://localhost:8000/docs in your browser

Key Code Sections

Model Loading (model_loader.py)

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Move to device (CPU or GPU)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

Generation (app.py)

# Tokenize input
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate
outputs = model.generate(
    **inputs,
    max_length=max_length,
    temperature=temperature
)

# Decode
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

Understanding the Output

When you call the API, you get:

• generated_text: The complete generated text
• prompt: Your input (echoed back)
• num_tokens: Number of tokens generated
• latency_ms: Time taken in milliseconds
• model_name: Which model was used

Common Issues

Out of Memory

• Problem: Model too large for available RAM/VRAM
• Solution: Use a smaller model (gpt2-small) or reduce batch size

Slow Inference

• Problem: Running on CPU
• Solution: Use GPU if available, or use a smaller model

Import Errors

• Problem: Missing dependencies
• Solution: pip install -r requirements.txt

Exercises

1. Change the model: Try different HuggingFace models (gpt2, distilgpt2, etc.)
2. Modify parameters: Experiment with temperature, top_p, max_length
3. Add logging: Log every request with timing information
4. Error handling: Add try/except for different error cases
5. Batch requests: Modify to handle multiple prompts at once

Next Steps

Once you understand this, move to:

• Topic 2: Containerization with Docker
• Topic 3: Kubernetes deployment

Further Reading

• HuggingFace Transformers Docs
• FastAPI Documentation
• GPT-2 Paper

Basic LLM Serving — A Deep Dive

Standing up a language model behind an HTTP API from first principles, and understanding why the naive version is slow — then hardening it into something you could actually put in front of traffic, and defending every decision in an interview.

Why this matters

Every production LLM system — ChatGPT, a support bot, a code assistant — is at bottom a loop that turns text into tokens, runs a forward pass, and turns tokens back into text, wrapped in a network server. If you understand that loop end to end, and understand the two costs that dominate it (compute and GPU memory), the rest of this book is just engineering to make the loop cheaper and more concurrent.

This chapter builds the loop the obvious way: one model, one process, one request at a time. That version works, and it is a perfect teaching tool precisely because it is slow. By the end you will be able to say, with numbers, exactly where the time and the memory go — and that motivates batching, KV-cache management, and dedicated engines (vLLM, TGI, Triton) in the chapters that follow.

We then go further than “it works on my laptop”: we add concurrency limits, structured error handling, and health/readiness endpoints, load-test the result, and walk through two real incidents that this kind of naive-but-hardened server either causes or prevents. By the end of this chapter you should be able to build a small serving stack yourself and survive a senior interviewer asking “walk me through what happens when this GPU gets 50 requests per second.”

We keep the intuition first and the mechanism precise. Where there is a tradeoff, we name it honestly.

How to use this chapter. Read “Core intuition” through “Failure modes” in order the first time — it is one continuous argument from the autoregressive loop to why naive serving breaks. “Production case studies,” “The 2025–2026 landscape,” and “Interview mastery” are meant to be revisited independently: before a design review, before an interview, or after an incident, to re-anchor on the mechanism that explains it.

Saying it out loud. So every LLM product you’ve ever used is, underneath, the same small loop: text goes in, gets chopped into tokens, the model runs a forward pass, and a token comes back out — over and over until it decides to stop. Everything that makes serving hard is just that loop being expensive in two specific ways: it burns GPU compute, and it eats GPU memory. If you can say where the time goes and where the bytes go, you can explain batching, KV caching, and why anybody bothers with vLLM. The reason I’d build the naive version first is that it’s slow in a diagnosable way — you can point at the exact millisecond and the exact gigabyte, and that’s what turns “vLLM is faster” into an argument instead of a slogan.

Core intuition: an LLM is an autoregressive next-token loop

A decoder-only transformer computes one thing: given a sequence of tokens, a probability distribution over the next token. Generation is just calling that repeatedly.

prompt: "The capital of France is"
   -> tokenizer -> [464, 3139, 286, 4881, 318]
   -> model -> logits over ~50k vocab -> pick "Paris" (token 6342)
   -> append -> [464, 3139, 286, 4881, 318, 6342]
   -> model -> pick "." -> append -> ...
   -> stop on EOS or max_new_tokens
   -> tokenizer.decode(...) -> " Paris."

Two things follow immediately, and they structure everything:

1. Generation is sequential. Token N+1 depends on token N. You cannot decode a 200-token answer in one shot; you do (at least) 200 forward passes. This is why latency scales with output length.
2. The model re-reads its own context every step — unless you cache. The naive loop re-processes the whole sequence on every token, which is quadratic waste. The fix is the KV cache (below), and the KV cache is what eats your GPU memory.

Hold those two facts. The whole performance story is a consequence of them.

Saying it out loud. The one thing to internalize is that a language model only ever predicts the next token — generation is just calling that in a loop and feeding the output back in. Two consequences fall straight out. First, it’s inherently sequential: a 200-token answer means at least 200 forward passes, so your latency scales with how long the answer is, not how clever the model is. Second, without a cache the model re-reads its entire context on every single step, which is quadratic waste — and the fix for that, the KV cache, is exactly what ends up eating your GPU memory. So the two costs that dominate serving, latency and memory, both fall out of that one sentence about next-token prediction.

Loading a model: weights, dtype, device, tokenizer

Before serving anything you load four things. Each has a failure mode.

Saying it out loud. Loading a model looks like one line of code, but there are really four decisions in it and each one has a way of biting you. You’re picking which weights, at what numeric precision, on what device, with which tokenizer. The precision choice is the biggest single memory lever you’ll ever pull — BF16 instead of FP32 literally halves your weight footprint. And the tokenizer is the sneaky one, because a mismatched tokenizer or a skipped chat template doesn’t throw an error, it just quietly produces fluent garbage. The failure mode I’d name is forgetting torch_dtype: you silently load in FP32, double your weights, and OOM on a model that would have fit fine.

Weights and where they live

A model is a set of tensors (the parameters) plus a config describing the architecture. With Hugging Face Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    torch_dtype="bfloat16",   # precision — see below
    device_map="cuda",        # placement
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

The weights download once to a local cache and are memory-mapped from safetensors shards on subsequent loads. Cold start = download + load + allocate; warm start = load + allocate. This distinction matters for autoscaling (Chapter 6): a cold pod may take minutes.

safetensors is not an incidental detail — it is the format almost every serious model ships in today, and it is worth knowing why (we return to this with dates and sources in “The 2025–2026 landscape” below): the older PyTorch default, pickle (.bin/.pt checkpoints), can execute arbitrary code on load because pickle.load deserializes by calling constructors named in the file — a well-known supply-chain attack vector for downloaded model weights. safetensors stores only raw tensor bytes plus a JSON header of shapes/dtypes, so loading it can never execute code, and because it is a flat memory-mappable layout, loading is also faster (mmap + lazy paging instead of unpickling). If you see pytorch_model.bin instead of model.safetensors in a repo today, treat it as a legacy artifact.

dtype / precision — the single biggest memory lever

Parameters are stored as floating-point numbers, and the bytes per parameter is a choice you make at load time.

BF16 (bfloat16) is usually preferred over FP16 on modern GPUs: same 2 bytes, but a wider exponent range, so it is less prone to overflow/NaN during the forward pass. FP32 doubles your memory for almost no inference quality gain — do not serve in FP32 by accident (it is the default if you forget torch_dtype).

Saying it out loud. Precision is just how many bytes you spend storing each parameter, and it maps directly to gigabytes on the card. FP32 is four bytes, FP16 and BF16 are two, INT8 is one, INT4 is about a half — so a 7-billion-parameter model is 28 GB, 14 GB, 7 GB, or 3.5 GB depending purely on that one choice. BF16 is the modern default over FP16 because it costs the same two bytes but has a wider exponent range, so you’re far less likely to hit overflow or NaNs mid forward pass. The tradeoff to name: dropping to INT8 or INT4 buys you memory and concurrency, but it’s a quality hit you have to measure on your own eval set — it’s never free, and “quantization is lossless” is the wrong answer.

Device placement

Weights must sit in GPU memory (VRAM) for fast inference. device_map="cuda" puts everything on one GPU; device_map="auto" will shard across multiple GPUs or spill to CPU/disk if the model does not fit — convenient, but CPU offload is catastrophically slow for serving. For a serving path you want the whole model resident on the GPU and you want to know it fits (memory math below).

Tokenizer — small, and a classic source of silent bugs

The tokenizer maps text <-> integer IDs. It must be the exact one the model was trained with; a mismatch produces garbage output with no error. Two things to get right in a server:

• Chat template. Instruct/chat models expect a specific formatting of roles (<|user|>, <|assistant|>, etc.). Use tokenizer.apply_chat_template(messages, add_generation_prompt=True) rather than hand-concatenating strings — getting the special tokens wrong quietly degrades quality.
• Padding side and pad token. For batched generation, decoder-only models must left-pad (tokenizer.padding_side = "left"), and many models ship without a pad_token — set tokenizer.pad_token = tokenizer.eos_token. Right-padding a decoder batch corrupts the generation. (We revisit padding when we build real batching.)

Saying it out loud. The tokenizer is the boring part that causes the scariest bugs, because when it’s wrong nothing crashes — the model just gets nonsense and confidently answers it. Two things I’d check every time. One, use the model’s own chat template rather than gluing role strings together by hand, because instruct models are trained on very specific special tokens and getting them subtly wrong degrades quality invisibly. Two, for batched generation on a decoder-only model you must left-pad, and lots of models ship without a pad token so you set it to the EOS token. The named failure mode is right-padding a decoder batch: no error, no warning, just quietly corrupted output for every request in the batch.

Mechanism in depth: prefill vs decode, and the KV cache

This is the heart of the chapter. A single generation request has two phases with completely different performance characteristics.

Saying it out loud. If there’s one thing worth saying unprompted in an interview, it’s that a request has two phases with completely different bottlenecks, not one. Prefill runs your whole prompt through the model in a single parallel pass — that’s compute-bound, and it’s what sets time-to-first-token. Then decode generates one token at a time, and each of those tiny steps still has to drag the entire model weights and the whole growing KV cache across GPU memory, so decode is memory-bandwidth-bound. That asymmetry is the reason your GPU can sit at low utilization while your latency is terrible, and it’s why the fix is batching rather than a bigger card.

Prefill (the prompt pass)

You feed the entire prompt (say 500 tokens) through the model in one forward pass. Because all prompt tokens are known up front, they are processed in parallel — the GPU does one big matrix-multiply-heavy pass over all 500 positions at once. This is compute-bound: it saturates the GPU’s arithmetic units. Prefill is what you pay for time-to-first-token (TTFT), and its cost grows with prompt length.

During prefill the model computes, for every layer and every attention head, a key (K) and value (V) vector for each prompt token, and stores them — that is the KV cache.

Decode (the generation loop)

Now you generate one token at a time. Each decode step feeds only the single newest token through the model. Its attention needs the K/V of all previous tokens — but those are already in the cache, so you do not recompute them. Each decode step is therefore tiny in arithmetic (one token’s worth of matmuls) but must read the entire KV cache and all model weights from GPU memory. Decode is memory-bandwidth-bound, not compute-bound: the GPU spends its time moving data, and its expensive tensor cores sit mostly idle.

This is the central asymmetry of LLM serving:

The decode phase being memory-bound and low-utilization is exactly why one-request-at-a-time wastes the GPU, and exactly why batching helps: multiple requests can share the same weight read. Hold that thought for the tradeoff section.

Saying it out loud. Here’s the thing that surprises people: decode does almost no math. Each step only pushes one new token through the model, because every earlier token’s keys and values are already cached — so the arithmetic is trivial, but you still have to read all fourteen-plus gigabytes of weights plus the whole KV cache out of GPU memory to do it. That’s what “memory-bandwidth-bound” means: the tensor cores are idle, waiting on the memory bus. Practically, that’s why a single request wastes an H100 — you’re paying for roughly 989 teraflops of dense BF16 compute and using a sliver of it — and it’s exactly why batching works, because one memory read can serve every request in the batch at once.

Why the KV cache exists

Without a cache, generating token N would re-run attention over all N prior tokens from scratch — an (O(N^2)) blowup over a full sequence. The KV cache trades memory for compute: store each token’s K and V once, reuse them for every future step. It turns per-step attention cost from “re-read and recompute everything” into “read the cache.” The price is GPU memory that grows linearly with every token in every active request — which becomes the binding constraint on how many requests you can serve at once.

Saying it out loud. The KV cache is a straight memory-for-compute trade. Without it, generating token N means re-running attention over all N previous tokens from scratch, which is quadratic over a full sequence and completely wasteful, since those keys and values never change once computed. So you compute each token’s key and value once and keep them around. The catch is that the cache grows linearly with every token in every active request and it only ever grows during a request — so on a typical 7B model it’s about half a megabyte per token, meaning one 4K-context request is roughly 2 GB. That’s why concurrency in LLM serving is almost always capped by memory, not by FLOPs.

The 60-second version (say this in an interview)

If you only remember one paragraph, make it this one — it is the answer to “explain prefill and decode” under time pressure:

“A request has two phases. Prefill processes the whole prompt in one parallel forward pass — it’s compute-bound, and its cost sets time-to-first-token. Decode then generates one token at a time, each step only running the new token through the model because past tokens’ key/value vectors are cached — that’s the KV cache. Decode is memory-bandwidth-bound, not compute-bound, because each tiny step still has to read the full model weights and the whole growing KV cache off the GPU. That’s why decode is inefficient for a single request — the GPU’s compute sits idle waiting on memory — and it’s exactly why batching multiple requests’ decode steps together helps: one memory read now produces tokens for many requests at once. And it’s why KV cache memory, which grows every step, is usually the real limit on concurrency, not raw compute.”

That is roughly 60 seconds spoken aloud and it hits every point an interviewer is listening for: the two phases, their bottlenecks, what each determines (TTFT vs TPOT), why batching works, and why memory — not FLOPs — usually caps you.

Worked example 1: a minimal FastAPI + Transformers server

Here is a complete, correct, single-file server. It is deliberately naive — synchronous generation, one request at a time — so it exposes every pitfall we then discuss. This is the “before” picture for the whole book.

# server.py
#   pip install fastapi "uvicorn[standard]" transformers torch accelerate
#   uvicorn server:app --host 0.0.0.0 --port 8000 --workers 1
import time
from contextlib import asynccontextmanager

import torch
from fastapi import FastAPI
from fastapi.concurrency import run_in_threadpool
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"
STATE = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the model ONCE at startup, not per request.
    tok = AutoTokenizer.from_pretrained(MODEL_ID)
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    tok.padding_side = "left"
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID, torch_dtype=torch.bfloat16, device_map="cuda"
    )
    model.eval()
    STATE["tok"], STATE["model"] = tok, model
    yield
    STATE.clear()


app = FastAPI(lifespan=lifespan)


class GenRequest(BaseModel):
    prompt: str
    max_new_tokens: int = 128
    temperature: float = 0.7
    top_p: float = 0.9
    do_sample: bool = True


@torch.inference_mode()
def _generate(req: GenRequest) -> dict:
    tok, model = STATE["tok"], STATE["model"]
    messages = [{"role": "user", "content": req.prompt}]
    inputs = tok.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    prompt_len = inputs.shape[1]

    t0 = time.perf_counter()
    out = model.generate(
        inputs,
        max_new_tokens=req.max_new_tokens,
        do_sample=req.do_sample,
        temperature=req.temperature,
        top_p=req.top_p,
        pad_token_id=tok.pad_token_id,
    )
    dt = time.perf_counter() - t0

    new_tokens = out[0, prompt_len:]
    text = tok.decode(new_tokens, skip_special_tokens=True)
    n_out = new_tokens.shape[0]
    return {
        "text": text,
        "prompt_tokens": int(prompt_len),
        "output_tokens": int(n_out),
        "latency_s": round(dt, 3),
        "tokens_per_s": round(n_out / dt, 1),
    }


@app.post("/generate")
async def generate(req: GenRequest):
    # Blocking, CPU/GPU-bound work goes to a thread so it does not
    # freeze the async event loop (see pitfalls).
    return await run_in_threadpool(_generate, req)


@app.get("/healthz")
async def healthz():
    return {"ok": "model" in STATE}

Test it:

curl -s localhost:8000/generate \
  -H 'content-type: application/json' \
  -d '{"prompt": "Explain KV cache in one sentence.", "max_new_tokens": 64}'

What this server gets right, and what it deliberately does not:

• Right: model loaded once at startup (not per request); @torch.inference_mode() disables gradient bookkeeping; blocking work offloaded off the event loop; chat template + pad token set correctly; returns real token counts and throughput.
• Deliberately wrong / naive: it serves one request at a time per worker (the model is a shared object and generate holds the GPU), it does not stream tokens (no TTFT benefit for the client), and it does no batching. Two simultaneous callers queue behind each other. That is the motivation for everything after this chapter.

Streaming, when you add it, uses TextIteratorStreamer + a background thread and a FastAPI StreamingResponse, so the client gets the first token as soon as prefill finishes rather than waiting for the whole answer — a large perceived latency win with no throughput change.

Saying it out loud. The minimal server is maybe forty lines and the important thing is what it gets right versus what it deliberately doesn’t. It loads the model once at startup instead of per request, it wraps generation in inference mode so PyTorch isn’t tracking gradients, and — the one people miss — it pushes the blocking generate call off to a thread pool. That last one matters because model.generate() is a long synchronous call, and if you run it directly inside an async handler it freezes the entire event loop, including your own health check endpoint. What it deliberately doesn’t do is batch or stream, so two callers just queue behind each other — and that serial bottleneck is the whole motivation for every chapter after this one.

Generation parameters (what the knobs actually do)

generate is controlled by a GenerationConfig. The ones that matter for serving:

A subtle correctness trap: if a caller passes do_sample=False and a non-default temperature, recent Transformers will warn that the sampling flags are ignored. Decide your server’s contract explicitly rather than passing user knobs through blindly.

A related, increasingly common knob that does not appear in the table above because it is not a GenerationConfig field: constrained / structured decoding, where the server forces every generated token to come from a grammar (a JSON Schema, a regex, a context-free grammar) rather than the raw vocabulary. We cover why this matters and how it works mechanically in “The 2025–2026 landscape” below, because it has become a default expectation for tool-calling and JSON-emitting endpoints, not a niche feature.

Saying it out loud. Most of the sampling knobs are quality dials, but one of them is a cost dial and that’s the one I’d lead with: max_new_tokens is your single biggest latency and money lever, because decode time is basically linear in output length. Temperature and top-p just reshape the probability distribution you sample from, and if you set do_sample=False they’re ignored entirely — you get deterministic greedy output. The stop condition is the quiet danger: a wrong or missing EOS token means the model runs all the way to the cap on every request, burning GPU time and blocking other callers. So the rule is always bound output length, and bound it server-side, because a client’s max_new_tokens should be a request, not a command.

Adding streaming (TTFT the user can feel)

The naive server returns the whole answer at once. Streaming emits tokens as they are decoded, so the client sees output right after prefill:

from threading import Thread
from transformers import TextIteratorStreamer
from fastapi.responses import StreamingResponse

@app.post("/generate/stream")
async def generate_stream(req: GenRequest):
    tok, model = STATE["tok"], STATE["model"]
    inputs = tok.apply_chat_template(
        [{"role": "user", "content": req.prompt}],
        add_generation_prompt=True, return_tensors="pt",
    ).to(model.device)
    streamer = TextIteratorStreamer(tok, skip_prompt=True, skip_special_tokens=True)
    kwargs = dict(inputs=inputs, streamer=streamer,
                  max_new_tokens=req.max_new_tokens, do_sample=req.do_sample,
                  temperature=req.temperature, top_p=req.top_p,
                  pad_token_id=tok.pad_token_id)
    Thread(target=model.generate, kwargs=kwargs).start()  # runs off the event loop

    def emit():
        for piece in streamer:      # yields decoded text as tokens arrive
            yield piece
    return StreamingResponse(emit(), media_type="text/plain")

generate runs in a background thread and pushes tokens into the streamer; the handler yields them to the client. Same total work, dramatically better perceived latency — but note it still occupies the GPU serially. Streaming improves TTFT, not throughput.

Saying it out loud. Streaming means you push each token to the client as it’s decoded instead of waiting for the whole answer, and the honest framing is that it’s a perceived latency win, not a real one. Same total work, same throughput, same moment the last token lands — but the user sees something at, say, 120 milliseconds instead of staring at a spinner for three seconds. Mechanically you run generate on a background thread that pushes tokens into a streamer, and the handler yields them out as a streaming response. The tradeoff to name: streaming improves TTFT and nothing else — it does not free up the GPU, so under load a streaming server serializes exactly as badly as a non-streaming one.

Constrained decoding by hand (what “structured outputs” actually costs you without an engine)

Section A below explains that hosted APIs and dedicated engines now offer schema-guaranteed JSON as a first-class feature, implemented as grammar-constrained decoding: at every decode step, mask out logits for any token that would violate the grammar. It is worth seeing the mechanism on the raw server, because it makes concrete exactly how much an engine is doing for you for free. Here is the simplest possible version — forcing the model to only ever emit digits and a decimal point (a toy “grammar,” but the mechanism generalizes to a full JSON Schema automaton):

import torch
from transformers import LogitsProcessor, LogitsProcessorList

class DigitsOnlyLogitsProcessor(LogitsProcessor):
    # Mask every token whose decoded text contains a character outside
    # "0123456789." -- a minimal stand-in for a compiled JSON-Schema/regex
    # automaton. Real constrained decoding (Outlines, XGrammar) precomputes
    # which token IDs are valid at each automaton state so this mask is a
    # cheap lookup, not a per-step string scan like this toy version.

    def __init__(self, tokenizer, allowed_chars=set("0123456789. ")):
        self.tokenizer = tokenizer
        self.allowed_chars = allowed_chars
        self._valid_ids = None  # lazily computed once, then reused every step

    def _compute_valid_ids(self, vocab_size: int) -> torch.Tensor:
        valid = torch.zeros(vocab_size, dtype=torch.bool)
        for tok_id in range(vocab_size):
            text = self.tokenizer.decode([tok_id])
            if all(c in self.allowed_chars for c in text) and text != "":
                valid[tok_id] = True
        return valid

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
        if self._valid_ids is None:
            self._valid_ids = self._compute_valid_ids(scores.shape[-1]).to(scores.device)
        scores = scores.masked_fill(~self._valid_ids, float("-inf"))
        return scores


# Wire it into generate() as an extra constraint on top of everything else:
processors = LogitsProcessorList([DigitsOnlyLogitsProcessor(tok)])
out = model.generate(inputs, max_new_tokens=16, logits_processor=processors,
                      pad_token_id=tok.pad_token_id)

This toy processor decodes every candidate token’s text on every step to check it, which is far too slow for a real vocabulary of 50k+ tokens at production latency — that per-step cost is exactly the engineering problem XGrammar and outlines solve, by precompiling the grammar into an automaton once and reducing each decode step’s mask computation to following one transition and reading a precomputed bitmask of valid token IDs for the current state, rather than re-deriving validity from scratch. The takeaway to carry into an interview: structured output is not prompt engineering, it is a LogitsProcessor (or the engine’s equivalent) applied every single decode step, and its cost and correctness both hinge on how the grammar-to-token-mask compilation is done.

Saying it out loud. When an API promises you guaranteed valid JSON, that’s not a better prompt — it’s constrained decoding. At every single decode step, the schema gets compiled into a state machine over the vocabulary, and every token that would break the grammar has its logit set to negative infinity before you sample. So schema-valid output is true by construction, not by retrying until the JSON parses. The cost is real, though: you’re computing a valid-token mask on every step, and if you do that naively by decoding all fifty thousand vocabulary entries and string-matching, you’ve just made yourself the bottleneck. That’s exactly the problem XGrammar and Outlines solve — precompile the grammar once, then each step is a state transition and a bitmask lookup.

Build it in practice — extended: from “works on my laptop” to “survives real traffic”

The server above is correct but has three gaps between it and something you would actually deploy: it has no concurrency control (a hundred simultaneous callers all get admitted and race for the GPU, and the process has no idea how loaded it is), no structured error handling (a CUDA OOM or a bad request produces an ugly 500 with a stack trace instead of a machine-readable error the caller can act on), and no readiness signal separate from liveness (a load balancer cannot tell “the process is up” from “the process is ready to take more work”). None of these require an inference engine to fix — they are ordinary backend engineering, and skipping them is exactly what produces the incidents in the war-stories section below.

Saying it out loud. There are three things standing between a correct server and a deployable one, and none of them need an inference engine — they’re just ordinary backend engineering. You need admission control, so a hundred simultaneous callers don’t all get let in to race for a GPU that fits four. You need structured errors, so a caller can tell “retry me later” from “your request is malformed.” And you need liveness and readiness as two separate signals, so a load balancer can stop sending traffic to a busy pod without Kubernetes killing it. Skip these and you get exactly the two incidents at the end of this chapter: a healthy pod restarted by its own health check, and an unbounded OOM that takes down unrelated requests with it.

1. Bound concurrency to what the GPU can actually hold

Do not let every accepted request race for the GPU unbounded. Gate concurrent generations with a semaphore sized from the memory math you will do in Worked Example 2 below — not a guess — and shed load past a bounded queue depth instead of accepting requests you cannot serve in time.

# server.py (extended)
#   pip install fastapi "uvicorn[standard]" transformers torch accelerate httpx
#   uvicorn server:app --host 0.0.0.0 --port 8000 --workers 1
import asyncio
import time
from contextlib import asynccontextmanager
from enum import Enum

import torch
from fastapi import FastAPI
from fastapi.concurrency import run_in_threadpool
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"
# Sized from the KV-cache math in Worked Example 2, NOT a guess: on a 24 GB
# GPU with this model we computed room for ~4 full-length concurrent
# requests before OOM. Leave headroom for shorter, cheaper requests too.
MAX_CONCURRENT_GENERATIONS = 3
MAX_QUEUE_DEPTH = 16          # requests allowed to wait for a GPU slot
REQUEST_TIMEOUT_S = 30.0      # end-to-end deadline, queue + generation
SERVER_MAX_NEW_TOKENS = 512   # hard server-side cap — never trust the client's value alone

STATE: dict = {}


class ErrorCode(str, Enum):
    OVERLOADED = "overloaded"
    TIMEOUT = "timeout"
    OUT_OF_MEMORY = "out_of_memory"
    BAD_REQUEST = "bad_request"
    INTERNAL = "internal_error"


@asynccontextmanager
async def lifespan(app: FastAPI):
    tok = AutoTokenizer.from_pretrained(MODEL_ID)
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    tok.padding_side = "left"
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID, torch_dtype=torch.bfloat16, device_map="cuda"
    )
    model.eval()
    STATE["tok"], STATE["model"] = tok, model
    STATE["gpu_gate"] = asyncio.Semaphore(MAX_CONCURRENT_GENERATIONS)
    STATE["in_flight"] = 0
    STATE["queued"] = 0
    STATE["ready"] = True     # flips readyz on; healthz is independent (below)
    yield
    STATE["ready"] = False
    STATE.clear()


app = FastAPI(lifespan=lifespan)


class GenRequest(BaseModel):
    prompt: str
    max_new_tokens: int = 128
    temperature: float = 0.7
    top_p: float = 0.9
    do_sample: bool = True

Saying it out loud. The rule is that concurrency should be a number you derived, not a number you guessed. You do the memory math — weights plus KV cache per request plus a margin — and that tells you how many generations actually fit; then a semaphore holds you at or below it, and a bounded queue holds a few more waiting. Anything past that gets a fast 503 instead of being admitted into a fight it can’t win. The reason to reject rather than queue forever is that an unbounded queue doesn’t remove overload, it just hides it — it turns a capacity problem into runaway process memory, and the client’s timeout fires anyway. On a 24 GB card running a 7B model in BF16, that number lands around three or four, not thirty.

2. Structured error handling — turn crashes into contracts

A caller that gets a bare 500 with a Python traceback cannot tell “retry me” from “your request is malformed” from “the server is out of capacity forever.” Classify failures and return a small, stable error shape:

def _generate(req: GenRequest) -> dict:
    tok, model = STATE["tok"], STATE["model"]
    n = min(req.max_new_tokens, SERVER_MAX_NEW_TOKENS)  # server has the last word

    messages = [{"role": "user", "content": req.prompt}]
    inputs = tok.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    prompt_len = inputs.shape[1]

    t0 = time.perf_counter()
    with torch.inference_mode():
        out = model.generate(
            inputs, max_new_tokens=n, do_sample=req.do_sample,
            temperature=req.temperature, top_p=req.top_p,
            pad_token_id=tok.pad_token_id,
        )
    dt = time.perf_counter() - t0

    new_tokens = out[0, prompt_len:]
    text = tok.decode(new_tokens, skip_special_tokens=True)
    n_out = new_tokens.shape[0]
    return {
        "text": text, "prompt_tokens": int(prompt_len), "output_tokens": int(n_out),
        "latency_s": round(dt, 3), "tokens_per_s": round(n_out / dt, 1) if dt > 0 else 0.0,
    }


def _generate_safe(req: GenRequest) -> dict:
    """Runs in the threadpool. Classify every failure into a stable error code
    so callers can distinguish 'retry later' from 'fix your request'."""
    try:
        return _generate(req)
    except torch.cuda.OutOfMemoryError:
        # Free whatever we can; do NOT try to keep serving this request.
        torch.cuda.empty_cache()
        return {"_error": ErrorCode.OUT_OF_MEMORY.value,
                "detail": "GPU ran out of memory; retry with a shorter prompt or lower max_new_tokens"}
    except ValueError as e:
        return {"_error": ErrorCode.BAD_REQUEST.value, "detail": str(e)}
    except Exception as e:  # last-resort classification, still structured
        return {"_error": ErrorCode.INTERNAL.value, "detail": str(e)}

Saying it out loud. A bare 500 with a Python traceback tells the caller nothing useful — they can’t tell whether to retry, to fix their input, or to give up. So you classify every failure into a small, stable set of codes and map them onto meaningful HTTP statuses: out-of-memory is different from a bad request, which is different from “we’re at capacity, come back with backoff.” Concretely, you catch torch.cuda.OutOfMemoryError explicitly, free what you can, and return something the client can act on rather than letting it escape as a generic 500. The payoff is that your retry logic, your load balancer, and your alerting all get an honest signal — and the failure mode you’re avoiding is clients hammering retries against an error that will never succeed.

3. The gated, backpressured endpoint

Wire the semaphore, the bounded queue, and the deadline together. The key design decision: a full queue fails fast with 503 rather than growing without bound. An unbounded queue does not remove the overload — it just hides it, converts it into ballooning process memory, and lets client timeouts fire anyway once the queue is long enough. Bounded queues plus fast rejection is a deliberate load-shedding choice, and it is what lets the rest of your system (a load balancer, a retry-with-backoff client, an autoscaler) actually react to the overload signal.

@app.post("/generate")
async def generate(req: GenRequest):
    if STATE["queued"] >= MAX_QUEUE_DEPTH:
        return JSONResponse(status_code=503, content={
            "error": ErrorCode.OVERLOADED.value,
            "detail": "server at capacity, retry with backoff",
        })

    STATE["queued"] += 1
    acquired = False
    try:
        async with asyncio.timeout(REQUEST_TIMEOUT_S):  # Python 3.11+
            await STATE["gpu_gate"].acquire()
            acquired = True
            STATE["queued"] -= 1
            STATE["in_flight"] += 1
            result = await run_in_threadpool(_generate_safe, req)
    except TimeoutError:
        return JSONResponse(status_code=504, content={
            "error": ErrorCode.TIMEOUT.value,
            "detail": f"exceeded {REQUEST_TIMEOUT_S}s waiting for a GPU slot or generating",
        })
    finally:
        if acquired:
            STATE["in_flight"] -= 1
            STATE["gpu_gate"].release()
        else:
            STATE["queued"] -= 1

    if "_error" in result:
        status = 507 if result["_error"] == ErrorCode.OUT_OF_MEMORY.value else \
                  400 if result["_error"] == ErrorCode.BAD_REQUEST.value else 500
        return JSONResponse(status_code=status, content=result)
    return result

Walk through the accounting carefully, because it is easy to get this subtly wrong (and a subtly wrong version is worse than none — it lies about server load): queued is incremented the moment a request is admitted past the queue-depth check, and decremented either when it successfully acquires a GPU slot (acquired = True path) or in the finally block if it times out or errors out before acquiring. in_flight is only ever touched once a slot is actually held, and is always released in finally, so a timeout, an exception inside _generate_safe, or a clean return all leave the semaphore and counters consistent. This is the difference between a load number you can trust for autoscaling and alerting, and one that slowly drifts until a restart papers over it.

Saying it out loud. Backpressure is just being honest about capacity: when the queue is full you say no immediately instead of accepting work you can’t finish. The bookkeeping is where people get it subtly wrong — you increment “queued” the moment you admit a request past the depth check, decrement it either when it grabs a GPU slot or in the finally block if it times out first, and you only ever touch “in flight” once a slot is actually held. Get that wrong and the counters drift upward until a restart papers over it, and now your autoscaler and your dashboards are lying to you. The tradeoff worth naming: a fast 503 looks worse on an error-rate graph than a slow success, but it’s the only thing that gives a retrying client, a load balancer, and an autoscaler something real to react to.

4. Health vs readiness — two different questions

Kubernetes (and any sane load balancer) asks two different questions and expects two different answers. Liveness (“is the process alive enough to not be killed and restarted?”) should be cheap and should not depend on the model or the GPU — otherwise a temporarily saturated but perfectly healthy process gets killed mid-batch, which is its own incident (see war stories). Readiness (“should traffic be routed to this pod right now?”) should reflect real capacity: model loaded, and — ideally — not already saturated.

@app.get("/healthz")
async def healthz():
    # Liveness: process can respond at all. Deliberately does NOT touch the
    # model, the GPU, or the semaphore — must stay cheap even while the GPU
    # is fully busy, or a liveness probe timeout kills a perfectly healthy pod.
    return {"status": "alive"}


@app.get("/readyz")
async def readyz():
    # Readiness: is there a loaded model AND spare capacity worth routing to?
    if not STATE.get("ready", False):
        return JSONResponse(status_code=503, content={"ready": False, "reason": "model not loaded"})
    saturated = STATE["queued"] >= MAX_QUEUE_DEPTH
    body = {
        "ready": not saturated,
        "in_flight": STATE["in_flight"],
        "queued": STATE["queued"],
        "capacity": MAX_CONCURRENT_GENERATIONS,
    }
    return JSONResponse(status_code=200 if not saturated else 503, content=body)

Point the Kubernetes livenessProbe at /healthz with a generous period, and the readinessProbe at /readyz so the Service stops sending new traffic to a saturated pod (letting the autoscaler add capacity) without killing it.

Saying it out loud. These are two genuinely different questions and conflating them causes outages. Liveness asks “is this process broken enough that I should kill and restart it?” — so it must be dirt cheap and must not touch the model or the GPU, because a saturated-but-perfectly-healthy pod that fails its liveness probe gets killed mid-request. Readiness asks “should I route new traffic here right now?” — so that one should reflect real load: model loaded, queue not full. The crucial asymmetry is what happens when each fails: a readiness failure just stops new traffic and lets in-flight work finish, while a liveness failure destroys everything currently running on that pod. If you take one thing: never let your liveness endpoint depend on GPU state.

5. Load-test it — see the bottleneck, don’t just believe in it

A claim like “this server serializes under concurrency” should be a measurement, not folklore. This is a minimal concurrent load test using httpx; it fires N_REQUESTS through CONCURRENCY simultaneous clients and reports the standard percentiles from the metrics section below.

# loadtest.py
#   pip install httpx
import asyncio
import time

import httpx

URL = "http://localhost:8000/generate"
N_REQUESTS = 30
CONCURRENCY = 10


async def one_request(client: httpx.AsyncClient, i: int) -> dict:
    t0 = time.perf_counter()
    r = await client.post(
        URL, json={"prompt": f"Count to five. (req {i})", "max_new_tokens": 64}, timeout=60.0
    )
    return {"status": r.status_code, "latency_s": time.perf_counter() - t0}


async def main():
    sem = asyncio.Semaphore(CONCURRENCY)

    async def bound(client, i):
        async with sem:
            return await one_request(client, i)

    async with httpx.AsyncClient() as client:
        t0 = time.perf_counter()
        results = await asyncio.gather(*(bound(client, i) for i in range(N_REQUESTS)))
        wall = time.perf_counter() - t0

    lat = sorted(r["latency_s"] for r in results)
    ok = sum(r["status"] == 200 for r in results)
    shed = sum(r["status"] == 503 for r in results)

    def pct(p):
        return lat[min(len(lat) - 1, int(p * len(lat)))]

    print(f"requests={N_REQUESTS} concurrency={CONCURRENCY} wall_s={wall:.2f} "
          f"observed_rps={N_REQUESTS / wall:.2f}")
    print(f"ok={ok} shed_503={shed}")
    print(f"p50={pct(0.50):.2f}s  p95={pct(0.95):.2f}s  p99={pct(0.99):.2f}s")


if __name__ == "__main__":
    asyncio.run(main())

Run the server, then in another shell python loadtest.py. With MAX_CONCURRENT_GENERATIONS = 3 and CONCURRENCY = 10, expect to see the serialization directly: p50 will be close to a single request’s latency, but p95/p99 will be roughly (3\times) to (4\times) higher, because the 4th-through-10th concurrent caller queue behind the semaphore for one or more full generation cycles before their own request even starts. If you push N_REQUESTS/CONCURRENCY high enough to exceed MAX_QUEUE_DEPTH, you will start seeing shed_503 > 0 — the bounded queue doing exactly its job instead of the process quietly ballooning. That gap between p50 and p99 is the naive server’s serial bottleneck, measured instead of asserted — and it is the number continuous batching (Chapter 5) exists to close.

Saying it out loud. “This server serializes under load” should be a measurement, not folklore — and it takes about thirty lines to prove. You fire N requests through a fixed number of concurrent clients and report p50, p95, and p99, never the average, because the average is exactly the statistic that hides a starved tail. What you’ll see with a concurrency gate of three and ten simultaneous callers is a p50 near a single request’s latency and a p99 roughly three to four times higher, because callers four through ten are waiting through entire generation cycles before theirs even starts. That gap between p50 and p99 is the serial bottleneck, and closing it is precisely what continuous batching exists to do.

6. Confirm the error contract by hand before trusting the load test

Before trusting the load test’s numbers, confirm each error path returns the structured shape you designed rather than a bare stack trace — this is a five-minute check that catches most of Section B’s wiring mistakes immediately:

# Healthy request -- expect 200 with text/prompt_tokens/output_tokens/latency_s
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/generate \
  -H 'content-type: application/json' \
  -d '{"prompt": "Say hi.", "max_new_tokens": 8}'

# Readiness under normal load -- expect 200 with in_flight/queued/capacity
curl -s localhost:8000/readyz

# Liveness -- expect 200 instantly, even while /generate is busy elsewhere
curl -s localhost:8000/healthz

# Force capacity exhaustion: fire more concurrent requests than
# MAX_CONCURRENT_GENERATIONS + MAX_QUEUE_DEPTH allow, and expect some 503s
# with {"error": "overloaded", ...} rather than hung connections
for i in $(seq 1 40); do
  curl -s -o /dev/null -w '%{http_code} ' localhost:8000/generate \
    -H 'content-type: application/json' \
    -d '{"prompt": "Write a long story.", "max_new_tokens": 512}' &
done; wait; echo

If the overload run above returns only 200s and hung connections instead of a mix of 200 and 503, the bounded-queue accounting in Section B’s /generate handler is not actually shedding load — go back and re-check the queued/in_flight bookkeeping before you trust anything the load test reports.

Worked example 2: GPU memory budget (weights + KV cache + activations)

You cannot reason about serving without the memory math. The GPU must simultaneously hold model weights, the KV cache for every in-flight request, and transient activations. Run out and you get a CUDA OOM — the most common production failure, and the exact one Section B’s MAX_CONCURRENT_GENERATIONS and Section C’s second war story both revolve around.

Saying it out loud. The memory budget is three terms and you should be able to do it on a whiteboard. Weights are parameters times bytes per parameter — roughly two gigabytes per billion parameters in BF16, so a 7B model is about 14 GB. Then KV cache, which is per token, per request, and only grows: about half a megabyte per token on a 7B model, so a 4K-context request costs around 2 GB all by itself. Then a gigabyte or two for activations and CUDA overhead. On a 24 GB card that leaves you about eight gigabytes of headroom, which is four concurrent full-length requests — and that number, not intuition, is where your concurrency limit comes from. The takeaway an interviewer wants: memory caps concurrency, not compute.

Weights

[ \text{weight bytes} = (\text{number of parameters}) \times (\text{bytes per parameter}) ]

For a 7-billion-parameter model in BF16:

[ 7 \times 10^{9} \ \text{params} \times 2 \ \text{bytes} = 14 \times 10^{9} \ \text{bytes} \approx 14 \ \text{GB} ]

The rule of thumb “~2 GB per billion params in FP16/BF16” (and ~1 GB/B in INT8, ~0.5 GB/B in INT4) falls straight out of this.

KV cache (per token, then per request)

The KV cache stores a key and a value vector for every token, every layer, every KV head:

[ \text{bytes per token} = 2 \times L \times H_{kv} \times D_{h} \times b ]

where the leading (2) covers K and V, (L) is the number of transformer layers, (H_{kv}) the number of key/value heads, (D_{h}) the head dimension, and (b) the bytes per element. For a LLaMA-style 7B model with (L = 32), (H_{kv} = 32), (D_{h} = 128), BF16 ((b = 2)):

[ 2 \times 32 \times 32 \times 128 \times 2 = 524{,}288 \ \text{bytes} \approx 0.5 \ \text{MB per token} ]

For a request with a full context of 4,096 tokens:

[ 4096 \times 524{,}288 \ \text{bytes} = 2{,}147{,}483{,}648 \ \text{bytes} = 2 \ \text{GB} ]

So a single 4K-context request costs ~2 GB of KV cache on top of the 14 GB of weights. Note that models using grouped-query attention (GQA) have far fewer KV heads (H_{kv}) than query heads, which is specifically a trick to shrink this number — one reason modern models are cheaper to serve. As a concrete comparison: if the same architecture used full multi-head attention with (H_{kv}) equal to 32 query heads unchanged, doubling (H_{kv}) doubles KV-cache bytes per token linearly — GQA with, say, 8 KV heads instead of 32 shrinks the same request’s KV cache fourfold, from 2 GB to 0.5 GB, which is the difference between fitting 4 concurrent long requests and fitting 16.

Saying it out loud. The KV cache formula is two — for keys and values — times layers, times KV heads, times head dimension, times bytes per element, and that gives you bytes per token. For a classic 7B model with 32 layers, 32 KV heads, head dim 128, in BF16, that works out to about half a megabyte per token, so a 4,096-token request is roughly 2 GB. The interesting term is the KV head count, because grouped-query attention deliberately shrinks it — dropping from 32 KV heads to 8 cuts that request’s cache fourfold, from 2 GB down to 0.5 GB. That’s the difference between fitting four concurrent long requests and fitting sixteen on the same card, which is why GQA is a serving-cost decision, not a model-card footnote.

Activations and overhead

Beyond weights and KV cache, each forward pass allocates transient activation tensors, and the CUDA context / allocator reserves a fixed slab (often 1–2 GB). Activation memory scales with batch size and sequence length but is freed between steps, so it is usually a smaller, bounded term than the KV cache — which only grows. When you size a GPU, budget weights + peak KV + a safety margin for activations and fragmentation; do not plan to use the last gigabyte.

Putting it together on a 24 GB GPU

[ \text{free for KV + activations} \approx 24 \ \text{GB} - 14 \ \text{GB (weights)} - \sim 1\text{–}2 \ \text{GB (activations, CUDA ctx)} \approx 8 \ \text{GB} ]

At ~2 GB per 4K-token request, that GPU holds on the order of 4 concurrent full-length requests before OOM — and fewer if prompts are longer. This is precisely where MAX_CONCURRENT_GENERATIONS = 3 in Section B’s server came from — a deliberately conservative choice below the theoretical ceiling of ~4, to leave headroom for activation spikes and fragmentation rather than run at the edge. This single calculation is why KV-cache efficiency (PagedAttention, Chapter 5) is the highest-leverage optimization in LLM serving: memory, not compute, usually caps your concurrency.

Saying it out loud. Put the numbers together on a real card: 24 GB total, minus 14 for BF16 weights, minus a gig or two for activations and the CUDA context, leaves you about eight gigabytes for KV cache. At roughly 2 GB per 4K-token request, that’s about four concurrent full-length requests — and you’d actually set your limit at three, because running at the theoretical edge means fragmentation and activation spikes will OOM you eventually. That’s the whole argument for PagedAttention in one calculation: if memory is what caps concurrency, then using memory more efficiently is the highest-leverage optimization in the entire stack, worth more than any scheduler tuning you’ll do.

Quantization changes the concurrency budget, not just the weight size

The same 24 GB-GPU calculation above assumed BF16 weights. Quantization is the other big lever, and it is worth running the numbers once so the tradeoff is concrete rather than a slogan (“quantize to fit more”). Take the same 7B model, same 24 GB GPU, same ~2 GB-per-4K-token-request KV cache (quantizing weights does not by itself shrink KV-cache dtype unless you separately quantize the cache):

[ \text{INT8 weights} = 7 \times 10^{9} \times 1 \ \text{byte} = 7 \ \text{GB}, \qquad \text{free for KV} \approx 24 - 7 - 1.5 \approx 15.5 \ \text{GB} ]

At ~2 GB per request, that is room for roughly 7–8 concurrent full-length requests, up from ~4 in BF16 — nearly double the concurrency on identical hardware, at the cost of some generation quality (INT8 post-training quantization is usually a small, workload-dependent quality hit; validate it against your own eval set rather than assuming it is free). Push to INT4 and weights drop to ~3.5 GB, freeing room for on the order of 15+ concurrent requests on the same GPU. This is the concrete version of the “quantization vs. smaller model” interview question below: quantizing a 7B model to INT8 and running a smaller, unquantized 3B model in BF16 can land at similar total memory, but they are not equivalent — quantization keeps the larger model’s capability at a precision cost, while a smaller model changes the capability itself. Which one wins depends on whether your bottleneck is quality or throughput per request, and the only way to know is to measure both against your task.

Saying it out loud. People pitch quantization as “make the model smaller,” but the interesting effect is second-order: shrinking the weights frees room for KV cache, and KV cache is what caps concurrency. Same 24 GB card, same 7B model — BF16 weights take 14 GB and leave room for about four concurrent requests; INT8 takes 7 GB and leaves room for seven or eight; INT4 takes about 3.5 GB and gets you past fifteen. So you roughly quadrupled your concurrency on identical hardware. The tradeoff you have to name honestly is quality: INT8 post-training quantization is usually a small hit, but it’s workload-dependent, and the only way to know is your own eval set. And note that quantizing weights doesn’t shrink the KV cache unless you separately quantize the cache too.

Metrics and the latency–throughput tradeoff

You cannot improve what you do not measure. Four numbers define an LLM endpoint.

Definitions and formulas follow the Anyscale benchmarking guide (see Further Reading). Report percentiles (p50/p95/p99), never just the mean — tail latency is what users feel and what SLOs are written against. This is exactly what the loadtest.py script in Section B reports, and exactly why it reports percentiles rather than an average: an average latency can look fine while p99 callers are being starved behind the semaphore.

Saying it out loud. Four numbers define an LLM endpoint and you should rattle them off. Time to first token — how long until the user sees anything, dominated by queueing plus prefill. Time per output token, the gap between successive tokens once it’s streaming. End-to-end latency, which is just TTFT plus generation. And throughput, tokens or requests per second across everyone. The rule that separates a senior answer from a junior one is: always report p50, p95, and p99, never the mean — because an average latency can look perfectly healthy while your p99 callers are starving in a queue, and your SLO is written against the tail, not the middle.

The tradeoff

Here is the crux, and it is why “just add batching” is not free:

• A single request in isolation gets the lowest possible latency: the whole GPU is devoted to it. But decode is memory-bound, so the GPU’s compute units are ~idle — terrible throughput per dollar.
• Batching many requests together amortizes each weight/KV read across all of them: one memory pass serves (B) requests. Throughput (tokens/s, req/s) rises sharply — you use the idle compute. But any individual request may wait to be batched and shares GPU cycles, so its TTFT and TPOT rise.

So batch size is a dial between latency (small batch, low utilization, expensive per token) and throughput (large batch, high utilization, cheap per token, worse tail latency). There is no single right setting — it depends on your SLO. Real serving engines make this dial dynamic (continuous batching), which we introduce next and detail in Chapter 5.

Saying it out loud. Batch size is a dial between latency and throughput, and there’s no universally right setting — it depends entirely on your SLO. Turn it down and a single request gets the whole GPU to itself: lowest possible latency, but since decode is memory-bound the compute units sit idle, so it’s terrible throughput per dollar. Turn it up and one read of the weights produces a token for every request in the batch, so throughput climbs sharply — but any individual request now waits to be batched and shares cycles, so its TTFT and per-token latency get worse. The important nuance: naive batching genuinely hurts single-request latency, which is why real engines make the dial dynamic with continuous batching instead of picking one static number.

Worked example 3: reading a latency timeline

Numbers make the asymmetry concrete. Suppose a request has a 500-token prompt and generates 200 tokens, and we measure:

• prefill takes 120 ms (this is the TTFT, ignoring queueing),
• each decode step takes 15 ms.

Then:

[ \text{TTFT} = 120 \ \text{ms}, \qquad \text{generation} = 199 \times 15 \ \text{ms} \approx 2985 \ \text{ms} ]

[ \text{E2E} = 120 + 2985 \approx 3.1 \ \text{s}, \qquad \text{TPOT} = \frac{3105 - 120}{200 - 1} \approx 15 \ \text{ms}, \qquad \text{user TPS} = \frac{200}{2.985} \approx 67 \ \text{tok/s} ]

Two lessons. First, decode dominates E2E here (~3 s vs ~120 ms) — output length is your biggest latency lever, which is why capping max_new_tokens matters so much. Second, if you stream, the user sees a token at 120 ms instead of waiting 3.1 s: identical work, far better perceived latency. Streaming trades nothing in throughput; it only reshapes when the user first sees output.

Saying it out loud. Put real numbers on it: 500-token prompt, 200 tokens out, prefill takes 120 milliseconds and each decode step takes 15. So your TTFT is 120 milliseconds, but generation is 199 more steps at 15 milliseconds each — about three seconds. End to end you’re at roughly 3.1 seconds, and decode is over ninety-five percent of it. Two things fall out. Output length is by far your biggest latency lever, which is why capping max_new_tokens matters more than almost any tuning. And if you stream, the user sees a token at 120 milliseconds instead of three seconds — identical work, completely different experience.

Why one-request-at-a-time is wasteful — the road to batching

The naive server above processes requests serially. During each request’s decode phase the GPU is memory-bandwidth-bound and its tensor cores are mostly idle — you are paying for an A100/H100 and using a fraction of its FLOPs. Meanwhile a second caller just waits.

Batching fixes this by running multiple sequences through the model together, so a single read of the weights (and a single scheduling step) produces a token for every request in the batch. Because decode was memory-bound, adding more requests is nearly free on compute up to a point — you convert idle compute into throughput.

There are three flavors, in increasing sophistication (full treatment in Chapter 5):

The mental model: static/dynamic batching batches requests; continuous batching batches token-generation steps. The latter keeps the GPU full even when requests have wildly different lengths — which is the normal case. This is the single biggest reason a dedicated engine outperforms the naive server, often by an order of magnitude in throughput at the same latency. Section B’s concurrency gate is a poor person’s version of this: it prevents catastrophe, but it still runs each admitted request’s decode loop independently rather than sharing steps across requests — the semaphore buys you safety, not the throughput win of continuous batching.

Saying it out loud. Serving one request at a time means that during decode your expensive GPU is mostly waiting on memory while a second caller just sits there — you’re renting tensor cores and using a fraction of them. Batching fixes it because one read of the weights produces a token for every sequence in the batch, and since decode was memory-bound, adding sequences is nearly free on compute up to a point. There are three flavors: static batching pads a fixed group and runs it to completion, dynamic batching buffers arrivals for a few milliseconds then does the same, and continuous batching schedules at the level of a single decode step, so a finished sequence leaves and a new one joins every iteration. The failure mode the first two share is head-of-line blocking — one 2,000-token answer holds the whole batch hostage — and continuous batching is what eliminates it, often for an order of magnitude more throughput at the same latency.

Failure modes and pitfalls

The naive server fails in predictable ways. Know them cold.

• CUDA out of memory (OOM). The #1 killer. Causes: model too big for the GPU in the chosen dtype; too many concurrent requests inflating the KV cache; a single very long prompt/output. Symptoms: CUDA out of memory. Tried to allocate .... Fixes: smaller dtype/quantization, cap max_new_tokens and context length, limit concurrency, use an engine with paged KV. Do the memory math before deploying.
• Blocking the async event loop. FastAPI is async, but model.generate() is a long, synchronous, GPU-bound call. If you await it directly in the handler (or call it inline), it freezes the entire event loop — health checks time out, every other connection stalls. Fix: run_in_threadpool (as above) or a dedicated worker/queue. This bug looks like “the server randomly hangs under load” — see the first war story below for exactly how this plays out in production.
• No batching / serial serving. Two users -> the second waits for the first. Throughput is capped at one request’s worth of decode, and the expensive GPU sits underutilized. This is not a bug to fix in this server — it is the reason to graduate to vLLM/TGI.
• Tokenizer mismatch / wrong chat template. Using a tokenizer from a different model, skipping apply_chat_template, or missing special tokens produces fluent-looking garbage with no error. Always pair the exact tokenizer with the model and use the official chat template.
• Padding-side and pad-token mistakes. Right-padding a decoder-only batch, or a missing pad_token, silently corrupts batched generation. Left-pad; set pad_token = eos_token if absent.
• FP32 by accident. Forgetting torch_dtype loads in FP32 and doubles weight memory — an instant OOM on models that would fit fine in BF16.
• Unbounded generation. No max_new_tokens and a wrong/missing EOS -> the model runs until it hits some default cap, burning GPU time and blocking others. Always bound output — and bound it server-side, not just as a client-supplied default (Section B’s SERVER_MAX_NEW_TOKENS).
• Unbounded concurrency / unbounded queues. Accepting every connection and either racing them all for the GPU or queueing them forever converts a capacity problem into a memory problem or a client-timeout problem, one level removed. Gate concurrency from real memory math and shed load explicitly (Section B).
• Cold-start latency ignored. First request after a scale-up waits for model download + load (seconds to minutes). Add readiness probes (/healthz gating on model-loaded) so traffic is not routed to a not-yet-ready pod (Chapters 3, 6) — and keep liveness probes independent of load, or a saturated-but-healthy pod gets killed (Section B, war story one).

Saying it out loud. If you asked me what actually breaks these servers, I’d name three in order. CUDA out-of-memory is number one, and it’s almost always unbounded concurrency or unbounded output length rather than the model itself being too big. Second is blocking the async event loop by calling model.generate() directly in an async handler — that freezes every other coroutine in the process including your health checks, and it presents as “the server randomly hangs under load.” Third is the silent class: a mismatched tokenizer, a skipped chat template, or right-padding a decoder batch, all of which produce fluent-looking garbage with no error at all. The pattern is that the loud failures are the easy ones — it’s the silent ones that ship to production.

Production case studies & war stories

Theory is cheap; here are two incidents shaped exactly like ones teams hit in production, and the lesson each one teaches. Both are avoidable with the hardening in Section B, which is why that section exists.

War story 1: the health check that killed a healthy pod

Setup. A team shipped a version of the naive server close to Worked Example 1, but with one shortcut: the /generate handler called model.generate() directly inside the async def function instead of routing it through run_in_threadpool. It passed every test — a single curl request worked fine, latency looked right, and it sailed through staging, which only ever sent one request at a time.

What happened in production. Once real traffic arrived — five or six concurrent users, nothing exotic — requests started stacking up, and every few minutes the pod would restart. On-call saw 502s from the load balancer and Kubernetes events showing Liveness probe failed followed by a container restart, which looked like a memory leak or a crash. It was neither. model.generate() is a long-running, synchronous, CPU-orchestrated call; running it directly in an async def handler blocks Python’s single event loop for the entire generation — often several seconds. While it was blocked, the event loop could not service any other coroutine, including the /healthz liveness endpoint on the same process. Kubelet’s liveness probe timed out, decided the process was unresponsive, and killed it mid-request — dropping every in-flight generation, including ones that had nothing to do with the blocked request.

Diagnosis. A py-spy dump against the running process during a slow period showed every worker thread’s Python-level stack sitting inside torch.nn.functional deep under model.generate, with the asyncio event loop task for /healthz never getting scheduled. That is the smoking gun for a blocked event loop: the liveness endpoint’s code is fine, it simply never runs.

Lesson. Never run long, synchronous, CPU/GPU-bound work directly inside an async def handler — always route it through run_in_threadpool (as in Worked Example 1) or a separate process/worker pool. Just as important: keep liveness probes independent of the workload. /healthz should answer instantly regardless of GPU saturation (Section B’s version does, deliberately); it is /readyz that should reflect load, and unlike a liveness failure, a readiness failure only stops new traffic — it does not kill in-flight work. Conflating the two, or letting the workload block the endpoint that answers either, turns ordinary load into a self-inflicted outage.

Saying it out loud. This one’s my favorite because the bug and the symptom look nothing alike. A team called model.generate() directly inside an async handler — worked perfectly in staging, which only ever sent one request at a time. In production with five or six concurrent users, pods started restarting every few minutes with liveness probe failures, which everyone read as a memory leak. It wasn’t: generate is a long synchronous call, so it blocked Python’s single event loop for seconds at a time, and the health endpoint living on that same loop simply never got scheduled. Kubelet saw no response, concluded the process was dead, and killed it mid-generation — taking down every unrelated in-flight request too. Two lessons: never run blocking work inside an async handler, and never let your liveness probe depend on the workload.

War story 2: the silent OOM from an unbounded batch

Setup. A different team’s server accepted arbitrary concurrent requests with no admission control, and let clients pass their own max_new_tokens and prompt length with no server-side cap — reasoning that “the model will just run a bit slower” under load. There was no memory math behind that assumption; nobody had run the calculation in Worked Example 2 for their actual GPU and model.

What happened in production. A marketing push drove a burst of traffic, much of it from a batch-import script sending long documents with max_new_tokens: 4096 set on every call, all fired concurrently. Each concurrent request’s KV cache grew independently and kept growing every decode step; there was nothing gating how many of these could run at once. Partway through the spike, the CUDA allocator ran out of memory mid-batch. Because there was no isolation between requests, the failure did not stay contained to the offending calls: some in-flight requests raised CUDA out of memory immediately, others returned corrupted or truncated output as the allocator scrambled to free fragmented memory, and a few hung entirely until the process was restarted — losing every request that happened to be in flight at that moment, including well-behaved ones from unrelated callers.

Diagnosis. Post-incident, the team ran the memory math from Worked Example 2 against their actual model and GPU for the first time and discovered the theoretical safe concurrency for full-length, full-max_new_tokens requests was around 4 — they had been letting 30-plus requests race for the GPU simultaneously with no cap, and it had simply not yet coincided with a burst large and long enough to blow the budget.

Lesson. Two independent fixes, both from Section B, and both necessary: (1) cap max_new_tokens and effective context length server-side, regardless of what the client requests — SERVER_MAX_NEW_TOKENS in Worked Example 1’s extended server exists precisely so a client’s number is a request, not a command; and (2) gate concurrency using the KV-cache math from Worked Example 2, not intuition — the semaphore in Section B turns “the model will just run a bit slower” into “the fourth-plus concurrent caller waits in a bounded queue or gets a clean 503,” which is a survivable, observable failure mode instead of an unbounded, correlated one. As a standing practice: monitor KV-cache/GPU-memory utilization as a first-class metric next to request count and latency — request count alone hid the real signal here until the GPU had already run out.

Saying it out loud. A team let clients pass their own max_new_tokens with no server-side cap and no admission control, on the theory that “it’ll just run a bit slower.” Then a batch-import script fired long documents with max_new_tokens: 4096 all at once. Each request’s KV cache grew independently every decode step until the allocator ran dry mid-batch — and because nothing isolated requests from each other, it wasn’t contained: some raised OOM, some returned truncated output, some hung until restart, and well-behaved callers went down with them. Post-mortem, they ran the memory math for the first time and found their safe concurrency was about four; they’d been running thirty-plus. Two fixes, both necessary: cap output length server-side, and derive your concurrency limit from the KV-cache math rather than intuition.

Tools comparison (brief — pointers to later chapters)

Rule of thumb: build the raw server once to understand the loop, then never ship it as-is. For anything real, reach for an engine that does continuous batching and paged KV memory. TorchServe is another general-purpose model server in this space, but for LLMs specifically the KV-cache-aware engines (vLLM, TGI, TensorRT-LLM) win decisively. The hardened version in Section B narrows — but does not close — that gap: it makes the naive server safe to run under real traffic (bounded concurrency, structured errors, correct health signals), which is table stakes for any service; it does not give you continuous batching’s throughput, which requires the engine-level scheduling covered in Chapter 5.

Saying it out loud. My honest rule is: build the raw server once so you understand the loop, then never ship it as-is. For anything with real concurrency you want an engine that does continuous batching and paged KV memory — vLLM if you want throughput and an OpenAI-compatible API, TGI if you’re deep in the Hugging Face ecosystem, Triton if you’re serving many models across frameworks in an NVIDIA shop. Hardening the raw server the way we just did makes it safe — bounded concurrency, structured errors, honest health signals — but safe isn’t the same as fast. It still runs one request’s decode loop per slot, so it never gets the throughput win, and that gap is the actual reason you migrate, not “vLLM is industry standard.”

The 2025–2026 landscape

Everything above is timeless mechanism — prefill/decode, the KV cache, memory math — but the tooling and API surface around it moves fast. Here is what “basic serving” looks like against the current landscape, with sources, so you can reason about where the raw-server pattern in this chapter still fits and where the industry has moved on.

The default artifact format is safetensors, and the reason is security, not just speed

By 2025–2026, safetensors (https://github.com/huggingface/safetensors) is the default distribution format on the Hugging Face Hub and in production runtimes (TGI’s own docs describe why: https://huggingface.co/docs/text-generation-inference/main/en/conceptual/safetensors). The mechanism, worth restating precisely because it comes up in interviews: PyTorch’s legacy checkpoint format is a pickle blob, and pickle.load is not a passive data format — it can invoke arbitrary Python callables encoded in the file, which is a documented remote-code-execution vector for a checkpoint downloaded from an untrusted source. safetensors stores only tensor bytes and a JSON metadata header, so there is no code path from “load this file” to “execute arbitrary code,” and because the layout is flat and contiguous, it also loads faster via mmap. If your serving pipeline still ingests raw .bin/.pt files from third parties without conversion, that is a supply-chain gap worth flagging, not a stylistic preference.

Saying it out loud. Everyone assumes safetensors won on speed, and the speed is real, but the actual reason is security. PyTorch’s old checkpoint format is a pickle blob, and unpickling isn’t passive parsing — it can call arbitrary Python constructors named inside the file, which is a documented remote-code-execution path for any weights you downloaded from someone else. Safetensors stores only raw tensor bytes plus a JSON header of shapes and dtypes, so there’s simply no code path from “load this file” to “execute something.” The speed comes free as a side effect, because a flat contiguous layout can be memory-mapped instead of deserialized. Practical version: if your pipeline still ingests third-party .bin or .pt files without conversion, that’s a supply-chain gap, not a style preference.

Native structured outputs are now a first-class API feature, not a prompting trick

As of 2024–2026, “ask nicely for JSON” has been replaced by API-level guarantees. OpenAI’s Structured Outputs (announced 2024, current docs at https://developers.openai.com/api/docs/guides/structured-outputs, original post at https://openai.com/index/introducing-structured-outputs-in-the-api/) let you pass a JSON Schema and get a response guaranteed to satisfy it — no missing keys, no hallucinated enum values, no manual retry-on-invalid-JSON loop. Anthropic and Google expose comparable schema-constrained/tool-argument guarantees on their current APIs. Mechanically, this is not a smarter prompt — it is constrained (grammar-guided) decoding: the schema is compiled into a finite-state or pushdown automaton over the tokenizer’s vocabulary, and at every decode step the set of grammatically-valid next tokens is computed and every other token’s logit is masked to (-\infty) before sampling. The open-source engines implement this directly — vLLM’s structured-outputs backends (https://docs.vllm.ai/en/latest/features/structured_outputs/) support both the outlines library and XGrammar, a purpose-built constrained-decoding engine (paper: https://arxiv.org/pdf/2411.15100) chosen for compiling grammars fast enough to mask logits every single decode step without becoming the bottleneck. The practical implication for this chapter’s naive server: adding structured outputs to model.generate() directly means wiring a LogitsProcessor that does this masking yourself (Transformers exposes the hook, but you own the FSM); a dedicated engine gives you response_format={"type": "json_schema", ...} for free. This is a concrete, common reason teams move off the raw server well before they need continuous batching.

Saying it out loud. “Please respond in JSON” is dead as an engineering technique — every major API now takes a JSON Schema and guarantees the response satisfies it. Under the hood it’s constrained decoding: the schema compiles into an automaton over the tokenizer’s vocabulary, and at each decode step every token that would violate the grammar gets its logit masked to negative infinity before sampling. So you get no missing keys, no invented enum values, and no retry-until-it-parses loop. The catch is that this cost lands on every decode step, which is why engines use purpose-built compilers like XGrammar rather than re-validating from scratch — and it’s a very common reason teams leave the raw server long before they ever need continuous batching.

Prompt / context caching is now standard, and it changes the economics of “basic serving”

All three major hosted APIs now cache repeated prompt prefixes so you do not pay full price to re-process a system prompt or a large shared context on every call:

• Anthropic prompt caching (https://platform.claude.com/docs/en/build-with-claude/prompt-caching): you mark a cache_control breakpoint; a cache write costs a multiplier over the base input rate (roughly 1.25× for a 5-minute TTL, 2× for a 1-hour TTL), while a cache read costs roughly 0.1× the base rate — a ~90% discount on tokens the model has already “seen” in cache. Minimum cacheable prefix length is model-dependent (from roughly 1,024 tokens up to 4,096 on smaller models); below that, caching is silently skipped, so check the response’s cache_read_input_tokens / cache_creation_input_tokens fields rather than assuming.
• OpenAI automatic prompt caching (https://developers.openai.com/api/docs/guides/prompt-caching, announced at https://openai.com/index/api-prompt-caching/): caching is automatic above roughly a 1,024-token prefix, keyed by an exact-prefix hash of (typically) the first portion of the prompt, and routed by a prompt_cache_key so repeated calls land on a machine likely already holding the cached prefix; cached content is retained on the order of minutes to about an hour depending on traffic and model.
• Gemini context caching (https://ai.google.dev/gemini-api/docs/caching): supports both implicit caching (automatic, enabled by default on current Gemini models) and explicit caching (you create and manage a cache object yourself, useful for a large shared document or system context reused across many calls), again with a per-model minimum token threshold before caching activates.

Why this matters for a chapter about a raw, self-hosted server: this is precisely the prefix-reuse problem that a naive model.generate() loop does nothing about — every call re-runs prefill from scratch, even if 90% of the prompt (a shared system prompt, a long set of tool definitions, a large retrieved context) is byte-identical to the previous call. Self-hosted engines solve the same problem under a different name — prefix caching / automatic prefix reuse (vLLM’s implementation is often discussed alongside RadixAttention-style prefix trees) — by keeping the KV cache for a previously-seen prefix resident and reusing it instead of recomputing prefill. If your workload has long, repeated system prompts or tool schemas, prefix caching is frequently a bigger win than raw batching throughput, and it is a capability the raw server in this chapter simply does not have.

Saying it out loud. If your prompts share a long prefix — a big system prompt, a pile of tool definitions, a retrieved document — you’re re-running prefill on identical bytes every single call, and everybody now has a fix for that. On hosted APIs it’s prompt caching: Anthropic charges roughly 1.25x base rate to write a cache entry and about 0.1x to read it, so a cache hit is around a ninety percent discount on those tokens; OpenAI does it automatically above about a 1,024-token prefix. Self-hosted engines call the same idea prefix caching — they keep the previously computed KV entries resident instead of recomputing them. The thing to say: this is a completely separate axis from batching, and for prefix-heavy workloads it’s often the bigger win — and it’s a capability the raw generate() loop simply does not have.

The shift from raw generate() to production runtimes is now the norm, not the exception

The direction of travel across 2025–2026 write-ups on production LLM serving (see, e.g., the state-of-the-field surveys and deployment guides in Further Reading) is consistent: teams prototype against transformers.generate() and move to a dedicated runtime — vLLM, TGI, TensorRT-LLM, or SGLang — as soon as concurrency, latency SLOs, or cost per token matter, because continuous batching plus paged KV memory is very hard to reproduce well by hand, and because these engines have absorbed structured-output support, prefix caching, speculative decoding, and multi-GPU sharding as built-in features rather than bespoke code. That does not make this chapter’s raw server obsolete knowledge — it is the mental model every one of those engines optimizes against — but it does mean the honest scope of “basic serving” in 2026 is: understand the loop and the memory math cold, harden it enough to survive real traffic if you must run it (Section B), and know precisely which of its gaps (batching, prefix reuse, structured decoding) are the reasons you reach for vLLM/TGI/Triton rather than vague “it’s faster” hand-waving.

Saying it out loud. The industry pattern is consistent: prototype on transformers.generate(), then move to a real runtime — vLLM, TGI, TensorRT-LLM, SGLang — the moment concurrency, latency SLOs, or cost per token start to matter. The reason isn’t that engines are magically faster; it’s that continuous batching plus paged KV memory is genuinely hard to reproduce by hand, and those engines have already absorbed structured outputs, prefix caching, speculative decoding, and multi-GPU sharding as built-in features instead of bespoke code you maintain. That doesn’t make the raw server useless knowledge — it’s the mental model those engines are optimizing against. What it does mean is you should be able to say exactly which gap pushed you over, not hand-wave that “it’s faster.”

Where “basic serving” still fits

None of the above obsoletes this chapter — it contextualizes it. A raw, hardened generate() server is still the right answer when: you need custom, non-standard generation logic an engine does not expose; you are serving a small number of concurrent users where continuous batching’s throughput gains do not matter; you are prototyping or teaching; or you are running a model on hardware/software combinations the major engines do not yet support well. The 2025–2026 baseline expectation, though, is that you can articulate exactly why you are not using vLLM/TGI in a given case — “we don’t need it yet, here’s the math” — rather than not knowing the alternative exists.

Saying it out loud. The raw hardened server is still the right call in a few real cases: you need custom generation logic no engine exposes, you’re serving a handful of concurrent users where batching gains don’t materialize, you’re prototyping or teaching, or you’re on hardware the big engines don’t support well. What’s changed by 2026 isn’t that this pattern is wrong — it’s that the burden of proof moved. The expectation now is that you can articulate why you’re not on vLLM with actual numbers, something like “our peak is four concurrent users, continuous batching buys us nothing at that scale, here’s the math.” Not knowing the alternative exists is the answer that loses you the interview; deliberately deferring it with a reason is the answer that wins.

Interview mastery

This section replaces and substantially expands the old “production checklist.” It is organized as: a bank of Q&A with model answers, a system-design prompt worked end to end, and a red-flags-vs-green-flags table you can use to grade yourself or a candidate.

Q&A bank

1. “Walk me through a request.” Request -> tokenize (+ chat template) -> prefill (compute-bound, sets TTFT) -> decode loop reusing the KV cache (memory-bound, sets TPOT) -> detokenize -> response. Naming the prefill/decode split unprompted is the single strongest signal you understand serving; see the 60-second answer above for the full version.

2. “How much GPU memory does model X need?” Weights = params × bytes/param (~2 GB/B in FP16/BF16); plus KV cache = (2 \times L \times H_{kv} \times D_h \times b) per token per request; plus activations/overhead. Know the order of magnitude — ~0.5 MB/token, ~2 GB per 4K-token request for a 7B model — and be able to say why GQA shrinks the KV term specifically.

3. “Your p99 latency is bad but the GPU is at 30% util — why?” Decode is memory-bandwidth-bound and you are serving serially or with tiny batches; the compute units sit idle waiting on memory traffic. Add continuous batching to convert that idle compute into throughput; a semaphore-gated raw server (Section B) prevents overload but does not fix this — it still runs one request’s decode loop at a time per slot.

4. “How do you trade latency for throughput?” Batch size is the dial: larger batches amortize weight reads across more requests (higher throughput) at the cost of per-request TTFT/TPOT, because a given request now waits on or shares cycles with others. Pick the setting from your SLO; continuous batching gets most of the throughput without static batching’s head-of-line blocking.

5. “What’s your OOM story?” Memory math up front (Worked Example 2), server-side caps on max_new_tokens/context (never trust client values alone), concurrency gated to a number derived from that math (not intuition), quantization as a lever, paged KV in a real engine. Monitor KV-cache/GPU-memory utilization as a first-class metric — see war story 2 for what happens when nobody does.

6. “Why not just await model.generate() in the handler?” It blocks the single-threaded asyncio event loop for the whole generation, freezing every other coroutine on that process — including your own health checks. Offload to a threadpool/worker, or use an engine with its own scheduler. See war story 1 for the exact failure this produces in production.

7. “Which metrics do you alert on?” TTFT, TPOT/ITL, E2E — all at p50/p95/p99 — plus throughput (tok/s, req/s), queue depth, KV-cache/GPU-memory utilization, and error/OOM rate. Percentiles, not means; a healthy-looking average can hide starved tail requests.

8. “When do you reach for vLLM/TGI/Triton over your own server?” As soon as you need real concurrency: continuous batching plus paged KV are hard to build well and are the whole point of those engines. Roll your own to learn, for genuinely custom logic, or for small enough traffic that batching’s gains don’t matter — and be able to say which of those applies to you.

9. “Static vs dynamic vs continuous batching — what’s the actual difference?” Static and dynamic batching both batch at the request level and run the batch to completion, so a short request can be stuck behind a long one (head-of-line blocking). Continuous batching operates at the decode-step level: a finished sequence leaves the batch and a new one joins on the very next iteration, so the GPU stays full even when request lengths vary wildly — which is the normal case in production.

10. “Explain GQA’s effect on serving, not just training.” Grouped-query attention reduces the number of KV heads (H_{kv}) relative to query heads. Since KV-cache bytes per token scale linearly in (H_{kv}), this directly shrinks the KV cache per token — e.g., dropping from 32 to 8 KV heads cuts KV-cache memory 4×, which multiplies directly into how many concurrent requests fit on a given GPU. It is a serving-cost decision baked into the architecture, not just an efficiency footnote from the model card.

11. “Does speculative decoding belong in ‘basic serving’?” It’s an optimization layered on top of the decode loop: a small draft model proposes several tokens, the full model verifies them in one parallel forward pass, and correct guesses are accepted for free, cutting the number of full-model decode steps needed. It doesn’t change the memory story (you still need the KV cache for both models) and it adds real scheduling complexity, so in practice it shows up in dedicated engines, not the raw server in this chapter — know what it is and why it doesn’t fit here yet.

12. “How does prompt caching change things at the API-provider level, and does the raw server in this chapter get that benefit?” No — a naive model.generate() loop re-runs prefill from scratch on every call, even for a byte-identical shared system prompt or tool schema. Hosted APIs (Anthropic, OpenAI, Gemini) and self-hosted engines both solve this via prefix reuse — either provider-side prompt caching or engine-side automatic prefix caching — by keeping a previously-computed KV-cache prefix resident and skipping its recomputation. It’s a distinct optimization axis from batching, and workloads with long, repeated prefixes often benefit from it more.

13. “What is ‘structured output’ / JSON mode under the hood?” Grammar-constrained (constrained) decoding: the JSON Schema or grammar is compiled into an automaton over the vocabulary, and at every decode step the logits for grammatically-invalid tokens are masked to (-\infty) before sampling, so every generated token is schema-valid by construction — not by asking nicely and retrying on failure. This is a decode-time cost (computing/re-checking the valid token set every step), which is why engines use purpose-built compilers (e.g., XGrammar) rather than naive per-step re-validation.

14. “Why bound the request queue instead of letting it grow to absorb bursts?” An unbounded queue doesn’t remove overload, it hides it — it turns a capacity problem into unbounded process memory growth and lets client-side timeouts fire anyway once wait time exceeds them, except now invisibly. A bounded queue with a fast, explicit rejection (503) gives callers, load balancers, and autoscalers an honest signal to react to.

15. “What’s the difference between your liveness and readiness checks, and why does it matter for a GPU pod specifically?” Liveness asks “is the process alive enough to not be killed and restarted” and must stay cheap and independent of GPU load, or a temporarily saturated-but-healthy pod gets killed mid-request (war story 1). Readiness asks “should traffic be routed here right now” and should reflect real capacity — model loaded, not already at its concurrency cap — so a load balancer stops sending new work without killing in-flight work.

16. “Quantization or a smaller model — how do you decide?” They are not interchangeable levers even when they land at similar memory footprints. Quantizing a larger model (e.g., 7B to INT8) keeps its underlying capability at a precision cost; swapping to a smaller model in full precision changes the capability itself. Run the memory math for both (Worked Example 2’s quantization variant), then validate quality against your own eval set — never assume either choice is quality-neutral by default.

17. “What would you monitor in week one of running this in production that you might not think of on day one?” Beyond the obvious latency/throughput dashboards: KV-cache/GPU-memory utilization over time (not just point-in-time), queue-depth and 503-rate as a leading indicator of undersized concurrency limits, p99-to-p50 latency ratio as a serialization signal, and cold-start duration on new pods — each of these is a metric that stayed invisible right up until the two war stories above turned into incidents.

18. “If you could only add one thing to the naive server before it sees production traffic, what would it be?” A defensible answer names the concurrency gate sized from real memory math (Worked Example 2 + Section B) over any other single change — it is the one guard that converts an unbounded-failure incident (war story 2) into a bounded, observable one, and most other hardening (structured errors, readiness checks) matters most in service of that same goal: knowing and respecting the server’s actual capacity.

System design prompt: “One GPU, 50 requests per second — walk me through your design”

This is a common live prompt because it forces you to reconcile everything above under a concrete constraint. A worked sketch:

Step 1 — clarify the workload. Ask: what’s the model size, typical prompt/output length, and latency SLO? Assume, for concreteness: a 7B model in BF16 on a single 80 GB H100, average prompt 300 tokens, average output 150 tokens, target p95 E2E under ~2s.

Step 2 — do the memory math first, not last. Weights: (7\text{B} \times 2\text{ bytes} \approx 14\text{ GB}). KV cache per token (from Worked Example 2’s formula, LLaMA-style with GQA, roughly ~0.15–0.5 MB/token depending on (H_{kv})) times a typical active sequence length of ~450 tokens (300 prompt + 150 output) gives on the order of 100–200 MB of KV cache per request. On an 80 GB GPU with ~60 GB free after weights and overhead, that is theoretical room for hundreds of concurrent sequences purely on memory — the real constraint at 50 req/s is compute and scheduling, not raw KV capacity, which is itself a useful thing to say out loud (it shows you check both constraints instead of assuming one).

Step 3 — pick the serving layer. At 50 req/s sustained, this is squarely continuous-batching territory: reach for vLLM (or TGI/TensorRT-LLM) rather than a hand-rolled server — the naive per-request server from this chapter cannot get anywhere close to 50 req/s on one GPU because it never overlaps requests’ decode steps. State this directly: “I would not build this on raw transformers.generate(); I’d deploy vLLM with continuous batching and PagedAttention on this GPU.”

Step 4 — size the batch/concurrency for the SLO. With continuous batching, throughput scales with how many sequences’ decode steps you can interleave before per-step latency (which grows mildly with batch size) blows the p95 SLO. Talk through the dial explicitly: push batch size up until TPOT growth threatens the 2s target, then back off — this is the latency/throughput tradeoff from the metrics section, now as a concrete scheduling decision rather than an abstract tradeoff.

Step 5 — add the operational layer. Structured error handling and admission control (Section B) still apply even in front of vLLM — you still want a bounded request queue in your API gateway, health/readiness endpoints that don’t depend on GPU load, and metrics on TTFT/TPOT/queue depth/GPU utilization. If a burst pushes sustained demand past what continuous batching on one GPU can sustain at your SLO, the answer is horizontal scaling (more GPU replicas behind a load balancer) or admission control (shed load past capacity), not trying to fit more into one GPU than the SLO tolerates.

Step 6 — name the next lever if traffic grows. Prefix/prompt caching if requests share system prompts or tools; quantization (INT8/FP8/INT4) to shrink weights and free more room for KV cache and larger batches; speculative decoding to cut decode steps for latency-sensitive traffic. Naming these as considered and deferred, with a reason, is stronger than listing them as things you’d “probably also do.”

The throughline an interviewer is grading: did you reach for memory math before guessing, did you correctly identify that a single naive server cannot hit this target, and did you reason about the latency/throughput dial explicitly rather than asserting “vLLM is fast.”

Saying it out loud. The move on a question like this is to clarify first, then do the memory math before you pick any technology. So: what’s the model size, typical prompt and output length, and what’s the latency SLO? Say it’s a 7B in BF16 on an 80 GB H100, 300-token prompts, 150-token outputs, p95 under two seconds. Weights are 14 GB, leaving around 60 free; at roughly 100 to 200 megabytes of KV cache per request that’s theoretical room for hundreds of sequences — so memory isn’t the binding constraint here, scheduling and compute are, and saying that out loud shows you checked both. Then you pick continuous batching, because 50 requests per second is flatly impossible on a serial server. What they’re grading is whether you reached for arithmetic before you reached for a tool name.

Red flags vs green flags

Glossary — quick reference for review

Further reading

Core mechanism, generation, and this chapter’s server

• Hugging Face — Text generation tutorial (generate): https://huggingface.co/docs/transformers/en/llm_tutorial
• Hugging Face — Generation config / GenerationConfig reference: https://huggingface.co/docs/transformers/en/main_classes/text_generation
• FastAPI documentation: https://fastapi.tiangolo.com/
• FastAPI — concurrency and async/await: https://fastapi.tiangolo.com/async/
• httpx (used for the load test): https://www.python-httpx.org/
• Kubernetes — configuring liveness, readiness, and startup probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

KV cache, memory, and batching

• NVIDIA — Mastering LLM Techniques: Inference Optimization (KV cache, batching): https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
• KV cache memory calculation (worked example): https://mbrenndoerfer.com/writing/kv-cache-memory-calculation-llm-inference-gpu
• Pierre Lienhart — LLM Inference Series: KV caching, a deeper look: https://medium.com/@plienhar/llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8
• vLLM — PagedAttention & continuous batching overview: https://www.runpod.io/articles/guides/vllm-pagedattention-continuous-batching
• Anyscale — Understand LLM latency and throughput metrics (TTFT/TPOT/ITL/TPS/RPS): https://docs.anyscale.com/llm/serving/benchmarking/metrics

Model formats and safety

• safetensors — project and rationale: https://github.com/huggingface/safetensors
• Text Generation Inference docs — why safetensors: https://huggingface.co/docs/text-generation-inference/main/en/conceptual/safetensors

Structured / constrained decoding

• OpenAI — Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/
• OpenAI — Structured model outputs guide: https://developers.openai.com/api/docs/guides/structured-outputs
• vLLM — Structured Outputs feature docs: https://docs.vllm.ai/en/latest/features/structured_outputs/
• XGrammar — Flexible and Efficient Structured Generation Engine for LLMs (paper): https://arxiv.org/pdf/2411.15100
• Red Hat Developer — Structured outputs in vLLM: guiding AI responses: https://developers.redhat.com/articles/2025/06/03/structured-outputs-vllm-guiding-ai-responses

Prompt / context caching (2025–2026 provider docs)

• Anthropic — Prompt caching: https://platform.claude.com/docs/en/build-with-claude/prompt-caching
• OpenAI — Prompt caching guide: https://developers.openai.com/api/docs/guides/prompt-caching
• OpenAI — Prompt Caching in the API (announcement): https://openai.com/index/api-prompt-caching/
• Google — Gemini API context caching: https://ai.google.dev/gemini-api/docs/caching

Production serving engines

• Text Generation Inference (TGI) docs: https://huggingface.co/docs/text-generation-inference/en/index
• TGI source and features: https://github.com/huggingface/text-generation-inference
• NVIDIA Triton Inference Server: https://github.com/triton-inference-server/server
• vLLM documentation: https://docs.vllm.ai/

Where this goes next: Chapter 2 containerizes this server; Chapter 4 load-tests it to see the serial bottleneck at larger scale than Section B’s local script; Chapter 5 replaces it with vLLM and continuous batching to fix everything this chapter exposed — and revisits prefix caching and structured decoding as engine-native features rather than DIY additions.

Topic 2: Containerization with Docker

What You’ll Learn

This topic teaches you how to:

• Containerize LLM serving applications
• Create efficient Docker images for ML workloads
• Handle large model files
• Use multi-stage builds
• Configure environment variables
• Optimize image size

Why We Need This

Business Need

• Consistency: Same model behavior in dev, staging, production
• Speed to market: Deploy faster without environment setup
• Cost reduction: Standardized deployment reduces ops overhead
• Compliance: Reproducible deployments for audit trails

Technical Need

• Environment consistency: Python version, dependencies, system libraries
• Isolation: No conflicts between different ML projects
• Portability: Run anywhere (local, cloud, edge)
• Versioning: Tag images with model versions for rollback

Real-World Impact

Without containerization:

• ❌ “Works on my machine” problems
• ❌ Difficult to reproduce production issues
• ❌ Slow deployments (manual environment setup)
• ❌ Can’t scale easily (each server needs manual setup)

Industry Use Cases

1. Multi-Cloud Deployment

Company: Enterprise ML platforms Use Case:

• Same Docker image runs on AWS, GCP, Azure
• No vendor lock-in, easy migration

Example:

# Build once, deploy anywhere
docker build -t llm-model:v1.0 .
# Deploy to AWS
docker push llm-model:v1.0
# Deploy to GCP
docker push llm-model:v1.0

2. CI/CD Pipelines

Company: All tech companies Use Case:

• Automated testing in containers
• Consistent environments across pipeline stages

Example:

# GitHub Actions / GitLab CI
- name: Test model
  run: |
    docker build -t test-model .
    docker run test-model pytest

3. Edge Deployment

Company: IoT, autonomous vehicles Use Case:

• Deploy models to edge devices
• Same container on server and edge

Example:

# Deploy to edge device
docker save llm-model:v1.0 | ssh edge-device docker load

4. Model Versioning & Rollback

Company: ML platforms (MLflow, Weights & Biases) Use Case:

• Tag images with model versions
• Quick rollback to previous version

Example:

docker tag llm-model:latest llm-model:v1.2.3
docker tag llm-model:latest llm-model:v1.2.2  # Rollback

5. Development Teams

Company: All companies with ML teams Use Case:

• New developers can run models immediately
• No “works on my machine” issues

Example:

# New developer setup
git clone repo
docker-compose up  # Everything works!

Industry-Standard Boilerplate Code

Production Dockerfile (Industry Standard)

# Multi-stage build for production (used by: Google, AWS, Microsoft)
# Stage 1: Builder
FROM python:3.9-slim as builder

WORKDIR /app

# Install build dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements and install
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# Stage 2: Runtime (minimal image)
FROM python:3.9-slim

WORKDIR /app

# Copy Python dependencies from builder
COPY --from=builder /root/.local /root/.local

# Make sure scripts are in PATH
ENV PATH=/root/.local/bin:$PATH

# Create non-root user (security best practice)
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser

# Copy application
COPY --chown=appuser:appuser . .

# Health check (for K8s, load balancers)
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
    CMD python -c "import requests; requests.get('http://localhost:8000/health')" || exit 1

# Expose port
EXPOSE 8000

# Run application
CMD ["python", "app.py"]

Docker Compose for Development (Industry Standard)

# docker-compose.yml
# Used by: Development teams, local testing
version: '3.8'

services:
  llm-serving:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    environment:
      - MODEL_NAME=gpt2
      - DEVICE=cpu
      - LOG_LEVEL=INFO
    volumes:
      # Mount code for development (hot reload)
      - .:/app
      # Mount models directory
      - ./models:/app/models
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  # Optional: Add monitoring
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

Build & Deploy Script (Industry Standard)

#!/bin/bash
# build-and-deploy.sh
# Used by: CI/CD pipelines, deployment automation

set -e  # Exit on error

IMAGE_NAME="llm-serving"
VERSION="${1:-latest}"
REGISTRY="your-registry.com"  # Docker Hub, ECR, GCR, etc.

echo "Building image: ${IMAGE_NAME}:${VERSION}"

# Build image
docker build -t ${IMAGE_NAME}:${VERSION} .

# Tag for registry
docker tag ${IMAGE_NAME}:${VERSION} ${REGISTRY}/${IMAGE_NAME}:${VERSION}
docker tag ${IMAGE_NAME}:${VERSION} ${REGISTRY}/${IMAGE_NAME}:latest

# Push to registry
docker push ${REGISTRY}/${IMAGE_NAME}:${VERSION}
docker push ${REGISTRY}/${IMAGE_NAME}:latest

echo "Image pushed: ${REGISTRY}/${IMAGE_NAME}:${VERSION}"

# Deploy (example for K8s)
# kubectl set image deployment/llm-serving llm-serving=${REGISTRY}/${IMAGE_NAME}:${VERSION}

GPU Dockerfile (Industry Standard)

# GPU-enabled Dockerfile
# Used by: Production ML serving, high-performance inference
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04

# Install Python
RUN apt-get update && apt-get install -y \
    python3.9 \
    python3-pip \
    curl \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install PyTorch with CUDA
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install other dependencies
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

# Copy application
COPY . .

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

EXPOSE 8000

CMD ["python3", "app.py"]

Usage in Production

# Build for production
docker build -t llm-serving:prod -f Dockerfile.prod .

# Run with GPU
docker run --gpus all -p 8000:8000 llm-serving:prod

# Run with resource limits
docker run \
  --memory="4g" \
  --cpus="2.0" \
  -p 8000:8000 \
  llm-serving:prod

# Run in production (with restart policy)
docker run -d \
  --name llm-serving \
  --restart unless-stopped \
  -p 8000:8000 \
  llm-serving:prod

Key Concepts

Dockerfile Basics

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Multi-Stage Builds

Split build into stages to reduce final image size:

1. Builder stage: Install dependencies, download models
2. Runtime stage: Copy only what’s needed

Layer Caching

Order matters! Put frequently changing files last:

# Good: Dependencies change less often
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .  # Code changes more often

Dockerfile Examples

Basic Dockerfile

See Dockerfile.basic for a simple example.

Optimized Dockerfile

See Dockerfile.optimized for:

• Multi-stage build
• Model caching
• Smaller image size
• Better layer caching

GPU Dockerfile

See Dockerfile.gpu for CUDA/GPU support.

Building Images

Basic Build

docker build -t llm-serving:latest .

With Build Args

docker build \
  --build-arg MODEL_NAME=gpt2 \
  --build-arg PYTHON_VERSION=3.9 \
  -t llm-serving:gpt2 .

Multi-stage Build

docker build -f Dockerfile.optimized -t llm-serving:optimized .

Running Containers

CPU Mode

docker run -p 8000:8000 llm-serving:latest

GPU Mode (NVIDIA Docker)

docker run --gpus all -p 8000:8000 llm-serving:gpu

With Environment Variables

docker run \
  -p 8000:8000 \
  -e MODEL_NAME=gpt2 \
  -e MAX_LENGTH=100 \
  llm-serving:latest

With Volume Mounts (for models)

docker run \
  -p 8000:8000 \
  -v ./models:/app/models \
  llm-serving:latest

Docker Compose

For multi-container setups, use docker-compose.yml:

• Application container
• Model storage
• Monitoring
• Load balancer

Image Optimization

Reduce Image Size

1. Use slim base images: python:3.9-slim instead of python:3.9
2. Multi-stage builds: Don’t include build tools in final image
3. Remove cache: pip install --no-cache-dir
4. Combine RUN commands: Fewer layers = smaller image

Speed Up Builds

1. Layer caching: Order Dockerfile commands by change frequency
2. Build cache: Use --cache-from for CI/CD
3. Parallel builds: Build multiple images simultaneously

Handling Large Models

Option 1: Download in Image

RUN python -c "from transformers import AutoModel; AutoModel.from_pretrained('gpt2')"

Pros: Self-contained Cons: Large image, slow builds

Option 2: Volume Mount

docker run -v ./models:/app/models ...

Pros: Small image, fast builds Cons: Need models on host

Option 3: Model Registry

Download from S3/GCS at runtime Pros: Flexible, versioned Cons: Requires network access

Best Practices

1. Use .dockerignore: Exclude unnecessary files
2. Tag images: Use semantic versioning
3. Health checks: Add HEALTHCHECK instruction
4. Non-root user: Run as non-root for security
5. Resource limits: Set memory/CPU limits
6. Logging: Configure proper logging

Exercises

1. Build basic image: Create Dockerfile for basic serving
2. Optimize image: Reduce image size by 50%
3. Multi-stage build: Create optimized multi-stage Dockerfile
4. GPU support: Add CUDA support to Dockerfile
5. Docker Compose: Create compose file for full stack

Common Issues

Out of Memory

• Problem: Container runs out of memory
• Solution: Increase Docker memory limit or use smaller model

Slow Builds

• Problem: Building takes forever
• Solution: Use layer caching, download models in separate stage

GPU Not Available

• Problem: CUDA not available in container
• Solution: Install nvidia-docker, use --gpus all flag

Large Image Size

• Problem: Image is several GB
• Solution: Use multi-stage build, slim base images

Next Steps

• Topic 3: Deploy containers to Kubernetes
• Topic 4: Load test your containerized app
• Topic 5: Optimize for production with vLLM

Further Reading

• Docker Best Practices
• Multi-stage Builds
• NVIDIA Docker

Docker for GPU LLM Serving

Containerizing an LLM inference service so it runs the same on your laptop, in CI, and on a rented H100 — without a 40 GB image, a leaked token, or a CUDA driver version is insufficient at 3 a.m.

Why this matters

An LLM server is not a normal web app. It links against CUDA, cuDNN, NCCL, and a specific PyTorch build; it needs a physical GPU exposed into the container; and it depends on multi-gigabyte model weights that you do not want to redownload on every restart. Get the containerization wrong and you hit one of a dozen classic failures: the image balloons to tens of gigabytes, the container can’t see the GPU, the CUDA version mismatches the host driver, your Hugging Face token ends up baked into a layer, or the server runs as root with no healthcheck and the orchestrator can’t tell it’s wedged.

Containers are also the unit that Kubernetes, Nomad, ECS, and every autoscaler schedule. A well-built image is the foundation for everything in later chapters (K8s, canary, autoscaling). This chapter is about getting that foundation right — and about what “right” means as of 2026, since this corner of the ecosystem (NVIDIA Container Toolkit, official framework images, and how weights get distributed) has moved substantially in the last two years.

The intuition first, then the exact mechanisms.

Saying it out loud. The reason containerizing an LLM server is different from containerizing a normal web app comes down to two things: it needs a physical GPU handed into the container, and it depends on tens of gigabytes of model weights you really don’t want to redownload on every restart. Get either wrong and you hit a very predictable set of failures — a 40 GB image, a container that can’t see the GPU, a CUDA driver version is insufficient on some nodes but not others, or a Hugging Face token baked permanently into a layer. And this matters beyond convenience, because the image is the unit that Kubernetes and every autoscaler schedules, so a bad image is a bad foundation for everything downstream. The one thing I’d lead with: the driver lives on the host, the container only ships CUDA userspace, and that contract is the source of most GPU container pain.

Core intuition

Three ideas carry most of the weight.

1. The container shares the host’s GPU driver, not its own. You never install an NVIDIA driver inside the image. The driver is a kernel module and lives on the host. The container ships userspace CUDA libraries. At docker run time, the NVIDIA Container Toolkit injects the host driver’s device files and libraries into the container. So the contract is: host driver must be new enough for the container’s CUDA userspace. This is the single most important mental model in GPU containers.

2. Model weights are data, not code. Code changes daily and is tiny; a 14 GB weights file changes rarely and is huge. Baking weights into an image layer couples the two lifecycles badly. The default for production is to keep weights out of the image and bring them in as a mounted volume, a startup download into a persistent cache, or — increasingly, as of 2025–2026 — a separate OCI artifact pulled alongside the image (see “The 2025–2026 landscape” below).

3. Build image ≠ runtime image. The tools you need to compile CUDA kernels (nvcc, headers, build-essential) are hundreds of MB you never need to run the server. Multi-stage builds let you compile in a fat stage and copy only the artifacts into a lean runtime stage.

Hold these three and the rest is detail.

Saying it out loud. Three ideas do most of the work here. First, the container does not have its own GPU driver — the driver is a kernel module on the host, and the container only ships the userspace CUDA libraries, so the host driver has to be new enough for whatever CUDA version you built against. Second, weights are data, not code: your code changes daily and is tiny, a 14 GB checkpoint changes rarely and is huge, so coupling them into one artifact means every one-line code fix redistributes the whole model. Third, the image you build in is not the image you ship — nvcc and headers are gigabytes you need to compile and never need to run. Hold those three and everything else in this chapter is detail.

Mechanism 1 — GPU base images and the CUDA/driver contract

The nvidia/cuda image family

NVIDIA publishes nvidia/cuda images on Docker Hub and NGC. Tags follow the pattern:

nvidia/cuda:<cuda_version>-<flavor>-<os>
# e.g.
nvidia/cuda:12.4.1-runtime-ubuntu22.04
nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
nvidia/cuda:12.4.1-devel-ubuntu22.04

There are three flavors, and picking the wrong one is a common source of bloat:

Rule of thumb: build in devel, ship on runtime. If your framework ships prebuilt wheels (most do), you may not even need devel.

Saying it out loud. NVIDIA publishes three flavors of CUDA base image and picking the wrong one is the most common source of bloat. base is minimal CUDA runtime libraries, a couple hundred megabytes. runtime adds the math libraries like cuBLAS and optionally cuDNN — call it two to three gigabytes, and that’s what you actually ship. devel adds nvcc, headers, and static libraries, which puts you at five to seven gigabytes, and you only need that if you’re compiling CUDA kernels. So the rule is: build in devel, ship on runtime — and pin the minor version explicitly, 12.4.1 not 12, because 12 is a moving target that will silently change under you.

Driver/runtime compatibility

The host driver exposes a maximum supported CUDA version. CUDA has forward compatibility within a major version and minor-version compatibility so that, e.g., a driver supporting CUDA 12.2 can generally run CUDA 12.4 userspace on data-center GPUs via the compat package — but do not rely on this casually. The safe posture:

• Check the host: nvidia-smi prints Driver Version and CUDA Version (the max CUDA the driver supports).
• Pick a container CUDA version ≤ that, or confirm forward-compat coverage.
• Pin the container CUDA minor version explicitly (12.4.1, not 12).

The failure you’re avoiding looks like:

CUDA driver version is insufficient for CUDA runtime version

That means the container’s CUDA userspace is newer than the host driver can serve. Fix by upgrading the host driver or downgrading the image’s CUDA version — you cannot fix it inside the image. (See the “driver skew across a mixed GPU fleet” case study later in this chapter — this exact error is what took down one team’s rollout.)

Saying it out loud. This is the single most important contract in GPU containers: the host’s driver has to be new enough for the CUDA userspace inside your image, and you cannot fix a violation from inside the image. Check the host with nvidia-smi — it reports a driver version and the maximum CUDA version that driver supports — then pick a container CUDA version at or below it, pinned to the minor. If you get it wrong you see exactly one error: CUDA driver version is insufficient for CUDA runtime version, and the only fixes are upgrade the host driver or ship a lower-CUDA image. The nasty version of this failure is a mixed-generation fleet, where a third of your pods come up healthy and two-thirds crash-loop, so it looks like a flaky partial outage rather than a version mismatch.

GPU partitioning: MIG and time-slicing

Two GPUs is not always the right unit of allocation — sometimes you want to run several smaller inference workloads on one physical GPU. NVIDIA data-center GPUs (A100/H100-class) support two partitioning mechanisms, and interviewers like to check you know the difference:

• MIG (Multi-Instance GPU) — hardware-level partitioning. A single A100/H100 can be split into up to 7 fully isolated instances, each with its own dedicated slice of SMs, memory, and memory bandwidth, exposed to the container runtime as a distinct device. Because the isolation is in hardware, one MIG instance’s workload cannot starve or interfere with another’s — the strongest isolation option, but the partition sizes are fixed at profile granularity and set outside the container (via nvidia-smi mig -cgi ... on the host) before containers ever start.
• Time-slicing — software-level sharing. Multiple containers share the same full GPU, and the driver time-multiplexes compute across them (similar in spirit to CPU time-sharing). No memory isolation: any container can, in principle, allocate all the GPU’s VRAM and starve its neighbors. Simpler to set up (no MIG profile management) but a weaker isolation guarantee — appropriate for dev/test or trusted-tenant workloads, not for hard multi-tenant isolation.

For a single dedicated LLM server per GPU (the common case in this chapter’s examples), neither matters — you’re using the whole device. They become relevant once you’re packing multiple smaller models or replicas onto shared GPUs, which is a natural follow-up question after “how do you containerize a model server” in a systems-design interview.

Saying it out loud. If you want to run several small workloads on one physical GPU, there are two ways and they give you very different guarantees. MIG — Multi-Instance GPU — is hardware partitioning: an A100 or H100 splits into up to seven fully isolated instances, each with its own dedicated slice of SMs, memory, and memory bandwidth, and one instance genuinely cannot starve another. Time-slicing is software: the driver just multiplexes compute between containers sharing the whole card, with no memory isolation at all, so one container can allocate all the VRAM and OOM its neighbors. So the tradeoff is real isolation with fixed, host-configured partition sizes versus trivial setup with no guarantees. For hard multi-tenancy you use MIG; time-slicing is for dev, test, or workloads you already trust.

The CUDA forward-compatibility package

The “pin container CUDA ≤ host driver” rule earlier in this section has one documented escape hatch worth knowing by name: NVIDIA ships a CUDA forward-compatibility package for data-center GPUs that lets a container built against a newer CUDA toolkit run on an older driver than would normally be required, by shipping a compatible driver shim inside the container image itself. It is intentionally scoped — it applies to data-center GPU driver branches, not consumer GPUs, and it does not make every combination of “any CUDA version on any driver” work. Treat it as a documented exception for a specific, narrow upgrade-sequencing problem (e.g., you need to ship a newer CUDA-versioned image before every node’s driver has been upgraded yet), not as a reason to stop tracking the driver/CUDA contract explicitly.

Saying it out loud. There’s one documented escape hatch from the “container CUDA must be no newer than the host driver” rule, and it’s worth knowing by name: NVIDIA’s CUDA forward-compatibility package ships a driver shim inside the image so a newer CUDA toolkit can run on an older driver. It’s deliberately narrow, though — it’s scoped to data-center GPU driver branches, not consumer cards, and it does not make arbitrary CUDA-version-on-arbitrary-driver combinations work. The right way to think about it is as a solution to an upgrade-sequencing problem: you need to ship a newer image before every node’s driver has been rolled. It is not a reason to stop tracking the driver contract explicitly, and treating it as one is how you end up debugging a partial outage across a mixed fleet.

Mechanism 2 — The NVIDIA Container Toolkit and --gpus

A plain docker run gives the container no GPU. Two pieces make it work.

Saying it out loud. A plain docker run gives your container zero GPUs — you have to explicitly wire it up, and there are two pieces. On the host you install the NVIDIA Container Toolkit once and register it with the Docker daemon; that’s the thing that mounts the host driver’s libraries and device nodes into the container at startup. Then at run time --gpus all or --gpus '"device=0,1"' selects which devices to expose. Note what you install on the host: the driver, and the toolkit — not the CUDA toolkit, that lives in your image. And the very first diagnostic when a container “can’t see the GPU” is docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi; if that fails, the problem is host-level and you should stop debugging your application.

The toolkit

The NVIDIA Container Toolkit is a set of host packages (nvidia-container-toolkit, the nvidia-ctk CLI, and a runtime shim) that automatically configure a container to use NVIDIA GPUs by mounting the driver libraries and device nodes at startup. You install it on the host, once:

# 1. Add NVIDIA's apt repo (see official install guide for the current key/URL)
sudo apt-get install -y nvidia-container-toolkit

# 2. Wire it into the Docker daemon (writes the "nvidia" runtime into
#    /etc/docker/daemon.json)
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Prerequisite: the NVIDIA driver is already installed on the host. You do not install the CUDA toolkit on the host — only the driver.

Verify the whole chain end-to-end:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If that prints your GPU table, host driver + toolkit + runtime are all good. This is the first thing to run when a GPU container “can’t see the GPU.”

Saying it out loud. The toolkit is a set of host packages plus a runtime shim, and what it actually does is inject the host’s driver libraries and /dev/nvidia* device nodes into your container when it starts. You install it once per host with the package manager, then run nvidia-ctk runtime configure --runtime=docker to write the nvidia runtime into the Docker daemon config, and restart Docker. The prerequisite people trip on is that the NVIDIA driver must already be installed on the host — the toolkit doesn’t install it. And the verification is one command: run nvidia-smi inside a stock CUDA base image with --gpus all. If that prints your GPU table, then driver, toolkit, and runtime registration are all correct, and any remaining problem is yours.

--gpus

Once the toolkit is installed, --gpus selects which GPUs to expose:

--gpus all                 # all GPUs
--gpus '"device=0,1"'      # only GPU 0 and 1 (note the nested quoting)
--gpus 2                   # any 2 GPUs

Under the hood this sets NVIDIA_VISIBLE_DEVICES and triggers the toolkit’s injection hook. Older stacks used --runtime=nvidia plus that env var directly; --gpus is the modern, preferred flag. Some tools (vLLM’s docs) still show --runtime nvidia — it’s equivalent when the runtime is registered.

Forward pointer — CDI. The legacy hook-based injection above is being superseded by the Container Device Interface (CDI), a vendor-neutral, Kubernetes/Podman/Docker-portable way to describe “how to expose device X into a container.” The Container Toolkit has generated CDI specs since v1.14 (2024) and it’s now the recommended path for Podman and for rootless setups. Full detail, with commands, is in “The 2025–2026 landscape” below.

Mechanism 3 — Handling large model weights

This is where architecture decisions bite hardest. You have three classic options, plus a fourth that emerged in 2025.

Saying it out loud. Where the weights live is the biggest architectural decision in this chapter, and there are basically four answers. You can bake them into the image — hermetic and air-gap-friendly, but now your image is 15 to 150 gigabytes and every code change redistributes the whole checkpoint. You can mount them as a volume, which is what the official vLLM and TGI images assume and what most production does. You can download them at startup into a persistent cache, which is smallest and most flexible but pays a multi-gigabyte cold start. Or, newer, you can ship them as a separate OCI artifact versioned independently of the engine. The failure mode I’d name for option C: if the cache volume isn’t actually persistent, you redownload the whole model on every single restart, and that bill is real.

Option A — Bake weights into the image

Copy the weights in during build (COPY ./model /model). The image is fully self-contained: pull it and run, no network, no external volume.

• Pro: hermetic, reproducible, air-gap friendly, one artifact to sign/scan.
• Con: the image is now 15–150 GB. Every push/pull moves all of it. Layer caching is useless once weights change. Registry storage costs balloon. Build context upload is slow.
• Verdict: reserve for small models, air-gapped/regulated deployments, or when the exact weights are part of your release contract.

Option B — Mount weights as a volume

Keep weights on the host / network storage and bind-mount at runtime (-v /data/models:/models). This is what the vLLM and TGI official images assume.

• Pro: small image, weights shared across containers and versions, swap models without rebuilding, fast cold builds.
• Con: image is no longer self-contained; you must provision and pre-populate the volume; in K8s you need a PersistentVolume / hostPath / CSI mount.
• Verdict: the default for most production on a fixed node pool or shared filesystem.

Option C — Download at startup into a cache

The container downloads weights from Hugging Face (or S3/GCS) on first boot into a persistent cache directory, then reuses the cache on restart.

• Pro: smallest image, model chosen by env var, trivially swappable.
• Con: cold start pays a multi-GB download; needs network egress + a token for gated models; if the cache volume isn’t persistent you redownload every restart (a classic and expensive bug); registry outages ≠ HF outages now both matter.
• Verdict: great for dev, experimentation, and autoscaling where a warm cache volume (or a pre-baked node image) hides the download.

Option D (emerging, 2025+) — weights as a separate OCI artifact. Docker’s Model Runner and Hugging Face’s OCI push path distribute weights as an OCI artifact (not an image layer) alongside the serving image, so the model can be docker pull-ed, versioned, and content-addressed independently of the engine — while staying uncompressed on disk for fast mmap loading. This is a genuine fourth point on the spectrum: image-registry ergonomics with volume-mount-like decoupling. Details and citations are in the landscape section below; treat it as complementary to, not a replacement for, Options B/C in most production fleets as of this writing.

The Hugging Face cache — make it persist

The huggingface_hub library caches downloads under HF_HOME (default ~/.cache/huggingface; the hub cache is $HF_HOME/hub). The env vars that matter:

• HF_HOME — root of all HF caches. Set this and everything follows.
• HF_HUB_CACHE (older: HUGGINGFACE_HUB_CACHE) — the model blob cache specifically.
• HF_TOKEN — auth for gated/private models.

The whole point of Options B/C is to mount a volume at the cache path so weights survive container restarts:

# vLLM: cache lives at /root/.cache/huggingface inside the image
-v ~/.cache/huggingface:/root/.cache/huggingface

# TGI: the official image sets HUGGINGFACE_HUB_CACHE=/data, so mount /data
-v $PWD/data:/data

Miss this mount and every docker run redownloads the model — slow, costly, and rate-limit-prone.

Saying it out loud. This is a two-line fix for a genuinely expensive bug. The huggingface_hub library caches downloads under HF_HOME, defaulting to ~/.cache/huggingface, and if you don’t mount a persistent volume at that path, every docker run redownloads the entire model from scratch. The exact path differs by image — vLLM’s official image caches at /root/.cache/huggingface, while TGI sets the cache to /data — so you mount to whichever one your image actually uses. Set HF_HOME explicitly and everything else follows, and pass HF_TOKEN at runtime for gated models rather than baking it into a layer. The cost of missing this isn’t just slow starts: it’s bandwidth charges and getting rate-limited by the Hub at exactly the moment you’re scaling up.

Quantization formats and their effect on image and weight size

Weight size is not a fixed property of a model — the quantization format you choose shifts it by 2–4x, which feeds directly back into the bake/mount/download decision above. A 70B-parameter model is roughly 140 GB in fp16, roughly 70 GB in int8, and commonly 35–40 GB in 4-bit formats (GPTQ, AWQ, or GGUF’s Q4_K_M-style schemes). This matters for containerization in three concrete ways:

• Bake-into-image (Option A) becomes far more viable for a 4-bit-quantized model than for the fp16 original — a 35 GB image is still large, but it is a very different operational proposition than a 140 GB one, and may cross the threshold into “acceptable for our registry and node bandwidth.”
• The serving engine constrains the format. vLLM, TGI, and TensorRT-LLM each support a different subset of quantization schemes with different kernel-level performance, so “which quantization format” and “which base image” are coupled decisions — you can’t pick a format your engine’s image doesn’t have kernels for and expect the speed benefit to materialize.
• Quantized weights still deserve the same weights-vs-code separation as the fp16 case — the file is smaller, but it is still large, slow-changing data, and belongs on a mounted volume or content-addressed artifact rather than baked into a layer that gets rebuilt every time the application code changes, for the same CI/registry-cost reasons covered in the war stories below.

Saying it out loud. Weight size isn’t a fixed property of a model — quantization moves it by two to four times, and that feeds straight back into the bake-versus-mount decision. A 70B model is roughly 140 GB in fp16, about 70 GB in int8, and commonly 35 to 40 GB in 4-bit schemes like GPTQ, AWQ, or GGUF’s Q4_K_M. That difference can genuinely flip “bake into the image” from absurd to merely large. Two catches worth naming. Your serving engine constrains the format — vLLM, TGI, and TensorRT-LLM each support different subsets with different kernel performance, so format and base image are a coupled decision. And even at 35 GB, weights are still slow-changing bulk data, so they still belong on a volume or an artifact rather than in a layer your CI rebuilds on every commit.

Mechanism 4 — Multi-stage builds, layer caching, and .dockerignore

Multi-stage

A multi-stage build uses multiple FROM statements. Early stages compile; the final stage copies only what’s needed. For an LLM server that means: compile custom kernels / install a heavy build toolchain in a devel stage, then COPY --from=builder the installed environment into a slim runtime stage. The devel layers never ship.

Saying it out loud. A multi-stage build just means multiple FROM statements in one Dockerfile: an early stage does the heavy work, and the final stage copies only the finished artifacts. For an LLM server that’s concrete — install into a virtualenv in a devel base that has nvcc and build-essential, then COPY --from=builder /opt/venv into a slim runtime base and never ship the compilers. The win is straightforward: you’re dropping three to four gigabytes of build toolchain that has no business being on a production node, and every one of those packages is CVE surface you’d otherwise have to answer for in a security review. Cost is essentially nothing — a slightly longer Dockerfile.

Targeting specific GPU architectures: TORCH_CUDA_ARCH_LIST

When a build stage compiles CUDA kernels from source (custom ops, or building a framework from source rather than installing a prebuilt wheel), it by default may compile for a broad list of GPU compute capabilities so the resulting wheel works everywhere — L4, L40S, A100, H100, and older architectures alike. That breadth costs real build time and binary size, and most of it is wasted if you know exactly which GPU SKU the image will run on. Setting TORCH_CUDA_ARCH_LIST (PyTorch’s build-time env var) to only the architectures you actually deploy on narrows the compiled kernel set accordingly:

# Example: building only for A100 (8.0) and H100 (9.0) compute capabilities,
# instead of the full default list PyTorch would otherwise target.
ENV TORCH_CUDA_ARCH_LIST="8.0 9.0"
RUN pip install --no-binary :all: torch

This is a build-stage-only concern — it has no effect once you’re installing prebuilt wheels (the common, recommended path from Mechanism 4) — but it’s worth knowing by name for the case where you are compiling from source, since it’s a direct, fast answer to “why is our custom-kernel build so slow” or “why is this wheel so large” in a design discussion.

Saying it out loud. If you’re compiling CUDA kernels from source, PyTorch defaults to targeting a broad list of GPU compute capabilities so the resulting wheel works on everything from an older card to an H100. That breadth costs real build time and binary size, and it’s mostly wasted if you know exactly which SKUs you deploy on. Setting TORCH_CUDA_ARCH_LIST to just the architectures you actually run — say "8.0 9.0" for A100 and H100 — narrows the compiled kernel set to those. The important caveat: this only matters in a build stage that actually compiles. If you’re installing prebuilt wheels, which is the recommended path, it does nothing. But it’s a fast, specific answer to “why is our custom-kernel build twenty minutes long.”

Layer caching order

Docker caches each instruction as a layer and reuses it until an input changes; everything after a changed layer is rebuilt. So order from least-to-most volatile:

1. Base image
2. System packages (apt-get)
3. Dependency manifests (requirements.txt / pyproject.toml) and pip install
4. Application source code

Copy requirements.txt and install before copying your source. Then a one-line code change reuses the (slow) dependency layer instead of reinstalling PyTorch every build. Use BuildKit cache mounts for the pip/uv cache to speed rebuilds further:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

Saying it out loud. Docker caches each instruction as a layer and reuses it until an input changes — and crucially, everything after a changed layer gets rebuilt. So you order your Dockerfile from least volatile to most volatile: base image, then system packages, then dependency manifests and pip install, then finally your application source. The concrete win is copying requirements.txt and installing before you copy your code, so a one-line code change reuses the cached PyTorch install instead of reinstalling several gigabytes. Add a BuildKit cache mount on the pip cache and even a genuine dependency change gets faster. Get the order backwards and every trivial commit reinstalls your entire CUDA-linked dependency tree — that’s minutes per build, on every build.

Registry-backed BuildKit cache in CI

Layer caching (above) only helps within a single machine’s local Docker cache. CI runners are frequently ephemeral or shared across many jobs, so the local cache that made your dependency layer fast on your laptop may not exist on the runner that picks up the next PR — which is exactly the trap that made War story 2’s 40 GB image so painful in CI. BuildKit’s registry cache backend fixes this by pushing/pulling the build cache itself as a registry artifact, so any runner can warm its cache from the last successful build regardless of which machine ran it:

docker buildx build \
  --cache-from type=registry,ref=myregistry.example.com/my-llm-server:buildcache \
  --cache-to   type=registry,ref=myregistry.example.com/my-llm-server:buildcache,mode=max \
  -t my-llm-server:ci .

mode=max caches every intermediate layer (not just the final stage), which matters for multi-stage Dockerfiles like the one in this chapter — otherwise the builder stage’s expensive pip install layer isn’t reusable across runners even though the final runtime-stage layers are. Combined with keeping weights out of the build context entirely (Mechanism 3), this is what keeps CI build times in the range War story 2’s fix achieved (under two minutes) rather than the 25-plus minutes the baked-weights version suffered.

Saying it out loud. Layer caching only helps on a machine that still has the cache, and CI runners are typically ephemeral or shared — so the dependency layer that’s instant on your laptop is a cold build on whichever runner picks up the next PR. The fix is BuildKit’s registry cache backend: push the build cache itself to your registry as an artifact with --cache-to type=registry, and any runner can warm from the last successful build with --cache-from. Use mode=max so intermediate stages get cached too, otherwise your builder stage’s expensive pip install isn’t reusable at all in a multi-stage Dockerfile. Combined with keeping weights out of the build context, this is the difference between the 25-plus-minute CI builds in war story two and builds under two minutes.

.dockerignore

The build context is everything Docker uploads to the daemon before building. Without a .dockerignore, a stray ./models, .git, __pycache__, or a 30 GB checkpoint gets shipped into the build — slow, and a vector for secrets and bloat. A minimal one:

.git
__pycache__/
*.pyc
*.pt
*.safetensors
models/
data/
.env
*.log
.venv/

This is also your first line of defense against COPY . . accidentally baking weights or a .env file into a layer.

Saying it out loud. The build context is everything Docker uploads to the daemon before it even starts building, and without a .dockerignore that includes your .git directory, your __pycache__, your .env file, and any stray checkpoint sitting in models/. So it’s slow and it’s a secret-leak vector. A minimal one excludes .git, *.pyc, *.pt, *.safetensors, models/, data/, and .env. Think of it as the first line of defense against a careless COPY . . baking a 30 GB checkpoint or a live Hugging Face token into a layer that then gets pushed to a registry — and remember that once a secret is in image history, deleting the file in a later layer does not remove it.

Mechanism 5 — Reproducibility, security, healthchecks, config

Pin everything. python:3.11-slim is a moving target. Prefer a digest for the base and pinned versions for packages:

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04@sha256:<digest>

Pin your Python deps (lockfile or ==), and tag your own images with an immutable version, never rely on :latest in production.

Run as non-root. By default containers run as UID 0. A container escape from root is worse than from an unprivileged user. Create a user and drop to it:

RUN useradd --create-home --uid 10001 appuser
USER appuser

Note some GPU stacks and cache paths assume /root; if you run non-root, point HF_HOME at a directory that user can write.

Keep secrets out of layers. Never ENV HF_TOKEN=hf_xxx or COPY .env — both persist in the image history for anyone who pulls it. Pass secrets at runtime (-e HF_TOKEN=..., Docker/K8s secrets) or use BuildKit --secret mounts for build-time-only credentials.

Add a HEALTHCHECK. Orchestrators restart containers that fail their health probe. For an OpenAI-compatible server, probe the health/models endpoint:

HEALTHCHECK --interval=30s --timeout=5s --start-period=180s --retries=3 \
  CMD curl -fsS http://localhost:8000/health || exit 1

The long start-period matters: model load can take minutes, and you don’t want the container killed during warmup.

Config via env, not baked files. Model id, tensor-parallel size, port, and dtype should be env vars / CLI args so one image serves many configs.

Saying it out loud. Four habits separate a demo image from a production one. Pin everything — base image by digest, Python deps by lockfile, your own image by an immutable version tag, never :latest. Run as non-root, because a container escape from UID 0 is a much worse day than one from an unprivileged user. Keep secrets out of layers entirely: no ENV HF_TOKEN=, no COPY .env — pass them at runtime or use BuildKit secret mounts, because image history is forever. And add a HEALTHCHECK with a generous start-period, something like 180 to 300 seconds, because model load genuinely takes minutes and the classic self-inflicted failure is your orchestrator killing the container while it’s still legitimately warming up.

Mechanism 6 — Operating the container: logs, exec, monitoring, and shutdown

Building the image is half the job; running it in production for months is the other half. A few operational mechanics are specific to GPU containers and worth knowing cold.

Saying it out loud. Building the image is half the job; running it for months is the other half, and a few things here are GPU-specific. Log to stdout in structured JSON so an aggregator can read it, not to a file inside the container nobody mounts out. When something’s wrong, docker exec in and run nvidia-smi from inside the container — it shows exactly which processes in this container are holding GPU memory. Watch GPU memory utilization, not just compute, because a server can look completely idle on the compute graph while sitting at 95% VRAM. And give yourself a real shutdown grace period, because Docker’s default is ten seconds and that’s not enough to drain an in-flight generation.

Logs and exec

docker logs -f <container> streams stdout/stderr — make sure your server logs there (not to a file inside the container that nobody mounts out) and emits structured (JSON) lines so they’re parseable by whatever aggregator sits downstream. For live debugging, docker exec -it <container> bash drops you inside the running container, where the single most useful diagnostic is running nvidia-smi from inside the container: it reflects the same driver and devices as the host (since the toolkit injected them), and its process list shows exactly which processes inside this container are holding GPU memory — invaluable when a server reports OOM but you’re not sure if it’s fragmentation, a leaked previous request’s KV cache, or another process entirely.

docker exec -it my-llm-server nvidia-smi        # GPU state as seen by *this* container
docker exec -it my-llm-server nvidia-smi pmon   # per-process GPU utilization inside the container

Saying it out loud. Two everyday tools. docker logs -f streams stdout and stderr, which is why your server should log there rather than to a file inside the container that no aggregator will ever see — and structured JSON lines make that downstream parsing actually work. Then docker exec -it <container> bash drops you inside a running container, and the single most useful thing to run there is nvidia-smi. It reflects the same driver and devices as the host because the toolkit injected them, and its process list tells you precisely which processes in this container are holding GPU memory. That’s what lets you distinguish a genuine OOM from fragmentation from some other process on the card — a distinction you cannot make from the host view alone.

Monitoring GPU utilization and health from outside the container

For host-level and fleet-level observability, don’t rely on shelling into every container. Two standard options:

• nvidia-smi dmon on the host gives a live per-GPU utilization/memory/temperature stream — useful for a quick manual check, not for durable metrics.
• DCGM (Data Center GPU Manager) exporter — NVIDIA’s dcgm-exporter runs as a sidecar or daemonset and exposes GPU utilization, memory, ECC errors, power, and temperature as Prometheus metrics, scoped per physical GPU and (with the right labels) attributable back to the container/pod using it. This is the standard way to get GPU metrics into the same dashboards and alerting as the rest of your fleet, rather than parsing nvidia-smi text output on a cron job.

The metric to alert on that’s easy to miss: GPU memory utilization, not just compute utilization. A server can show low SM utilization (looks “idle”) while sitting at 95% VRAM usage from an oversized KV cache or a memory leak across requests — the next allocation OOMs with no warning if you were only watching compute.

Saying it out loud. Don’t build your GPU observability on shelling into containers. For a quick manual look, nvidia-smi dmon on the host gives you a live per-GPU stream. For anything durable, you run NVIDIA’s dcgm-exporter as a sidecar or daemonset and it exposes utilization, memory, ECC errors, power, and temperature as Prometheus metrics you can label back to the owning pod. The metric that’s easy to miss, and the one I’d insist on alerting: GPU memory utilization, separately from compute. A leaking KV cache will push you to 95% VRAM while SM utilization stays boringly flat, and then the next allocation OOMs with zero warning on the dashboard everyone was watching.

Resource limits: what cgroups do and do not control for GPUs

The deploy.resources.limits/reservations block shown earlier in this chapter’s compose file controls CPU and memory through the host’s cgroups — Docker genuinely enforces those. It does not give you an equivalent hard limit on GPU compute or GPU memory the way it does for CPU shares or RAM. --gpus controls which GPUs a container can see, not how much of a shared GPU’s compute or memory it’s capped at using cgroups semantics. Practically:

• If you run one container per GPU (the common pattern for LLM serving, since a large model typically wants a whole device or several via tensor-parallel), this limitation doesn’t bite — the container has the whole GPU and there’s nothing else to contend with.
• If you deliberately share a GPU across containers (time-slicing, or just running two processes on one device without MIG), nothing at the container-runtime layer stops one from allocating all the VRAM and OOM-ing the other. NVIDIA’s MPS (Multi-Process Service) can give more predictable compute sharing between cooperating processes, and MIG (above) gives hardware-enforced isolation — but plain Docker resource limits do not extend to the GPU the way they do to CPU/RAM. Don’t assume mem_limit or cpus in your compose file constrains GPU memory; it doesn’t.

Saying it out loud. This one catches people: the CPU and memory limits in your compose file are enforced by the host’s cgroups and are genuinely real, but there is no cgroup equivalent for the GPU. --gpus controls which devices a container can see, not how much of a shared device it’s allowed to consume. So if you run one container per GPU — the normal pattern for LLM serving, since a big model wants a whole card or several — this never bites you, because there’s nothing to contend with. But if you deliberately share a card between containers, nothing at the runtime layer stops one from allocating all the VRAM and OOM-ing the other. MPS gives more predictable compute sharing between cooperating processes; MIG gives hardware-enforced isolation. mem_limit gives you nothing.

Graceful shutdown and restart policy

docker stop sends SIGTERM, waits a grace period (default 10s, configurable with docker stop -t <seconds> or stop_grace_period in compose), then SIGKILLs. For an LLM server mid-generation, 10 seconds is often not enough to drain in-flight streaming requests cleanly. Two adjustments matter:

• Increase the grace period (stop_grace_period: 60s in compose, or the orchestrator’s equivalent — e.g., Kubernetes terminationGracePeriodSeconds) to comfortably exceed your longest expected generation, so in-flight requests finish rather than being cut off mid-stream.
• Handle SIGTERM in the application itself if the framework allows it — stop accepting new requests immediately (so the load balancer’s health/readiness check can flip and stop routing new traffic) while letting in-flight requests complete within the grace window, rather than dropping everything the instant SIGTERM arrives.

Restart policy (restart: unless-stopped used in this chapter’s compose file, or on-failure with a max retry count) determines what happens after a crash. For a GPU server, prefer a policy with a capped retry count or backoff over unconditional restart-forever: a driver mismatch or an OOM that recurs deterministically on every restart will otherwise crash-loop indefinitely, burning GPU-node scheduling slots and generating alert noise, when what you actually want after N failures is for the orchestrator to mark the replica unhealthy and stop retrying until a human looks at it.

Saying it out loud. docker stop sends SIGTERM, waits ten seconds by default, then SIGKILLs — and ten seconds is often not enough for an LLM server to finish streaming an in-flight generation. So you do two things. Raise the grace period past your longest expected generation, stop_grace_period in compose or terminationGracePeriodSeconds in Kubernetes. And handle SIGTERM in the app: immediately stop accepting new work so the readiness check flips and the load balancer drains you, while letting in-flight requests finish inside the window. On restart policy, prefer a capped retry or backoff over restart-forever, because a driver mismatch or a deterministic OOM will crash-loop indefinitely, burning scarce GPU scheduling slots and generating alert noise instead of getting a human’s attention.

A debugging playbook for “the container won’t come up”

When a GPU container fails on a node and the cause isn’t obvious from the first log line, work through this order — cheapest checks first:

1. Does the toolkit even see the GPU? docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi on the same host. If this fails, the problem is host-level (toolkit/runtime/driver), not your image — stop debugging the application.
2. Does the driver support the image’s CUDA version? Compare nvidia-smi’s reported CUDA Version against the image’s CUDA tag. A driver-insufficient error here means fix the host or ship a lower-CUDA image; it is never fixed inside the container.
3. Is /dev/shm large enough? If the crash is a Bus error or an NCCL hang rather than a CUDA-init failure, suspect the 64 MB default shared-memory size before suspecting the model or the framework — check with docker exec <c> df -h /dev/shm.
4. Is the cache volume actually mounted, and writable by the running user? A silent full redownload on every restart, or a permission-denied on first write, both trace back to a missing or misowned mount at the HF_HOME/cache path.
5. Is the healthcheck timing out during legitimate model load, or is the process actually wedged? Check docker logs -f for load-progress output and compare elapsed time against start-period; don’t assume a failed healthcheck means a hung process without checking whether it’s simply still loading.
6. Is this node’s driver actually the one you tested against? On a mixed-generation fleet, confirm the specific node’s driver version rather than assuming fleet-wide uniformity — this is the failure mode from War story 1 below, and it is easy to lose an hour to before checking it directly.

Saying it out loud. When a GPU container won’t start, work cheapest-check-first instead of reading the application logs. Step one: does a stock CUDA base image with --gpus all print nvidia-smi on this host? If not, it’s host-level — toolkit, runtime, or driver — and your image is irrelevant. Step two: compare the driver’s reported CUDA version against your image’s CUDA tag. Step three: if the symptom is a Bus error or an NCCL hang rather than a CUDA init failure, suspect the 64-megabyte default /dev/shm before you suspect the model. Step four: check the cache volume is actually mounted and writable by the running user. Step five: check whether the healthcheck is failing because the process is wedged or because it’s simply still loading — those look identical from the outside and only the logs distinguish them.

The 2025–2026 landscape

The mechanics above (driver contract, toolkit, multi-stage) haven’t changed. What has changed since roughly 2024 is how much of this you have to build yourself versus what ships as a hardened, official artifact — and how seriously the industry now treats GPU images as a supply-chain surface. Four threads matter for a working engineer today.

Saying it out loud. The mechanics — driver contract, toolkit, multi-stage builds — haven’t changed. What’s changed since about 2024 is how much of this you build yourself. The toolkit is moving from its old hook-based injection to CDI, a vendor-neutral device spec that Docker, Podman, containerd, and Kubernetes all understand, which finally makes rootless GPU containers practical. Most teams now start from an official vendor image — vLLM’s, TGI’s, or NVIDIA NIM — instead of hand-rolling a Dockerfile. And GPU images are now treated as a supply-chain surface in their own right, because a CUDA plus PyTorch base drags in a huge dependency graph. The interview-relevant version: know when a vendor image is the right answer, and know that “custom Dockerfile” now needs a justification.

1. The NVIDIA Container Toolkit has moved to CDI, and rootless is now a real option

The toolkit itself keeps shipping — v1.19.0 was released March 12, 2026 (see the release list at the project’s GitHub, cited below). The bigger shift is architectural: the legacy nvidia-container-runtime “hook” approach (patch the OCI runtime spec at container-start time) is being superseded by the Container Device Interface (CDI), a CNCF-adjacent, vendor-neutral spec for describing “how to inject device X into this container” that Docker, Podman, containerd, and Kubernetes (via kubelet device plugins) all understand the same way. NVIDIA’s toolkit has generated CDI specs since v1.14 (2024), and NVIDIA’s own docs now present CDI as the forward-looking path, especially for Podman and rootless setups.

Rootless GPU containers were awkward for years because /dev/nvidia* device nodes and the injection hook both assumed a privileged (rootful) daemon. CDI plus nvidia-ctk closes that gap. The rootless recipe (Podman, but the same idea applies to Docker’s rootless mode) is:

# Generate a CDI spec into user space (not /etc/cdi, which needs root)
mkdir -p ~/.config/cdi
nvidia-ctk cdi generate --output=$HOME/.config/cdi/nvidia.yaml

# Sanity-check what devices the spec exposes
nvidia-ctk cdi list

# Run rootless, referencing the CDI device by its vendor.com/class=name identifier
podman --cdi-spec-dir=$HOME/.config/cdi run --rm \
  --device nvidia.com/gpu=all \
  --security-opt=label=disable \
  nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Two caveats that bite in practice: (1) rootless containers still need the invoking user to have read/write on the /dev/nvidia* device files — usually via membership in the host’s video (or render) group, passed through with --group-add keep-groups; and (2) on SELinux hosts (Fedora/RHEL-family) you may need sudo setsebool -P container_use_devices on or GPU access is silently denied. Rootless matters for the same reason it always has — a compromised process inside the container can’t leverage root-equivalent privileges on the host — and it is now something you can actually put in a hardening checklist rather than an aspiration.

Saying it out loud. CDI — the Container Device Interface — is a vendor-neutral way of describing how to inject a device into a container, and Docker, Podman, containerd, and Kubernetes all understand the same spec. That replaces the old approach where NVIDIA’s runtime patched the OCI spec via a hook at container start. The practical payoff is rootless: for years rootless GPU containers were awkward because the device nodes and the injection hook both assumed a privileged daemon, and nvidia-ctk cdi generate into your user’s config directory closes that gap. Two gotchas that will waste your afternoon: the invoking user still needs read-write on /dev/nvidia*, usually via the video or render group, and on SELinux hosts you need container_use_devices on or GPU access is silently denied.

2. Official, hardened images are the default path for the mainstream servers

Five years ago most teams hand-rolled a Dockerfile like the one in this chapter. Today, for the two most common serving stacks, you usually start from the vendor’s image and only write a custom Dockerfile when you have a genuinely custom serving path:

• vLLM publishes vllm/vllm-openai and documents the docker run --runtime nvidia --gpus all ... invocation directly (docs.vllm.ai/en/latest/deployment/docker/). The known trade-off: the image is large — community reports and vLLM’s own GitHub issue tracker put a recent tag at roughly 12.6 GB (vLLM Forums thread “Current vLLM docker image size is 12.64Gb”; GitHub issue vllm-project/vllm#27154, “How to reduce the vllm image”). A vLLM maintainer’s response to a from-source slimming attempt was blunt: “Probably not as of yet, too many changes to manually work. Even then not guaranteed to work yet.” Translation for your own builds: don’t assume you can easily out-slim the officially supported image without maintenance burden — budget registry storage and pull-time accordingly, or accept the size as the cost of using the maintained artifact.
• Hugging Face TGI ships ghcr.io/huggingface/text-generation-inference with GPU install docs at huggingface.co/docs/text-generation-inference/en/installation_nvidia, following the same “official image + mounted volume” pattern shown earlier in this chapter.
• NVIDIA NIM (developer.nvidia.com/nim) goes a step further: it packages pre-optimized inference engines — TensorRT-LLM, vLLM, SGLang builds tuned per GPU SKU — behind an OpenAI-compatible API, distributed as containers you self-host (via NGC) or consume as managed endpoints on Hugging Face. NIM trades some flexibility (you’re consuming a curated engine build, often gated behind an NGC API key / NVIDIA AI Enterprise entitlement for production use) for meaningfully less Dockerfile authoring and tuning work; it’s worth evaluating before writing a bespoke image for a well-known model architecture.

The interview-relevant takeaway: know when to reach for an official/vendor image (the common case now) versus when a custom multi-stage build is actually warranted (custom serving logic, a framework without an official image, or hard constraints on image size/content that the vendor image doesn’t meet).

Saying it out loud. Five years ago everyone hand-rolled a Dockerfile like the one in this chapter; today you usually start from the vendor’s image and only write your own when you have genuinely custom serving logic. vLLM publishes vllm/vllm-openai, Hugging Face publishes TGI on GHCR, and NVIDIA NIM goes further by packaging pre-optimized engines tuned per GPU SKU behind an OpenAI-compatible API. The honest tradeoff is size — a recent vLLM tag runs around 12.6 gigabytes, and a maintainer’s own answer to “can I slim it” was essentially “not really, and not without ongoing maintenance burden.” So budget registry storage and pull time rather than fighting it. The thing to be able to say in an interview is which specific constraint would push you off the official image.

3. Image-size reduction has two live approaches: harden the base, or stop shipping weights as layers

Two independent techniques are gaining traction for the “these images are enormous” problem:

• Hardened, minimal base images. Chainguard publishes CUDA/PyTorch-family images built to a zero-known-CVE target, rebuilt daily, each with an SBOM and reproducible from signed build configs. Their own comparison (chainguard.dev/unchained/securing-the-foundations-of-ai-applications-with-chainguard-images) found the official PyTorch Docker Hub image carried 1 critical, 23 high, 1,189 medium, and 72 low CVEs (as measured July 24, 2024) against zero in their equivalent image at the same time, driven mostly by stripping unnecessary OS packages rather than by removing CUDA/PyTorch functionality. This is the same “runtime not devel, --no-install-recommends, clean apt lists” discipline from Mechanism 4/5 in this chapter, taken to its logical extreme by a vendor who productizes it.
• Stop treating weights as image layers at all. Docker’s rationale for packaging AI models as OCI artifacts rather than image layers (docker.com/blog/oci-artifacts-for-ai-model-packaging) is worth understanding even if you don’t adopt Docker Model Runner: model weight files are high-entropy, so compressing them into a tar layer (the normal image-layer behavior) buys negligible size reduction while costing real (de)compression time, and it prevents the inference engine from mmap-ing the file directly off disk. Docker’s model-artifact manifest instead stores the weights as an uncompressed layer under model-specific media types (e.g., application/vnd.docker.ai.gguf.v3) alongside a JSON config carrying architecture/quantization/parameter-count metadata — decoupling “which weights” from “which engine” so you don’t duplicate a 15 GB checkpoint across every framework’s image. It is early days for this pattern in mainstream LLM-serving production, but the direction — weights as a distinct, content-addressed, uncompressed OCI artifact rather than baked into devel/runtime layers — is the one to watch, and it’s a clean answer to “is there a better option than bake/mount/download?” if it comes up in an interview.

Saying it out loud. Two different attacks on “these images are enormous.” One is hardening the base: Chainguard builds CUDA and PyTorch images to a zero-known-CVE target with daily rebuilds and SBOMs, and their comparison against the official PyTorch Docker Hub image found 1 critical, 23 high, and over 1,100 medium CVEs there versus zero in theirs, mostly by stripping unnecessary OS packages rather than removing functionality. The other is refusing to ship weights as layers at all. Weight files are high-entropy, so compressing them into a tar layer buys almost nothing in size while costing real decompression time and preventing the engine from mmap-ing them straight off disk. That’s Docker’s argument for OCI model artifacts: uncompressed, content-addressed, and decoupled from whichever engine you’re running.

4. Supply-chain scanning is now expected on AI images specifically, not just on your app images

Because a CUDA + PyTorch + framework image drags in an unusually large OS + Python dependency graph, it accumulates CVEs faster than a typical microservice image — which is exactly why the Chainguard comparison above is so stark. The practical response teams are standardizing on in 2025–2026:

• Docker Scout (docker scout cves <image>, docs.docker.com/reference/cli/docker/scout/cves/) or Trivy as a CI gate — fail the build (or at least the “promote to prod” step) above a critical/high CVE threshold. Docker’s own “Docker Hardened Images” program (docs.docker.com/dhi/how-to/scan/) packages this scanning workflow for a curated base-image catalog.
• SBOM generation (docker sbom, syft, or Chainguard’s built-in SBOMs) attached to the image so a security team can answer “are we exposed to CVE-XXXX” without re-scanning every running container.
• Image signing (cosign / Sigstore) so the deployment pipeline can verify the image it’s about to run on a GPU node actually came from your CI, not a tampered registry mirror.
• Practically, because GPU images are large and slow to scan/pull, teams increasingly scan once at build/push time and verify signature + digest at deploy time, rather than re-scanning on every node — the same “shift left” idea as regular container security, adjusted for the fact these images are 10–100x the size of a typical service image.

None of this replaces the fundamentals earlier in this chapter (multi-stage, .dockerignore, non-root, pinned digests). It’s the layer on top: assume your CUDA/PyTorch base has a nontrivial CVE surface, and have an explicit story — vendor-hardened base, CI scanning gate, or both — for managing it, rather than discovering it during a customer security questionnaire.

Saying it out loud. A CUDA plus PyTorch plus framework image pulls in a far bigger OS and Python dependency graph than a typical microservice, so it accumulates CVEs faster — which is why that Chainguard comparison is so lopsided. The standard response has three parts: a scanning gate in CI with Docker Scout or Trivy that fails the promote step above a critical or high threshold, an SBOM attached to the image so security can answer “are we exposed to this CVE” without re-scanning running containers, and cosign signing so the deploy pipeline verifies the image actually came from your CI. The practical adaptation for GPU images specifically: because they’re ten to a hundred times the size of a normal service image, you scan once at push time and verify signature and digest at deploy, rather than re-scanning on every node.

5. Kubernetes-adjacent: the GPU Operator abstracts the host-side setup

Everything in Mechanism 2 (install the toolkit, configure the Docker/containerd runtime, verify with nvidia-smi) is manual host configuration. On a Kubernetes fleet, the NVIDIA GPU Operator packages that entire host-side setup — driver installation/management, container toolkit, device plugin, DCGM monitoring, and (as of the 25.10 release line) CDI support — as a set of components the cluster manages itself, so individual node bootstrap scripts stop being where GPU readiness lives. This chapter deliberately stays at the single-host docker run/compose level; the GPU Operator (and the rest of the Kubernetes-specific device-plugin and scheduling story) belongs to this guide’s Kubernetes chapter, but it’s worth knowing the name and roughly what it replaces before that chapter goes deep, since an interviewer moving from “containerize this” to “now put it in a cluster” is testing whether you know the boundary between the two layers.

Saying it out loud. Everything in the toolkit section — install the packages, configure the container runtime, verify with nvidia-smi — is manual host configuration, and on a Kubernetes fleet you stop doing that by hand. The NVIDIA GPU Operator packages the whole host-side story: driver management, the container toolkit, the device plugin, DCGM monitoring, and CDI support, all managed by the cluster itself, so GPU readiness stops living in per-node bootstrap scripts. It’s worth knowing the name and roughly what it replaces before you get to the Kubernetes chapter, because an interviewer moving from “containerize this” to “now put it in a cluster” is specifically testing whether you know where the boundary between those two layers is.

Build it in practice — extended

The Dockerfile below is the same pattern shown earlier in this chapter (build in devel, ship on runtime, non-root, healthcheck, exec-form entrypoint), reproduced here as the anchor for the compose and CI additions that follow.

# syntax=docker/dockerfile:1.7

###############################################################################
# Stage 1: builder — has nvcc + build tools, compiles/installs the env.
###############################################################################
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builder

ENV DEBIAN_FRONTEND=noninteractive \
    PIP_NO_CACHE_DIR=0 \
    PYTHONDONTWRITEBYTECODE=1

# System build deps. Pin, clean apt lists to keep the layer lean.
RUN apt-get update && apt-get install -y --no-install-recommends \
        python3.11 python3.11-venv python3-pip build-essential git \
    && rm -rf /var/lib/apt/lists/*

# Isolated virtualenv so we can copy the whole thing to the runtime stage.
RUN python3.11 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Dependency layer FIRST — cached across code changes.
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --upgrade pip && pip install -r requirements.txt

###############################################################################
# Stage 2: runtime — slim CUDA runtime, no compilers, non-root, healthcheck.
###############################################################################
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 AS runtime

ENV DEBIAN_FRONTEND=noninteractive \
    PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    # Persist weights here; mount a volume at this path (see docker run).
    HF_HOME=/models/hf

# Only the runtime OS deps: python + curl (for healthcheck). No build-essential.
RUN apt-get update && apt-get install -y --no-install-recommends \
        python3.11 curl \
    && rm -rf /var/lib/apt/lists/*

# Copy the fully-built virtualenv from the builder stage — no nvcc ships.
COPY --from=builder /opt/venv /opt/venv

# Non-root user that can write the cache dir.
RUN useradd --create-home --uid 10001 appuser \
    && mkdir -p /models/hf && chown -R appuser:appuser /models
COPY --chown=appuser:appuser ./app /app
WORKDIR /app
USER appuser

EXPOSE 8000

# Config comes from env / CLI at runtime, not baked in.
ENV MODEL_ID=meta-llama/Llama-3.1-8B-Instruct \
    TENSOR_PARALLEL_SIZE=1

HEALTHCHECK --interval=30s --timeout=5s --start-period=300s --retries=3 \
    CMD curl -fsS http://localhost:8000/health || exit 1

# Exec form so signals reach the process (clean shutdown).
ENTRYPOINT ["python3.11", "-m", "vllm.entrypoints.openai.api_server"]
CMD ["--host", "0.0.0.0", "--port", "8000"]

Build and run:

# Build (BuildKit on for cache mounts + syntax directive)
DOCKER_BUILDKIT=1 docker build -t my-llm-server:1.0.0 .

# Run: expose GPUs, mount the HF cache volume, pass secrets at runtime.
docker run --rm \
  --gpus all \                                  # expose all GPUs (toolkit required)
  --ipc=host \                                  # shared mem for NCCL / TP; see note
  -p 8000:8000 \
  -v $HOME/.cache/hf:/models/hf \               # persist weights across restarts
  -e HF_TOKEN=$HF_TOKEN \                        # gated-model auth, NOT baked in
  my-llm-server:1.0.0 \
  --model meta-llama/Llama-3.1-8B-Instruct \    # config via CLI
  --tensor-parallel-size 1

Why --ipc=host? vLLM (and PyTorch tensor-parallel generally) uses shared memory (/dev/shm) for inter-process/GPU communication. Docker’s default /dev/shm is 64 MB, which causes cryptic crashes or hangs under load. --ipc=host (or --shm-size=1g) gives it room. TGI uses --shm-size 1g for the same reason.

Reference: the official one-liner most teams actually start from —

# vLLM official image
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env HF_TOKEN=$HF_TOKEN -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest --model mistralai/Mistral-7B-Instruct-v0.2

# TGI official image
docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id teknium/OpenHermes-2.5-Mistral-7B

Saying it out loud. If I had to describe the production Dockerfile in one breath: build in a devel CUDA base into an isolated virtualenv, copy just that virtualenv into a slim cudnn-runtime stage, create a non-root user that owns the cache directory, set a healthcheck with a five-minute start period, and use exec-form ENTRYPOINT so signals actually reach the process. Then at run time you pass --gpus all, mount the weights cache volume, and inject the token as an environment variable rather than baking it. The one flag people forget is --ipc=host or --shm-size=1g — Docker’s default shared memory is 64 megabytes, and tensor-parallel NCCL communication needs far more, so without it you get cryptic hangs and Bus error crashes that look nothing like a shared-memory problem.

The full stack: LLM server + reverse proxy + resource limits

A single docker run is fine for a dev box. A production compose file needs at minimum: the model server, a reverse proxy in front of it (TLS termination, request buffering, and a stable port even if you swap the backend image), and explicit resource limits so one runaway container can’t starve its neighbors on a shared host. Here is a more complete docker-compose.yaml:

services:
  llm:
    image: my-llm-server:1.0.0
    restart: unless-stopped
    expose:
      - "8000"                      # only reachable from other compose services, not the host
    ipc: host                       # equivalent to --ipc=host
    environment:
      - HF_TOKEN=${HF_TOKEN}        # sourced from host env / .env, not in image
      - MODEL_ID=meta-llama/Llama-3.1-8B-Instruct
    volumes:
      - hf-cache:/models/hf         # persistent named volume for weights
    command: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
    deploy:
      resources:
        limits:
          cpus: "8"                 # cap host CPU this container can use
          memory: 32g                # cap host RAM (guards against OOM-killing neighbors)
        reservations:
          cpus: "4"
          memory: 16g
          devices:
            - driver: nvidia
              count: all             # or `device_ids: ["0","1"]`
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      start_period: 300s
      retries: 3

  proxy:
    image: caddy:2.8-alpine
    restart: unless-stopped
    depends_on:
      llm:
        condition: service_healthy   # don't take traffic until the model is loaded
    ports:
      - "443:443"                    # only the proxy is exposed to the host/internet
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy-data:/data
    deploy:
      resources:
        limits:
          cpus: "1"
          memory: 512m

volumes:
  hf-cache:
  caddy-data:

A minimal Caddyfile fronting the model server — TLS, a request timeout longer than your typical generation latency, and a size cap on request bodies:

llm.example.com {
    reverse_proxy llm:8000 {
        # Streaming responses (SSE) need this off, or you buffer the whole stream.
        flush_interval -1
    }
    timeout 300s
    request_body {
        max_size 2MB
    }
}

Two details that matter more than they look: expose (not ports) on the llm service means the model server is reachable only from the proxy service on the compose network, not directly from the host or internet — the proxy is the only public entry point, which is where you’d add auth, rate limiting, and TLS. And depends_on: condition: service_healthy means the proxy won’t route traffic to the model server until its HEALTHCHECK passes — closing the “requests arrive during the multi-minute model load and get 502s” gap.

Saying it out loud. A single docker run is fine on a dev box; production wants at least three things in the compose file. The model server itself, exposed only internally rather than published to the host. A reverse proxy in front for TLS termination, request buffering, and a stable port so you can swap the backend image underneath. And explicit CPU and memory limits so one runaway container can’t starve its neighbors. The detail that saves you an incident: gate the proxy on the model server’s healthcheck with depends_on: condition: service_healthy, so callers get a clean connection refusal instead of a wall of 502s during the several minutes the model is loading. And remember those resource limits cover CPU and RAM only — they do nothing for GPU memory.

Adding GPU observability to the stack

The compose file above is functionally complete but operationally blind — nothing exports GPU metrics. Adding NVIDIA’s DCGM exporter as a sidecar makes GPU utilization, memory, and temperature visible to Prometheus/Grafana without touching the llm service:

  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
    restart: unless-stopped
    ports:
      - "9400:9400"                 # scrape target for Prometheus
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Point Prometheus at dcgm-exporter:9400/metrics and alert on GPU memory utilization specifically, not just SM/compute utilization — a server can look idle on compute while sitting dangerously close to an out-of-memory KV-cache allocation, and compute-only dashboards miss that entirely (see Mechanism 6’s monitoring note above).

Saying it out loud. A working compose file is still operationally blind — nothing in it exports a single GPU metric. Adding NVIDIA’s DCGM exporter as a sidecar container fixes that without touching your serving service at all: it exposes GPU utilization, memory, temperature, power, and ECC errors on a port Prometheus can scrape. The reason to do this at compose time rather than later is that the interesting failures are gradual — a KV-cache leak creeping up over days, thermal throttling under sustained load — and you cannot see a trend you never started recording. The specific alert I’d add first is GPU memory utilization with a warning threshold well under 100%, so you get runway before a hard out-of-memory rather than an alert that arrives with the crash.

CI smoke-test stage

GPU-backed CI runners are expensive and often unavailable, so most teams cannot run a full inference smoke test on every commit. The pragmatic middle ground: build the image, verify it starts and imports correctly on a CPU-only runner (catching the large class of bugs that have nothing to do with the GPU — bad pip install, broken entrypoint, missing env var, syntax errors), and reserve full GPU inference smoke tests for a nightly job or a pre-deploy gate on a real GPU runner.

# .github/workflows/docker-build.yml
name: build-and-smoke-test

on: [pull_request]

jobs:
  build:
    runs-on: ubuntu-latest         # CPU-only runner: no --gpus here
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: DOCKER_BUILDKIT=1 docker build -t my-llm-server:ci .

      - name: Scan image (fail on critical/high CVEs)
        run: docker scout cves my-llm-server:ci --exit-code --only-severity critical,high

      - name: Smoke test — container starts, imports resolve, CLI parses args
        run: |
          # No GPU on this runner: --help exits before touching CUDA, so this
          # catches broken installs / bad entrypoints without needing a GPU.
          docker run --rm my-llm-server:ci --help

      - name: Smoke test — process boots far enough to hit the arg parser
        run: |
          docker run --rm --entrypoint python3.11 my-llm-server:ci \
            -c "import vllm; print(vllm.__version__)"

  gpu-smoke-test:
    if: github.event_name == 'workflow_dispatch'   # manual / nightly, on a real GPU runner
    runs-on: [self-hosted, gpu]
    needs: build
    steps:
      - name: Full smoke test — load a tiny model, hit /health and /v1/completions
        run: |
          docker run -d --rm --gpus all --ipc=host -p 8000:8000 \
            --name smoke my-llm-server:ci --model facebook/opt-125m
          for i in $(seq 1 60); do
            curl -fsS http://localhost:8000/health && break
            sleep 5
          done
          curl -fsS http://localhost:8000/v1/models
          docker stop smoke

The split matters: the CPU job runs on every PR and catches most regressions cheaply; the GPU job runs on a real GPU runner against a tiny model (opt-125m, not your 70B production model) so the full request path — container start, healthcheck, an actual generation call — is exercised without needing an expensive GPU for every commit.

Saying it out loud. GPU CI runners are expensive and often unavailable, so the pragmatic split is two jobs. On every pull request, build the image and verify it starts and imports correctly on a plain CPU runner — that catches the large class of bugs that have nothing to do with the GPU: a broken pip install, a bad entrypoint, a missing environment variable, a syntax error. Then reserve a real GPU runner for a nightly or pre-promote job that runs an actual generation call against a tiny model like opt-125m, not your 70B production checkpoint. That exercises the full path — container start, healthcheck, real inference — for a couple of cents instead of an expensive GPU-hour on every commit.

Weights-handling comparison

Pin the model revision (commit SHA) or artifact digest, not just the repo name or tag, when reproducibility matters for any of these options.

A note that cuts across every row of this table: the quantization format (see Mechanism 3’s earlier note) shifts where each option’s numbers land — a 4-bit-quantized 70B model bakes into an image at a size that would have been unthinkable in fp16, which is why “bake vs. mount vs. download” is a decision you should revisit per model/format combination, not settle once for the whole fleet.

Saying it out loud. If someone asks me to compare the weight-handling options, I’d frame it as a tradeoff between self-containment and lifecycle coupling. Baking into the image gives you the best reproducibility and works air-gapped, but you’re at 15 to 150 gigabytes and every code change redistributes the model. Mounting a volume is the production default — small image, fast start, models shared across containers — but the image is no longer self-contained and you have to provision the volume. Downloading at startup is the most flexible and smallest, but a cold start pays a multi-gigabyte download and now Hugging Face’s uptime is in your critical path. The rule that applies to all of them: pin the model revision by commit SHA or artifact digest, not the repo name — a tag can move under you.

Failure modes and pitfalls

• Driver/runtime mismatch — CUDA driver version is insufficient. Container CUDA newer than host driver supports. Fix host driver or lower image CUDA; can’t be patched in the image. On a mixed-node fleet, this can appear on some nodes and not others — see the case study below.
• GPU invisible — torch.cuda.is_available() is False, no nvidia-smi in container. Toolkit not installed, --gpus omitted, or runtime not registered. Test with docker run --gpus all nvidia/cuda:...-base nvidia-smi.
• Giant images — shipped the devel base, apt lists left behind, weights baked in, or pip cache retained. Use multi-stage, --no-install-recommends, clean /var/lib/apt/lists, and keep weights out. If you still land at 10+ GB (common with official framework images), that may simply be the cost of the maintained artifact — budget registry/pull time rather than fighting it alone.
• Redownloading weights every restart — cache path not mounted to a persistent volume, or HF_HOME points at an ephemeral dir. Mount a named volume at the cache path.
• Secrets in layers — ENV HF_TOKEN=... or COPY .env persists in image history forever. Pass at runtime or use BuildKit secrets.
• Running as root — default UID 0; a bad default for security and for shared-filesystem permissions. Create and drop to a non-root user. On rootless Podman/CDI setups, also check video/render group membership and SELinux container_use_devices — GPU access can silently fail even when the container itself is configured correctly.
• No / bad healthcheck — orchestrator can’t detect a wedged server, or kills it during a 3-minute model load. Add a healthcheck with a generous start-period, and gate your reverse proxy on it (depends_on: condition: service_healthy) so requests don’t 502 during warmup.
• /dev/shm too small — default 64 MB causes NCCL/tensor-parallel hangs and Bus error. Use --ipc=host or --shm-size.
• :latest everywhere — non-reproducible builds and surprise upgrades. Pin base by digest, deps by version, your image by semver.
• Signals ignored — shell-form CMD runs under /bin/sh which doesn’t forward SIGTERM; use exec-form ENTRYPOINT/CMD so shutdown is graceful and requests drain.
• Unscanned CVE surface — CUDA/PyTorch images pull in a much larger OS + Python dependency graph than a typical service image, so they accumulate CVEs faster; shipping without a CI scanning gate (Docker Scout / Trivy) or a hardened base (e.g., Chainguard) is a common 2025–2026-era gap that shows up in security reviews, not in functional testing.
• CI “passed” but the image never boots on GPU — a CPU-only CI runner that only checks --help/import succeeds doesn’t exercise CUDA initialization at all. Pair it with a periodic real-GPU smoke test (see the CI section above) or you’ll ship a CUDA-init bug straight to prod.
• no kernel image is available for execution on the device — a from-source build’s TORCH_CUDA_ARCH_LIST didn’t include the compute capability of the GPU you’re actually running on (e.g., built for 8.0/9.0, deployed on an older 7.5-class card). Fixed by rebuilding with the right architecture list, or by using a prebuilt wheel that already covers it — not a runtime-configurable option.
• GPU memory creep goes unnoticed — dashboards only track SM/compute utilization, which can look healthy (or idle) right up until an allocation fails. Track GPU memory utilization per container (DCGM exporter) as a first-class metric, not an afterthought.
• Crash-loop storm from an unconditional restart policy — a deterministic failure (driver mismatch, OOM on every boot) combined with restart: always and no backoff burns node scheduling slots and pages on-call repeatedly instead of failing fast. Cap retries or use on-failure with a backoff, and treat “restarted N times in M minutes” as its own alert.

Saying it out loud. The failure list here is short and repeats across every team. Driver mismatch — container CUDA newer than the host driver supports, unfixable from inside the image. GPU invisible, which means the toolkit isn’t installed or --gpus was omitted. Giant images from shipping the devel base or baking weights. Redownloading weights every restart because the cache path isn’t on a persistent volume. Secrets in layers, which live in image history forever. Running as root by default. And /dev/shm too small at 64 megabytes, which produces NCCL hangs and Bus error crashes that look nothing like a shared-memory problem. The pattern worth naming: almost all of these are silent or misleading at the symptom level — the error you see is rarely the layer where the bug lives.

Tools and options comparison