LLM Serving & Inference: Complete Learning Repository

A comprehensive, hands-on guide to learning LLM serving and inference from fundamentals to production deployment.

🎯 What You’ll Learn

This repository teaches you everything about serving Large Language Models in production:

• How LLM inference works (tokenization, attention, generation)
• Basic serving (FastAPI, model loading, endpoints)
• Containerization (Docker for ML workloads)
• Kubernetes deployment (production infrastructure)
• Performance optimization (vLLM, batching, GPU utilization)
• Scaling (autoscaling, load balancing)
• Production practices (monitoring, versioning, drift detection)

📁 Repository Structure

mlops_serving/
├── LEARNING_PATH.md          # Start here! Complete learning guide
├── docs/                     # Detailed concept explanations
│   ├── llm_inference_fundamentals.md
│   ├── serving_architectures.md
│   └── optimization_techniques.md
├── 01_basic_serving/         # FastAPI + HuggingFace serving
├── 02_docker/                # Containerization
├── 03_kubernetes/            # K8s deployment
├── 04_load_testing/          # Performance testing
├── 05_vllm_serving/          # High-performance serving
├── 06_autoscaling/           # K8s autoscaling
├── 07_canary_deployments/    # Gradual rollouts
├── 08_monitoring/            # Prometheus + Grafana
├── 09_model_versioning/      # Version management
├── 10_drift_detection/       # Evidently integration
└── 11_triton/                # Multi-model serving

🚀 Quick Start

👉 START HERE: Read HOW_TO_START.md for a complete step-by-step guide!

Quick Commands

# 1. Set up environment
cd 01_basic_serving
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

# 2. Start the server
python app.py

# 3. Test it (in another terminal)
python test_api.py
# OR visit http://localhost:8000/docs for interactive API docs

📚 Learning Approach

This is a hands-on, topic-based learning path:

1. Each topic is self-contained with code, docs, and exercises
2. Build incrementally - each topic builds on previous concepts
3. Learn by doing - run code, modify it, break it, fix it
4. Focus on understanding - not just copying code

🎓 Prerequisites

• Python 3.9+
• Basic Python knowledge
• Understanding of REST APIs
• Docker (for containerization topics)
• Kubernetes cluster (minikube/kind for K8s topics)
• GPU (optional, helpful for vLLM)

📖 Topics Covered

Core Concepts

• LLM architecture and inference pipeline
• Tokenization and vocabulary
• Attention mechanisms and KV caching
• Memory and computation requirements

Serving Basics

• Model loading and initialization
• API endpoint design
• Request/response handling
• Error handling and validation

Infrastructure

• Docker containerization
• Kubernetes deployment
• Health checks and probes
• Resource management

Performance

• Load testing and benchmarking
• Latency vs throughput
• vLLM optimization
• GPU utilization

Production

• Autoscaling strategies
• Canary deployments
• Monitoring and observability
• Model versioning
• Drift detection

🔧 Technologies

• FastAPI: Web framework
• HuggingFace Transformers: Model loading
• vLLM: High-performance inference
• Docker: Containerization
• Kubernetes: Orchestration
• Locust: Load testing
• Prometheus: Metrics
• Grafana: Visualization
• Evidently: Drift detection
• Triton: Multi-model serving

📝 How to Use This Repository

1. Start with LEARNING_PATH.md - Understand the structure
2. Read the fundamentals - docs/llm_inference_fundamentals.md
3. Work through topics in order - Each numbered topic builds on the last
4. Experiment and modify - Don’t just run code, change it and learn
5. Read the code comments - They explain the “why” not just the “what”
6. Check solutions - See EXERCISE_SOLUTIONS.md for exercise answers
7. Prepare for interviews - Review INTERVIEW_QA.md for common questions

🎯 Learning Goals

By completing this repository, you’ll be able to:

• Understand how LLM inference works internally
• Build production-ready serving APIs
• Deploy models to Kubernetes
• Optimize for performance and cost
• Monitor and debug serving systems
• Handle model updates and drift

🤝 Contributing

This is a learning repository. Feel free to:

• Add more examples
• Improve documentation
• Fix bugs
• Share your learnings

📄 License

MIT License - Feel free to use this for learning and teaching.

📖 Getting Started

1. New to this? → Read HOW_TO_START.md for step-by-step instructions
2. Want overview? → Read LEARNING_PATH.md to see all topics
3. Ready to code? → Start with 01_basic_serving/
4. Need help? → Check EXERCISE_SOLUTIONS.md for exercise answers
5. Interview prep? → Review INTERVIEW_QA.md for common questions

Ready to start? Open HOW_TO_START.md and begin your journey! 🚀

📚 Additional Resources

• Exercise Solutions: EXERCISE_SOLUTIONS.md - Answers to all exercises
• Interview Q&A: INTERVIEW_QA.md - 23+ interview questions with detailed answers
• Quick Reference: QUICK_REFERENCE.md - Quick commands and tips
• Complete Topics: COMPLETE_TOPICS.md - Overview of all 12 topics
• Industry Boilerplate: Each topic README includes industry-standard code and use cases

🏭 Industry Focus

Each topic now includes:

• Why We Need This: Business and technical justification
• Industry Use Cases: Real-world applications (5+ examples per topic)
• Boilerplate Code: Production-ready, industry-standard implementations
• Company Examples: How major companies use these technologies

Topics updated with industry content:

• ✅ Topic 1: Basic Serving (Customer support, content generation)
• ✅ Topic 2: Docker (Multi-cloud, CI/CD, edge deployment)
• ✅ Topic 3: Kubernetes (Large-scale platforms, enterprise ML)
• ✅ Topic 5: vLLM (High-throughput APIs, cost optimization)
• ✅ Topic 8: Monitoring (Production observability, incident response)

👋 Welcome! Start Here

This repository teaches you everything about LLM serving and inference through hands-on, practical examples.

🎯 What You’ll Learn

You’ll learn how to:

• Serve LLMs in production
• Optimize for performance
• Deploy to Kubernetes
• Monitor and debug systems
• Handle model updates safely
• Detect and handle drift

📚 How This Repository is Organized

Learning Structure

• Topics are numbered (01, 02, 03…) - work through them in order
• Each topic is self-contained - has its own code, docs, and examples
• Builds incrementally - each topic builds on previous concepts

Key Files

🚀 Your First Steps

1. Read the Start Guide

cat HOW_TO_START.md

This has everything you need to begin.

2. Understand the Fundamentals

cat docs/llm_inference_fundamentals.md

Learn how LLM inference works internally.

3. Run Your First Server

cd 01_basic_serving
pip install -r requirements.txt
python app.py

4. Test It

# In another terminal
python test_api.py

📖 Learning Topics

1. LLM Inference Fundamentals - How LLMs work
2. Basic Serving - FastAPI + HuggingFace
3. Containerization - Docker
4. Kubernetes - K8s deployment
5. Load Testing - Performance measurement
6. vLLM Serving - High-performance serving
7. Autoscaling - Scale automatically
8. Canary Deployments - Safe rollouts
9. Monitoring - Metrics and dashboards
10. Model Versioning - Version management
11. Drift Detection - Detect issues
12. Triton - Multi-model serving

💡 Learning Approach

1. Read the documentation
2. Study the code
3. Run the examples
4. Modify and experiment
5. Move to the next topic

✅ Prerequisites

• Python 3.9+
• Basic Python knowledge
• Understanding of REST APIs
• (Optional) Docker and Kubernetes

Don’t have Docker/K8s? No problem - you can learn the basics without them!

🎓 Ready to Start?

👉 Open HOW_TO_START.md and follow the step-by-step guide!

Questions? Check the README.md in each topic directory for detailed explanations.

Stuck? Read error messages carefully, check the docs, and experiment with simpler examples first.

Let’s learn! 🚀

How to Start Learning LLM Serving & Inference

🎯 Your Learning Journey Starts Here

This guide will walk you through exactly how to start learning LLM serving and inference, step by step.

📋 Prerequisites Check

Before you start, make sure you have:

• Python 3.9 or higher (python --version)
• pip installed
• Basic understanding of Python
• Basic understanding of REST APIs
• (Optional) Docker installed
• (Optional) Kubernetes cluster (minikube/kind)

Don’t worry if you don’t have Docker/K8s yet - you can learn the basics without them!

🚀 Step-by-Step Learning Path

Step 1: Understand the Fundamentals (30-60 minutes)

Read this first: docs/llm_inference_fundamentals.md

This explains:

• How LLMs work internally
• What happens during inference
• Tokenization, attention, generation
• Memory and computation requirements

Why this matters: You need to understand what’s happening under the hood before you can serve models effectively.

Action: Open the file and read through it. Don’t worry if you don’t understand everything - you’ll learn more as you build.

Step 2: Set Up Your Environment (10 minutes)

# Navigate to the project
cd /Users/faisal/Projects/mlops_serving

# Create a virtual environment (recommended)
python -m venv venv

# Activate it
source venv/bin/activate  # On Mac/Linux
# OR
venv\Scripts\activate  # On Windows

# Install dependencies for basic serving
cd 01_basic_serving
pip install -r requirements.txt

What this does: Sets up an isolated Python environment with all the libraries you need.

Step 3: Run Your First LLM Server (5 minutes)

# Make sure you're in 01_basic_serving directory
cd 01_basic_serving

# Start the server
python app.py

You should see:

INFO:     Started server process
INFO:     Waiting for application startup.
🚀 Starting up: Loading LLM model...
Loading model: gpt2 on device: cpu
✅ Model loaded successfully!
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000

What’s happening:

• The server is loading a GPT-2 model (small, fast model for learning)
• It’s starting a FastAPI web server
• The model is now ready to serve requests

Step 4: Test the API (5 minutes)

Option A: Use the test script

# In a new terminal (keep server running)
cd 01_basic_serving
python test_api.py

Option B: Use curl

# Health check
curl http://localhost:8000/health

# Generate text
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_length": 50,
    "temperature": 0.7
  }'

Option C: Use the interactive docs Open http://localhost:8000/docs in your browser - FastAPI provides an interactive API explorer!

Step 5: Understand What Just Happened (15 minutes)

Read: 01_basic_serving/README.md

This explains:

• What each file does
• How the model loading works
• How the API endpoints work
• Key concepts you just used

Then explore the code:

• model_loader.py - How models are loaded and used
• app.py - How the API is structured

Try modifying:

• Change the model name (try “distilgpt2” for a smaller model)
• Change temperature (0.1 = more deterministic, 2.0 = more creative)
• Add logging to see what’s happening

Step 6: Learn Each Topic in Order

Now that you’ve run your first server, work through each topic:

1. 01_basic_serving ✅ (You just did this!)
2. 02_docker - Containerize your app
3. 03_kubernetes - Deploy to K8s
4. 04_load_testing - Test performance
5. 05_vllm_serving - High-performance serving
6. 06_autoscaling - Scale automatically
7. 07_canary_deployments - Safe rollouts
8. 08_monitoring - Track metrics
9. 09_model_versioning - Manage versions
10. 10_drift_detection - Detect issues
11. 11_triton - Multi-model serving

For each topic:

1. Read the README.md
2. Study the code
3. Run the examples
4. Modify and experiment
5. Move to the next topic

🎓 Learning Tips

1. Don’t Rush

Understanding is more important than speed. Take time to:

• Read error messages carefully
• Experiment with parameters
• Break things and fix them

2. Experiment

After running each example:

• Change parameters
• Modify the code
• See what breaks
• Understand why

3. Use the Documentation

Each topic has:

• README.md explaining concepts
• Code comments explaining “why”
• Examples you can run

4. Ask Questions

As you learn, ask yourself:

• “Why does this work this way?”
• “What happens if I change X?”
• “How does this scale?”
• “What could go wrong?”

🐛 Common Issues & Solutions

Issue: “Model not found” or download errors

Solution: Check your internet connection. Models are downloaded from HuggingFace on first use.

Issue: “Out of memory”

Solution:

• Use a smaller model (distilgpt2 instead of gpt2)
• Reduce max_length
• Close other applications

Issue: “Port already in use”

Solution:

• Stop the previous server (Ctrl+C)
• Or change the port in app.py

Issue: “Import errors”

Solution:

• Make sure virtual environment is activated
• Run pip install -r requirements.txt again

📊 What You’ll Learn

By the end of this journey, you’ll understand:

Core Concepts

• ✅ How LLM inference works internally
• ✅ Tokenization and vocabulary
• ✅ Attention mechanisms
• ✅ Memory and computation requirements

Serving Skills

• ✅ Building serving APIs
• ✅ Containerization
• ✅ Kubernetes deployment
• ✅ Performance optimization

Production Skills

• ✅ Monitoring and observability
• ✅ Scaling strategies
• ✅ Safe deployments
• ✅ Drift detection

🎯 Next Steps

1. Right now: Complete Steps 1-5 above
2. Today: Read through 01_basic_serving/README.md and understand the code
3. This week: Work through topics 2-4 (Docker, K8s, Load Testing)
4. This month: Complete topics 5-8 (vLLM, Scaling, Monitoring)
5. Ongoing: Topics 9-11 (Advanced production topics)

❓ Questions?

If you get stuck:

1. Check the README.md in each topic
2. Read error messages carefully
3. Check the docs/ directory for detailed explanations
4. Experiment with simpler examples first

Remember: Learning by doing is the best way. Don’t just read - run the code, modify it, break it, fix it!

Ready? Let’s start! 🚀

Begin with Step 1: Read docs/llm_inference_fundamentals.md

LLM Serving & Inference: Complete Learning Guide

🎯 How to Use This Guide

This guide is organized by learning topics, not time periods. Work through each topic at your own pace. Each topic builds on the previous one, but you can also jump to specific areas you want to learn.

📚 Learning Topics (In Order)

1. LLM Inference Fundamentals

What you’ll learn: How LLMs actually work under the hood

• Tokenization and vocabulary
• Forward pass through transformer layers
• Autoregressive generation
• Attention mechanism and KV caching
• Memory and computation requirements

Start here: docs/llm_inference_fundamentals.md

2. Basic Model Serving

What you’ll learn: Serve a model with a simple API

• Loading models from HuggingFace
• Building FastAPI endpoints
• Request/response handling
• Basic error handling

Practice: 01_basic_serving/

3. Containerization

What you’ll learn: Package your serving application

• Docker basics for ML models
• Multi-stage builds
• Handling large model files
• Environment configuration

Practice: 02_docker/

4. Kubernetes Deployment

What you’ll learn: Deploy to production infrastructure

• K8s manifests for ML workloads
• Health checks and probes
• Resource limits and requests
• ConfigMaps and Secrets

Practice: 03_kubernetes/

5. Load Testing & Performance

What you’ll learn: Measure and understand performance

• Latency vs throughput
• Load testing with Locust
• Performance profiling
• Identifying bottlenecks

Practice: 04_load_testing/

6. High-Performance Serving (vLLM)

What you’ll learn: Optimize for production throughput

• vLLM architecture
• Continuous batching
• PagedAttention
• GPU optimization

Practice: 05_vllm_serving/

7. Autoscaling

What you’ll learn: Scale based on demand

• Horizontal Pod Autoscaling (HPA)
• Request-based scaling
• Concurrency metrics
• Scaling strategies

Practice: 06_autoscaling/

8. Monitoring & Observability

What you’ll learn: Track your serving system

• Prometheus metrics
• Grafana dashboards
• GPU monitoring
• Logging and tracing

Practice: 08_monitoring/

9. Canary Deployments

What you’ll learn: Safely roll out model updates

• Traffic splitting
• A/B testing models
• Gradual rollouts
• Rollback procedures

Practice: 09_canary_deployments/

10. Model Versioning

What you’ll learn: Manage multiple model versions

• Version management strategies
• Model registry
• Rollback procedures
• A/B testing infrastructure

Practice: 10_model_versioning/

11. Drift Detection

What you’ll learn: Detect when models degrade

• Data drift detection
• Concept drift
• Evidently AI integration
• Alerting on anomalies

Practice: 11_drift_detection/

12. Multi-Model Serving (Triton)

What you’ll learn: Serve multiple models efficiently

• NVIDIA Triton Inference Server
• Dynamic batching
• Model ensembles
• Multi-framework support

Practice: 12_triton/

🚀 Quick Start Guide

Step 1: Understand the Basics

Read docs/llm_inference_fundamentals.md to understand how LLM inference works.

Step 2: Set Up Your Environment

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install base dependencies
pip install -r requirements.txt

Step 3: Start with Basic Serving

cd 01_basic_serving
python app.py
# Test it: curl http://localhost:8000/health

Step 4: Progress Through Topics

Work through each numbered topic in order. Each includes:

• README.md: Explanation of concepts
• Code examples: Working implementations
• Exercises: Hands-on practice

🎓 Learning Approach

For Each Topic:

1. Read the documentation - Understand the concepts
2. Study the code - See how it’s implemented
3. Run the examples - Get hands-on experience
4. Modify and experiment - Break things, fix them, learn
5. Move to next topic - Build on what you learned

Tips:

• Don’t rush: Understanding > Speed
• Experiment: Change parameters, break things, learn why
• Read error messages: They teach you a lot
• Use the docs: Each topic has detailed explanations

📖 Prerequisites

Required:

• Python 3.9+
• Basic Python knowledge
• Understanding of REST APIs
• Docker basics

Helpful but not required:

• Kubernetes experience
• ML/AI background
• GPU access (CPU works for learning)

🔧 Technology Stack

You’ll learn these tools:

• FastAPI: Web framework
• HuggingFace Transformers: Model loading
• vLLM: High-performance inference
• Docker: Containerization
• Kubernetes: Orchestration
• Locust: Load testing
• Prometheus/Grafana: Monitoring
• Evidently: Drift detection
• Triton: Multi-model serving

❓ Common Questions

Q: Do I need a GPU? A: Not for the basics. GPU helps with vLLM and production workloads, but you can learn on CPU.

Q: How long will this take? A: Depends on your pace. Each topic can take a few hours to a few days. Focus on understanding, not speed.

Q: Can I skip topics? A: The basics (1-5) should be done in order. Advanced topics (6-12) can be done based on interest.

Q: What if I get stuck? A: Check the docs, read error messages carefully, experiment with simpler examples first.

🎯 Learning Goals

By the end, you’ll be able to:

• ✅ Serve LLMs in production
• ✅ Optimize for performance
• ✅ Deploy to Kubernetes
• ✅ Monitor and debug serving systems
• ✅ Handle model updates safely
• ✅ Detect and handle model drift

Let’s start learning! 🚀

Quick Reference: vLLM, Latency, and Grafana

🚀 Quick Start Guide

1. vLLM Serving

Install vLLM

# Requires CUDA/GPU
pip install vllm

Start vLLM Server

cd 05_vllm_serving
pip install -r requirements.txt
python vllm_server.py

Or use vLLM’s built-in server

python -m vllm.entrypoints.openai.api_server \
    --model gpt2 \
    --port 8000

Test vLLM API

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="gpt2",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=50
)

print(response.choices[0].message.content)

2. Measure Latency

Option A: Using Locust (Interactive)

cd 04_load_testing
pip install -r requirements.txt

# Start your server first
# Then run Locust
locust -f locust_test.py --host=http://localhost:8000

# Open http://localhost:8089 in browser

Option B: Using Python Script (Headless)

cd 04_load_testing
python measure_latency.py --url http://localhost:8000 --requests 100

What You Get

• P50 (Median): 50% of requests faster
• P95: 95% of requests faster
• P99: 99% of requests faster
• Throughput: Requests per second

Compare Endpoints

python measure_latency.py --compare
# Compares basic serving vs vLLM

3. Grafana Monitoring

Start Monitoring Stack

cd 08_monitoring
docker-compose up -d

This starts:

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Add Metrics to Your App

from prometheus_exporter import add_prometheus_middleware

app = FastAPI()
add_prometheus_middleware(app)  # That's it!

Your app now exposes /metrics endpoint.

Configure Prometheus

Edit prometheus.yml to scrape your app:

scrape_configs:
  - job_name: 'llm-serving'
    static_configs:
      - targets: ['localhost:8000']

Create Grafana Dashboard

1. Open Grafana: http://localhost:3000
2. Login: admin/admin
3. Add data source: Prometheus (http://prometheus:9090)
4. Create dashboard with these queries:

Request Rate:

rate(llm_requests_total[5m])

Latency (P95):

histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m]))

Error Rate:

rate(llm_requests_total{status="500"}[5m]) / rate(llm_requests_total[5m])

GPU Utilization:

llm_gpu_utilization_percent

📊 Complete Workflow Example

Step 1: Start vLLM Server

cd 05_vllm_serving
python vllm_server.py
# Server running on http://localhost:8000

Step 2: Measure Latency

cd 04_load_testing
python measure_latency.py --url http://localhost:8000 --requests 100

Output:

P50:  150ms
P95:  450ms
P99:  800ms
Throughput: 25.3 req/s

Step 3: Start Monitoring

cd 08_monitoring
docker-compose up -d

Step 4: View Metrics

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000

Step 5: Load Test with Monitoring

# Terminal 1: Watch Grafana
open http://localhost:3000

# Terminal 2: Run load test
cd 04_load_testing
locust -f locust_test.py --host=http://localhost:8000 --headless --users 10 --spawn-rate 2

🔍 Key Metrics to Monitor

Application Metrics

• llm_requests_total - Total requests
• llm_request_duration_seconds - Request latency
• llm_active_requests - Currently processing
• llm_tokens_generated_total - Tokens generated

System Metrics

• llm_gpu_utilization_percent - GPU usage
• llm_gpu_memory_used_bytes - GPU memory
• llm_gpu_temperature_celsius - GPU temperature

Important Queries

Requests per second:

rate(llm_requests_total[5m])

95th percentile latency:

histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m]))

Error percentage:

rate(llm_requests_total{status="500"}[5m]) / rate(llm_requests_total[5m]) * 100

GPU memory usage:

llm_gpu_memory_used_bytes / llm_gpu_memory_total_bytes * 100

🎯 Performance Targets

Good Performance

• P50 latency: < 200ms
• P95 latency: < 500ms
• P99 latency: < 1s
• Error rate: < 0.1%
• GPU utilization: 70-90%

Warning Signs

• P99 > 2s: System struggling
• Error rate > 1%: Problems occurring
• GPU utilization < 50%: Underutilized
• GPU utilization > 95%: Overloaded

🐛 Troubleshooting

vLLM won’t start

• Check CUDA: python -c "import torch; print(torch.cuda.is_available())"
• Install correct CUDA version
• Use smaller model for testing

High latency

• Check GPU utilization
• Reduce batch size
• Use smaller model
• Check for bottlenecks (CPU, memory, I/O)

No metrics in Grafana

• Check Prometheus is scraping: http://localhost:9090/targets
• Verify /metrics endpoint works: curl http://localhost:8000/metrics
• Check Prometheus config points to correct host

GPU metrics missing

• Install pynvml: pip install pynvml
• Check GPU is accessible
• Verify NVIDIA drivers installed

📚 Learn More

• vLLM: 05_vllm_serving/README.md
• Latency: 04_load_testing/README.md
• Monitoring: 08_monitoring/README.md

Quick tip: Start with basic serving, measure latency, then optimize with vLLM, and monitor with Grafana!

Complete Learning Topics Overview

All 12 learning topics are now complete! Here’s what you have:

✅ Completed Topics

1. LLM Inference Fundamentals ✅

• Location: docs/llm_inference_fundamentals.md
• What it covers: Tokenization, attention, generation, memory, computation
• Status: Complete with detailed explanations

2. Basic Model Serving ✅

• Location: 01_basic_serving/
• What it covers: FastAPI, HuggingFace, model loading, endpoints
• Status: Complete with working code

3. Containerization ✅

• Location: 02_docker/
• What it covers: Docker basics, multi-stage builds, GPU support
• Status: Complete with Dockerfiles and examples

4. Kubernetes Deployment ✅

• Location: 03_kubernetes/
• What it covers: K8s manifests, health checks, resources, ConfigMaps
• Status: Complete with deployment YAMLs

5. Load Testing & Performance ✅

• Location: 04_load_testing/
• What it covers: Latency measurement, Locust, performance profiling
• Status: Complete with test scripts

6. High-Performance Serving (vLLM) ✅

• Location: 05_vllm_serving/
• What it covers: vLLM setup, continuous batching, PagedAttention
• Status: Complete with server implementation

7. Autoscaling ✅

• Location: 06_autoscaling/
• What it covers: HPA, CPU/memory scaling, custom metrics
• Status: Complete with HPA configurations

8. Monitoring & Observability ✅

• Location: 08_monitoring/
• What it covers: Prometheus, Grafana, GPU monitoring, metrics
• Status: Complete with full monitoring stack

9. Canary Deployments ✅

• Location: 09_canary_deployments/
• What it covers: Traffic splitting, gradual rollouts, rollback
• Status: Complete with deployment examples

10. Model Versioning ✅

• Location: 10_model_versioning/
• What it covers: Version management, model registry, rollback
• Status: Complete with strategies and examples

11. Drift Detection ✅

• Location: 11_drift_detection/
• What it covers: Data drift, concept drift, Evidently AI
• Status: Complete with detector implementation

12. Multi-Model Serving (Triton) ✅

• Location: 12_triton/
• What it covers: Triton server, dynamic batching, model ensembles
• Status: Complete with configuration examples

📚 Learning Path

Follow this order for best learning experience:

1. Start: Read HOW_TO_START.md
2. Fundamentals: Read docs/llm_inference_fundamentals.md
3. Basic Serving: Work through 01_basic_serving/
4. Containerize: Learn Docker in 02_docker/
5. Deploy: Deploy to K8s with 03_kubernetes/
6. Test: Measure performance with 04_load_testing/
7. Optimize: Use vLLM in 05_vllm_serving/
8. Scale: Set up autoscaling in 06_autoscaling/
9. Monitor: Add monitoring in 08_monitoring/
10. Deploy Safely: Learn canary in 09_canary_deployments/
11. Version: Manage versions in 10_model_versioning/
12. Detect Issues: Add drift detection in 11_drift_detection/
13. Advanced: Use Triton in 12_triton/

🎯 Quick Reference

• Quick Start: HOW_TO_START.md
• Learning Path: LEARNING_PATH.md
• Quick Commands: QUICK_REFERENCE.md
• Overview: README.md

📖 Each Topic Includes

• README.md: Detailed explanations and concepts
• Code examples: Working implementations
• Configuration files: YAMLs, Dockerfiles, etc.
• Exercises: Hands-on practice suggestions

🚀 Ready to Learn!

Everything is set up and ready. Start with HOW_TO_START.md and work through each topic at your own pace.

Happy learning! 🎓

Industry Boilerplate Code & Use Cases

This document provides industry-standard boilerplate code and real-world use cases for each topic.

Quick Reference

• Topic 1: Basic Serving → Customer support, content generation
• Topic 2: Docker → Multi-cloud, CI/CD, edge deployment
• Topic 3: Kubernetes → Large-scale platforms, enterprise ML
• Topic 4: Load Testing → Performance validation, capacity planning
• Topic 5: vLLM → High-throughput production serving
• Topic 6: Autoscaling → Cost optimization, traffic handling
• Topic 7: Canary → Safe deployments, A/B testing
• Topic 8: Monitoring → Production observability, alerting
• Topic 9: Canary (same as 7)
• Topic 10: Versioning → Model management, rollback
• Topic 11: Drift Detection → Model health, quality assurance
• Topic 12: Triton → Multi-model serving, model pipelines

Common Industry Patterns

Pattern 1: API Gateway → Serving Layer

[API Gateway] → [Load Balancer] → [K8s Service] → [LLM Pods]

Used by: OpenAI, Anthropic, HuggingFace

Pattern 2: Model Registry → Serving

[Model Registry] → [CI/CD] → [K8s Deployment] → [Serving Pods]

Used by: MLflow, Weights & Biases, custom platforms

Pattern 3: Monitoring → Alerting → Auto-remediation

[Prometheus] → [Grafana] → [Alertmanager] → [PagerDuty/Slack]

Used by: All production ML systems

See individual topic READMEs for detailed boilerplate code and use cases.

LLM Serving & Inference Interview Q&A

Comprehensive interview questions and answers for LLM serving, inference, and MLOps roles.

Table of Contents

1. LLM Inference Fundamentals
2. Model Serving
3. Performance Optimization
4. Kubernetes & Deployment
5. Monitoring & Observability
6. Production Best Practices
7. System Design

LLM Inference Fundamentals

Q1: Explain how LLM inference works step-by-step.

Answer:

1. Tokenization: Input text is converted to token IDs using the model’s tokenizer
2. Embedding: Token IDs are converted to dense vectors (embeddings)
3. Forward Pass:
   • Input passes through transformer layers
   • Each layer applies self-attention and feed-forward networks
   • Attention mechanism computes relationships between tokens
4. Output Projection: Final layer projects to vocabulary size
5. Sampling: Next token is sampled from probability distribution
6. Autoregressive Generation: Process repeats with new token until stop condition

Key Points:

• Inference is autoregressive (one token at a time)
• KV cache stores attention key-values to avoid recomputation
• Each token generation requires a full forward pass

Q2: What is KV caching and why is it important?

Answer: KV (Key-Value) caching stores the attention key and value matrices for previously processed tokens.

How it works:

• First token: Compute full Q, K, V for all tokens
• Subsequent tokens: Only compute Q for new token, reuse cached K, V

Benefits:

• Speed: Avoids recomputing attention for previous tokens
• Memory trade-off: Uses more memory but much faster
• Critical for performance: Without it, each token would recompute all previous tokens

Example:

• Without cache: Generate 100 tokens = 100 forward passes, each processing all 100 tokens
• With cache: Generate 100 tokens = 100 forward passes, but each only processes 1 new token

Q3: What is the difference between training and inference?

Answer:

Key Differences:

• Training: Updates weights, needs gradients
• Inference: Uses fixed weights, generates predictions

Q4: Explain attention mechanism in the context of inference.

Answer: Attention determines which tokens to focus on when generating the next token.

Formula:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

Components:

• Q (Query): “What am I looking for?”
• K (Key): “What information do I have?”
• V (Value): “What is the actual information?”

In Inference:

• Computes relationships between current token and all previous tokens
• Allows model to “attend” to relevant context
• KV cache stores K and V to avoid recomputation

Complexity:

• O(n²) where n is sequence length
• This is why longer sequences are slower and more memory-intensive

Model Serving

Q5: How would you design an LLM serving API?

Answer:

API Design:

POST /v1/completions
{
  "prompt": "The future of AI is",
  "max_tokens": 100,
  "temperature": 0.7,
  "top_p": 0.9
}

Key Components:

1. Request Validation: Validate inputs (prompt length, parameters)
2. Model Loading: Load model once at startup (not per request)
3. Tokenization: Convert text to tokens
4. Generation: Run inference with parameters
5. Response: Return generated text + metadata (latency, tokens)

Considerations:

• Async support: Handle concurrent requests
• Streaming: Support streaming responses
• Error handling: Graceful error responses
• Rate limiting: Prevent abuse
• Health checks: For Kubernetes probes

Q6: What are the differences between HuggingFace Transformers and vLLM?

Answer:

When to use HuggingFace:

• Development and prototyping
• Small-scale deployments
• Need maximum flexibility

When to use vLLM:

• Production high-throughput
• Need maximum GPU utilization
• Many concurrent requests

Q7: How does continuous batching work in vLLM?

Answer: Continuous batching allows adding/removing requests dynamically during batch processing.

Traditional Batching:

1. Wait for batch to fill (e.g., 8 requests)
2. Process entire batch
3. Wait for all to complete
4. Start next batch

Continuous Batching:

1. Start processing batch
2. Add new requests as they arrive
3. Remove completed requests
4. Continue processing remaining requests
5. GPU always busy

Benefits:

• Higher GPU utilization
• Lower latency (no waiting for batch to fill)
• Better throughput

Example:

Time 0: [Req1, Req2, Req3] → Processing
Time 1: [Req1, Req2, Req3, Req4] → Req4 added
Time 2: [Req2, Req3, Req4] → Req1 completed, removed

Performance Optimization

Q8: How would you optimize LLM inference latency?

Answer:

1. Model Optimization:

• Quantization: FP16, INT8, INT4 (trade accuracy for speed)
• Model pruning: Remove unnecessary weights
• Knowledge distillation: Use smaller model

2. Inference Optimization:

• KV caching: Cache attention key-values
• Batching: Process multiple requests together
• Continuous batching: vLLM’s approach

3. Hardware:

• GPU: Use GPU instead of CPU (10-100x faster)
• Tensor cores: Use specialized hardware
• Model parallelism: Split across multiple GPUs

4. System:

• Pre-warming: Load model before first request
• Connection pooling: Reuse connections
• CDN: Cache responses when appropriate

5. Architecture:

• Async processing: Don’t block on I/O
• Request queuing: Handle bursts gracefully
• Load balancing: Distribute requests

Q9: What is PagedAttention and why does it matter?

Answer: PagedAttention is vLLM’s memory management technique for KV cache.

Problem it solves:

• Traditional KV cache: Fixed-size blocks, memory fragmentation
• Wastes memory when sequences have different lengths
• Can’t support very long sequences efficiently

How it works:

1. Divide KV cache into fixed-size “pages” (like OS memory pages)
2. Allocate pages on-demand as tokens are generated
3. Free pages when sequences complete
4. Reuse freed pages for new sequences

Benefits:

• Efficient memory: No fragmentation
• Longer sequences: Support sequences up to model max
• Higher throughput: More sequences fit in memory

Analogy: Like virtual memory in operating systems - pages allocated as needed.

Q10: Explain the trade-offs between latency and throughput.

Answer:

Latency: Time for one request (ms) Throughput: Requests processed per second

Trade-offs:

1. Batching:

   • Larger batches → Higher throughput, higher latency
   • Smaller batches → Lower latency, lower throughput

2. Model Size:

   • Larger model → Higher quality, higher latency
   • Smaller model → Lower latency, potentially lower quality

3. Quantization:

   • Lower precision → Faster, lower memory, potential accuracy loss
   • Higher precision → Slower, more memory, better accuracy

4. Hardware:

   • More GPUs → Higher throughput, higher cost
   • Fewer GPUs → Lower cost, lower throughput

Optimization Strategy:

• Low latency: Small batches, optimized model, fast hardware
• High throughput: Large batches, continuous batching, multiple GPUs

Kubernetes & Deployment

Q11: How would you deploy an LLM model to Kubernetes?

Answer:

1. Containerize:

FROM python:3.9-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

2. Create Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: llm-serving
        image: llm-serving:v1.0
        resources:
          requests:
            memory: "4Gi"
            cpu: "2000m"
          limits:
            memory: "8Gi"
            cpu: "4000m"
            nvidia.com/gpu: 1
        livenessProbe:
          httpGet:
            path: /health
            port: 8000

3. Create Service:

apiVersion: v1
kind: Service
metadata:
  name: llm-serving
spec:
  selector:
    app: llm-serving
  ports:
  - port: 80
    targetPort: 8000

4. Deploy:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

Key Considerations:

• Resource limits (GPU, memory)
• Health checks
• Rolling updates
• ConfigMaps for configuration
• Secrets for API keys

Q12: How does Horizontal Pod Autoscaling (HPA) work?

Answer: HPA automatically scales the number of pod replicas based on metrics.

How it works:

1. HPA checks metrics every 15 seconds (default)
2. Compares current metric value to target
3. Calculates desired number of replicas
4. Updates deployment replica count
5. Kubernetes creates/destroys pods

Example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Metrics:

• CPU/Memory: Built-in resource metrics
• Custom metrics: Requests per second, latency (requires Prometheus Adapter)

Scaling Behavior:

• Scale up: When metric > target (add pods)
• Scale down: When metric < target (remove pods)
• Stabilization window: Prevents flapping

Q13: Explain canary deployments for model updates.

Answer: Canary deployment gradually rolls out new model version to a small percentage of traffic.

Process:

1. Deploy canary: Deploy new version alongside stable
2. Route traffic: Split traffic (e.g., 90% stable, 10% canary)
3. Monitor: Compare metrics (latency, errors, quality)
4. Gradual increase: If good, increase canary traffic (25%, 50%, 100%)
5. Rollback: If issues, route all traffic back to stable

Implementation:

• Kubernetes: Multiple deployments with different replica counts
• Service Mesh (Istio): Fine-grained traffic control
• Application-level: Route based on headers/parameters

Benefits:

• Risk reduction: Test on small traffic first
• Quick rollback: Revert if issues detected
• A/B testing: Compare model versions

Monitoring:

• Compare latency (P50, P95, P99)
• Error rates
• Business metrics (user satisfaction)

Monitoring & Observability

Q14: What metrics should you monitor for LLM serving?

Answer:

Application Metrics:

• Request rate: Requests per second
• Latency: P50, P95, P99 response times
• Error rate: Failed requests percentage
• Queue size: Pending requests
• Throughput: Tokens per second

Model Metrics:

• Generation time: Time to generate tokens
• Tokens generated: Average tokens per request
• Model version: Which model is running

System Metrics:

• GPU utilization: GPU usage percentage
• GPU memory: Used/total memory
• CPU usage: CPU utilization
• Memory usage: RAM usage

Business Metrics:

• Cost per request: Compute cost
• User satisfaction: Quality metrics
• API usage: Requests by endpoint

Key Dashboards:

1. Performance: Latency, throughput, errors
2. Resource: GPU, CPU, memory usage
3. Model: Version performance comparison

Q15: How would you detect model drift in production?

Answer:

Types of Drift:

1. Data drift: Input distribution changes
2. Concept drift: Input-output relationship changes
3. Prediction drift: Output distribution changes

Detection Methods:

1. Statistical Tests:

• PSI (Population Stability Index): Compare distributions
• Kolmogorov-Smirnov: Test distribution differences
• Chi-square: Test categorical distributions

2. Monitoring Tools:

• Evidently AI: Open-source drift detection
• Prometheus: Custom metrics
• Custom scripts: Compare reference vs current

3. Implementation:

from evidently import Report, DataDriftTable

report = Report(metrics=[DataDriftTable()])
report.run(
    reference_data=train_data,
    current_data=production_data
)

if report.get_metric(DataDriftTable()).drift_detected:
    alert("Data drift detected!")

4. Alerting:

• Set thresholds (e.g., PSI > 0.2)
• Monitor continuously
• Alert on drift detection
• Investigate causes

Actions:

• Retrain model: If drift significant
• Investigate: Understand why drift occurred
• Update baseline: Update reference data if appropriate

Production Best Practices

Q16: What are the key considerations for production LLM serving?

Answer:

1. Performance:

• Latency: P95 < 500ms for most use cases
• Throughput: Handle expected load
• Scalability: Auto-scale based on demand

2. Reliability:

• Health checks: Liveness and readiness probes
• Error handling: Graceful degradation
• Circuit breakers: Prevent cascade failures
• Retries: With exponential backoff

3. Monitoring:

• Metrics: Comprehensive observability
• Logging: Structured logging
• Alerting: Proactive issue detection
• Tracing: Request tracing

4. Security:

• Authentication: API keys, OAuth
• Rate limiting: Prevent abuse
• Input validation: Sanitize inputs
• Secrets management: Secure API keys

5. Cost:

• Resource optimization: Right-size instances
• Auto-scaling: Scale down when not needed
• Model optimization: Use efficient models

6. Model Management:

• Versioning: Track model versions
• A/B testing: Compare model performance
• Rollback: Quick revert capability
• Drift detection: Monitor model degradation

Q17: How would you handle a sudden spike in traffic?

Answer:

Immediate Actions:

1. Auto-scaling: HPA should scale up automatically
2. Load balancing: Distribute across pods
3. Queue management: Queue requests if needed
4. Rate limiting: Protect backend from overload

Prevention:

1. Capacity planning: Understand max capacity
2. Load testing: Test under expected load
3. Auto-scaling: Configure HPA properly
4. Circuit breakers: Prevent cascade failures

Monitoring:

• Watch pod count
• Monitor latency (P95, P99)
• Check error rates
• GPU utilization

If Overwhelmed:

1. Degrade gracefully: Return cached responses
2. Rate limit: Reject excess requests
3. Scale manually: If auto-scaling insufficient
4. Add capacity: More nodes/GPUs

Post-Incident:

• Analyze what happened
• Improve auto-scaling config
• Increase baseline capacity if needed
• Document learnings

System Design

Q18: Design a system to serve LLMs at scale.

Answer:

Architecture:

[Load Balancer] 
    ↓
[API Gateway] (Rate limiting, Auth)
    ↓
[Kubernetes Cluster]
    ├── [LLM Serving Pods] (vLLM)
    ├── [Monitoring] (Prometheus, Grafana)
    └── [Model Registry] (S3/GCS)

Components:

1. Load Balancer:

• Distribute traffic
• Health checks
• SSL termination

2. API Gateway:

• Authentication/Authorization
• Rate limiting
• Request routing
• API versioning

3. Serving Layer:

• vLLM servers: High-performance inference
• Auto-scaling: HPA based on metrics
• GPU nodes: Dedicated GPU instances

4. Model Storage:

• Model registry: S3/GCS for model files
• Versioning: Track model versions
• Caching: Cache models on nodes

5. Monitoring:

• Metrics: Prometheus
• Dashboards: Grafana
• Logging: Centralized logging
• Alerting: PagerDuty/Slack

6. Data Pipeline:

• Request logging: Store inputs/outputs
• Drift detection: Monitor data drift
• A/B testing: Compare model versions

Scaling Strategy:

• Horizontal: Add more pods (HPA)
• Vertical: Larger GPUs for bigger models
• Multi-region: Geographic distribution

Key Metrics:

• Latency (P50, P95, P99)
• Throughput (req/s)
• Error rate
• GPU utilization
• Cost per request

Q19: How would you implement model versioning and rollback?

Answer:

Versioning Strategy:

1. Semantic versioning: v1.0.0, v1.1.0, v2.0.0
2. Model registry: Store models with metadata
3. Metadata tracking: Training date, metrics, dataset

Implementation:

1. Model Registry:

models/
  v1.0.0/
    model.bin
    tokenizer.json
    metadata.json
  v1.1.0/
    model.bin
    tokenizer.json
    metadata.json

2. Deployment:

env:
- name: MODEL_VERSION
  value: "v1.0.0"
- name: MODEL_PATH
  value: "/models/v1.0.0"

3. Rollback:

# Update to previous version
kubectl set env deployment/llm-serving \
  MODEL_VERSION=v0.9.0

# Or use canary deployment
# Route traffic back to stable version

4. API:

GET /api/v1/models/versions
POST /api/v1/models/rollback
{
  "target_version": "v1.0.0"
}

Best Practices:

• Test before deploy: Validate new version
• Gradual rollout: Use canary deployment
• Monitor: Track version performance
• Document: Changelog for each version

Q20: Explain how you would optimize costs for LLM serving.

Answer:

Cost Components:

1. Compute: GPU/CPU instances
2. Storage: Model storage
3. Network: Data transfer
4. Monitoring: Observability tools

Optimization Strategies:

1. Right-sizing:

• Use appropriate instance types
• Don’t over-provision
• Match workload to instance

2. Auto-scaling:

• Scale down during low traffic
• Scale up only when needed
• Use spot instances for non-critical workloads

3. Model Optimization:

• Quantization: INT8/INT4 (smaller, faster)
• Pruning: Remove unnecessary weights
• Distillation: Use smaller models

4. Caching:

• Cache model weights
• Cache common responses (if applicable)
• Use CDN for static content

5. Batch Processing:

• Batch requests when possible
• Use continuous batching (vLLM)
• Higher GPU utilization = lower cost per request

6. Monitoring:

• Track cost per request
• Identify expensive operations
• Optimize based on data

7. Reserved Instances:

• Commit to usage for discounts
• Use for predictable workloads

Example:

• Before: 10 GPUs, 50% utilization = $5000/month
• After: 5 GPUs, 90% utilization (with batching) = $2500/month
• Savings: 50%

Additional Quick Questions

Q21: What is the difference between batch size and sequence length?

Answer:

• Batch size: Number of requests processed together
• Sequence length: Number of tokens in a single request

Example:

• Batch size = 8: Process 8 requests simultaneously
• Sequence length = 512: Each request has up to 512 tokens

Impact:

• Larger batch size → Higher throughput, more memory
• Longer sequence length → More computation, more memory (KV cache)

Q22: How does quantization affect model performance?

Answer: Quantization reduces model precision to save memory and speed up inference.

Types:

• FP32 → FP16: 2x smaller, 2x faster, minimal accuracy loss
• FP16 → INT8: 2x smaller, 2x faster, small accuracy loss
• INT8 → INT4: 2x smaller, 2x faster, larger accuracy loss

Trade-offs:

• Pros: Faster inference, less memory, lower cost
• Cons: Potential accuracy loss, may need calibration

When to use:

• Production when speed/cost matters
• After validating accuracy is acceptable
• For edge deployment (limited resources)

Q23: What is the difference between model parallelism and data parallelism?

Answer:

Data Parallelism:

• Same model on multiple GPUs
• Different data on each GPU
• Used in training (gradient sync)
• Inference: Not commonly used (each request needs full model)

Model Parallelism:

• Split model across multiple GPUs
• Each GPU holds part of model
• Used for large models that don’t fit on one GPU
• Inference: Common for very large models (70B+)

Example:

• Data parallel: 8 GPUs, each running full GPT-2
• Model parallel: 8 GPUs, each holding 1/8 of GPT-3

Tips for Interviews

1. Be specific: Use numbers and examples
2. Show trade-offs: Understand pros/cons
3. Think system-wide: Consider all components
4. Ask clarifying questions: Understand requirements
5. Draw diagrams: Visualize architecture
6. Discuss monitoring: Always mention observability
7. Talk about failures: How to handle edge cases

Resources

• This repository’s documentation
• vLLM documentation
• Kubernetes documentation
• Prometheus/Grafana guides
• Evidently AI documentation

Good luck with your interviews! 🚀

Exercise Solutions

This document provides solutions to exercises in each learning topic.

Topic 1: Basic Model Serving

Exercise 1: Change the Model

Solution:

# In model_loader.py, change:
model = LLMModel(
    model_name="distilgpt2",  # Smaller, faster model
    device="cpu"
)

Exercise 2: Modify Parameters

Solution:

# In app.py, test different temperatures:
# Temperature 0.1 = more deterministic
# Temperature 2.0 = more creative
payload = {
    "prompt": "The future of AI is",
    "temperature": 0.1,  # Try different values
    "max_length": 100
}

Exercise 3: Add Logging

Solution:

import logging
from datetime import datetime

logger = logging.getLogger(__name__)

@app.post("/generate")
async def generate_text(request: TextGenerationRequest):
    start_time = time.time()
    logger.info(f"Request received: prompt='{request.prompt[:50]}...'")
    
    # ... generation code ...
    
    latency = (time.time() - start_time) * 1000
    logger.info(f"Request completed: latency={latency:.2f}ms, tokens={num_tokens}")
    return response

Exercise 4: Error Handling

Solution:

@app.post("/generate")
async def generate_text(request: TextGenerationRequest):
    try:
        if model is None:
            raise HTTPException(status_code=503, detail="Model not loaded")
        
        if len(request.prompt) > 1000:
            raise HTTPException(status_code=400, detail="Prompt too long")
        
        # Generation code...
        
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Unexpected error: {e}", exc_info=True)
        raise HTTPException(status_code=500, detail="Internal server error")

Exercise 5: Batch Requests

Solution:

class BatchGenerationRequest(BaseModel):
    prompts: List[str] = Field(..., min_items=1, max_items=10)

@app.post("/generate_batch")
async def generate_batch(request: BatchGenerationRequest):
    results = []
    for prompt in request.prompts:
        generated = model.generate(prompt=prompt, max_length=50)
        results.append({"prompt": prompt, "generated": generated})
    return {"results": results}

Topic 2: Docker

Exercise 1: Build Basic Image

Solution:

cd 01_basic_serving
docker build -f ../02_docker/Dockerfile.basic -t llm-serving:basic .
docker run -p 8000:8000 llm-serving:basic

Exercise 2: Optimize Image Size

Solution:

# Use multi-stage build
FROM python:3.9-slim as builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

FROM python:3.9-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "app.py"]

Exercise 3: Multi-stage Build

Solution: See 02_docker/Dockerfile.optimized

Exercise 4: GPU Support

Solution: See 02_docker/Dockerfile.gpu

Exercise 5: Docker Compose

Solution: See 02_docker/docker-compose.yml

Topic 3: Kubernetes

Exercise 1: Basic Deployment

Solution:

kubectl apply -f 03_kubernetes/deployment.yaml
kubectl get pods -n llm-serving
kubectl logs -f deployment/llm-serving -n llm-serving

Exercise 2: Health Checks

Solution:

# Already in deployment.yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30

Exercise 3: Resource Limits

Solution:

resources:
  requests:
    memory: "2Gi"
    cpu: "1000m"
  limits:
    memory: "4Gi"
    cpu: "2000m"

Exercise 4: ConfigMap

Solution:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-config
data:
  model_name: "gpt2"
  max_length: "100"
---
env:
- name: MODEL_NAME
  valueFrom:
    configMapKeyRef:
      name: llm-config
      key: model_name

Exercise 5: Scaling

Solution:

kubectl scale deployment llm-serving --replicas=3 -n llm-serving
kubectl get pods -n llm-serving

Exercise 6: Rolling Update

Solution:

# Update image
kubectl set image deployment/llm-serving \
  llm-serving=llm-serving:v2 -n llm-serving

# Watch rollout
kubectl rollout status deployment/llm-serving -n llm-serving

# Rollback if needed
kubectl rollout undo deployment/llm-serving -n llm-serving

Topic 4: Load Testing

Exercise 1: Baseline Test

Solution:

python measure_latency.py --url http://localhost:8000 --requests 10

Exercise 2: Ramp-up Test

Solution:

# In Locust UI, use "Spawn rate" to gradually increase users
# Or use command line:
locust -f locust_test.py \
  --host=http://localhost:8000 \
  --headless \
  --users 50 \
  --spawn-rate 5 \
  --run-time 5m

Exercise 3: Compare Models

Solution:

# Test model A
python measure_latency.py --url http://localhost:8000 --requests 100 > model_a.txt

# Change model, restart server
# Test model B
python measure_latency.py --url http://localhost:8001 --requests 100 > model_b.txt

# Compare results
diff model_a.txt model_b.txt

Exercise 4: Find Bottleneck

Solution:

# Add profiling to app.py
import cProfile
import pstats

@app.post("/generate")
async def generate_text(request: TextGenerationRequest):
    profiler = cProfile.Profile()
    profiler.enable()
    
    # ... generation code ...
    
    profiler.disable()
    stats = pstats.Stats(profiler)
    stats.sort_stats('cumulative')
    stats.print_stats(10)  # Top 10 slowest functions

Exercise 5: Stress Test

Solution:

# Keep increasing users until system breaks
locust -f locust_test.py \
  --host=http://localhost:8000 \
  --headless \
  --users 100 \
  --spawn-rate 10 \
  --run-time 10m

Topic 5: vLLM Serving

Exercise 1: Compare Performance

Solution:

# Test HuggingFace
import time
start = time.time()
# ... HuggingFace generation ...
hf_time = time.time() - start

# Test vLLM
start = time.time()
# ... vLLM generation ...
vllm_time = time.time() - start

print(f"HuggingFace: {hf_time:.2f}s")
print(f"vLLM: {vllm_time:.2f}s")
print(f"Speedup: {hf_time/vllm_time:.2f}x")

Exercise 2: Tune Parameters

Solution:

# Test different GPU memory utilization
for util in [0.7, 0.8, 0.9, 0.95]:
    llm = LLM(
        model="gpt2",
        gpu_memory_utilization=util
    )
    # Measure throughput
    # Find optimal value

Exercise 3: Test Continuous Batching

Solution:

# Send requests at different rates
import asyncio

async def send_requests(rate_per_second=10):
    for i in range(100):
        # Send request
        await asyncio.sleep(1/rate_per_second)
        # Monitor GPU utilization

Exercise 4: Monitor Metrics

Solution:

# Add Prometheus metrics to vLLM server
from prometheus_client import Counter, Histogram

REQUEST_COUNT = Counter('vllm_requests_total', 'Total requests')
REQUEST_DURATION = Histogram('vllm_request_duration_seconds', 'Request duration')

@app.post("/complete")
async def complete(request: CompletionRequest):
    REQUEST_COUNT.inc()
    with REQUEST_DURATION.time():
        # ... generation ...
        pass

Topic 6: Autoscaling

Exercise 1: Basic HPA

Solution:

kubectl apply -f 06_autoscaling/hpa.yaml
kubectl get hpa -n llm-serving
kubectl describe hpa llm-serving-hpa -n llm-serving

Exercise 2: Custom Metrics

Solution:

# Requires Prometheus Adapter
metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "10"

Exercise 3: Scaling Policies

Solution:

behavior:
  scaleUp:
    policies:
    - type: Percent
      value: 50  # Increase by 50%
      periodSeconds: 30
  scaleDown:
    stabilizationWindowSeconds: 300  # Wait 5 min
    policies:
    - type: Percent
      value: 25  # Decrease by 25%

Exercise 4: Load Test

Solution:

# Terminal 1: Watch HPA
kubectl get hpa -w -n llm-serving

# Terminal 2: Generate load
locust -f locust_test.py --host=http://<service-url> --users 50

# Terminal 3: Watch pods
kubectl get pods -w -n llm-serving

Topic 7: Canary Deployments

Exercise 1: Basic Canary

Solution:

kubectl apply -f 07_canary_deployments/canary-deployment.yaml
# Traffic splits 90/10 based on replica count

Exercise 2: Gradual Rollout

Solution:

# Phase 1: 10% (already done)
# Phase 2: 25%
kubectl scale deployment llm-serving-canary --replicas=3 -n llm-serving
kubectl scale deployment llm-serving-stable --replicas=7 -n llm-serving

# Phase 3: 50%
kubectl scale deployment llm-serving-canary --replicas=5 -n llm-serving
kubectl scale deployment llm-serving-stable --replicas=5 -n llm-serving

# Phase 4: 100%
kubectl scale deployment llm-serving-canary --replicas=10 -n llm-serving
kubectl delete deployment llm-serving-stable -n llm-serving

Exercise 3: Monitoring

Solution:

# Compare latency
histogram_quantile(0.95, 
  rate(llm_request_duration_seconds_bucket{version="canary"}[5m])
) / 
histogram_quantile(0.95, 
  rate(llm_request_duration_seconds_bucket{version="stable"}[5m])
)

Exercise 4: Rollback

Solution:

# Scale canary to 0
kubectl scale deployment llm-serving-canary --replicas=0 -n llm-serving

# Or update service to only route to stable

Topic 8: Monitoring

Exercise 1: Set Up Monitoring

Solution:

cd 08_monitoring
docker-compose up -d
# Access Grafana at http://localhost:3000

Exercise 2: Create Dashboard

Solution:

1. Open Grafana
2. Add Prometheus data source
3. Create new dashboard
4. Add panels with queries from 08_monitoring/README.md

Exercise 3: Set Up Alerts

Solution:

# In Prometheus alerts.yml
groups:
- name: llm_alerts
  rules:
  - alert: HighLatency
    expr: histogram_quantile(0.95, rate(llm_request_duration_seconds_bucket[5m])) > 1
    for: 5m
    annotations:
      summary: "P95 latency > 1s"

Exercise 4: GPU Monitoring

Solution:

# Use prometheus_exporter.py
# GPU metrics are automatically collected if pynvml is installed

Topic 9: Canary Deployments

(Same as Topic 7 - see above)

Topic 10: Model Versioning

Exercise 1: Create Registry

Solution:

# models/registry.py
class ModelRegistry:
    def __init__(self, base_path="models"):
        self.base_path = base_path
    
    def register(self, version, model_path, metadata):
        version_dir = f"{self.base_path}/{version}"
        os.makedirs(version_dir, exist_ok=True)
        # Copy model files
        # Save metadata

Exercise 2: Version Model

Solution:

# Tag model version
mkdir -p models/v1.0.0
cp model.bin models/v1.0.0/
cp tokenizer.json models/v1.0.0/
echo '{"version": "v1.0.0", "created_at": "2024-01-15"}' > models/v1.0.0/metadata.json

Exercise 3: Deploy Version

Solution:

# Update deployment
env:
- name: MODEL_VERSION
  value: "v1.0.0"
- name: MODEL_PATH
  value: "/models/v1.0.0"

Exercise 4: Rollback

Solution:

# Update to previous version
kubectl set env deployment/llm-serving \
  MODEL_VERSION=v0.9.0 \
  -n llm-serving

Topic 11: Drift Detection

Exercise 1: Set Up Evidently

Solution:

pip install evidently pandas

Exercise 2: Detect Data Drift

Solution:

from drift_detector import DriftDetector
import pandas as pd

# Load reference data
reference = pd.read_csv("train_data.csv")

# Initialize detector
detector = DriftDetector(reference_data=reference)

# Load current data
current = pd.read_csv("production_data.csv")

# Detect drift
result = detector.detect_data_drift(current_data=current)
print(result)

Exercise 3: Create Dashboard

Solution:

from evidently.ui.dashboards import Dashboard

report = detector.generate_report(current_data)
dashboard = Dashboard("Drift Monitoring")
dashboard.add_report(report)
dashboard.show()

Exercise 4: Set Up Alerts

Solution:

if result["drift_detected"]:
    send_email_alert("Data drift detected!")
    send_slack_alert("Drift score: {result['drift_score']}")

Topic 12: Triton

Exercise 1: Deploy Model

Solution:

# Create model repository
mkdir -p model_repository/gpt2/1
# Copy model files
cp model.pt model_repository/gpt2/1/

# Start Triton
docker run --gpus all \
  -v $(pwd)/model_repository:/models \
  nvcr.io/nvidia/tritonserver:23.10-py3 \
  tritonserver --model-repository=/models

Exercise 2: Multiple Models

Solution:

model_repository/
  gpt2/
    config.pbtxt
    1/model.pt
  distilgpt2/
    config.pbtxt
    1/model.pt

Exercise 3: Dynamic Batching

Solution:

dynamic_batching {
  max_queue_delay_microseconds: 100000
  preferred_batch_size: [ 4, 8, 16 ]
  max_batch_size: 32
}

Exercise 4: Model Ensemble

Solution: See 12_triton/README.md for ensemble configuration

General Tips

1. Test locally first: Always test changes locally before deploying
2. Monitor metrics: Watch metrics during exercises
3. Read errors: Error messages often contain solutions
4. Experiment: Try different values and see what happens
5. Document: Note what works and what doesn’t

Topic 1: Basic LLM Serving

What You’ll Learn

This topic teaches you the fundamentals of serving an LLM:

• How to load a pre-trained model from HuggingFace
• Building a FastAPI endpoint for inference
• Understanding the inference pipeline
• Handling requests and responses
• Basic error handling

Why We Need This

Business Need

Companies need to expose LLM capabilities as APIs to:

• Integrate AI into applications: Chatbots, content generation, code completion
• Serve multiple clients: Web apps, mobile apps, internal tools
• Scale independently: Separate model serving from application logic
• Enable monetization: API-based business models

Technical Need

• Separation of concerns: Model serving separate from application code
• Resource management: Dedicated servers for compute-intensive inference
• Standardization: REST APIs are universal, language-agnostic
• Testing: Easy to test model independently

Real-World Impact

Without proper serving infrastructure:

• ❌ Models can’t be used in production applications
• ❌ No way to integrate AI into existing systems
• ❌ Difficult to scale and maintain
• ❌ Hard to version and update models

Industry Use Cases

1. Customer Support Chatbots

Company: E-commerce, SaaS platforms Use Case:

• Customer asks question → API call → LLM generates response
• Handles 24/7 support, reduces human agent workload

Example Request:

POST /generate
{
  "prompt": "Customer: How do I return an item?\nAssistant:",
  "max_length": 100
}

2. Content Generation

Company: Marketing agencies, content platforms Use Case:

• Generate blog posts, social media content, product descriptions
• API called from CMS or marketing tools

Example Request:

POST /generate
{
  "prompt": "Write a product description for a wireless headphone:",
  "temperature": 0.8,  # More creative
  "max_length": 200
}

3. Code Completion & Assistance

Company: GitHub Copilot, IDEs Use Case:

• Developer types code → API suggests completions
• Real-time code generation in editor

Example Request:

POST /generate
{
  "prompt": "def calculate_total(items):\n    ",
  "max_length": 50,
  "temperature": 0.2  # More deterministic
}

4. Translation Services

Company: Google Translate, DeepL Use Case:

• Translate text between languages
• API integrated into websites, apps

Example Request:

POST /generate
{
  "prompt": "Translate to French: Hello, how are you?",
  "max_length": 50
}

5. Sentiment Analysis

Company: Social media platforms, review sites Use Case:

• Analyze customer reviews, social media posts
• Real-time sentiment detection

Example Request:

POST /generate
{
  "prompt": "Sentiment: This product is amazing!",
  "max_length": 10
}

Industry-Standard Boilerplate Code

Complete FastAPI Serving Application

"""
Industry-standard LLM serving application
Used by: OpenAI API, Anthropic Claude, HuggingFace Inference API
"""

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import time
import logging
from contextlib import asynccontextmanager

# Configure logging (industry standard)
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

# Global model (loaded once at startup)
model = None
tokenizer = None
device = "cuda" if torch.cuda.is_available() else "cpu"


@asynccontextmanager
async def lifespan(app: FastAPI):
    """Load model at startup, cleanup at shutdown"""
    global model, tokenizer
    
    logger.info(f"Loading model on {device}...")
    model_name = "gpt2"  # In production: from environment variable
    
    try:
        tokenizer = AutoTokenizer.from_pretrained(model_name)
        model = AutoModelForCausalLM.from_pretrained(model_name)
        model.to(device)
        model.eval()  # Set to evaluation mode
        logger.info("Model loaded successfully")
    except Exception as e:
        logger.error(f"Failed to load model: {e}")
        raise
    
    yield  # Application runs here
    
    # Cleanup
    model = None
    tokenizer = None
    logger.info("Model unloaded")


app = FastAPI(
    title="LLM Serving API",
    description="Production-ready LLM inference endpoint",
    version="1.0.0",
    lifespan=lifespan
)


# Request/Response Models (Industry standard structure)
class GenerationRequest(BaseModel):
    """Standard request format (similar to OpenAI API)"""
    prompt: str = Field(..., min_length=1, max_length=2000)
    max_tokens: int = Field(50, ge=1, le=500)
    temperature: float = Field(1.0, ge=0.0, le=2.0)
    top_p: float = Field(1.0, ge=0.0, le=1.0)
    stop: list[str] = Field(default_factory=list)


class GenerationResponse(BaseModel):
    """Standard response format"""
    text: str
    model: str
    usage: dict
    latency_ms: float


@app.get("/health")
async def health_check():
    """Health check for load balancers and K8s probes"""
    return {
        "status": "healthy" if model is not None else "unhealthy",
        "model_loaded": model is not None
    }


@app.post("/v1/completions", response_model=GenerationResponse)
async def generate_completion(request: GenerationRequest):
    """
    Main generation endpoint
    Industry standard: /v1/completions (OpenAI-compatible)
    """
    if model is None:
        raise HTTPException(status_code=503, detail="Model not loaded")
    
    start_time = time.time()
    
    try:
        # Tokenize
        inputs = tokenizer(
            request.prompt,
            return_tensors="pt",
            truncation=True,
            max_length=1024
        ).to(device)
        
        # Generate
        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=request.max_tokens,
                temperature=request.temperature,
                top_p=request.top_p,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id
            )
        
        # Decode
        generated_text = tokenizer.decode(
            outputs[0][inputs['input_ids'].shape[1]:],
            skip_special_tokens=True
        )
        
        # Calculate metrics
        latency_ms = (time.time() - start_time) * 1000
        num_tokens = len(outputs[0]) - inputs['input_ids'].shape[1]
        
        logger.info(
            f"Generated {num_tokens} tokens in {latency_ms:.2f}ms"
        )
        
        return GenerationResponse(
            text=generated_text,
            model="gpt2",
            usage={
                "prompt_tokens": inputs['input_ids'].shape[1],
                "completion_tokens": num_tokens,
                "total_tokens": len(outputs[0])
            },
            latency_ms=latency_ms
        )
        
    except Exception as e:
        logger.error(f"Generation error: {e}", exc_info=True)
        raise HTTPException(status_code=500, detail=str(e))


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(
        app,
        host="0.0.0.0",
        port=8000,
        log_level="info"
    )

Usage Example (Industry Standard)

# Client code (how applications use the API)
import requests

def generate_text(prompt: str, max_tokens: int = 50) -> str:
    """
    Call LLM serving API
    Used by: Web applications, mobile apps, microservices
    """
    response = requests.post(
        "http://llm-api:8000/v1/completions",
        json={
            "prompt": prompt,
            "max_tokens": max_tokens,
            "temperature": 0.7
        },
        timeout=30
    )
    response.raise_for_status()
    return response.json()["text"]


# Example: Customer support chatbot
def handle_customer_query(query: str) -> str:
    prompt = f"Customer: {query}\nAssistant:"
    return generate_text(prompt, max_tokens=100)


# Example: Content generation
def generate_product_description(product_name: str) -> str:
    prompt = f"Write a compelling product description for {product_name}:"
    return generate_text(prompt, max_tokens=200, temperature=0.8)

Concepts Explained

Model Loading

When you load a model, you’re:

1. Downloading weights (if not cached) - These are the learned parameters
2. Loading into memory - CPU RAM or GPU VRAM
3. Initializing tokenizer - Converts text ↔ tokens
4. Setting up device - CPU or CUDA (GPU)

Why this matters: Model loading is expensive. You do it once at startup, not per request.

Tokenization

• Text → Tokens: “Hello” → [15496]
• Tokens → Text: [15496] → “Hello”
• Special tokens: <BOS>, <EOS>, <PAD>, <UNK>

Inference Process

1. Tokenize input prompt
2. Run forward pass through model (autoregressive)
3. Sample next token
4. Repeat until max_length or stop token
5. Decode tokens back to text

Autoregressive Generation

LLMs generate one token at a time:

• Input: “The weather is”
• Step 1: Generate “nice”
• Step 2: Generate “today”
• Step 3: Generate “.”
• Output: “The weather is nice today.”

Each step uses the previous tokens as context.

Code Structure

01_basic_serving/
├── README.md           # This file
├── app.py              # FastAPI application
├── model_loader.py     # Model loading logic
├── requirements.txt    # Dependencies
└── test_api.py        # Simple test script

Running the Code

1. Install Dependencies

pip install -r requirements.txt

2. Start the Server

python app.py

3. Test the API

# Health check
curl http://localhost:8000/health

# Generate text
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_length": 50,
    "temperature": 0.7
  }'

4. View API Docs

Open http://localhost:8000/docs in your browser

Key Code Sections

Model Loading (model_loader.py)

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Move to device (CPU or GPU)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

Generation (app.py)

# Tokenize input
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate
outputs = model.generate(
    **inputs,
    max_length=max_length,
    temperature=temperature
)

# Decode
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

Understanding the Output

When you call the API, you get:

• generated_text: The complete generated text
• prompt: Your input (echoed back)
• num_tokens: Number of tokens generated
• latency_ms: Time taken in milliseconds
• model_name: Which model was used

Common Issues

Out of Memory

• Problem: Model too large for available RAM/VRAM
• Solution: Use a smaller model (gpt2-small) or reduce batch size

Slow Inference

• Problem: Running on CPU
• Solution: Use GPU if available, or use a smaller model

Import Errors

• Problem: Missing dependencies
• Solution: pip install -r requirements.txt

Exercises

1. Change the model: Try different HuggingFace models (gpt2, distilgpt2, etc.)
2. Modify parameters: Experiment with temperature, top_p, max_length
3. Add logging: Log every request with timing information
4. Error handling: Add try/except for different error cases
5. Batch requests: Modify to handle multiple prompts at once

Next Steps

Once you understand this, move to:

• Topic 2: Containerization with Docker
• Topic 3: Kubernetes deployment

Further Reading

• HuggingFace Transformers Docs
• FastAPI Documentation
• GPT-2 Paper

Basic LLM Serving — A Deep Dive

Standing up a language model behind an HTTP API from first principles, and understanding why the naive version is slow.

Why this matters

Every production LLM system — ChatGPT, a support bot, a code assistant — is at bottom a loop that turns text into tokens, runs a forward pass, and turns tokens back into text, wrapped in a network server. If you understand that loop end to end, and understand the two costs that dominate it (compute and GPU memory), the rest of this book is just engineering to make the loop cheaper and more concurrent.

This chapter builds the loop the obvious way: one model, one process, one request at a time. That version works, and it is a perfect teaching tool precisely because it is slow. By the end you will be able to say, with numbers, exactly where the time and the memory go — and that motivates batching, KV-cache management, and dedicated engines (vLLM, TGI, Triton) in the chapters that follow.

We keep the intuition first and the mechanism precise. Where there is a tradeoff, we name it honestly.

Core intuition: an LLM is an autoregressive next-token loop

A decoder-only transformer computes one thing: given a sequence of tokens, a probability distribution over the next token. Generation is just calling that repeatedly.

prompt: "The capital of France is"
   -> tokenizer -> [464, 3139, 286, 4881, 318]
   -> model -> logits over ~50k vocab -> pick "Paris" (token 6342)
   -> append -> [464, 3139, 286, 4881, 318, 6342]
   -> model -> pick "." -> append -> ...
   -> stop on EOS or max_new_tokens
   -> tokenizer.decode(...) -> " Paris."

Two things follow immediately, and they structure everything:

1. Generation is sequential. Token N+1 depends on token N. You cannot decode a 200-token answer in one shot; you do (at least) 200 forward passes. This is why latency scales with output length.
2. The model re-reads its own context every step — unless you cache. The naive loop re-processes the whole sequence on every token, which is quadratic waste. The fix is the KV cache (below), and the KV cache is what eats your GPU memory.

Hold those two facts. The whole performance story is a consequence of them.

Loading a model: weights, dtype, device, tokenizer

Before serving anything you load four things. Each has a failure mode.

Weights and where they live

A model is a set of tensors (the parameters) plus a config describing the architecture. With Hugging Face Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    torch_dtype="bfloat16",   # precision — see below
    device_map="cuda",        # placement
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

The weights download once to a local cache and are memory-mapped from safetensors shards on subsequent loads. Cold start = download + load + allocate; warm start = load + allocate. This distinction matters for autoscaling (Chapter 6): a cold pod may take minutes.

dtype / precision — the single biggest memory lever

Parameters are stored as floating-point numbers, and the bytes per parameter is a choice you make at load time.

BF16 (bfloat16) is usually preferred over FP16 on modern GPUs: same 2 bytes, but a wider exponent range, so it is less prone to overflow/NaN during the forward pass. FP32 doubles your memory for almost no inference quality gain — do not serve in FP32 by accident (it is the default if you forget torch_dtype).

Device placement

Weights must sit in GPU memory (VRAM) for fast inference. device_map="cuda" puts everything on one GPU; device_map="auto" will shard across multiple GPUs or spill to CPU/disk if the model does not fit — convenient, but CPU offload is catastrophically slow for serving. For a serving path you want the whole model resident on the GPU and you want to know it fits (memory math below).

Tokenizer — small, and a classic source of silent bugs

The tokenizer maps text <-> integer IDs. It must be the exact one the model was trained with; a mismatch produces garbage output with no error. Two things to get right in a server:

• Chat template. Instruct/chat models expect a specific formatting of roles (<|user|>, <|assistant|>, etc.). Use tokenizer.apply_chat_template(messages, add_generation_prompt=True) rather than hand-concatenating strings — getting the special tokens wrong quietly degrades quality.
• Padding side and pad token. For batched generation, decoder-only models must left-pad (tokenizer.padding_side = "left"), and many models ship without a pad_token — set tokenizer.pad_token = tokenizer.eos_token. Right-padding a decoder batch corrupts the generation. (We revisit padding when we build real batching.)

Mechanism in depth: prefill vs decode, and the KV cache

This is the heart of the chapter. A single generation request has two phases with completely different performance characteristics.

Prefill (the prompt pass)

You feed the entire prompt (say 500 tokens) through the model in one forward pass. Because all prompt tokens are known up front, they are processed in parallel — the GPU does one big matrix-multiply-heavy pass over all 500 positions at once. This is compute-bound: it saturates the GPU’s arithmetic units. Prefill is what you pay for time-to-first-token (TTFT), and its cost grows with prompt length.

During prefill the model computes, for every layer and every attention head, a key (K) and value (V) vector for each prompt token, and stores them — that is the KV cache.

Decode (the generation loop)

Now you generate one token at a time. Each decode step feeds only the single newest token through the model. Its attention needs the K/V of all previous tokens — but those are already in the cache, so you do not recompute them. Each decode step is therefore tiny in arithmetic (one token’s worth of matmuls) but must read the entire KV cache and all model weights from GPU memory. Decode is memory-bandwidth-bound, not compute-bound: the GPU spends its time moving data, and its expensive tensor cores sit mostly idle.

This is the central asymmetry of LLM serving:

The decode phase being memory-bound and low-utilization is exactly why one-request-at-a-time wastes the GPU, and exactly why batching helps: multiple requests can share the same weight read. Hold that thought for the tradeoff section.

Why the KV cache exists

Without a cache, generating token N would re-run attention over all N prior tokens from scratch — an (O(N^2)) blowup over a full sequence. The KV cache trades memory for compute: store each token’s K and V once, reuse them for every future step. It turns per-step attention cost from “re-read and recompute everything” into “read the cache.” The price is GPU memory that grows linearly with every token in every active request — which becomes the binding constraint on how many requests you can serve at once.

Worked example 1: a minimal FastAPI + Transformers server

Here is a complete, correct, single-file server. It is deliberately naive — synchronous generation, one request at a time — so it exposes every pitfall we then discuss. This is the “before” picture for the whole book.

# server.py
#   pip install fastapi "uvicorn[standard]" transformers torch accelerate
#   uvicorn server:app --host 0.0.0.0 --port 8000 --workers 1
import time
from contextlib import asynccontextmanager

import torch
from fastapi import FastAPI
from fastapi.concurrency import run_in_threadpool
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"
STATE = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the model ONCE at startup, not per request.
    tok = AutoTokenizer.from_pretrained(MODEL_ID)
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    tok.padding_side = "left"
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID, torch_dtype=torch.bfloat16, device_map="cuda"
    )
    model.eval()
    STATE["tok"], STATE["model"] = tok, model
    yield
    STATE.clear()


app = FastAPI(lifespan=lifespan)


class GenRequest(BaseModel):
    prompt: str
    max_new_tokens: int = 128
    temperature: float = 0.7
    top_p: float = 0.9
    do_sample: bool = True


@torch.inference_mode()
def _generate(req: GenRequest) -> dict:
    tok, model = STATE["tok"], STATE["model"]
    messages = [{"role": "user", "content": req.prompt}]
    inputs = tok.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    prompt_len = inputs.shape[1]

    t0 = time.perf_counter()
    out = model.generate(
        inputs,
        max_new_tokens=req.max_new_tokens,
        do_sample=req.do_sample,
        temperature=req.temperature,
        top_p=req.top_p,
        pad_token_id=tok.pad_token_id,
    )
    dt = time.perf_counter() - t0

    new_tokens = out[0, prompt_len:]
    text = tok.decode(new_tokens, skip_special_tokens=True)
    n_out = new_tokens.shape[0]
    return {
        "text": text,
        "prompt_tokens": int(prompt_len),
        "output_tokens": int(n_out),
        "latency_s": round(dt, 3),
        "tokens_per_s": round(n_out / dt, 1),
    }


@app.post("/generate")
async def generate(req: GenRequest):
    # Blocking, CPU/GPU-bound work goes to a thread so it does not
    # freeze the async event loop (see pitfalls).
    return await run_in_threadpool(_generate, req)


@app.get("/healthz")
async def healthz():
    return {"ok": "model" in STATE}

Test it:

curl -s localhost:8000/generate \
  -H 'content-type: application/json' \
  -d '{"prompt": "Explain KV cache in one sentence.", "max_new_tokens": 64}'

What this server gets right, and what it deliberately does not:

• Right: model loaded once at startup (not per request); @torch.inference_mode() disables gradient bookkeeping; blocking work offloaded off the event loop; chat template + pad token set correctly; returns real token counts and throughput.
• Deliberately wrong / naive: it serves one request at a time per worker (the model is a shared object and generate holds the GPU), it does not stream tokens (no TTFT benefit for the client), and it does no batching. Two simultaneous callers queue behind each other. That is the motivation for everything after this chapter.

Streaming, when you add it, uses TextIteratorStreamer + a background thread and a FastAPI StreamingResponse, so the client gets the first token as soon as prefill finishes rather than waiting for the whole answer — a large perceived latency win with no throughput change.

Generation parameters (what the knobs actually do)

generate is controlled by a GenerationConfig. The ones that matter for serving:

A subtle correctness trap: if a caller passes do_sample=False and a non-default temperature, recent Transformers will warn that the sampling flags are ignored. Decide your server’s contract explicitly rather than passing user knobs through blindly.

Adding streaming (TTFT the user can feel)

The naive server returns the whole answer at once. Streaming emits tokens as they are decoded, so the client sees output right after prefill:

from threading import Thread
from transformers import TextIteratorStreamer
from fastapi.responses import StreamingResponse

@app.post("/generate/stream")
async def generate_stream(req: GenRequest):
    tok, model = STATE["tok"], STATE["model"]
    inputs = tok.apply_chat_template(
        [{"role": "user", "content": req.prompt}],
        add_generation_prompt=True, return_tensors="pt",
    ).to(model.device)
    streamer = TextIteratorStreamer(tok, skip_prompt=True, skip_special_tokens=True)
    kwargs = dict(inputs=inputs, streamer=streamer,
                  max_new_tokens=req.max_new_tokens, do_sample=req.do_sample,
                  temperature=req.temperature, top_p=req.top_p,
                  pad_token_id=tok.pad_token_id)
    Thread(target=model.generate, kwargs=kwargs).start()  # runs off the event loop

    def emit():
        for piece in streamer:      # yields decoded text as tokens arrive
            yield piece
    return StreamingResponse(emit(), media_type="text/plain")

generate runs in a background thread and pushes tokens into the streamer; the handler yields them to the client. Same total work, dramatically better perceived latency — but note it still occupies the GPU serially. Streaming improves TTFT, not throughput.

Worked example 2: GPU memory budget (weights + KV cache + activations)

You cannot reason about serving without the memory math. The GPU must simultaneously hold model weights, the KV cache for every in-flight request, and transient activations. Run out and you get a CUDA OOM — the most common production failure.

Weights

[ \text{weight bytes} = (\text{number of parameters}) \times (\text{bytes per parameter}) ]

For a 7-billion-parameter model in BF16:

[ 7 \times 10^{9} \ \text{params} \times 2 \ \text{bytes} = 14 \times 10^{9} \ \text{bytes} \approx 14 \ \text{GB} ]

The rule of thumb “~2 GB per billion params in FP16/BF16” (and ~1 GB/B in INT8, ~0.5 GB/B in INT4) falls straight out of this.

KV cache (per token, then per request)

The KV cache stores a key and a value vector for every token, every layer, every KV head:

[ \text{bytes per token} = 2 \times L \times H_{kv} \times D_{h} \times b ]

where the leading (2) covers K and V, (L) is the number of transformer layers, (H_{kv}) the number of key/value heads, (D_{h}) the head dimension, and (b) the bytes per element. For a LLaMA-style 7B model with (L = 32), (H_{kv} = 32), (D_{h} = 128), BF16 ((b = 2)):

[ 2 \times 32 \times 32 \times 128 \times 2 = 524{,}288 \ \text{bytes} \approx 0.5 \ \text{MB per token} ]

For a request with a full context of 4,096 tokens:

[ 4096 \times 524{,}288 \ \text{bytes} = 2{,}147{,}483{,}648 \ \text{bytes} = 2 \ \text{GB} ]

So a single 4K-context request costs ~2 GB of KV cache on top of the 14 GB of weights. Note that models using grouped-query attention (GQA) have far fewer KV heads (H_{kv}) than query heads, which is specifically a trick to shrink this number — one reason modern models are cheaper to serve.

Activations and overhead

Beyond weights and KV cache, each forward pass allocates transient activation tensors, and the CUDA context / allocator reserves a fixed slab (often 1–2 GB). Activation memory scales with batch size and sequence length but is freed between steps, so it is usually a smaller, bounded term than the KV cache — which only grows. When you size a GPU, budget weights + peak KV + a safety margin for activations and fragmentation; do not plan to use the last gigabyte.

Putting it together on a 24 GB GPU

[ \text{free for KV + activations} \approx 24 \ \text{GB} - 14 \ \text{GB (weights)} - \sim 1\text{–}2 \ \text{GB (activations, CUDA ctx)} \approx 8 \ \text{GB} ]

At ~2 GB per 4K-token request, that GPU holds on the order of 4 concurrent full-length requests before OOM — and fewer if prompts are longer. This single calculation is why KV-cache efficiency (PagedAttention, Chapter 5) is the highest-leverage optimization in LLM serving: memory, not compute, usually caps your concurrency.

Metrics and the latency–throughput tradeoff

You cannot improve what you do not measure. Four numbers define an LLM endpoint.

Definitions and formulas follow the Anyscale benchmarking guide (see Further Reading). Report percentiles (p50/p95/p99), never just the mean — tail latency is what users feel and what SLOs are written against.

The tradeoff

Here is the crux, and it is why “just add batching” is not free:

• A single request in isolation gets the lowest possible latency: the whole GPU is devoted to it. But decode is memory-bound, so the GPU’s compute units are ~idle — terrible throughput per dollar.
• Batching many requests together amortizes each weight/KV read across all of them: one memory pass serves (B) requests. Throughput (tokens/s, req/s) rises sharply — you use the idle compute. But any individual request may wait to be batched and shares GPU cycles, so its TTFT and TPOT rise.

So batch size is a dial between latency (small batch, low utilization, expensive per token) and throughput (large batch, high utilization, cheap per token, worse tail latency). There is no single right setting — it depends on your SLO. Real serving engines make this dial dynamic (continuous batching), which we introduce next and detail in Chapter 5.

Worked example 3: reading a latency timeline

Numbers make the asymmetry concrete. Suppose a request has a 500-token prompt and generates 200 tokens, and we measure:

• prefill takes 120 ms (this is the TTFT, ignoring queueing),
• each decode step takes 15 ms.

Then:

[ \text{TTFT} = 120 \ \text{ms}, \qquad \text{generation} = 199 \times 15 \ \text{ms} \approx 2985 \ \text{ms} ]

[ \text{E2E} = 120 + 2985 \approx 3.1 \ \text{s}, \qquad \text{TPOT} = \frac{3105 - 120}{200 - 1} \approx 15 \ \text{ms}, \qquad \text{user TPS} = \frac{200}{2.985} \approx 67 \ \text{tok/s} ]

Two lessons. First, decode dominates E2E here (~3 s vs ~120 ms) — output length is your biggest latency lever, which is why capping max_new_tokens matters so much. Second, if you stream, the user sees a token at 120 ms instead of waiting 3.1 s: identical work, far better perceived latency. Streaming trades nothing in throughput; it only reshapes when the user first sees output.

Why one-request-at-a-time is wasteful — the road to batching

The naive server above processes requests serially. During each request’s decode phase the GPU is memory-bandwidth-bound and its tensor cores are mostly idle — you are paying for an A100/H100 and using a fraction of its FLOPs. Meanwhile a second caller just waits.

Batching fixes this by running multiple sequences through the model together, so a single read of the weights (and a single scheduling step) produces a token for every request in the batch. Because decode was memory-bound, adding more requests is nearly free on compute up to a point — you convert idle compute into throughput.

There are three flavors, in increasing sophistication (full treatment in Chapter 5):

The mental model: static/dynamic batching batches requests; continuous batching batches token-generation steps. The latter keeps the GPU full even when requests have wildly different lengths — which is the normal case. This is the single biggest reason a dedicated engine outperforms the naive server, often by an order of magnitude in throughput at the same latency.

Failure modes and pitfalls

The naive server fails in predictable ways. Know them cold.

• CUDA out of memory (OOM). The #1 killer. Causes: model too big for the GPU in the chosen dtype; too many concurrent requests inflating the KV cache; a single very long prompt/output. Symptoms: CUDA out of memory. Tried to allocate .... Fixes: smaller dtype/quantization, cap max_new_tokens and context length, limit concurrency, use an engine with paged KV. Do the memory math before deploying.
• Blocking the async event loop. FastAPI is async, but model.generate() is a long, synchronous, GPU-bound call. If you await it directly in the handler (or call it inline), it freezes the entire event loop — health checks time out, every other connection stalls. Fix: run_in_threadpool (as above) or a dedicated worker/queue. This bug looks like “the server randomly hangs under load.”
• No batching / serial serving. Two users -> the second waits for the first. Throughput is capped at one request’s worth of decode, and the expensive GPU sits underutilized. This is not a bug to fix in this server — it is the reason to graduate to vLLM/TGI.
• Tokenizer mismatch / wrong chat template. Using a tokenizer from a different model, skipping apply_chat_template, or missing special tokens produces fluent-looking garbage with no error. Always pair the exact tokenizer with the model and use the official chat template.
• Padding-side and pad-token mistakes. Right-padding a decoder-only batch, or a missing pad_token, silently corrupts batched generation. Left-pad; set pad_token = eos_token if absent.
• FP32 by accident. Forgetting torch_dtype loads in FP32 and doubles weight memory — an instant OOM on models that would fit fine in BF16.
• Unbounded generation. No max_new_tokens and a wrong/missing EOS -> the model runs until it hits some default cap, burning GPU time and blocking others. Always bound output.
• Cold-start latency ignored. First request after a scale-up waits for model download + load (seconds to minutes). Add readiness probes (/healthz gating on model-loaded) so traffic is not routed to a not-yet-ready pod (Chapters 3, 6).

Tools comparison (brief — pointers to later chapters)

Rule of thumb: build the raw server once to understand the loop, then never ship it. For anything real, reach for an engine that does continuous batching and paged KV memory. TorchServe is another general-purpose model server in this space, but for LLMs specifically the KV-cache-aware engines (vLLM, TGI, TensorRT-LLM) win decisively.

Production checklist — what an interviewer probes

1. “Walk me through a request.” Expected: request -> tokenize (+ chat template) -> prefill (compute-bound, sets TTFT) -> decode loop reusing KV cache (memory-bound, sets TPOT) -> detokenize -> response. Naming the prefill/decode split is the tell that you actually understand serving.
2. “How much GPU memory does model X need?” Expected: weights = params × bytes/param (~2 GB/B in FP16); plus KV cache = (2 \times L \times H_{kv} \times D_h \times b) per token per request; plus activations/overhead. Know the ~0.5 MB/token, ~2 GB-per-4K-request order of magnitude.
3. “Your p99 latency is bad but the GPU is at 30% util — why?” Expected: decode is memory-bandwidth-bound and you are serving serially / with tiny batches; add continuous batching to convert idle compute into throughput.
4. “How do you trade latency for throughput?” Expected: batch size is the dial; larger batches amortize weight reads (higher throughput) at the cost of per-request TTFT/TPOT; pick per SLO; use continuous batching to get most of the throughput without static batching’s head-of-line blocking.
5. “What’s your OOM story?” Expected: memory math up front; cap max_new_tokens and context; limit concurrency; quantize; paged KV; monitor KV-cache utilization, not just GPU memory.
6. “Why not just await model.generate() in the handler?” Expected: it blocks the event loop; offload to a threadpool/worker or use an engine with its own scheduler.
7. “Which metrics do you alert on?” Expected: TTFT, TPOT/ITL, E2E — all at p50/p95/p99 — plus throughput (tok/s, req/s), queue depth, KV-cache utilization, and error/OOM rate. Percentiles, not means.
8. “When do you reach for vLLM/TGI/Triton over your own server?” Expected: as soon as you need concurrency — continuous batching + paged KV are hard to build well and are the whole point; roll your own only to learn or for genuinely custom logic.

Further reading

• Hugging Face — Text generation tutorial (generate): https://huggingface.co/docs/transformers/en/llm_tutorial
• Hugging Face — Generation config / GenerationConfig reference: https://huggingface.co/docs/transformers/en/main_classes/text_generation
• FastAPI documentation: https://fastapi.tiangolo.com/
• Text Generation Inference (TGI) docs: https://huggingface.co/docs/text-generation-inference/en/index
• TGI source and features: https://github.com/huggingface/text-generation-inference
• vLLM — PagedAttention & continuous batching overview: https://www.runpod.io/articles/guides/vllm-pagedattention-continuous-batching
• NVIDIA — Mastering LLM Techniques: Inference Optimization (KV cache, batching): https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
• Anyscale — Understand LLM latency and throughput metrics (TTFT/TPOT/ITL/TPS/RPS): https://docs.anyscale.com/llm/serving/benchmarking/metrics
• KV cache memory calculation (worked example): https://mbrenndoerfer.com/writing/kv-cache-memory-calculation-llm-inference-gpu
• Pierre Lienhart — LLM Inference Series: KV caching, a deeper look: https://medium.com/@plienhar/llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8
• NVIDIA Triton Inference Server: https://github.com/triton-inference-server/server

Where this goes next: Chapter 2 containerizes this server; Chapter 4 load-tests it to see the serial bottleneck; Chapter 5 replaces it with vLLM and continuous batching to fix everything this chapter exposed.

Topic 2: Containerization with Docker

What You’ll Learn

This topic teaches you how to:

• Containerize LLM serving applications
• Create efficient Docker images for ML workloads
• Handle large model files
• Use multi-stage builds
• Configure environment variables
• Optimize image size

Why We Need This

Business Need

• Consistency: Same model behavior in dev, staging, production
• Speed to market: Deploy faster without environment setup
• Cost reduction: Standardized deployment reduces ops overhead
• Compliance: Reproducible deployments for audit trails

Technical Need

• Environment consistency: Python version, dependencies, system libraries
• Isolation: No conflicts between different ML projects
• Portability: Run anywhere (local, cloud, edge)
• Versioning: Tag images with model versions for rollback

Real-World Impact

Without containerization:

• ❌ “Works on my machine” problems
• ❌ Difficult to reproduce production issues
• ❌ Slow deployments (manual environment setup)
• ❌ Can’t scale easily (each server needs manual setup)

Industry Use Cases

1. Multi-Cloud Deployment

Company: Enterprise ML platforms Use Case:

• Same Docker image runs on AWS, GCP, Azure
• No vendor lock-in, easy migration

Example:

# Build once, deploy anywhere
docker build -t llm-model:v1.0 .
# Deploy to AWS
docker push llm-model:v1.0
# Deploy to GCP
docker push llm-model:v1.0

2. CI/CD Pipelines

Company: All tech companies Use Case:

• Automated testing in containers
• Consistent environments across pipeline stages

Example:

# GitHub Actions / GitLab CI
- name: Test model
  run: |
    docker build -t test-model .
    docker run test-model pytest

3. Edge Deployment

Company: IoT, autonomous vehicles Use Case:

• Deploy models to edge devices
• Same container on server and edge

Example:

# Deploy to edge device
docker save llm-model:v1.0 | ssh edge-device docker load

4. Model Versioning & Rollback

Company: ML platforms (MLflow, Weights & Biases) Use Case:

• Tag images with model versions
• Quick rollback to previous version

Example:

docker tag llm-model:latest llm-model:v1.2.3
docker tag llm-model:latest llm-model:v1.2.2  # Rollback

5. Development Teams

Company: All companies with ML teams Use Case:

• New developers can run models immediately
• No “works on my machine” issues

Example:

# New developer setup
git clone repo
docker-compose up  # Everything works!

Industry-Standard Boilerplate Code

Production Dockerfile (Industry Standard)

# Multi-stage build for production (used by: Google, AWS, Microsoft)
# Stage 1: Builder
FROM python:3.9-slim as builder

WORKDIR /app

# Install build dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements and install
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# Stage 2: Runtime (minimal image)
FROM python:3.9-slim

WORKDIR /app

# Copy Python dependencies from builder
COPY --from=builder /root/.local /root/.local

# Make sure scripts are in PATH
ENV PATH=/root/.local/bin:$PATH

# Create non-root user (security best practice)
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser

# Copy application
COPY --chown=appuser:appuser . .

# Health check (for K8s, load balancers)
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
    CMD python -c "import requests; requests.get('http://localhost:8000/health')" || exit 1

# Expose port
EXPOSE 8000

# Run application
CMD ["python", "app.py"]

Docker Compose for Development (Industry Standard)

# docker-compose.yml
# Used by: Development teams, local testing
version: '3.8'

services:
  llm-serving:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    environment:
      - MODEL_NAME=gpt2
      - DEVICE=cpu
      - LOG_LEVEL=INFO
    volumes:
      # Mount code for development (hot reload)
      - .:/app
      # Mount models directory
      - ./models:/app/models
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  # Optional: Add monitoring
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

Build & Deploy Script (Industry Standard)

#!/bin/bash
# build-and-deploy.sh
# Used by: CI/CD pipelines, deployment automation

set -e  # Exit on error

IMAGE_NAME="llm-serving"
VERSION="${1:-latest}"
REGISTRY="your-registry.com"  # Docker Hub, ECR, GCR, etc.

echo "Building image: ${IMAGE_NAME}:${VERSION}"

# Build image
docker build -t ${IMAGE_NAME}:${VERSION} .

# Tag for registry
docker tag ${IMAGE_NAME}:${VERSION} ${REGISTRY}/${IMAGE_NAME}:${VERSION}
docker tag ${IMAGE_NAME}:${VERSION} ${REGISTRY}/${IMAGE_NAME}:latest

# Push to registry
docker push ${REGISTRY}/${IMAGE_NAME}:${VERSION}
docker push ${REGISTRY}/${IMAGE_NAME}:latest

echo "Image pushed: ${REGISTRY}/${IMAGE_NAME}:${VERSION}"

# Deploy (example for K8s)
# kubectl set image deployment/llm-serving llm-serving=${REGISTRY}/${IMAGE_NAME}:${VERSION}

GPU Dockerfile (Industry Standard)

# GPU-enabled Dockerfile
# Used by: Production ML serving, high-performance inference
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04

# Install Python
RUN apt-get update && apt-get install -y \
    python3.9 \
    python3-pip \
    curl \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install PyTorch with CUDA
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install other dependencies
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

# Copy application
COPY . .

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

EXPOSE 8000

CMD ["python3", "app.py"]

Usage in Production

# Build for production
docker build -t llm-serving:prod -f Dockerfile.prod .

# Run with GPU
docker run --gpus all -p 8000:8000 llm-serving:prod

# Run with resource limits
docker run \
  --memory="4g" \
  --cpus="2.0" \
  -p 8000:8000 \
  llm-serving:prod

# Run in production (with restart policy)
docker run -d \
  --name llm-serving \
  --restart unless-stopped \
  -p 8000:8000 \
  llm-serving:prod

Key Concepts

Dockerfile Basics

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Multi-Stage Builds

Split build into stages to reduce final image size:

1. Builder stage: Install dependencies, download models
2. Runtime stage: Copy only what’s needed

Layer Caching

Order matters! Put frequently changing files last:

# Good: Dependencies change less often
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .  # Code changes more often

Dockerfile Examples

Basic Dockerfile

See Dockerfile.basic for a simple example.

Optimized Dockerfile

See Dockerfile.optimized for:

• Multi-stage build
• Model caching
• Smaller image size
• Better layer caching

GPU Dockerfile

See Dockerfile.gpu for CUDA/GPU support.

Building Images

Basic Build

docker build -t llm-serving:latest .

With Build Args

docker build \
  --build-arg MODEL_NAME=gpt2 \
  --build-arg PYTHON_VERSION=3.9 \
  -t llm-serving:gpt2 .

Multi-stage Build

docker build -f Dockerfile.optimized -t llm-serving:optimized .

Running Containers

CPU Mode

docker run -p 8000:8000 llm-serving:latest

GPU Mode (NVIDIA Docker)

docker run --gpus all -p 8000:8000 llm-serving:gpu

With Environment Variables

docker run \
  -p 8000:8000 \
  -e MODEL_NAME=gpt2 \
  -e MAX_LENGTH=100 \
  llm-serving:latest

With Volume Mounts (for models)

docker run \
  -p 8000:8000 \
  -v ./models:/app/models \
  llm-serving:latest

Docker Compose

For multi-container setups, use docker-compose.yml:

• Application container
• Model storage
• Monitoring
• Load balancer

Image Optimization

Reduce Image Size

1. Use slim base images: python:3.9-slim instead of python:3.9
2. Multi-stage builds: Don’t include build tools in final image
3. Remove cache: pip install --no-cache-dir
4. Combine RUN commands: Fewer layers = smaller image

Speed Up Builds

1. Layer caching: Order Dockerfile commands by change frequency
2. Build cache: Use --cache-from for CI/CD
3. Parallel builds: Build multiple images simultaneously

Handling Large Models

Option 1: Download in Image

RUN python -c "from transformers import AutoModel; AutoModel.from_pretrained('gpt2')"

Pros: Self-contained Cons: Large image, slow builds

Option 2: Volume Mount

docker run -v ./models:/app/models ...

Pros: Small image, fast builds Cons: Need models on host

Option 3: Model Registry

Download from S3/GCS at runtime Pros: Flexible, versioned Cons: Requires network access

Best Practices

1. Use .dockerignore: Exclude unnecessary files
2. Tag images: Use semantic versioning
3. Health checks: Add HEALTHCHECK instruction
4. Non-root user: Run as non-root for security
5. Resource limits: Set memory/CPU limits
6. Logging: Configure proper logging

Exercises

1. Build basic image: Create Dockerfile for basic serving
2. Optimize image: Reduce image size by 50%
3. Multi-stage build: Create optimized multi-stage Dockerfile
4. GPU support: Add CUDA support to Dockerfile
5. Docker Compose: Create compose file for full stack

Common Issues

Out of Memory

• Problem: Container runs out of memory
• Solution: Increase Docker memory limit or use smaller model

Slow Builds

• Problem: Building takes forever
• Solution: Use layer caching, download models in separate stage

GPU Not Available

• Problem: CUDA not available in container
• Solution: Install nvidia-docker, use --gpus all flag

Large Image Size

• Problem: Image is several GB
• Solution: Use multi-stage build, slim base images

Next Steps

• Topic 3: Deploy containers to Kubernetes
• Topic 4: Load test your containerized app
• Topic 5: Optimize for production with vLLM

Further Reading

• Docker Best Practices
• Multi-stage Builds
• NVIDIA Docker

Docker for GPU LLM Serving

Containerizing an LLM inference service so it runs the same on your laptop, in CI, and on a rented H100 — without a 40 GB image, a leaked token, or a CUDA driver version is insufficient at 3 a.m.

Why this matters

An LLM server is not a normal web app. It links against CUDA, cuDNN, NCCL, and a specific PyTorch build; it needs a physical GPU exposed into the container; and it depends on multi-gigabyte model weights that you do not want to redownload on every restart. Get the containerization wrong and you hit one of a dozen classic failures: the image balloons to tens of gigabytes, the container can’t see the GPU, the CUDA version mismatches the host driver, your Hugging Face token ends up baked into a layer, or the server runs as root with no healthcheck and the orchestrator can’t tell it’s wedged.

Containers are also the unit that Kubernetes, Nomad, ECS, and every autoscaler schedule. A well-built image is the foundation for everything in later chapters (K8s, canary, autoscaling). This chapter is about getting that foundation right.

The intuition first, then the exact mechanisms.

Core intuition

Three ideas carry most of the weight.

1. The container shares the host’s GPU driver, not its own. You never install an NVIDIA driver inside the image. The driver is a kernel module and lives on the host. The container ships userspace CUDA libraries. At docker run time, the NVIDIA Container Toolkit injects the host driver’s device files and libraries into the container. So the contract is: host driver must be new enough for the container’s CUDA userspace. This is the single most important mental model in GPU containers.

2. Model weights are data, not code. Code changes daily and is tiny; a 14 GB weights file changes rarely and is huge. Baking weights into an image layer couples the two lifecycles badly. The default for production is to keep weights out of the image and bring them in as a mounted volume or a startup download into a persistent cache.

3. Build image ≠ runtime image. The tools you need to compile CUDA kernels (nvcc, headers, build-essential) are hundreds of MB you never need to run the server. Multi-stage builds let you compile in a fat stage and copy only the artifacts into a lean runtime stage.

Hold these three and the rest is detail.

Mechanism 1 — GPU base images and the CUDA/driver contract

The nvidia/cuda image family

NVIDIA publishes nvidia/cuda images on Docker Hub and NGC. Tags follow the pattern:

nvidia/cuda:<cuda_version>-<flavor>-<os>
# e.g.
nvidia/cuda:12.4.1-runtime-ubuntu22.04
nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
nvidia/cuda:12.4.1-devel-ubuntu22.04

There are three flavors, and picking the wrong one is a common source of bloat:

Rule of thumb: build in devel, ship on runtime. If your framework ships prebuilt wheels (most do), you may not even need devel.

Driver/runtime compatibility

The host driver exposes a maximum supported CUDA version. CUDA has forward compatibility within a major version and minor-version compatibility so that, e.g., a driver supporting CUDA 12.2 can generally run CUDA 12.4 userspace on data-center GPUs via the compat package — but do not rely on this casually. The safe posture:

• Check the host: nvidia-smi prints Driver Version and CUDA Version (the max CUDA the driver supports).
• Pick a container CUDA version ≤ that, or confirm forward-compat coverage.
• Pin the container CUDA minor version explicitly (12.4.1, not 12).

The failure you’re avoiding looks like:

CUDA driver version is insufficient for CUDA runtime version

That means the container’s CUDA userspace is newer than the host driver can serve. Fix by upgrading the host driver or downgrading the image’s CUDA version — you cannot fix it inside the image.

Mechanism 2 — The NVIDIA Container Toolkit and --gpus

A plain docker run gives the container no GPU. Two pieces make it work.

The toolkit

The NVIDIA Container Toolkit is a set of host packages (nvidia-container-toolkit, the nvidia-ctk CLI, and a runtime shim) that automatically configure a container to use NVIDIA GPUs by mounting the driver libraries and device nodes at startup. You install it on the host, once:

# 1. Add NVIDIA's apt repo (see official install guide for the current key/URL)
sudo apt-get install -y nvidia-container-toolkit

# 2. Wire it into the Docker daemon (writes the "nvidia" runtime into
#    /etc/docker/daemon.json)
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Prerequisite: the NVIDIA driver is already installed on the host. You do not install the CUDA toolkit on the host — only the driver.

Verify the whole chain end-to-end:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If that prints your GPU table, host driver + toolkit + runtime are all good. This is the first thing to run when a GPU container “can’t see the GPU.”

--gpus

Once the toolkit is installed, --gpus selects which GPUs to expose:

--gpus all                 # all GPUs
--gpus '"device=0,1"'      # only GPU 0 and 1 (note the nested quoting)
--gpus 2                   # any 2 GPUs

Under the hood this sets NVIDIA_VISIBLE_DEVICES and triggers the toolkit’s injection hook. Older stacks used --runtime=nvidia plus that env var directly; --gpus is the modern, preferred flag. Some tools (vLLM’s docs) still show --runtime nvidia — it’s equivalent when the runtime is registered.

Mechanism 3 — Handling large model weights

This is where architecture decisions bite hardest. You have three options.

Option A — Bake weights into the image

Copy the weights in during build (COPY ./model /model). The image is fully self-contained: pull it and run, no network, no external volume.

• Pro: hermetic, reproducible, air-gap friendly, one artifact to sign/scan.
• Con: the image is now 15–150 GB. Every push/pull moves all of it. Layer caching is useless once weights change. Registry storage costs balloon. Build context upload is slow.
• Verdict: reserve for small models, air-gapped/regulated deployments, or when the exact weights are part of your release contract.

Option B — Mount weights as a volume

Keep weights on the host / network storage and bind-mount at runtime (-v /data/models:/models). This is what the vLLM and TGI official images assume.

• Pro: small image, weights shared across containers and versions, swap models without rebuilding, fast cold builds.
• Con: image is no longer self-contained; you must provision and pre-populate the volume; in K8s you need a PersistentVolume / hostPath / CSI mount.
• Verdict: the default for most production on a fixed node pool or shared filesystem.

Option C — Download at startup into a cache

The container downloads weights from Hugging Face (or S3/GCS) on first boot into a persistent cache directory, then reuses the cache on restart.

• Pro: smallest image, model chosen by env var, trivially swappable.
• Con: cold start pays a multi-GB download; needs network egress + a token for gated models; if the cache volume isn’t persistent you redownload every restart (a classic and expensive bug); registry outages ≠ HF outages now both matter.
• Verdict: great for dev, experimentation, and autoscaling where a warm cache volume (or a pre-baked node image) hides the download.

The Hugging Face cache — make it persist

The huggingface_hub library caches downloads under HF_HOME (default ~/.cache/huggingface; the hub cache is \(HF\_HOME\)/hub). The env vars that matter:

• HF_HOME — root of all HF caches. Set this and everything follows.
• HF_HUB_CACHE (older: HUGGINGFACE_HUB_CACHE) — the model blob cache specifically.
• HF_TOKEN — auth for gated/private models.

The whole point of Options B/C is to mount a volume at the cache path so weights survive container restarts:

# vLLM: cache lives at /root/.cache/huggingface inside the image
-v ~/.cache/huggingface:/root/.cache/huggingface

# TGI: the official image sets HUGGINGFACE_HUB_CACHE=/data, so mount /data
-v $PWD/data:/data

Miss this mount and every docker run redownloads the model — slow, costly, and rate-limit-prone.

Mechanism 4 — Multi-stage builds, layer caching, and .dockerignore

Multi-stage

A multi-stage build uses multiple FROM statements. Early stages compile; the final stage copies only what’s needed. For an LLM server that means: compile custom kernels / install a heavy build toolchain in a devel stage, then COPY --from=builder the installed environment into a slim runtime stage. The devel layers never ship.

Layer caching order

Docker caches each instruction as a layer and reuses it until an input changes; everything after a changed layer is rebuilt. So order from least-to-most volatile:

1. Base image
2. System packages (apt-get)
3. Dependency manifests (requirements.txt / pyproject.toml) and pip install
4. Application source code

Copy requirements.txt and install before copying your source. Then a one-line code change reuses the (slow) dependency layer instead of reinstalling PyTorch every build. Use BuildKit cache mounts for the pip/uv cache to speed rebuilds further:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

.dockerignore

The build context is everything Docker uploads to the daemon before building. Without a .dockerignore, a stray ./models, .git, __pycache__, or a 30 GB checkpoint gets shipped into the build — slow, and a vector for secrets and bloat. A minimal one:

.git
__pycache__/
*.pyc
*.pt
*.safetensors
models/
data/
.env
*.log
.venv/

This is also your first line of defense against COPY . . accidentally baking weights or a .env file into a layer.

Mechanism 5 — Reproducibility, security, healthchecks, config

Pin everything. python:3.11-slim is a moving target. Prefer a digest for the base and pinned versions for packages:

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04@sha256:<digest>

Pin your Python deps (lockfile or ==), and tag your own images with an immutable version, never rely on :latest in production.

Run as non-root. By default containers run as UID 0. A container escape from root is worse than from an unprivileged user. Create a user and drop to it:

RUN useradd --create-home --uid 10001 appuser
USER appuser

Note some GPU stacks and cache paths assume /root; if you run non-root, point HF_HOME at a directory that user can write.

Keep secrets out of layers. Never ENV HF_TOKEN=hf_xxx or COPY .env — both persist in the image history for anyone who pulls it. Pass secrets at runtime (-e HF_TOKEN=..., Docker/K8s secrets) or use BuildKit --secret mounts for build-time-only credentials.

Add a HEALTHCHECK. Orchestrators restart containers that fail their health probe. For an OpenAI-compatible server, probe the health/models endpoint:

HEALTHCHECK --interval=30s --timeout=5s --start-period=180s --retries=3 \
  CMD curl -fsS http://localhost:8000/health || exit 1

The long start-period matters: model load can take minutes, and you don’t want the container killed during warmup.

Config via env, not baked files. Model id, tensor-parallel size, port, and dtype should be env vars / CLI args so one image serves many configs.

Fully worked example

A real, multi-stage GPU Dockerfile for a vLLM-based OpenAI-compatible server. (If you just want vLLM, the official vllm/vllm-openai image is usually the right call — this shows the general pattern you’d use for a custom server or a framework without an official image.)

# syntax=docker/dockerfile:1.7

###############################################################################
# Stage 1: builder — has nvcc + build tools, compiles/installs the env.
###############################################################################
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builder

ENV DEBIAN_FRONTEND=noninteractive \
    PIP_NO_CACHE_DIR=0 \
    PYTHONDONTWRITEBYTECODE=1

# System build deps. Pin, clean apt lists to keep the layer lean.
RUN apt-get update && apt-get install -y --no-install-recommends \
        python3.11 python3.11-venv python3-pip build-essential git \
    && rm -rf /var/lib/apt/lists/*

# Isolated virtualenv so we can copy the whole thing to the runtime stage.
RUN python3.11 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Dependency layer FIRST — cached across code changes.
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --upgrade pip && pip install -r requirements.txt

###############################################################################
# Stage 2: runtime — slim CUDA runtime, no compilers, non-root, healthcheck.
###############################################################################
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 AS runtime

ENV DEBIAN_FRONTEND=noninteractive \
    PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    # Persist weights here; mount a volume at this path (see docker run).
    HF_HOME=/models/hf

# Only the runtime OS deps: python + curl (for healthcheck). No build-essential.
RUN apt-get update && apt-get install -y --no-install-recommends \
        python3.11 curl \
    && rm -rf /var/lib/apt/lists/*

# Copy the fully-built virtualenv from the builder stage — no nvcc ships.
COPY --from=builder /opt/venv /opt/venv

# Non-root user that can write the cache dir.
RUN useradd --create-home --uid 10001 appuser \
    && mkdir -p /models/hf && chown -R appuser:appuser /models
COPY --chown=appuser:appuser ./app /app
WORKDIR /app
USER appuser

EXPOSE 8000

# Config comes from env / CLI at runtime, not baked in.
ENV MODEL_ID=meta-llama/Llama-3.1-8B-Instruct \
    TENSOR_PARALLEL_SIZE=1

HEALTHCHECK --interval=30s --timeout=5s --start-period=300s --retries=3 \
    CMD curl -fsS http://localhost:8000/health || exit 1

# Exec form so signals reach the process (clean shutdown).
ENTRYPOINT ["python3.11", "-m", "vllm.entrypoints.openai.api_server"]
CMD ["--host", "0.0.0.0", "--port", "8000"]

Build and run:

# Build (BuildKit on for cache mounts + syntax directive)
DOCKER_BUILDKIT=1 docker build -t my-llm-server:1.0.0 .

# Run: expose GPUs, mount the HF cache volume, pass secrets at runtime.
docker run --rm \
  --gpus all \                                  # expose all GPUs (toolkit required)
  --ipc=host \                                  # shared mem for NCCL / TP; see note
  -p 8000:8000 \
  -v $HOME/.cache/hf:/models/hf \               # persist weights across restarts
  -e HF_TOKEN=$HF_TOKEN \                        # gated-model auth, NOT baked in
  my-llm-server:1.0.0 \
  --model meta-llama/Llama-3.1-8B-Instruct \    # config via CLI
  --tensor-parallel-size 1

Why --ipc=host? vLLM (and PyTorch tensor-parallel generally) uses shared memory (/dev/shm) for inter-process/GPU communication. Docker’s default /dev/shm is 64 MB, which causes cryptic crashes or hangs under load. --ipc=host (or --shm-size=1g) gives it room. TGI uses --shm-size 1g for the same reason.

Reference: the official one-liner most teams actually start from —

# vLLM official image
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env HF_TOKEN=$HF_TOKEN -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest --model mistralai/Mistral-7B-Instruct-v0.2

# TGI official image
docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id teknium/OpenHermes-2.5-Mistral-7B

Compose snippet

docker compose needs the deploy.resources.reservations.devices block to request GPUs:

services:
  llm:
    image: my-llm-server:1.0.0
    ports:
      - "8000:8000"
    ipc: host                       # equivalent to --ipc=host
    environment:
      - HF_TOKEN=${HF_TOKEN}        # sourced from host env / .env, not in image
      - MODEL_ID=meta-llama/Llama-3.1-8B-Instruct
    volumes:
      - hf-cache:/models/hf         # persistent named volume for weights
    command: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all            # or `device_ids: ["0","1"]`
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      start_period: 300s
      retries: 3

volumes:
  hf-cache:

Weights-handling comparison

Pin the model revision (commit SHA), not just the repo name, when reproducibility matters for Options B and C.

Failure modes and pitfalls

• Driver/runtime mismatch — CUDA driver version is insufficient. Container CUDA newer than host driver supports. Fix host driver or lower image CUDA; can’t be patched in the image.
• GPU invisible — torch.cuda.is_available() is False, no nvidia-smi in container. Toolkit not installed, --gpus omitted, or runtime not registered. Test with docker run --gpus all nvidia/cuda:...-base nvidia-smi.
• Giant images — shipped the devel base, apt lists left behind, weights baked in, or pip cache retained. Use multi-stage, --no-install-recommends, clean /var/lib/apt/lists, and keep weights out.
• Redownloading weights every restart — cache path not mounted to a persistent volume, or HF_HOME points at an ephemeral dir. Mount a named volume at the cache path.
• Secrets in layers — ENV HF_TOKEN=... or COPY .env persists in image history forever. Pass at runtime or use BuildKit secrets.
• Running as root — default UID 0; a bad default for security and for shared-filesystem permissions. Create and drop to a non-root user.
• No / bad healthcheck — orchestrator can’t detect a wedged server, or kills it during a 3-minute model load. Add a healthcheck with a generous start-period.
• /dev/shm too small — default 64 MB causes NCCL/tensor-parallel hangs and Bus error. Use --ipc=host or --shm-size.
• :latest everywhere — non-reproducible builds and surprise upgrades. Pin base by digest, deps by version, your image by semver.
• Signals ignored — shell-form CMD runs under /bin/sh which doesn’t forward SIGTERM; use exec-form ENTRYPOINT/CMD so shutdown is graceful and requests drain.

Tools and options comparison

Production checklist — what an interviewer probes

1. “How does a container get access to the GPU?” — Host driver + NVIDIA Container Toolkit + --gpus. You never install the driver in the image; the toolkit injects host driver libs at runtime. Verify with docker run --gpus all ... nvidia-smi.
2. “runtime vs devel base image — which do you ship?” — Build in devel, ship on runtime via multi-stage. Shipping devel is a multi-GB mistake.
3. “Where do the weights live and why?” — Articulate bake vs mount vs download and the cold-start/size/reproducibility tradeoffs; know that a mounted persistent HF cache (HF_HOME) is the usual answer.
4. “How do you keep the image small?” — Multi-stage, --no-install-recommends, clean apt lists, .dockerignore, don’t bake weights, BuildKit cache mounts.
5. “How do you handle the CUDA/driver version contract?” — Pin container CUDA ≤ host-driver-supported; understand minor-version/forward compatibility; recognize the “driver insufficient” error.
6. “How do secrets and config get in?” — Runtime env / orchestrator secrets and BuildKit --secret, never ENV/COPY .env (persists in layer history).
7. “Non-root, healthcheck, signals?” — Drop to an unprivileged UID, healthcheck with a long start-period for model load, exec-form entrypoint for graceful SIGTERM draining.
8. “Why --ipc=host / --shm-size?” — Tensor-parallel / NCCL uses /dev/shm; the 64 MB default causes hangs and bus errors under load.

Further reading

• NVIDIA Container Toolkit — repo and overview: https://github.com/NVIDIA/nvidia-container-toolkit
• NVIDIA Container Toolkit — install guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
• nvidia/cuda image tags (Docker Hub): https://hub.docker.com/r/nvidia/cuda/tags
• CUDA container supported tags & flavors: https://gitlab.com/nvidia/container-images/cuda/-/blob/master/doc/supported-tags.md
• vLLM — Using Docker: https://docs.vllm.ai/en/stable/deployment/docker/
• Hugging Face TGI — Nvidia GPU install: https://huggingface.co/docs/text-generation-inference/en/installation_nvidia
• Hugging Face TGI — Quick Tour: https://huggingface.co/docs/text-generation-inference/quicktour
• Hugging Face Hub — Understand caching (HF_HOME): https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache
• Docker — Multi-stage builds: https://docs.docker.com/build/building/multi-stage/
• Docker — Building best practices: https://docs.docker.com/build/building/best-practices/
• Docker — Build cache & BuildKit: https://docs.docker.com/build/cache/
• Docker Compose — GPU support: https://docs.docker.com/compose/how-tos/gpu-support/

Topic 3: Kubernetes Deployment

What You’ll Learn

This topic teaches you how to:

• Deploy LLM serving applications to Kubernetes
• Configure health checks and probes
• Set resource limits and requests
• Use ConfigMaps and Secrets
• Handle rolling updates
• Scale applications

Why We Need This

Business Need

• Cost optimization: Auto-scale down during low traffic, save money
• Reliability: 99.9% uptime SLA requires automatic recovery
• Speed: Deploy updates in seconds, not hours
• Compliance: Resource limits ensure fair resource usage

Technical Need

• Orchestration: Manage hundreds of containers automatically
• Resource isolation: Prevent one service from starving others
• Service discovery: Automatic load balancing across pods
• Self-healing: Restart failed containers automatically

Real-World Impact

Without Kubernetes:

• ❌ Manual scaling (slow, error-prone)
• ❌ Downtime during deployments
• ❌ Resource conflicts between services
• ❌ Difficult to manage at scale (100+ services)

Industry Use Cases

1. Large-Scale ML Platforms

Company: OpenAI, Anthropic, HuggingFace Use Case:

• Serve millions of API requests per day
• Auto-scale based on demand
• Zero-downtime deployments

Example:

# Handles traffic spikes automatically
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 10
  maxReplicas: 1000  # Scale to 1000 pods during peak

2. Enterprise ML Infrastructure

Company: Banks, healthcare, finance Use Case:

• Multi-tenant ML serving
• Resource quotas per team
• Compliance and audit trails

Example:

# Resource quotas per namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ml-team-quota
spec:
  hard:
    requests.cpu: "100"
    requests.memory: 200Gi
    limits.nvidia.com/gpu: "10"

3. Multi-Region Deployment

Company: Global SaaS companies Use Case:

• Deploy same model to multiple regions
• Low latency for global users
• Regional failover

Example:

# Deploy to multiple regions
kubectl apply -f deployment.yaml --context us-east
kubectl apply -f deployment.yaml --context eu-west
kubectl apply -f deployment.yaml --context asia-pacific

4. A/B Testing Infrastructure

Company: Tech companies (Netflix, Spotify) Use Case:

• Run multiple model versions simultaneously
• Split traffic between versions
• Compare performance

Example:

# Deploy model A and B
kubectl apply -f model-a-deployment.yaml
kubectl apply -f model-b-deployment.yaml
# Route 50% traffic to each

5. Development/Staging/Production

Company: All companies Use Case:

• Same deployment config for all environments
• Easy promotion from dev → staging → prod

Example:

# Deploy to dev
kubectl apply -f deployment.yaml -n dev

# Test, then promote to staging
kubectl apply -f deployment.yaml -n staging

# Finally, production
kubectl apply -f deployment.yaml -n production

Industry-Standard Boilerplate Code

Complete Production Deployment (Industry Standard)

# deployment.yaml
# Used by: Production ML serving at scale
apiVersion: v1
kind: Namespace
metadata:
  name: llm-serving
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-config
  namespace: llm-serving
data:
  model_name: "gpt2"
  max_length: "100"
  temperature: "0.7"
  log_level: "INFO"
---
apiVersion: v1
kind: Secret
metadata:
  name: llm-secrets
  namespace: llm-serving
type: Opaque
stringData:
  api_key: "your-api-key-here"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving
  namespace: llm-serving
  labels:
    app: llm-serving
    version: v1.0.0
spec:
  replicas: 3  # High availability
  selector:
    matchLabels:
      app: llm-serving
  template:
    metadata:
      labels:
        app: llm-serving
        version: v1.0.0
    spec:
      containers:
      - name: llm-serving
        image: your-registry/llm-serving:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
          name: http
        env:
        # From ConfigMap
        - name: MODEL_NAME
          valueFrom:
            configMapKeyRef:
              name: llm-config
              key: model_name
        - name: MAX_LENGTH
          valueFrom:
            configMapKeyRef:
              name: llm-config
              key: max_length
        # From Secret
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: llm-secrets
              key: api_key
        resources:
          requests:
            memory: "4Gi"
            cpu: "2000m"
            nvidia.com/gpu: 1  # GPU request
          limits:
            memory: "8Gi"
            cpu: "4000m"
            nvidia.com/gpu: 1  # GPU limit
        # Health checks (critical for production)
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60  # Model loading time
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
        startupProbe:
          httpGet:
            path: /health
            port: 8000
          failureThreshold: 30  # Allow 5 minutes for startup
          periodSeconds: 10
      # Node selector for GPU nodes
      nodeSelector:
        accelerator: nvidia-tesla-v100
      # Tolerations for GPU nodes
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
  name: llm-serving
  namespace: llm-serving
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8000
    protocol: TCP
    name: http
  selector:
    app: llm-serving
---
# Ingress for external access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-ingress
  namespace: llm-serving
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - llm-api.yourcompany.com
    secretName: llm-tls
  rules:
  - host: llm-api.yourcompany.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: llm-serving
            port:
              number: 80

Deployment Script (Industry Standard)

#!/bin/bash
# deploy.sh
# Used by: CI/CD pipelines, deployment automation

set -e

NAMESPACE="llm-serving"
IMAGE_TAG="${1:-latest}"
ENVIRONMENT="${2:-staging}"

echo "Deploying to ${ENVIRONMENT} with image tag ${IMAGE_TAG}"

# Create namespace if it doesn't exist
kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

# Update image in deployment
kubectl set image deployment/llm-serving \
  llm-serving=your-registry/llm-serving:${IMAGE_TAG} \
  -n ${NAMESPACE}

# Wait for rollout
kubectl rollout status deployment/llm-serving -n ${NAMESPACE} --timeout=5m

# Verify deployment
kubectl get pods -n ${NAMESPACE}
kubectl get svc -n ${NAMESPACE}

echo "Deployment complete!"

Rollback Script (Industry Standard)

#!/bin/bash
# rollback.sh
# Used when issues are detected in production

NAMESPACE="llm-serving"

echo "Rolling back deployment..."

kubectl rollout undo deployment/llm-serving -n ${NAMESPACE}
kubectl rollout status deployment/llm-serving -n ${NAMESPACE}

echo "Rollback complete!"

Key Concepts

• Pods: Smallest deployable unit (your container)
• Deployments: Manage pod replicas
• Services: Expose pods to network
• ConfigMaps: Configuration data
• Secrets: Sensitive data
• Ingress: External access

Prerequisites

Local Kubernetes

# Option 1: Minikube
minikube start

# Option 2: Kind
kind create cluster

# Option 3: Docker Desktop (has K8s built-in)

Verify Setup

kubectl cluster-info
kubectl get nodes

Basic Deployment

Deployment Manifest

See deployment.yaml for a complete example.

Key components:

• Deployment: Manages pods
• Service: Exposes pods
• ConfigMap: Configuration
• Resource limits: CPU/memory

Deploy

kubectl apply -f deployment.yaml

Check Status

kubectl get pods
kubectl get services
kubectl logs -f deployment/llm-serving

Health Checks

Liveness Probe

Detects if pod is alive. If fails, pod is restarted.

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 10

Readiness Probe

Detects if pod is ready to serve traffic.

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 5

Startup Probe

Gives pod time to start (useful for slow-starting apps).

startupProbe:
  httpGet:
    path: /health
    port: 8000
  failureThreshold: 30
  periodSeconds: 10

Resource Management

Requests

Minimum resources guaranteed to pod.

resources:
  requests:
    memory: "2Gi"
    cpu: "1000m"

Limits

Maximum resources pod can use.

resources:
  limits:
    memory: "4Gi"
    cpu: "2000m"
    nvidia.com/gpu: 1  # For GPU

Why This Matters

• Requests: Scheduler uses this to place pods
• Limits: Prevents pods from consuming all resources
• GPU: Must specify for GPU workloads

ConfigMaps and Secrets

ConfigMap

Store non-sensitive configuration.

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-config
data:
  model_name: "gpt2"
  max_length: "100"
  temperature: "0.7"

Secret

Store sensitive data (API keys, tokens).

apiVersion: v1
kind: Secret
metadata:
  name: llm-secrets
type: Opaque
data:
  api_key: <base64-encoded>

Using in Deployment

env:
  - name: MODEL_NAME
    valueFrom:
      configMapKeyRef:
        name: llm-config
        key: model_name
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: llm-secrets
        key: api_key

Scaling

Manual Scaling

kubectl scale deployment llm-serving --replicas=3

Auto-scaling (HPA)

See 06_autoscaling/ for detailed HPA setup.

Rolling Updates

Update Image

kubectl set image deployment/llm-serving \
  llm-serving=llm-serving:v2

Rollback

kubectl rollout undo deployment/llm-serving

Check Rollout Status

kubectl rollout status deployment/llm-serving

GPU Support

Node Labels

# Label nodes with GPU
kubectl label nodes <node-name> accelerator=nvidia-tesla-v100

GPU Resource Request

resources:
  limits:
    nvidia.com/gpu: 1

Device Plugin

Install NVIDIA device plugin:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.1/nvidia-device-plugin.yml

Ingress

Expose Service Externally

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-ingress
spec:
  rules:
  - host: llm.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: llm-serving
            port:
              number: 8000

Monitoring

View Logs

kubectl logs -f deployment/llm-serving
kubectl logs -f <pod-name>

Describe Resources

kubectl describe pod <pod-name>
kubectl describe deployment llm-serving

Get Events

kubectl get events --sort-by='.lastTimestamp'

Common Issues

Pod Not Starting

• Check logs: kubectl logs <pod-name>
• Describe pod: kubectl describe pod <pod-name>
• Check events: kubectl get events

Out of Resources

• Check node resources: kubectl describe node
• Adjust resource requests/limits
• Add more nodes

Image Pull Errors

• Check image name and tag
• Verify image registry access
• Check image pull secrets

Health Check Failures

• Verify health endpoint works
• Adjust probe timing
• Check application startup time

Best Practices

1. Always set resource limits: Prevent resource exhaustion
2. Use health checks: Enable automatic recovery
3. Use ConfigMaps: Don’t hardcode configuration
4. Tag images: Use semantic versioning
5. Test locally: Use minikube/kind before production
6. Monitor: Set up logging and metrics

Exercises

1. Basic deployment: Deploy app to local K8s
2. Health checks: Add liveness/readiness probes
3. Resource limits: Set appropriate CPU/memory
4. ConfigMap: Move config to ConfigMap
5. Scaling: Scale to 3 replicas
6. Rolling update: Update to new version

Next Steps

• Topic 4: Load test your K8s deployment
• Topic 6: Set up auto-scaling
• Topic 8: Add monitoring

Further Reading

• Kubernetes Documentation
• Kubernetes for ML
• GPU Support

Kubernetes for LLM Serving

Deploying and operating GPU inference on Kubernetes — the control plane that most production LLM stacks run on.

Why this matters

Once you have a working inference server (vLLM, TGI, Triton, SGLang), the next question is always the same: how do I run twelve of these across a fleet of GPU nodes, upgrade them without dropping traffic, and not set money on fire? Kubernetes is the de-facto answer. It gives you a declarative fleet, health-based traffic gating, rolling upgrades, and a scheduler that can place a pod on exactly the right GPU.

But LLM serving breaks several of Kubernetes’ defaults. Models take minutes to load, so naive probes will kill pods mid-startup in an infinite crash loop. GPUs are not overcommittable, so the usual CPU/memory bin-packing intuition is wrong. Container images and model weights are tens of gigabytes, so image pulls and cold starts dominate. This chapter walks the mechanisms and the sharp edges.

Autoscaling (HPA, KEDA, custom metrics, scale-to-zero) is deep enough to deserve its own chapter — see Autoscaling GPU Inference. Here we reference it but focus on deployment, scheduling, health, weights, and lifecycle.

Core intuition

Three mental models carry most of the chapter:

1. A GPU is an indivisible, non-overcommittable device. Unlike CPU (compressible) and memory (overcommittable at your peril), a GPU is handed to exactly one container by the device plugin. You can share it deliberately (time-slicing, MPS, MIG), but the scheduler still treats each advertised unit as an integer resource. There is no “burst above your GPU limit.”

2. Health is a function of time, not just liveness. A pod that has been alive for 40 seconds but hasn’t loaded a 140 GB model is not broken — it’s starting. Kubernetes has a dedicated primitive for exactly this distinction: the startup probe. Get this wrong and you get a crash loop that looks like a hardware failure.

3. The weights are the workload. For classic web services the container image is the app. For LLM serving the image is a runtime and the weights are the payload — often 10x larger than the image. Where the weights live and how they land on the node (baked into the image, PVC, initContainer, object storage, model cache) determines your cold-start time and your blast radius.

4. Voluntary disruption is the one you control. Nodes get drained for upgrades, spot reclamation, and autoscaler scale-down constantly. Kubernetes lets you bound that damage with a PodDisruptionBudget and a graceful-shutdown path. Unlike a crashed process (involuntary), these are scheduled, negotiable evictions — and the difference between a clean rolling drain and a full outage is a few lines of YAML you either wrote or didn’t.

Keep these four in mind and the rest of the chapter is mostly detail: GPUs are exclusive integers, health is time-aware, weights are the real payload, and disruptions are bounded on purpose.

Mechanisms in depth

1. The building blocks: Deployment, Service, Ingress

For a stateless replicated inference server the standard trio is:

• Deployment — declares N replicas of a pod template, handles rolling updates and self-healing.
• Service — a stable virtual IP + DNS name that load-balances across the ready pods (ClusterIP for in-cluster, LoadBalancer for cloud L4).
• Ingress (or Gateway API) — L7 routing, TLS termination, path/host rules, into the Service.

A subtlety unique to LLM serving: long request durations and streaming. Token-streaming responses (SSE) can run for tens of seconds to minutes. Make sure your Ingress/proxy timeouts (proxy-read-timeout on the NGINX ingress, backend request timeout on cloud LBs) are raised, and that buffering is disabled so tokens flush as they generate. Default 30–60s timeouts will cut long generations.

2. GPU scheduling: the NVIDIA device plugin

Kubernetes has no native notion of a GPU. The NVIDIA device plugin is a DaemonSet that runs on every GPU node, discovers the GPUs, and advertises them to the kubelet as an extended resource named nvidia.com/gpu. The scheduler then treats that resource like any countable resource.

You request GPUs under resources:

resources:
  limits:
    nvidia.com/gpu: 1   # request one whole GPU

Key rules that trip people up:

• Extended resources must be integers, and request must equal limit. Kubernetes requires that for any extended resource, if you set it at all, the request and limit are equal. You cannot request 0.5 of a nvidia.com/gpu and burst to 1. Practically you only ever specify it under limits (Kubernetes copies it to requests for you).
• GPUs are never overcommitted. Two pods cannot each hold nvidia.com/gpu: 1 on a node that advertises one GPU — the second stays Pending. This is by design: two processes fighting over one GPU’s memory would OOM each other unpredictably.
• Sharing is opt-in and explicit. If you want to pack multiple pods on one GPU you enable time-slicing (the plugin advertises, say, 4 “replicas” of each GPU — but note this is oversubscription with no memory isolation, so proportional compute is not guaranteed), MPS, or hardware MIG partitions. Each mechanism changes what the plugin advertises; the scheduler math stays “integer units.”

The device plugin is often installed as part of the NVIDIA GPU Operator, which additionally manages the driver, the container toolkit, DCGM metrics exporter, Node Feature Discovery, and MIG configuration — so you don’t hand-install drivers on every node.

2b. GPU sharing: MIG vs time-slicing vs MPS

One whole GPU per pod is wasteful for small models that use a few GB of a 80 GB card. Three mechanisms let you pack more, each changing what the device plugin advertises:

Critical caveat: time-slicing gives no memory isolation — two pods on one time-sliced GPU can OOM each other, and “requesting 2 shared GPUs” does not guarantee 2x the compute. For production multi-tenant serving, MIG is the safe choice; time-slicing/MPS are for dev or trusted, well-characterized co-tenancy. All three are configured via the device plugin / GPU Operator, not by the pod author.

3. Getting pods onto GPU nodes: selectors, taints, tolerations

You almost never want a random CPU workload landing on an expensive GPU node, and you want GPU pods to land only on GPU nodes. Two complementary mechanisms:

• Node labels + nodeSelector/affinity (attraction). GPU nodes carry labels — cloud pools add things like cloud.google.com/gke-accelerator=nvidia-l4, and the GPU Operator / NFD add labels such as nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB. You pull your pod toward them:

  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
  

• Taints + tolerations (repulsion). You taint GPU nodes so nothing schedules there unless it explicitly tolerates the taint. Cloud GPU pools often auto-apply a taint like nvidia.com/gpu=present:NoSchedule. Your inference pod must tolerate it:

  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  

Use both: the taint keeps freeloaders off, the selector/affinity ensures your pod picks the right GPU SKU. A GPU node pool is simply a node group with a fixed instance type (all A100, or all L4), its own taint, and often its own cluster-autoscaler settings so you can scale GPU capacity independently of the CPU fleet.

4. Probes done right for multi-minute model loads

This is the single most common LLM-on-k8s bug. Kubernetes has three probes:

The classic failure: you set a liveness probe with a short initialDelaySeconds, the model takes 4 minutes to load, the liveness probe fails during load, the kubelet kills the container, it restarts, tries to load again, gets killed again — an infinite crash loop that looks like the model is broken.

The fix is the startup probe. While a startup probe is configured and not yet successful, liveness and readiness are suppressed. So you give the startup probe a generous budget:

startupProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 10
  failureThreshold: 60   # 10s * 60 = up to 600s (10 min) to become healthy

The effective grace window is periodSeconds * failureThreshold. Size it to your worst-case cold load (weights download + load into VRAM + CUDA graph capture / warmup), then add margin. Once the startup probe passes once, the fast liveness and readiness probes take over.

• readiness should reflect “can serve a request” — many servers expose /health (up) vs a readiness endpoint that only returns 200 once the model is loaded and warmup is done. Gate traffic on the latter.
• liveness should be cheap and lenient — it exists to recover a genuinely wedged process (deadlock, CUDA error), not to police slow loads. Use a modest periodSeconds and a failureThreshold of 3+ so a single slow health check doesn’t kill a healthy pod mid-inference.

5. Resource requests/limits and why GPUs aren’t overcommitted

For CPU and memory you set requests (scheduling guarantee) and limits (cap). For LLM pods:

• CPU: set a request so the scheduler reserves headroom (tokenization, HTTP, scheduling loops are CPU-hungry), but be cautious with CPU limits — throttling the server’s event loop can tank throughput. Many teams set a CPU request and no CPU limit.
• Memory: set request and limit close together and generous. Host RAM is used to stage weights before they hit VRAM; an OOMKill mid-load looks like a probe failure but is actually the kernel.
• GPU: nvidia.com/gpu request == limit == integer, always. There is no overcommit. The reason is physical: GPU memory (VRAM) has no swap and no soft limit the scheduler understands. If two pods both assumed they had the whole 80 GB, the second allocation would fail at CUDA-malloc time, not at schedule time — an ugly runtime crash instead of a clean Pending. So Kubernetes refuses to double-book.

Corollary — GPU fragmentation: because allocation is integer and per-node, a cluster with eight 8-GPU nodes and lots of single-GPU pods can end up unable to schedule a pod that needs 4 GPUs on one node, even though 20 GPUs are free cluster-wide. Multi-GPU / tensor-parallel pods need whole nodes or careful bin-packing (topology-aware scheduling, podAffinity, or a scheduler like the NVIDIA/Volcano gang scheduler).

6. Serving the model weights

Where do the weights come from at pod start? Four common patterns, with tradeoffs:

Rules of thumb: bake weights into the image only for small models or when immutability matters more than pull time. For large models, keep the runtime image lean and fetch weights via initContainer or a pre-populated read-only PVC, and cache on the node so replicas 2..N start fast. Beware the thundering herd: ten pods cold-starting simultaneously all pulling 140 GB from the same bucket will saturate egress and each other.

7. Rolling updates, PodDisruptionBudgets, graceful shutdown

• Rolling updates: the Deployment’s RollingUpdate strategy with maxUnavailable / maxSurge controls how many pods are replaced at once. For GPU pods, maxSurge costs real extra GPUs — surging by 1 means the autoscaler must find another GPU node. Often you set maxSurge: 0, maxUnavailable: 1 to avoid needing spare GPUs, accepting slightly reduced capacity during the rollout. And remember: each new pod pays the full multi-minute cold-start, so rollouts of GPU fleets are slow. Budget for it.

• PodDisruptionBudget (PDB): protects against voluntary disruptions (node drains, cluster-autoscaler scale-down, upgrades). Without a PDB, a node drain can evict all your replicas at once and take the service down. Set minAvailable (or maxUnavailable) so the eviction API refuses to take down too many at once:

  apiVersion: policy/v1
  kind: PodDisruptionBudget
  spec:
    minAvailable: 2
    selector:
      matchLabels: { app: llm-inference }
  

• Graceful shutdown: on SIGTERM, a good inference server should stop accepting new requests, drain in-flight generations, then exit. Kubernetes gives it terminationGracePeriodSeconds (default 30s) before SIGKILL — raise this well above your longest expected generation (e.g. 120–300s) so streaming requests aren’t cut off. Pair it with a preStop hook or a readiness flip so the pod is pulled from Service endpoints before it starts draining, avoiding races where traffic hits a shutting-down pod.

8. Serving frameworks & operators

You don’t have to hand-roll Deployments. Higher-level tools add model-aware features (autoscaling on GPU/queue metrics, scale-to-zero, canary, standardized model formats):

• KServe — a CRD (InferenceService) on top of Knative/Kubernetes. Handles autoscaling (incl. scale-to-zero), canary rollout, and a standard prediction protocol. Has first-class support for LLM runtimes (vLLM) via ServingRuntime. Good when you want a platform abstraction over raw pods.
• NVIDIA NIM / Triton on k8s — NIM packages optimized model microservices as containers; the NIM Operator (and Triton) deploy them, and NIM integrates with KServe for the serving layer. Best when you’re standardized on NVIDIA’s optimized stack and want vendor-supported images.
• KubeAI — an open, k8s-native inference operator focused on OpenAI-compatible serving of LLMs (vLLM/Ollama), with built-in autoscaling and model management, no Istio/Knative dependency. Lighter-weight alternative to KServe.
• Ray Serve (KubeRay) — deploy via the RayService CRD. Shines for multi-model, model-composition, and distributed (multi-node tensor/pipeline-parallel) serving where a request fans across many actors/GPUs. More of a distributed compute framework than a thin serving layer.

9. Networking specifics: Gateway API, timeouts, affinity

• Ingress vs Gateway API. The classic Ingress resource works, but the newer Gateway API (Gateway + HTTPRoute) is the direction the ecosystem is moving and expresses timeouts, traffic splitting, and header routing more cleanly — useful for canarying model versions.
• Streaming timeouts. Token-by-token SSE/HTTP responses can run minutes. On the NGINX ingress set nginx.ingress.kubernetes.io/proxy-read-timeout and proxy-send-timeout to several hundred seconds and disable buffering (proxy-buffering: "off") so tokens flush live. Cloud L7 LBs have their own backend timeout you must raise.
• Session affinity for KV-cache reuse. With prefix/KV caching, routing a follow-up request to the same replica that holds the cache boosts throughput. Basic sessionAffinity: ClientIP on the Service helps; smarter setups use a cache-aware router (e.g. the vLLM production stack / router) instead of round-robin.
• Headless Services for multi-node. Distributed (tensor/pipeline-parallel across nodes) runtimes often need pod-to-pod addressing; a headless Service (clusterIP: None) plus a StatefulSet gives stable per-pod DNS.

10. Observability: know when a GPU pod is unhealthy

Standard pod metrics miss the GPU. Add:

• DCGM exporter (shipped by the GPU Operator) → Prometheus: GPU utilization, memory used, temperature, ECC errors, throttling. Alert on sustained 0% utilization on a “ready” pod (stuck), on VRAM near 100% (OOM risk), and on XID/ECC errors (failing hardware).
• Server-level metrics from the runtime: queue depth, time-to-first-token, tokens/sec, running vs waiting requests. These drive autoscaling (see the autoscaling chapter) and tell you why latency moved.
• Event/probe signals: watch for Unhealthy probe events, CrashLoopBackOff, and FailedScheduling (usually taint/selector or capacity). kubectl describe pod and kubectl get events are your first stop.

A “ready” pod pinned at 0% GPU utilization with a growing request queue is the classic silent failure — the health endpoint returns 200 but inference is wedged. Alert on the metric, not just the probe.

11. Spot / preemptible GPUs and cost

GPU nodes are the dominant cost, so many teams run inference on spot/preemptible instances at a large discount — accepting that the cloud can reclaim the node with ~30–120s notice.

• Spread replicas across on-demand and spot with topologySpreadConstraints so a spot reclamation storm can’t take the whole service down; keep a baseline of on-demand capacity protected by the PDB.
• The preemption signal arrives as a node drain → your graceful shutdown path (SIGTERM, drain, terminationGracePeriodSeconds) must fit inside the cloud’s notice window, or in-flight requests are lost.
• Cold-start time is your enemy here: a reclaimed spot pod must re-download weights and reload the model before serving. Node-local weight caches and pre-pulled images shrink the recovery gap.
• Right-size the GPU: an 8B model on an 80 GB H100 wastes the card — MIG-slice it or pick a smaller SKU (L4/L40S) and let the comparison table of frameworks + autoscaling do the packing.

Fully worked example: raw Deployment + Service

A complete, correct manifest for a vLLM server on a single A100, with GPU request, startup/readiness/liveness probes tuned for a slow load, weights fetched from object storage by an initContainer into a shared cache, a PDB, and graceful shutdown.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  labels: { app: llm-inference }
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0          # don't demand a spare GPU during rollout
      maxUnavailable: 1    # replace one pod at a time
  selector:
    matchLabels: { app: llm-inference }
  template:
    metadata:
      labels: { app: llm-inference }
    spec:
      terminationGracePeriodSeconds: 180   # let in-flight generations drain

      # --- placement: only land on the right GPU nodes ---
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

      # --- weights: download once into a shared emptyDir cache ---
      volumes:
      - name: model-cache
        emptyDir:
          sizeLimit: 200Gi
      initContainers:
      - name: fetch-weights
        image: amazon/aws-cli:2.15.0
        command:
        - sh
        - -c
        - |
          if [ ! -f /models/.done ]; then
            aws s3 sync s3://my-models/llama-3.1-70b /models/llama-3.1-70b
            touch /models/.done
          fi
        volumeMounts:
        - { name: model-cache, mountPath: /models }

      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.3
        args:
        - --model=/models/llama-3.1-70b
        - --served-model-name=llama-3.1-70b
        - --port=8000
        ports:
        - containerPort: 8000
        volumeMounts:
        - { name: model-cache, mountPath: /models, readOnly: true }

        resources:
          limits:
            nvidia.com/gpu: 1          # one whole GPU, request==limit, integer
            memory: 96Gi               # host RAM to stage weights
          requests:
            cpu: "8"
            memory: 96Gi
            nvidia.com/gpu: 1

        # --- probes: startup guards the slow load, then liveness/readiness ---
        startupProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 60         # up to 600s to load 70B weights + warmup
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 3          # pull from LB if it goes unhealthy
        livenessProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 20
          failureThreshold: 3          # only restart a genuinely wedged process

        lifecycle:
          preStop:
            exec:
              # flip out of rotation, give the LB time to notice before drain
              command: ["sh", "-c", "sleep 15"]
---
apiVersion: v1
kind: Service
metadata:
  name: llm-inference
spec:
  selector: { app: llm-inference }
  ports:
  - name: http
    port: 80
    targetPort: 8000
  type: ClusterIP
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-inference
spec:
  minAvailable: 2                      # keep >=2 replicas through drains/upgrades
  selector:
    matchLabels: { app: llm-inference }

Notes on the choices:

• startupProbe budget = 10s * 60 = 600s. If your model loads in ~90s, this is generous headroom; shrink failureThreshold if you want faster crash detection, but never below your real worst-case load time.
• The initContainer idempotency (.done sentinel) means a restarted pod on a node whose emptyDir survived (it won’t across reschedule) skips the re-download; for true cross-pod caching use a read-only RWX PVC or a node-local hostPath cache instead of emptyDir.
• maxSurge: 0 trades a little capacity during rollout for not needing an extra GPU.
• The preStop sleep + 180s grace period gives streaming requests time to finish and the load balancer time to stop routing before the process exits.

Brief KServe example

The same intent, far less YAML, using KServe’s InferenceService. KServe wires up autoscaling, routing, and the storage fetch for you.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-31-70b
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 4
    model:
      modelFormat: { name: vLLM }
      storageUri: s3://my-models/llama-3.1-70b   # KServe fetches the weights
      resources:
        limits:
          nvidia.com/gpu: "1"
        requests:
          nvidia.com/gpu: "1"
          memory: 96Gi

KServe pulls the weights from storageUri (S3/GCS/PVC/HTTP), applies a ServingRuntime for the vLLM format, and manages the Deployment/Service/autoscaler behind the CRD. You still tune probes and node placement via the ServingRuntime or pod overrides.

Debugging playbook: Pending and crash-looping GPU pods

Two symptoms cover most incidents. Work them like this:

Pod stuck Pending.

kubectl describe pod <pod> | sed -n '/Events/,$p'

Read the scheduler message:

• 0/12 nodes are available: 12 Insufficient nvidia.com/gpu → no free GPUs. Is the cluster-autoscaler adding a GPU node? Is your node pool at max? Is another pod holding the GPU?
• ... node(s) had untolerated taint {nvidia.com/gpu: present} → you’re missing the toleration.
• ... didn't match Pod's node affinity/selector → your nodeSelector/label is wrong (check exact label with kubectl get nodes --show-labels).
• Insufficient cpu/memory → GPU is free but the node can’t fit your CPU/RAM request.

Pod in CrashLoopBackOff during startup.

kubectl logs <pod> -c vllm --previous     # logs from the killed attempt
kubectl get events --field-selector involvedObject.name=<pod>

• Repeated Liveness probe failed events at ~the same age → probe is killing the model mid-load; add/extend the startup probe.
• OOMKilled in the container’s lastState → raise the memory limit (host RAM to stage weights).
• initContainer errors (S3 auth, disk full on emptyDir sizeLimit) → weights fetch is failing; the main container never starts.
• CUDA/driver errors in logs → driver/toolkit mismatch (a job for the GPU Operator), or the GPU was already claimed.

A note on cold-start math

Cold-start time for a scaled-up or rescheduled pod is roughly:

[ T_{cold} = T_{provision} + T_{pull} + T_{weights} + T_{load} + T_{warmup} ]

where ( T_{provision} ) is node acquisition (0 if a node is warm, minutes if the cluster-autoscaler must boot a GPU VM), ( T_{pull} ) is image pull, ( T_{weights} ) is fetching weights to the node, ( T_{load} ) is loading them into VRAM, and ( T_{warmup} ) is CUDA-graph capture / first-token warmup. Your startup probe budget must exceed ( T_{weights} + T_{load} + T_{warmup} ) (the init container covers ( T_{weights} ) separately if you split it out), and your autoscaling responsiveness is gated by the whole sum — which is why node-local caches and pre-pulled images matter so much.

Comparison: serving-on-Kubernetes options

Rule of thumb: start with a raw Deployment to understand the mechanics; graduate to KServe or KubeAI when you have many models and want autoscaling/canary for free; reach for Ray Serve when a single request must fan across multiple GPUs/nodes or you’re composing models; adopt NIM/Triton when NVIDIA’s optimized engines and support matter.

Failure modes & pitfalls

• Probes killing pods mid-load. No startup probe (or a liveness probe with a too-short initialDelaySeconds) turns a 4-minute model load into an infinite CrashLoopBackOff. Always use a startup probe sized to worst-case load; keep liveness lenient. This is the number-one LLM-on-k8s bug.
• GPU fragmentation. Integer, per-node GPU allocation strands capacity: 20 GPUs free cluster-wide but no single node has the 4 your tensor-parallel pod needs. Use topology-aware / gang scheduling and design node pools around your parallelism.
• Image pull of huge images. A 60 GB image (weights baked in) can take many minutes to pull on a cold node, and the cluster-autoscaler’s node-provision + pull time compounds it. Keep runtime images lean, pre-pull images to nodes, or use a node-local weight cache.
• Thundering-herd weight downloads. N pods cold-starting at once each pulling the full model from one bucket saturates egress and slows all of them. Pre-populate a read-only PVC or node cache; stagger scale-ups.
• No PDB → full outage on drain. A routine node upgrade or autoscaler scale-down can evict every replica simultaneously. Always ship a PodDisruptionBudget with minAvailable.
• Missing tolerations / wrong selectors → Pending forever. GPU nodes are tainted; a pod without the matching toleration silently stays Pending. Conversely, no selector and CPU pods squat on GPU nodes. kubectl describe pod shows the scheduling reason.
• CPU/memory misconfig masquerading as GPU failure. An OOMKill while staging weights into host RAM, or CPU throttling from a tight CPU limit, looks like a model/probe problem. Give generous memory limits; be careful with CPU limits.
• Ingress/proxy timeouts cutting streams. Default 30–60s proxy timeouts truncate long token streams. Raise read/backend timeouts and disable response buffering.
• Rollouts assuming spare GPUs. maxSurge > 0 on a full GPU pool blocks the rollout waiting for GPUs that don’t exist. Use maxSurge: 0 or ensure headroom.
• Ephemeral emptyDir cache re-downloads every reschedule. An emptyDir dies with the pod, so a rescheduled pod re-fetches the whole model. If cold-start matters, back the cache with a read-only RWX PVC or a node-local hostPath/CSI volume that survives pod churn.
• Graceful shutdown too short. Default 30s terminationGracePeriodSeconds SIGKILLs pods mid-generation. Raise it above your longest generation and flip readiness first.

Production checklist — what an interviewer probes

1. “A pod loads a 100 GB model in 5 minutes but keeps restarting. Why?” — Missing/short startup probe; liveness kills it mid-load. Fix with a startup probe whose periodSeconds * failureThreshold exceeds worst-case load, and suppress liveness until then.
2. “Why can’t you overcommit GPUs like memory?” — VRAM has no swap and the scheduler can’t reason about it; the device plugin advertises integer, exclusive units (request == limit). Sharing requires explicit time-slicing/MPS/MIG.
3. “How do you keep GPU pods on GPU nodes and everything else off?” — Taint GPU nodes, add matching tolerations, plus a nodeSelector/affinity on the GPU SKU label. Explain the attraction-vs-repulsion split.
4. “Where do the weights come from and how fast is a cold start?” — Articulate baked-image vs initContainer vs read-only PVC vs node cache, the thundering-herd risk, and how you make replicas 2..N start fast.
5. “How do you upgrade without an outage?” — RollingUpdate with maxSurge: 0/maxUnavailable: 1, a PDB with minAvailable, graceful drain via terminationGracePeriodSeconds + preStop + readiness flip.
6. “Readiness vs liveness vs startup — when does each fire and what does failure do?” — Startup gates the others and restarts on failure; readiness gates traffic (no kill); liveness restarts a wedged process. Traffic should ride on a readiness endpoint that only passes after warmup.
7. “When would you reach for KServe or Ray Serve over a raw Deployment?” — KServe/KubeAI for many models + autoscaling/canary/scale-to-zero for free; Ray Serve for distributed multi-GPU/multi-node or model composition; raw Deployment for control/simplicity.
8. “You have 20 free GPUs but a 4-GPU pod won’t schedule — explain.” — Fragmentation: allocation is per-node and integer. Needs topology-aware/gang scheduling or node pools sized to your parallelism.

Further reading

• NVIDIA k8s device plugin (README, resource requests, time-slicing/MPS): https://github.com/NVIDIA/k8s-device-plugin
• NVIDIA GPU Operator (drivers, toolkit, NFD, MIG): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
• Kubernetes — Configure Liveness, Readiness and Startup Probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
• Kubernetes — Probes concepts: https://kubernetes.io/docs/concepts/workloads/pods/probes/
• Kubernetes — Pod Disruption Budgets: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
• Kubernetes — Gateway API: https://gateway-api.sigs.k8s.io/
• Kubernetes — Schedule GPUs: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
• Kubernetes — Taints and Tolerations: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
• KServe documentation: https://kserve.github.io/website/
• KServe GitHub: https://github.com/kserve/kserve
• NVIDIA NIM — Kubernetes deployment & KServe: https://docs.nvidia.com/nim/large-language-models/latest/deployment/kubernetes-deployment/kserve.html
• NVIDIA NIM Operator: https://docs.nvidia.com/nim-operator/latest/index.html
• Ray Serve LLM on Kubernetes (KubeRay): https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayserve-llm-example.html
• KubeAI (k8s-native LLM inference operator): https://github.com/substratusai/kubeai
• EKS — Manage NVIDIA GPU devices: https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia.html

Next: autoscaling these deployments — HPA on custom/GPU metrics, KEDA, queue-depth scaling, and scale-to-zero — is covered in Autoscaling GPU Inference.

Topic 4: Load Testing & Latency Measurement

What You’ll Learn

This topic teaches you how to:

• Measure inference latency (P50, P95, P99)
• Test throughput (requests/second)
• Identify performance bottlenecks
• Use Locust for load testing
• Analyze performance metrics

Key Concepts

Latency Metrics

Latency = Time from request sent to response received

Common percentiles:

• P50 (Median): 50% of requests faster than this
• P95: 95% of requests faster than this
• P99: 99% of requests faster than this

Why P95/P99 matter: P50 might be 100ms, but P99 could be 5s. Users notice the slow requests!

Throughput

Throughput = Requests processed per second

• Single request: Measure latency
• Multiple requests: Measure throughput
• Trade-off: Higher throughput often means higher latency

Load Testing Types

1. Baseline: Single request, measure latency
2. Ramp-up: Gradually increase load
3. Sustained: Constant load for extended period
4. Spike: Sudden increase in load
5. Stress: Keep increasing until system breaks

Tools

Locust

• Python-based load testing
• Write tests in Python
• Web UI for monitoring
• Real-time statistics

Alternatives

• Apache Bench (ab): Simple, command-line
• wrk: High-performance, Lua scripting
• k6: JavaScript-based, modern

Installation

pip install locust requests

Running Load Tests

Option 1: Basic Locust Test

cd 04_load_testing
locust -f locust_test.py --host=http://localhost:8000

Then open http://localhost:8089 in your browser.

Option 2: Headless Mode (No UI)

locust -f locust_test.py \
    --host=http://localhost:8000 \
    --headless \
    --users 10 \
    --spawn-rate 2 \
    --run-time 60s

Option 3: Custom Python Script

python measure_latency.py

Understanding Results

Latency Distribution

P50:  150ms  (median)
P95:  450ms  (95% of requests faster)
P99:  800ms  (99% of requests faster)
Max:  2000ms (worst case)

Throughput

Requests/sec: 25.3
Total requests: 1518
Failures: 2 (0.13%)

What to Look For

• High P99: System struggling under load
• Increasing latency: Resource exhaustion
• Failures: System overloaded or errors
• Low throughput: Bottleneck somewhere

Performance Bottlenecks

Common Issues

1. CPU-bound: Model too large for CPU

   • Solution: Use GPU, smaller model, or quantization

2. Memory-bound: Out of memory

   • Solution: Reduce batch size, use smaller model

3. I/O-bound: Slow tokenization or network

   • Solution: Optimize tokenization, use faster network

4. GPU underutilized: Not batching efficiently

   • Solution: Use continuous batching (vLLM)

Exercises

1. Baseline Test: Measure single-request latency
2. Ramp-up Test: Gradually increase from 1 to 50 users
3. Compare Models: Test gpt2 vs distilgpt2 performance
4. Find Bottleneck: Identify what limits throughput
5. Stress Test: Find maximum capacity

Next Steps

• Topic 5: Use vLLM for better performance
• Topic 8: Set up monitoring dashboards
• Topic 6: Configure autoscaling based on load

Load Testing LLM Inference — Measuring Throughput and Latency Correctly Under Load

Why This Matters

You cannot capacity-plan, price, or SLA a serving system you have not load-tested. And LLM inference is unusually easy to benchmark wrong: a single request against an idle server tells you almost nothing about behavior at 200 concurrent users, because the whole point of a modern engine (vLLM, TGI, TensorRT-LLM) is continuous batching — throughput and latency both change as concurrency changes.

The stakes are concrete:

• Capacity planning. “How many GPUs do I need for 500 chat sessions?” is answerable only from a concurrency sweep.
• SLA definition. “p95 time-to-first-token under 500 ms” is meaningless without saying at what load.
• Cost. Output tokens/second/GPU is the number your finance model divides into. A 2× throughput win halves your bill.
• Regression gating. A benchmark you can rerun in CI catches the day someone flips --enable-chunked-prefill off.

A wrong benchmark is worse than none: it gives false confidence. Most of this chapter is about the ways benchmarks lie, and how to stop them.

Core Intuition

Why average latency lies

Latency distributions in a batched system are heavy-tailed and multi-modal. A request that lands in an empty batch returns fast; an identical request that lands when the batch is full waits for a scheduler slot, then shares GPU compute with 63 neighbors. Same input, wildly different latency.

Average that distribution and you get a number that describes no actual request. Consider ten requests with latencies (ms):

90, 95, 100, 100, 105, 110, 110, 120, 130, 2000

The mean is 296 ms. Nine of ten users saw ≤130 ms; one saw 2 s. The mean reports a latency nobody experienced and hides the 2 s tail that will dominate your support tickets. The p90 is 130 ms and the p100 (max) is 2000 ms — those describe reality. Tail latency is where user pain, timeouts, and retry storms live, so you report percentiles, never just the mean.

Rule: the mean is for throughput accounting; percentiles are for latency SLAs.

Why concurrency defines the operating point

There is no single “latency” or “throughput” for a serving system — there is a curve parameterized by load. As you push more concurrent requests:

1. Low load: GPU underutilized. Latency is flat and near-minimal. Throughput rises roughly linearly with concurrency.
2. The knee: GPU compute (or KV-cache memory) saturates. Throughput flattens — you’ve hit the roofline. Latency starts climbing because requests now queue.
3. Overload: Throughput is flat (or falls, from scheduling/paging overhead), but latency climbs without bound as the queue grows.

 throughput (tok/s)                    latency p95 (ms)
        |            ____________              |                 /
        |          /                           |               /
        |        /   <- knee                   |         _____/
        |      /                               |    ____/
        |    /                                 |___/
        |  /                                   |
        |/____________________ concurrency     |________________ concurrency

The engineering goal is to find the knee and operate just below it: that’s where you get near-peak throughput while latency is still bounded. A benchmark that reports one concurrency level has told you one point on a two-dimensional curve. Always sweep.

Metrics, Defined Precisely

Let a single streaming request produce output tokens at wall-clock times ( t_1 < t_2 < \dots < t_N ), with the request sent at ( t_0 ). Over a whole test, let ( R ) requests complete in wall-clock window ( T ) seconds, producing ( O ) total output tokens.

Time To First Token (TTFT)

[ \text{TTFT} = t_1 - t_0 ]

The latency until the first token appears. Dominated by prefill (processing the prompt) plus any queueing wait for a scheduler slot. This is what a user perceives as “responsiveness” — the cursor starting to move. TTFT is the metric most sensitive to load, because a queued request pays its wait entirely before ( t_1 ).

Micro-example: prompt sent at ( t_0 = 0 ), first token at ( t_1 = 0.18\text{ s} ) → TTFT = 180 ms.

Time Per Output Token (TPOT) / Inter-Token Latency (ITL)

TPOT is the average gap between output tokens after the first, for one request:

[ \text{TPOT} = \frac{t_N - t_1}{N - 1} ]

ITL is the per-gap version — the distribution of individual ( t_{i+1} - t_i ) values. TPOT is the mean of a request’s ITLs. (Tools differ: vLLM reports both TPOT and ITL; some tools call TPOT “inter-token latency.” Know which your tool means.) TPOT is governed by the decode phase; ( 1/\text{TPOT} ) is the per-user tokens/second — the perceived “typing speed.”

Micro-example: a request emits 201 tokens, ( t_1 = 0.18 ), ( t_{201} = 4.18 ). TPOT ( = (4.18 - 0.18)/200 = 20\text{ ms} ) → each user sees ~50 tokens/s.

End-to-End (E2E) Request Latency

[ \text{E2E} = t_N - t_0 = \text{TTFT} + (N-1)\cdot\text{TPOT} ]

Total time from send to last token. This is the number that scales with output length, so it is only comparable across runs if output-length distributions match. A “latency regression” is often just longer outputs.

Normalized latency (per-token E2E)

[ \text{normalized latency} = \frac{\text{E2E}}{N} ]

Divides out output length, making runs with different output distributions comparable. vLLM’s benchmark historically reported this.

Request throughput

[ \lambda_{\text{out}} = \frac{R}{T} \quad [\text{req/s}] ]

Completed requests per second across the whole run. The right top-line number when your unit of work is “a request” (e.g., classification).

Output-token throughput

[ X = \frac{O}{T} \quad [\text{tok/s}] ]

Generated tokens per second across all concurrent requests. This is the money metric for generative workloads and the one that goes up with better batching. Report output tokens (excludes the prompt); also report total token throughput (prompt + output) if prefill cost matters to you. Divide by GPU count for tok/s/GPU.

Micro-example: 64 concurrent requests each streaming at 50 tok/s → aggregate ( X \approx 3200 ) tok/s, even though each user still sees only 50 tok/s. Per-user speed and aggregate throughput are different axes.

Percentiles

For a metric with sorted samples, the p-th percentile ( P_p ) is the smallest value ( \geq p% ) of samples:

[ P_p = \text{value at rank } \lceil \tfrac{p}{100} \cdot n \rceil \text{ in sorted order} ]

Report p50 (median), p95, p99 for TTFT, TPOT, and E2E. p99 matters more than it looks: if a page makes 10 backend calls, the chance all 10 beat p99 is ( 0.99^{10} \approx 0.90 ) — so ~10% of page loads hit a p99 tail. Tail latency compounds.

Open-Loop vs Closed-Loop Load Generation

This is the single most important methodology decision, and the one most benchmarks get wrong.

Closed-loop

A fixed pool of ( C ) “virtual users.” Each sends a request, waits for the full response, then immediately sends the next. Concurrency is capped at ( C ) by construction. This models a fixed number of clients in a tight loop (e.g., a batch job, or exactly ( C ) synchronous callers).

Open-loop

Requests are launched on an arrival schedule (e.g., Poisson at rate ( \lambda )) independent of whether prior requests have finished. In-flight concurrency is an emergent property, free to grow if the server slows down. This models real traffic: users arrive whether or not your server is keeping up.

Coordinated omission — why closed-loop hides overload

Here is the trap. In a closed-loop test, when the server slows down, each virtual user’s loop stalls waiting for its response — so it stops sending new requests. The offered load automatically backs off exactly when the system is struggling. The load generator “coordinates” with the server’s slowness and omits the requests a real (open) world would have kept sending.

Consequences:

• Your measured request rate silently drops below target, and you may not notice.
• The latency samples you do collect exclude the requests that would have queued behind a slow one, so tail latency is dramatically underreported. The one 2-second stall in a real system would have delayed 40 requests behind it; closed-loop just… didn’t send them.
• You conclude the system is healthy at a load it actually cannot sustain.

The fix in an open-loop tool is to schedule requests on absolute wall-clock times and measure each request’s latency from its intended send time, not from when a freed-up worker got around to it. k6’s constant-arrival-rate/ramping-arrival-rate executors and Gatling’s open model do this; constant-vus/ramping-vus are closed. Corrected closed-loop tools (wrk2, Gatling) reconstruct the omitted samples by back-dating latency to the scheduled time.

When each is right

• Open-loop / fixed request-rate: validating an SLA against realistic traffic (“can we hold p95 TTFT < 500 ms at 30 req/s?”). This is the honest default for user-facing services.
• Closed-loop / fixed concurrency: finding maximum sustainable throughput and the saturation curve, where a bounded, known concurrency is exactly the independent variable you want to sweep. LLMPerf and GenAI-Perf’s concurrency mode work this way, and that is fine because you are deliberately measuring the throughput ceiling, not pretending to reproduce arrival traffic.

Use both: a concurrency sweep to find the knee, then an open-loop test at your chosen arrival rate to confirm the SLA holds with realistic tail behavior.

Little’s Law — Reasoning About Concurrency

Little’s Law relates the three quantities you care about, for any stable system in steady state:

[ L = \lambda , W ]

• ( L ) = average number of requests in the system (concurrency / in-flight).
• ( \lambda ) = average arrival = completion rate (req/s), in steady state.
• ( W ) = average time in system (E2E latency, seconds).

It is an identity — no assumptions about distributions. Uses:

Sanity-check a benchmark. If your open-loop test offers ( \lambda = 20 ) req/s and you measure mean E2E ( W = 4 ) s, then average in-flight concurrency is ( L = 20 \times 4 = 80 ). If your engine’s --max-num-seqs is 64, you are oversubscribed: requests are queuing, latency will keep rising, and the system is not in steady state. The law just told you the offered load exceeds capacity before you stare at a climbing latency graph.

Convert closed-loop to a rate. A closed-loop test with ( C = 100 ) virtual users measuring ( W = 2.5 ) s achieves ( \lambda = L/W = 100/2.5 = 40 ) req/s. That’s how you translate a concurrency-sweep point into “requests per second this config sustains.”

Token form. Apply it to tokens: aggregate output throughput ( X ) (tok/s) with ( n ) requests in flight each of length ( N ) tokens taking ( W ) seconds gives ( X = nN/W ) — decompose regressions into “fewer concurrent” vs “slower per request.”

Caveat: Little’s Law holds in steady state. During warmup, ramp, or overload it does not, which is one more reason to discard warmup and to hold each sweep point long enough to stabilize.

A Fully Worked Example — Async Open-Loop Client

Below is a self-contained async Python client that drives an OpenAI-compatible streaming endpoint (vLLM, TGI, etc.), generates a Poisson arrival schedule (true open-loop — arrivals do not wait for completions), records TTFT/TPOT/E2E per request from the intended send time (coordinated-omission-safe), discards warmup, and prints percentiles and throughput.

#!/usr/bin/env python3
"""Open-loop load test for an OpenAI-compatible LLM endpoint.

Launches requests on a Poisson schedule at a target rate, independent of
whether prior requests finished, and measures latency from the INTENDED
send time to avoid coordinated omission.

Usage:
  python load_test.py --url http://localhost:8000/v1/chat/completions \
      --model my-model --rate 20 --duration 60 --warmup 10
"""
import argparse, asyncio, json, random, statistics, time
import aiohttp

PROMPTS = [
    "Explain how continuous batching improves LLM throughput.",
    "Write a haiku about GPU memory fragmentation.",
    "Summarize the tradeoffs of speculative decoding in three sentences.",
    "What is time-to-first-token and why does it depend on load?",
]

async def one_request(session, url, model, prompt, max_tokens, sched_t, t0, results):
    """Fire one request. Latency is measured from sched_t (intended send),
    NOT from now — this is the coordinated-omission fix."""
    # Sleep until this request's scheduled arrival time (open-loop).
    delay = sched_t - (time.perf_counter() - t0)
    if delay > 0:
        await asyncio.sleep(delay)
    intended = t0 + sched_t                       # absolute intended send time
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": True,
        "temperature": 0.0,
    }
    token_times, first_t, err = [], None, None
    send = time.perf_counter()
    try:
        async with session.post(url, json=payload) as resp:
            async for raw in resp.content:
                line = raw.decode("utf-8").strip()
                if not line.startswith("data:"):
                    continue
                data = line[len("data:"):].strip()
                if data == "[DONE]":
                    break
                chunk = json.loads(data)
                delta = chunk["choices"][0]["delta"].get("content")
                if delta:
                    now = time.perf_counter()
                    if first_t is None:
                        first_t = now
                    token_times.append(now)
    except Exception as e:                          # noqa: BLE001
        err = repr(e)
    end = time.perf_counter()
    n = len(token_times)
    results.append({
        "intended": intended,
        "ttft": (first_t - send) if first_t else None,
        # E2E measured from INTENDED time, not send — captures scheduling debt.
        "e2e": end - send,                          # from actual send time
        "e2e_true": end - (t0 + sched_t),           # from intended arrival (CO-safe)
        "n_tokens": n,
        "tpot": ((token_times[-1] - first_t) / (n - 1)) if n > 1 else None,
        "error": err,
    })

def pct(xs, p):
    if not xs:
        return float("nan")
    xs = sorted(xs)
    k = max(0, min(len(xs) - 1, int(round(p / 100 * len(xs) + 0.5)) - 1))
    return xs[k]

async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", required=True)
    ap.add_argument("--model", required=True)
    ap.add_argument("--rate", type=float, default=10.0, help="req/s (Poisson mean)")
    ap.add_argument("--duration", type=float, default=60.0, help="seconds")
    ap.add_argument("--warmup", type=float, default=10.0, help="seconds to discard")
    ap.add_argument("--max-tokens", type=int, default=200)
    args = ap.parse_args()

    # Pre-build a Poisson arrival schedule: gaps ~ Exponential(rate).
    schedule, t = [], 0.0
    while t < args.duration:
        t += random.expovariate(args.rate)
        schedule.append(t)

    results = []
    conn = aiohttp.TCPConnector(limit=0)            # no client-side cap!
    timeout = aiohttp.ClientTimeout(total=None)
    async with aiohttp.ClientSession(connector=conn, timeout=timeout) as session:
        t0 = time.perf_counter()
        tasks = [
            asyncio.create_task(one_request(
                session, args.url, args.model,
                random.choice(PROMPTS), args.max_tokens, s, t0, results))
            for s in schedule
        ]
        await asyncio.gather(*tasks)
        wall = time.perf_counter() - t0

    # Drop warmup window and errored requests.
    ok = [r for r in results if r["error"] is None
          and r["intended"] - t0 >= args.warmup]
    measured_window = wall - args.warmup
    ttfts = [r["ttft"] * 1000 for r in ok if r["ttft"]]
    tpots = [r["tpot"] * 1000 for r in ok if r["tpot"]]
    e2es = [r["e2e_true"] * 1000 for r in ok]
    out_tokens = sum(r["n_tokens"] for r in ok)

    print(f"\n=== target rate {args.rate} req/s | wall {wall:.1f}s | "
          f"measured window {measured_window:.1f}s ===")
    print(f"requests ok: {len(ok)}  errors: {sum(1 for r in results if r['error'])}")
    print(f"achieved req/s:      {len(ok)/measured_window:8.2f}")
    print(f"output tok/s:        {out_tokens/measured_window:8.1f}")
    for name, xs in (("TTFT ms", ttfts), ("TPOT ms", tpots), ("E2E  ms", e2es)):
        if xs:
            print(f"{name:9s} p50 {pct(xs,50):8.1f}  p95 {pct(xs,95):8.1f}  "
                  f"p99 {pct(xs,99):8.1f}  mean {statistics.mean(xs):8.1f}")

if __name__ == "__main__":
    asyncio.run(main())

Two design points that matter:

• TCPConnector(limit=0) removes the client’s own connection cap. If you leave aiohttp’s default (100) or run one CPU core hot parsing SSE, the client becomes the bottleneck and you benchmark your laptop, not the server. (See pitfalls.)
• e2e_true measures from the intended arrival time (t0 + sched_t), so a request delayed because the event loop was busy still counts its full latency — coordinated-omission-safe. (Use the e2e_true field, end - (t0 + sched_t), for reporting; the plain e2e from actual send time will under-report latency when the event loop falls behind.)

Sample results — a concurrency/rate sweep

Run the client at increasing rates against one A100 serving an 8B model, fixed input ≈512 / output ≈200 tokens:

Reading the saturation curve

• 5 → 20 req/s: achieved rate tracks target, output tok/s scales ~linearly (1000→4000), TTFT p95 stays modest. Underloaded region — GPU has headroom.
• ~30 req/s is the knee. Output tok/s (5900) is close to the ceiling; achieved rate still ≈ target but TTFT p95 has jumped to 620 ms. This is the operating point you’d target for a latency-sensitive service, maybe backing off to ~25 for headroom.
• 40 → 50 req/s: overload. Achieved rate flatlines at ~33 req/s even as you offer more — that ~33 req/s (≈6600 tok/s) is the true saturation throughput. Meanwhile latency explodes (TTFT p95 2.4 s → 9 s; E2E p95 41 s) and Little’s-Law in-flight ( L ) blows past any sane max-num-seqs. Offering 50 doesn’t get you 50; it just grows the queue.

The signature of the knee: throughput stops rising while latency starts rising superlinearly. Peak throughput and acceptable latency are different points — publish both, and state which you’re operating at.

A Second Worked Example — Locust (open-model)

Locust is convenient for HTTP services and dashboards. Locust is closed-loop by default (each user loops), but the constant_throughput/constant_pacing shape plus a high user count approximates open arrivals. Here is a locustfile.py that measures streaming TTFT and records it as a custom metric:

# locustfile.py  —  run:  locust -f locustfile.py --host http://localhost:8000
import json, time
from locust import HttpUser, task, constant_throughput

class LLMUser(HttpUser):
    # Each user targets 1 req/s; scale arrivals via -u (number of users).
    # constant_throughput paces to a rate rather than back-to-back looping,
    # which is closer to open-loop than the default.
    wait_time = constant_throughput(1.0)

    @task
    def chat(self):
        payload = {
            "model": "my-model",
            "messages": [{"role": "user", "content": "Explain KV cache paging."}],
            "max_tokens": 200, "stream": True, "temperature": 0.0,
        }
        start = time.perf_counter()
        first_t = None
        n_tokens = 0
        with self.client.post("/v1/chat/completions", json=payload,
                              stream=True, catch_response=True,
                              name="chat-stream") as resp:
            for raw in resp.iter_lines():
                if not raw:
                    continue
                line = raw.decode("utf-8")
                if not line.startswith("data:"):
                    continue
                data = line[len("data:"):].strip()
                if data == "[DONE]":
                    break
                delta = json.loads(data)["choices"][0]["delta"].get("content")
                if delta:
                    if first_t is None:
                        first_t = time.perf_counter()
                    n_tokens += 1
            # Report TTFT as a named event so it shows in Locust stats/percentiles.
            if first_t is not None:
                ttft_ms = (first_t - start) * 1000
                self.environment.events.request.fire(
                    request_type="METRIC", name="TTFT_ms",
                    response_time=ttft_ms, response_length=n_tokens,
                    exception=None, context={})

Locust’s own percentile table then gives you p50/p95/p99 for both the full request and the synthetic TTFT_ms event. Drive concurrency with -u <users> and -r <ramp>; use the web UI’s charts to watch the knee live. Caveat: Locust workers are Python and can become the bottleneck — run distributed workers (--worker) and confirm client CPU isn’t saturated before trusting numbers at high load.

Tools Comparison

Rule of thumb: benchmark_serving.py/GenAI-Perf when you want LLM-native metrics and datasets out of the box; k6 when you want a rigorous open-loop SLA test; LLMPerf for cross-provider comparisons; Locust for bespoke multi-step traffic with a dashboard.

Failure Modes and Pitfalls

Closed-loop coordinated omission. Covered above — the big one. A closed-loop tool stops sending when the server stalls, so it under-reports tail latency and over-reports sustainable load. Fix: use open-loop arrival-rate executors, or a corrected tool that back-dates latency to the intended send time. If you must go closed-loop, only use it to measure the throughput ceiling, and never quote its latencies as an SLA.

No warmup. The first requests hit cold caches: CUDA graph capture, torch.compile / TRT-LLM engine warmup, cuBLAS autotune, KV-cache allocation, JIT. Cold TTFT can be 5–50× steady-state. Including warmup poisons your percentiles (a handful of huge samples wreck p99). Fix: send a warmup burst and discard the first N seconds/requests (the example’s --warmup). Also warm up the client (DNS, TLS, connection pool).

Unrealistic input/output lengths. A benchmark with 128-in/128-out tokens tells you nothing about a RAG workload with 4000-in/500-out. Prefill cost scales with input length; decode cost and E2E scale with output length; KV-cache pressure scales with both × concurrency. Fixed lengths also hide batching dynamics because every request finishes together. Fix: replay a realistic length distribution (ShareGPT, your own production logs, or --dataset-name random with a mean/std matching production). Report the distribution you used.

Measuring only averages. The mean hides the tail and can be dominated by a few slow requests. Always report p50/p95/p99 (and max) for TTFT, TPOT, and E2E separately. And never average latency across different output lengths without normalizing — longer outputs inflate E2E and masquerade as a regression.

Client-side bottleneck. The most insidious: your load generator, not the server, is the limit. Symptoms: achieved rate plateaus well below the server’s known capacity, client CPU pegged at 100%, or latency that scales with client concurrency. Causes: single-threaded Python parsing SSE, aiohttp/requests connection-pool caps, GIL contention, running the client on the same box as the server, or a network link between client and server that’s the actual bottleneck. Fixes: pin and monitor client CPU, remove connection caps (TCPConnector(limit=0)), use async or distributed workers (k6/Locust workers, multiple client hosts), and sanity-check with Little’s Law — if measured ( L = \lambda W ) can’t reach the server’s max-num-seqs, your client is starving it.

Reusing identical prompts / prefix caching artifacts. If every request sends the same prompt, prefix caching makes prefill nearly free and TTFT looks unrealistically good. Vary prompts (or explicitly test both with- and without-cache) so you measure the case you’ll actually run.

Not holding steady state / too-short runs. Little’s Law and stable percentiles need steady state. A 5-second run at 30 req/s is ~150 samples — too few for a trustworthy p99 (you want ≥1000+ post-warmup samples) and too short to reach queue equilibrium. Run each sweep point long enough that metrics stop drifting.

Production Checklist — What an Interviewer Probes

1. “What do you measure, and why not just average latency?” — Name TTFT, TPOT/ITL, E2E, request- and output-token throughput; report p50/p95/p99, not means, because latency is heavy-tailed and the tail is what users feel. Mean is only for throughput accounting.
2. “Open-loop or closed-loop, and what’s coordinated omission?” — Open-loop (arrival-rate) for SLA validation; closed-loop (fixed concurrency) only to find the throughput ceiling. Coordinated omission = closed-loop stops sending when the server stalls, so it hides the tail. Bonus: measure latency from the intended send time.
3. “How do you find the right operating point?” — Sweep concurrency/rate, plot throughput and p95 latency vs load, find the knee where throughput flattens and latency turns up; operate just below it with headroom.
4. “Apply Little’s Law here.” — ( L = \lambda W ). Given a rate and E2E latency, compute in-flight concurrency and compare to max-num-seqs to detect oversubscription; convert closed-loop concurrency to sustainable req/s.
5. “How do you make the workload realistic?” — Match production input/output length distributions (ShareGPT / replayed logs), vary prompts to avoid prefix-cache flattery, and report the distribution used.
6. “How do you know the client isn’t the bottleneck?” — Monitor client CPU, remove connection caps, use async/distributed generators, separate client and server hosts, and cross-check achieved ( L ) against server capacity.
7. “Warmup and steady state?” — Discard cold-start requests (CUDA graphs/compile/cache allocation); run long enough (≥~1000 post-warmup samples, metrics stable) for trustworthy p99.
8. “How do you gate regressions in CI?” — A reproducible benchmark (pinned tool/model/dataset/lengths) run at a fixed rate, asserting on p95 TTFT and output tok/s thresholds, so a config regression fails the build.

Further Reading

• vLLM — Benchmark CLI (vllm bench serve): https://docs.vllm.ai/en/latest/cli/bench/serve/
• vLLM — Benchmarking overview & datasets: https://docs.vllm.ai/en/latest/benchmarking/cli/
• LLMPerf (Ray) — benchmarking library and token_benchmark_ray.py: https://github.com/ray-project/llmperf
• LLMPerf Leaderboard: https://github.com/ray-project/llmperf-leaderboard
• NVIDIA GenAI-Perf (Triton Perf Analyzer): https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
• NVIDIA AIPerf — Metrics Reference (TTFT/ITL definitions): https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference
• k6 — Open vs closed workload models: https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/
• Locust — documentation: https://docs.locust.io/en/stable/
• Gil Tene — “How NOT to Measure Latency” (coordinated omission, source): https://www.infoq.com/presentations/latency-response-time/
• ScyllaDB — On Coordinated Omission: https://www.scylladb.com/2021/04/22/on-coordinated-omission/
• Marc Brooker — Open, Closed, Omission and Collapse: https://brooker.co.za/blog/2023/05/10/open-closed.html
• Little’s Law (overview): https://en.wikipedia.org/wiki/Little%27s_law
• Red Hat AI Inference Server — validating with key metrics (TTFT/TPOT/throughput): https://docs.redhat.com/en/documentation/red_hat_ai_inference_server/3.1/html/getting_started/validating-benefits-with-key-metrics_getting-started

Topic 5: High-Performance Serving with vLLM

What You’ll Learn

This topic teaches you how to use vLLM for high-performance LLM serving:

• What vLLM is and why it’s fast
• Setting up vLLM server
• Understanding continuous batching
• PagedAttention memory optimization
• GPU utilization and throughput optimization

Why We Need This

Business Need

• Cost reduction: 10x higher throughput = 10x lower cost per request
• User experience: Lower latency = better user satisfaction
• Scalability: Handle more users with same infrastructure
• Competitive advantage: Faster responses than competitors

Technical Need

• GPU efficiency: 80-95% utilization vs 20-40% with basic serving
• Memory efficiency: Support longer sequences with same memory
• Throughput: 50-200 req/s vs 1-5 req/s with basic serving
• Production-ready: Used by major companies in production

Real-World Impact

Without vLLM:

• ❌ 10x higher infrastructure costs
• ❌ Poor user experience (slow responses)
• ❌ Can’t scale to handle traffic
• ❌ Wasted GPU resources (expensive!)

Industry Use Cases

1. High-Volume API Services

Company: OpenAI, Anthropic, Cohere Use Case:

• Serve millions of requests per day
• Need maximum GPU utilization
• Cost-sensitive at scale

Example:

# vLLM handles 1000 req/s vs 10 req/s with basic serving
# Cost: $10,000/month vs $100,000/month

2. Real-Time Applications

Company: Chatbots, code completion tools Use Case:

• Sub-second response times required
• Many concurrent users
• Low latency critical

Example:

# GitHub Copilot, ChatGPT use continuous batching
# User types → immediate suggestions

3. Cost-Optimized ML Platforms

Company: ML infrastructure companies Use Case:

• Serve multiple customers on shared infrastructure
• Maximize GPU utilization = lower costs
• Pass savings to customers

Example:

# Shared GPU cluster
# vLLM allows 10x more customers per GPU

4. Long Context Windows

Company: Document processing, code analysis Use Case:

• Process long documents (10K+ tokens)
• PagedAttention enables longer sequences
• Memory-efficient

Example:

# Process entire codebase (100K tokens)
# Traditional: Out of memory
# vLLM: Works efficiently

5. Multi-Tenant Serving

Company: SaaS platforms, ML platforms Use Case:

• Serve multiple models/users simultaneously
• Efficient resource sharing
• Fair resource allocation

Example:

# 100 customers, each with different model
# vLLM manages memory efficiently

Industry-Standard Boilerplate Code

Production vLLM Server (Industry Standard)

"""
Production vLLM server
Used by: OpenAI-compatible APIs, high-throughput serving
"""
from vllm import LLM, SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import uvicorn
import os

# Configuration from environment (12-factor app)
MODEL_NAME = os.getenv("MODEL_NAME", "gpt2")
GPU_MEMORY_UTILIZATION = float(os.getenv("GPU_MEMORY_UTILIZATION", "0.9"))
TENSOR_PARALLEL_SIZE = int(os.getenv("TENSOR_PARALLEL_SIZE", "1"))
MAX_MODEL_LEN = int(os.getenv("MAX_MODEL_LEN", "2048"))

app = FastAPI(title="vLLM Serving API")

# Initialize vLLM engine
llm_engine = None

@app.on_event("startup")
async def startup():
    global llm_engine
    engine_args = AsyncEngineArgs(
        model=MODEL_NAME,
        tensor_parallel_size=TENSOR_PARALLEL_SIZE,
        gpu_memory_utilization=GPU_MEMORY_UTILIZATION,
        max_model_len=MAX_MODEL_LEN,
        dtype="float16",  # FP16 for speed
        trust_remote_code=True,
    )
    llm_engine = AsyncLLMEngine.from_engine_args(engine_args)

class CompletionRequest(BaseModel):
    prompt: str
    max_tokens: int = 100
    temperature: float = 1.0
    top_p: float = 1.0

@app.post("/v1/completions")
async def completions(request: CompletionRequest):
    """OpenAI-compatible endpoint"""
    from vllm.utils import random_uuid
    
    request_id = random_uuid()
    sampling_params = SamplingParams(
        temperature=request.temperature,
        top_p=request.top_p,
        max_tokens=request.max_tokens,
    )
    
    llm_engine.add_request(
        request_id=request_id,
        prompt=request.prompt,
        sampling_params=sampling_params,
    )
    
    final_output = None
    async for request_output in llm_engine.generate(
        request_id=request_id,
        prompt=request.prompt,
        sampling_params=sampling_params,
    ):
        final_output = request_output
    
    return {
        "choices": [{
            "text": final_output.outputs[0].text,
            "finish_reason": final_output.outputs[0].finish_reason
        }]
    }

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

vLLM with OpenAI Client (Industry Standard)

"""
Client code using vLLM server
Used by: Applications integrating with LLM APIs
"""
from openai import OpenAI

# vLLM provides OpenAI-compatible API
client = OpenAI(
    base_url="http://vllm-server:8000/v1",
    api_key="dummy"  # vLLM doesn't require real key
)

def generate_text(prompt: str, max_tokens: int = 100) -> str:
    """Generate text using vLLM server"""
    response = client.completions.create(
        model="gpt2",
        prompt=prompt,
        max_tokens=max_tokens,
        temperature=0.7
    )
    return response.choices[0].text

# Example: High-throughput batch processing
async def process_batch(prompts: list[str]):
    """Process multiple prompts concurrently"""
    import asyncio
    
    tasks = [
        generate_text(prompt) 
        for prompt in prompts
    ]
    return await asyncio.gather(*tasks)

Key Concepts

Continuous Batching

Traditional batching:

Request 1: [████████████████] (waiting)
Request 2: [████████████████] (waiting)
Request 3: [████████████████] (waiting)
→ Process all together
→ Wait for all to finish

vLLM continuous batching:

Request 1: [████████████] (done, remove)
Request 2: [████████████████] (processing)
Request 3: [████████] (processing)
Request 4: [██] (just added)
→ Process together, remove completed, add new
→ GPU always busy

PagedAttention

• Divides KV cache into fixed-size pages
• Allocates pages on-demand
• Reuses freed pages
• Result: Support for longer sequences, less memory waste

Installation

# vLLM requires CUDA (GPU)
pip install vllm

# Or with specific CUDA version
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu118

Running the Examples

Option 1: Using vLLM’s Built-in Server

# Start vLLM server
python -m vllm.entrypoints.openai.api_server \
    --model gpt2 \
    --port 8000 \
    --tensor-parallel-size 1

Option 2: Using Our Custom Server

cd 05_vllm_serving
pip install -r requirements.txt
python vllm_server.py

API Usage

vLLM provides OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # vLLM doesn't require real API key
)

# Chat completion
response = client.chat.completions.create(
    model="gpt2",
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
    max_tokens=100,
    temperature=0.7
)

print(response.choices[0].message.content)

Performance Comparison

Basic Serving (HuggingFace)

• Throughput: ~1-5 requests/second
• Latency: 100-500ms per request
• GPU Utilization: 20-40%

vLLM Serving

• Throughput: ~50-200 requests/second
• Latency: 50-200ms per request
• GPU Utilization: 80-95%

Why the difference?

• Continuous batching keeps GPU busy
• PagedAttention uses memory efficiently
• Optimized CUDA kernels

Configuration Options

Model Loading

from vllm import LLM

llm = LLM(
    model="gpt2",
    tensor_parallel_size=1,  # Number of GPUs
    gpu_memory_utilization=0.9,  # Use 90% of GPU memory
    max_model_len=2048,  # Maximum sequence length
    dtype="float16",  # Use FP16 for speed
)

Generation Parameters

outputs = llm.generate(
    prompts=["Hello"],
    sampling_params={
        "temperature": 0.7,
        "top_p": 0.9,
        "max_tokens": 100,
    }
)

Monitoring vLLM

vLLM exposes metrics you can monitor:

• Request queue size
• GPU utilization
• Throughput (tokens/second)
• Latency (P50, P95, P99)

See 08_monitoring/ for Grafana dashboards.

Common Issues

Out of Memory

• Problem: Model too large for GPU
• Solutions:
  • Use smaller model
  • Reduce gpu_memory_utilization
  • Use quantization (INT8, INT4)
  • Use tensor parallelism (split across GPUs)

Slow Performance

• Problem: Not using GPU
• Solution: Make sure CUDA is available: python -c "import torch; print(torch.cuda.is_available())"

Import Errors

• Problem: vLLM not installed correctly
• Solution: Install with correct CUDA version

Exercises

1. Compare Performance: Run same model with HuggingFace vs vLLM, measure throughput
2. Tune Parameters: Experiment with gpu_memory_utilization, max_model_len
3. Test Continuous Batching: Send requests at different rates, observe GPU utilization
4. Monitor Metrics: Set up Prometheus to scrape vLLM metrics

Next Steps

• Topic 6: Autoscaling with vLLM
• Topic 8: Monitoring vLLM with Grafana
• Topic 11: Multi-model serving with Triton

Further Reading

• vLLM Documentation
• PagedAttention Paper
• Continuous Batching Explained

vLLM Serving — PagedAttention, Continuous Batching, and Production Tuning

Why this matters

If you serve open-weight LLMs at any real scale, vLLM is very likely the engine underneath — or the baseline everything else is measured against. It became the default high-throughput inference server because it attacked the single biggest bottleneck in LLM serving: memory, specifically the KV cache. Its headline result, from the original paper, is a 2–4× throughput improvement at the same latency versus the prior state of the art (FasterTransformer and Orca).

The insight is almost embarrassingly simple in hindsight: LLM serving was wasting 60–80% of KV-cache memory to fragmentation and over-reservation. vLLM borrowed virtual memory and paging from operating systems, applied it to the KV cache, and turned that wasted memory back into batch capacity. More batch capacity means more requests share each expensive weight-load from GPU memory, which is exactly what raises throughput.

This chapter goes deep on the mechanisms (PagedAttention, continuous batching, prefix caching), the knobs that actually matter in production (gpu-memory-utilization, max-num-seqs, max-num-batched-tokens, chunked prefill), how to scale across GPUs, and how to trade quality for speed with quantization and speculative decoding. Then a fully worked tuning walkthrough, a comparison to TGI and TensorRT-LLM, and the failure modes that page you at 3am.

Core intuition: LLM inference is memory-bound, and the KV cache is the problem

Two facts drive everything:

1. Autoregressive decoding is memory-bandwidth-bound, not compute-bound. Generating one token touches the entire model weights but does very little arithmetic per token. The GPU spends most of its time reading weights from HBM, not doing math. The fix is batching: process many sequences at once so a single weight read serves many tokens. Bigger batch → higher throughput, until you run out of memory.

2. What runs you out of memory is the KV cache. For every token in every sequence, attention must remember the key and value vectors of all previous tokens. This “KV cache” grows linearly with sequence length and with the number of concurrent sequences. On a typical setup the weights are fixed, and whatever HBM is left is a fixed budget you spend on KV cache. The more efficiently you pack the KV cache, the bigger your batch, the higher your throughput.

So the game is: fit as many sequences’ KV caches into the leftover HBM as possible, and keep the GPU busy on all of them at once. PagedAttention wins the first half; continuous batching wins the second.

How big is the KV cache?

Per token, the KV cache size is:

[ \text{bytes/token} = 2 \times n_\text{layers} \times n_\text{kv_heads} \times d_\text{head} \times \text{dtype_bytes} ]

The leading (2) is for K and V. Note (n_\text{kv_heads}), not the number of query heads — models with grouped-query attention (GQA) share KV heads across query heads, which shrinks the cache dramatically.

Worked numbers (fp16, 2 bytes):

• OPT-13B (40 layers, hidden 5120, full MHA): (2 \times 40 \times 5120 \times 2 = 819{,}200) bytes ≈ 800 KB per token. A single 2048-token sequence needs ~1.6 GB of KV cache — this is the number from the vLLM paper.
• Llama-3-8B (32 layers, 8 KV heads, (d_\text{head}=128), GQA): (2 \times 32 \times 8 \times 128 \times 2 = 131{,}072) bytes = 128 KB per token. GQA makes it ~6× cheaper than a same-size MHA model.

At 800 KB/token, KV cache is enormous and dynamic — you don’t know a request’s final length in advance. That combination is exactly what classic allocators handle badly.

PagedAttention in depth

The problem it solves: fragmentation and over-reservation

Pre-vLLM systems (Orca, FasterTransformer) stored each sequence’s KV cache in one contiguous chunk of memory, sized to the maximum possible length. If your model supports 2048 tokens, every sequence reserved space for 2048 tokens the moment it started — even if it only ever generated 30. Three kinds of waste result, and the paper measures them directly:

• Internal fragmentation (13.3%–57.3%): the slot reserved for max length but never filled.
• Reservation waste (part of 25.2%–96.3%): space reserved for future tokens of a still-running sequence — technically “will be used” but idle now, so it can’t serve other requests.
• External fragmentation: gaps between contiguous chunks of different sizes that no new sequence fits into.

Net effect: measured effective KV utilization of only 20.4%–38.2%. Four out of five bytes wasted.

The idea: page the KV cache like virtual memory

Operating systems solved this decades ago. A process sees a contiguous virtual address space, but physically it’s scattered across fixed-size pages mapped by a page table. No process reserves all of physical RAM up front; pages are handed out on demand.

PagedAttention does the same for the KV cache:

• The KV cache of a sequence is split into fixed-size KV blocks, each holding the K and V vectors for a fixed number of tokens — the block size, default 16 tokens.
• Blocks live in a global pool of physical GPU memory and need not be contiguous.
• Each sequence has a block table mapping its logical block index → physical block number, exactly like a page table.
• The attention kernel is modified to gather K/V from these scattered blocks using the block table, so attention runs correctly over non-contiguous memory.

Diagram-in-words

Logical view (what the sequence "sees"):
  Seq A tokens: [ t0 t1 ... t15 | t16 t17 ... t31 | t32 t33 ... ]
                 logical block 0   logical block 1   logical block 2

Block table for Seq A:  [ 0 -> phys #7 ] [ 1 -> phys #3 ] [ 2 -> phys #11 ]

Physical KV block pool (16 tokens each, scattered in HBM):
  #0  #1  #2  #3(A1)  #4  #5  #6  #7(A0)  #8  #9  #10  #11(A2)  #12 ...
  free free free  used  free ... free  used   ...      used    free

A block is allocated only when the sequence’s current block fills up. A sequence generating 30 tokens uses 2 blocks (32 slots), wasting at most 15 token-slots in its last block — at most one block of internal fragmentation per sequence, and zero reservation waste. External fragmentation vanishes because all blocks are the same size and interchangeable. Effective utilization approaches ~96%.

Sharing and copy-on-write

Because blocks are indirected through a block table, two sequences can point their block tables at the same physical block. This is where paging pays a second dividend:

• Shared prompts. In parallel sampling or beam search, (n) outputs share the same prompt. Instead of (n) copies of the prompt’s KV cache, all (n) block tables point at one shared set of prompt blocks. The paper reports up to 55% memory savings on parallel sampling / beam search.
• Copy-on-write (CoW). When one sharer needs to diverge (e.g., append a different token into a shared block), vLLM copies just that one block, updates that sequence’s block table, and leaves the others untouched — the same trick fork() uses. Reference counts on each block track sharing.

This block-level sharing is the foundation that automatic prefix caching (below) builds on.

Worked memory example

Serve Llama-3-8B in fp16 on one A100-80GB.

1. Weights: (8\text{B} \times 2\ \text{bytes} = 16\ \text{GB}).
2. Budget: --gpu-memory-utilization 0.9 → vLLM may use (0.9 \times 80 = 72\ \text{GB}).
3. Non-KV overhead: CUDA context, activations, CUDA graphs — call it ~2 GB.
4. KV cache pool: (72 - 16 - 2 = 54\ \text{GB}).
5. Per token: 128 KB (from above). Per block (16 tokens): (16 \times 128\ \text{KB} = 2\ \text{MB}).
6. Total blocks: (54\ \text{GB} / 2\ \text{MB} \approx 27{,}600) blocks = ~442,000 tokens of KV capacity.

That single number, ~442k tokens, is your batch budget. It can be 54 sequences at the full 8192-context ((442000/8192)), or ~880 concurrent chatbot turns averaging 500 tokens each, or anything in between. vLLM logs this at startup as # GPU blocks: 27600 and reports “Maximum concurrency for 8192 tokens per request.” Watch that log line — it tells you exactly how much headroom you bought.

Continuous (in-flight) batching in depth

PagedAttention gives you the memory to run a big batch. Continuous batching keeps that batch full.

Static batching (the naive baseline)

Collect (N) requests, run them together, wait for all to finish, return, repeat. The problem: generations have wildly different lengths. If request A emits 20 tokens and request B emits 500, A’s slot sits idle for 480 steps while B finishes, because the batch can’t return or refill until the whole batch is done. GPU utilization craters, and latency for A is dictated by the slowest sibling.

Dynamic batching (a partial fix)

Servers like Triton’s dynamic batcher wait a few milliseconds to form a larger batch before launching, then still run it to completion as a unit. This improves batch size but does not solve the ragged-completion problem — it’s still batch-at-a-time.

Continuous batching (a.k.a. in-flight / iteration-level scheduling)

vLLM schedules at the granularity of a single decode step, not a whole request (the idea comes from Orca’s iteration-level scheduling). At every forward pass:

1. Any sequence that emitted its EOS/stop this step is evicted immediately, and its KV blocks are freed.
2. Waiting requests are admitted mid-flight to fill the vacated slots.
3. The next forward pass runs over the new mix of prefills and decodes.

No sequence waits on a slower sibling. The batch is continuously topped up, so the GPU stays saturated. Anyscale’s widely cited benchmark measured up to 23× throughput from continuous batching plus paging versus naive batching, while also reducing p50 latency — a rare win on both axes, because higher utilization means less queueing.

Why it raises throughput: decoding is memory-bound, so throughput scales with how many sequences you can run per weight-load. Static batching leaves the effective batch shrinking toward 1 as siblings finish; continuous batching holds it near its memory-limited maximum every single step.

Prefill vs decode, and chunked prefill

A request has two phases with opposite performance profiles:

• Prefill — process the whole prompt in one big parallel pass. Compute-bound, high FLOPs, fills the pipeline. A 4000-token prompt is one heavy step.
• Decode — generate tokens one at a time. Memory-bound, tiny per-step compute.

Mixing them is awkward. A giant prefill can monopolize a forward pass and stall every decoding sequence, spiking inter-token latency (ITL) for everyone already streaming. This is the classic prefill/decode interference.

Chunked prefill (--enable-chunked-prefill) splits a large prefill into token-sized chunks and co-schedules a prefill chunk alongside ongoing decodes in the same batch, bounded by max-num-batched-tokens. Benefits:

• Smooths out ITL — decodes no longer freeze behind a monster prompt.
• Improves GPU utilization — decode steps are compute-light, so padding the batch with prefill tokens uses otherwise-idle FLOPs.
• In modern vLLM (V1 engine) chunked prefill is on by default, and prefill/decode are unified in one scheduler.

Tuning: raise max-num-batched-tokens for throughput (bigger chunks, more prefill work per step); lower it to protect decode latency (smaller chunks yield to decodes more often).

Prefix caching (automatic KV reuse)

Many requests share a prefix: the same long system prompt, a shared few-shot preamble, a document everyone asks questions about, a multi-turn conversation where each turn re-sends the history.

Automatic prefix caching (--enable-prefix-caching) hashes KV blocks by their content (and the tokens preceding them). When a new request’s prefix hashes to blocks already in the cache, vLLM skips recomputing that prefill entirely and points the new sequence’s block table at the cached blocks. It’s the block-sharing / CoW machinery from PagedAttention, applied across requests and over time.

• When it wins big: long shared system prompts, RAG with a fixed instruction preamble, multi-turn chat (each turn reuses the whole prior conversation’s KV), agent loops that resend context.
• Cost: cached blocks occupy KV memory that could otherwise hold active batch. Under memory pressure, cached prefix blocks are evicted LRU. It’s a hit-rate bet — near-free when hits are common, mild overhead when they’re not.
• In current vLLM (V1) prefix caching is enabled by default.

The saving is real work avoided: a 2000-token shared system prompt cached across 1000 requests skips ~2,000,000 tokens of prefill compute.

Memory and the engine args that matter

These are the flags you actually turn in production. Names are the current vllm serve CLI form (dashes); the Python LLM(...) form uses underscores.

Startup log lines to watch: # GPU blocks: (your KV capacity), Maximum concurrency for N tokens, and any Sequence group ... is preempted warnings (you’re memory-starved).

Parallelism for big models

When a model (plus its KV cache) doesn’t fit on one GPU, or single-GPU latency is too high, split it.

Tensor parallelism (TP) — --tensor-parallel-size

Mechanism: shard every layer’s weight matrices across N GPUs; each GPU computes its slice, and an all-reduce combines partial results each layer. The KV cache is also sharded (by heads), so TP grows your KV budget too.

When: the model is too big for one GPU, or you want lower latency on a single request (more GPUs working the same forward pass). Best within one node over NVLink, because the per-layer all-reduce is bandwidth-hungry.

Tradeoff: communication overhead grows with N; going cross-node over slower interconnect (Ethernet/PCIe) tanks efficiency. Keep -tp ≤ GPUs-per-node. TP size must divide the number of attention heads.

Pipeline parallelism (PP) — --pipeline-parallel-size

Mechanism: assign contiguous stages of layers to different GPUs; activations flow stage → stage. Communication is a small point-to-point hand-off between stages, tolerant of slower links.

When: to span multiple nodes, or to fit truly huge models where even TP-across-a-node isn’t enough. Common pattern: -tp 8 within each node × -pp 2 across two nodes = 16 GPUs.

Tradeoff: introduces pipeline bubbles (stages idle waiting for the previous stage); throughput-friendly with enough in-flight requests, but adds latency per request. Combine TP (intra-node) + PP (inter-node) for the best of both.

Rule of thumb: TP first, up to one node; PP to cross nodes. vLLM also supports data parallelism / multi-replica behind a router for pure scale-out.

Quantization — trade quality for memory and speed

Quantization shrinks weights (and optionally activations/KV) to fewer bits. Smaller weights free HBM for KV cache and can speed up the memory-bound decode. All of it costs some accuracy; how much depends on scheme and model.

Guidance:

• Memory-constrained, throughput-focused, Ampere/Ada: AWQ 4-bit is the workhorse — cuts weight memory ~4×, freeing large KV headroom.
• H100 / Hopper: prefer FP8 — near-lossless and uses native tensor-core FP8 for real speedups; add --kv-cache-dtype fp8 to roughly double KV capacity.
• Quality-sensitive tasks (code, math, long reasoning): measure. 4-bit weight-only can visibly hurt; validate on your eval set, not just perplexity.
• Quantization reduces weight memory, not KV — for long-context blowup, --kv-cache-dtype fp8 and --max-model-len are the relevant levers.

Speculative decoding — lower latency, not more throughput

Mechanism: a cheap draft proposes several tokens ahead; the big target model verifies them all in one forward pass. Accepted tokens are kept; the first rejection resets to the target’s own token. Because verification is parallel, a good draft yields multiple tokens per target forward pass — output is provably identical in distribution to the target alone (it’s exact, not approximate).

Draft sources vLLM supports include a small draft model, n-gram / prompt-lookup (propose from repeated text — great for code and RAG where output echoes input), and EAGLE / Medusa-style self-speculative heads.

When to use: latency-sensitive, low-to-moderate batch serving where the GPU has spare compute (decode is memory-bound, so verification is nearly free). Interactive chat, single-user, or bursty low-QPS endpoints.

Tradeoff: the win depends entirely on acceptance rate. Low acceptance means you paid for drafting and got little back — it can reduce throughput. And under high batch load the GPU is already compute-saturated, so speculation’s “free” parallel verification isn’t free anymore — its benefit shrinks or reverses. Rule: speculate when you’re latency-bound and under-batched; skip it when you’re throughput-bound and saturated.

Fully worked example: serve Llama-3-8B on one A100-80GB, then tune

1. Baseline launch (OpenAI-compatible server)

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

At startup, read the log:

INFO ... # GPU blocks: 27600, # CPU blocks: 2048
INFO ... Maximum concurrency for 8192 tokens per request: 53.9x

That confirms the ~442k-token / ~54-concurrent budget we computed by hand.

2. Call it with the OpenAI client

The server speaks the OpenAI API, so existing SDKs work unchanged — just point base_url at vLLM:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a terse assistant."},
        {"role": "user", "content": "Explain PagedAttention in two sentences."},
    ],
    max_tokens=128,
    temperature=0.2,
    stream=True,  # tokens stream as they decode
)
for chunk in resp:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

curl sanity check:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Meta-Llama-3-8B-Instruct","prompt":"Hello","max_tokens":16}'

3. Tuning walkthrough (target: max throughput for a RAG service)

Symptoms to check first with a load test (vllm bench serve or a locust/k6 run): GPU util, tokens/s, p99 ITL, and any preempted warnings.

• Shared 1500-token system+retrieval preamble across requests → keep --enable-prefix-caching (default on). Hit rate is high; prefill work drops sharply.
• GPU shows headroom, no OOM → push memory:

  vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --gpu-memory-utilization 0.94 \
    --max-num-seqs 384 \
    --max-num-batched-tokens 16384 \
    --enable-chunked-prefill \
    --max-model-len 8192
  

  Bigger token budget = fatter prefill chunks and higher throughput; more seqs = more concurrency, backed by the enlarged KV pool.
• p99 inter-token latency too high (big prompts stalling decodes) → lower --max-num-batched-tokens (e.g. to 4096). Smaller chunks yield to decodes more often, smoothing ITL at a small throughput cost.
• Sequence group is preempted by ... warnings → you over-committed KV. Either drop --max-num-seqs, drop --max-model-len, or raise --swap-space so preempted sequences swap to CPU instead of recomputing.
• Need more KV for longer contexts → add --kv-cache-dtype fp8 (roughly doubles KV capacity) and/or --quantization fp8 on an H100 to also shrink weights.
• Model too big for one GPU (e.g. Llama-3-70B) → --tensor-parallel-size 4 (or 8) within the node; across nodes add --pipeline-parallel-size 2.

Iterate: change one knob, re-run the load test, compare tokens/s and p99. Stop when you’re memory-limited (preemptions appear) or latency SLO-limited.

Comparison: vLLM vs TGI vs TensorRT-LLM / Triton

Reality check: the three have converged — TGI and TensorRT-LLM adopted paged KV and in-flight batching. TensorRT-LLM often wins raw latency/throughput on NVIDIA thanks to ahead-of-time kernel compilation, at the cost of a per-model engine-build step and NVIDIA lock-in. vLLM wins on flexibility, ease, and hardware breadth, and is the usual default. Always benchmark on your model, hardware, and traffic shape before deciding.

Failure modes and pitfalls

• OOM at startup from gpu-memory-utilization too high. vLLM pre-allocates the KV pool; if you set 0.98 with no slack, activation spikes or CUDA-graph capture push you over and it crashes on load. Back off to ~0.90 and grow gradually. Co-located processes share the same HBM — vLLM only sees the fraction you give it.
• Preemption and recompute thrash. When admitted sequences collectively exceed KV capacity, vLLM preempts some — either swapping their KV to CPU (--swap-space) or discarding and recomputing it later. Frequent preempted warnings mean you over-committed max-num-seqs/max-model-len; throughput drops as work is redone. Fix by lowering concurrency, shortening max-model-len, or adding swap.
• Long-context KV blowup. KV grows linearly with context. A handful of 128k-token requests can consume the entire pool and starve everyone else. Bound it with --max-model-len, --kv-cache-dtype fp8, and admission limits; don’t advertise a context you can’t afford to serve concurrently.
• Quantization quality loss. 4-bit weight-only (AWQ/GPTQ) can degrade code/math/reasoning noticeably even when perplexity looks fine. Always validate on a task-specific eval, and prefer FP8 on Hopper where it’s near-lossless.
• Speculative decoding backfiring. Low draft acceptance, or high batch load, turns speculation into pure overhead. Measure acceptance rate; disable under saturation.
• max-num-batched-tokens too low with chunked prefill off. If a prompt exceeds the token budget and chunked prefill isn’t enabled, requests fail. Keep chunked prefill on, or set the budget ≥ max-model-len.
• Assuming max-num-seqs is the batch limit. Usually KV memory binds first. Raising max-num-seqs without KV headroom just causes preemption. Watch the # GPU blocks log, not just the seq cap.
• Prefix cache eviction under pressure. Cached prefixes compete with active KV; under load they’re evicted and hit rate falls — throughput quietly regresses. Size memory for both if prefix reuse is core to your workload.

Production checklist — what an interviewer probes

1. “Why is LLM decode memory-bound, and why does that make batching the key throughput lever?” — Weights are re-read every token; batching amortizes the read. Expect you to connect this to KV cache being the batch-size limiter.
2. “Explain PagedAttention and what waste it eliminates.” — Blocks + block table, non-contiguous KV, block size 16, kills internal/external fragmentation and reservation waste (from ~20–38% utilization to ~96%).
3. “Continuous vs static batching — why does continuous raise throughput and cut latency?” — Iteration-level scheduling evicts finished seqs and admits new ones each step; no slot idles behind a slow sibling.
4. “Walk me through sizing KV cache and picking gpu-memory-utilization / max-num-seqs / max-model-len for a given GPU.” — Do the bytes/token math, subtract weights, derive block count and concurrency; explain preemption when over-committed.
5. “When TP vs PP? What are the comms costs?” — TP intra-node over NVLink (per-layer all-reduce), PP across nodes (stage hand-off, pipeline bubbles).
6. “Quantization choices and their quality cost — AWQ vs GPTQ vs FP8, and when each?” — 4-bit weight-only for memory on Ampere; FP8 near-lossless on Hopper; validate on task evals.
7. “When does speculative decoding help vs hurt?” — Helps latency at low batch with high acceptance; hurts under saturation or low acceptance.
8. “How do you debug an OOM or a latency spike in production?” — Check # GPU blocks and preemption logs; lower gpu-memory-utilization/max-model-len; tune max-num-batched-tokens for ITL; add swap-space; confirm prefix-cache hit rate.

Bonus signals: knowing chunked prefill and prefix caching are on by default in the V1 engine, that max-num-seqs rarely binds before KV memory, and that you always benchmark on real traffic rather than trusting a spec sheet.

How the attention kernel actually reads paged blocks

It’s worth being precise about why PagedAttention needs a custom kernel, because interviewers probe it. Standard fused attention (FlashAttention) assumes K and V for a sequence live in one contiguous tensor it can stride through. Paged KV breaks that assumption: a sequence’s K/V are scattered across physical blocks in arbitrary order.

The PagedAttention kernel therefore takes the block table as an input. For a query at the current position it:

1. Reads the sequence’s block table (logical block → physical block number).
2. For each logical block, computes attention scores ( q \cdot k ) against the K vectors in that physical block, iterating block by block.
3. Accumulates the softmax-weighted sum of V vectors from the same blocks.

Because the block is the unit of gather, the kernel does a small indirection per block (once per 16 tokens), not per token — cheap relative to the matmul. The block table lives in GPU memory alongside the cache. Modern vLLM builds this on FlashAttention/FlashInfer backends that natively accept paged KV, so you keep FlashAttention’s IO-awareness and paging. This is the crux: paging costs almost nothing at kernel time, yet returns most of the wasted memory as usable batch.

The scheduler: waiting, running, swapped

vLLM’s scheduler maintains three queues and reconciles them every step — this is the machinery behind continuous batching, preemption, and swapping.

• Waiting — admitted requests not yet started (need KV blocks allocated for their prefill).
• Running — sequences actively decoding (or being prefilled) this step.
• Swapped — sequences preempted out of GPU KV, their blocks parked in CPU swap space.

Each iteration the scheduler:

1. Frees blocks of any sequence that finished last step.
2. Tries to admit waiting requests into running, subject to the KV block budget and max-num-seqs / max-num-batched-tokens.
3. If running collectively needs more blocks than exist (e.g., all sequences grew a token and a new block boundary was crossed), it preempts the lowest-priority sequences — either swap (copy their KV blocks to CPU, restore later) or recompute (drop KV, re-run prefill when readmitted). Recompute is the default for short sequences; swap wins for long ones where recompute is expensive.

The default policy is FCFS-ish with the newest/lowest-priority preempted first. The practical takeaway: preemption is the pressure-relief valve, and seeing it constantly in logs means your admission settings exceed your true KV capacity. It is correct behavior, not a bug — but it costs throughput, so tune it away.

Benchmarking vLLM properly

Never tune by feel. vLLM ships a benchmark harness that mirrors real serving:

# Start the server, then in another shell:
vllm bench serve \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000 \
  --request-rate 20

Metrics that matter, and what they mean:

• Throughput (tokens/s, requests/s) — the number to maximize for batch/offline workloads.
• TTFT (time to first token) — dominated by prefill and queueing; what a chat user feels as “lag before it starts.”
• ITL / TPOT (inter-token latency / time per output token) — decode smoothness; hurt by big un-chunked prefills.
• p50 vs p99 — always look at the tail. High p99 with fine p50 usually means preemption or prefill interference.

Sweep one knob at a time (gpu-memory-utilization, max-num-batched-tokens, max-num-seqs) and plot throughput vs p99 latency. The right operating point is the knee of that curve for your SLO — not the max-throughput point, which usually violates latency targets.

V1 engine and disaggregated serving (where vLLM is heading)

Two architectural notes worth knowing:

• The V1 engine (default in current vLLM) rewrote the core for lower CPU overhead and a unified scheduler where prefill and decode are co-scheduled by default. Chunked prefill and prefix caching are on by default there. If you read older tutorials that tell you to manually enable these, that advice is stale.
• Disaggregated prefill/decode separates the compute-bound prefill and memory-bound decode onto different GPU pools, streaming the KV cache between them. Because the two phases have opposite resource profiles, dedicating hardware to each — and scaling them independently — can beat co-locating them, especially at high scale with long prompts. This is an emerging pattern (KV transfer over NVLink/RDMA) that large deployments increasingly adopt.

Second worked example: Llama-3-70B across GPUs

An 8B model fits one card; a 70B does not. In fp16, weights alone are ( 70\text{B} \times 2 = 140\ \text{GB} ) — larger than a single 80 GB GPU. Options:

Option A — tensor-parallel across 4 GPUs (one node, NVLink):

vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

Weights shard to ( 140/4 = 35\ \text{GB} ) per GPU, leaving each card ~( 0.92 \times 80 - 35 \approx 38\ \text{GB} ) (minus overhead) for its KV shard. KV is also split by heads across the 4 GPUs, so aggregate KV capacity is roughly 4× a single card’s leftover — that is what makes big batches on 70B feasible.

Option B — quantize to AWQ 4-bit, fit on fewer GPUs:

vllm serve casperhansen/llama-3-70b-instruct-awq \
  --quantization awq \
  --tensor-parallel-size 2 \
  --max-model-len 8192

4-bit weights are ~( 70\text{B} \times 0.5 = 35\ \text{GB} ), fitting two 80 GB cards with room for KV. Fewer GPUs, lower cost, at some quality cost — validate on your eval set.

Option C — two nodes, TP×PP: --tensor-parallel-size 8 --pipeline-parallel-size 2 spreads a large model across 16 GPUs, TP within each node over NVLink and PP across the two nodes over the slower inter-node link. Launch via vLLM’s Ray-based multi-node path.

Decision order: fit on one node with TP first; quantize to shrink weights and cut GPU count; go multi-node with PP only when a single node genuinely cannot hold model + working KV.

Reading the startup logs (your first diagnostic)

Every launch prints the numbers that tell you whether your config is sane. Learn to read them before touching load tests:

INFO ... Available KV cache memory: 53.7 GiB
INFO ... GPU KV cache size: 442,368 tokens
INFO ... Maximum concurrency for 8192 tokens per request: 54.0x
INFO ... # GPU blocks: 27648, # CPU blocks: 2048

• Available KV cache memory — what’s left after weights + overhead. If this is tiny or negative-adjacent, lower max-model-len, quantize, or add GPUs.
• GPU KV cache size (tokens) — your total batch budget in tokens; divide by average request length to estimate real concurrency.
• Maximum concurrency — full-context sequences you can run at once. If it’s < your expected concurrency, you will preempt under load.
• # CPU blocks — swap capacity, sized by --swap-space.

If you never look at anything else, look at these four lines. They convert the abstract flags into the one number that governs throughput: how many tokens of KV you can hold.

Quick-reference config recipes

Further reading

• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023 — the vLLM paper: https://arxiv.org/abs/2309.06180
• ACM DL (SOSP ’23 proceedings): https://dl.acm.org/doi/10.1145/3600006.3613165
• vLLM PagedAttention design doc: https://docs.vllm.ai/en/latest/design/paged_attention/
• vLLM Engine Arguments reference: https://docs.vllm.ai/en/stable/configuration/engine_args/
• vLLM Optimization and Tuning (chunked prefill, batching knobs): https://docs.vllm.ai/en/latest/configuration/optimization/
• vLLM Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html
• vLLM Distributed Inference (TP/PP): https://docs.vllm.ai/en/latest/serving/distributed_serving.html
• vLLM Quantization overview: https://docs.vllm.ai/en/latest/features/quantization/
• vLLM Speculative Decoding: https://docs.vllm.ai/en/latest/features/spec_decode.html
• vLLM OpenAI-compatible server: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html
• Anyscale, “How continuous batching enables 23x throughput in LLM inference”: https://www.anyscale.com/blog/continuous-batching-llm-inference
• Orca (iteration-level scheduling), OSDI 2022: https://www.usenix.org/conference/osdi22/presentation/yu

Topic 6: Autoscaling

What You’ll Learn

This topic teaches you how to:

• Set up Horizontal Pod Autoscaling (HPA)
• Scale based on CPU/memory metrics
• Scale based on custom metrics (request rate, latency)
• Configure scaling policies
• Handle scale-up and scale-down events

Why Autoscaling?

Benefits

• Cost optimization: Scale down when not needed
• Performance: Scale up under load
• Automation: No manual intervention needed
• Efficiency: Right-size resources

When to Scale

• High CPU/memory: Pods are overloaded
• High request rate: Many incoming requests
• High latency: System struggling
• Queue building: Requests waiting

Types of Autoscaling

1. Horizontal Pod Autoscaler (HPA)

Scales number of pod replicas based on metrics.

2. Vertical Pod Autoscaler (VPA)

Adjusts resource requests/limits (advanced).

3. Cluster Autoscaler

Adds/removes nodes (cloud providers).

HPA Basics

Simple HPA (CPU-based)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-serving-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-serving
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

How It Works

1. HPA checks metrics every 15s (default)
2. Compares current vs target
3. Calculates desired replicas
4. Updates deployment
5. K8s creates/destroys pods

Custom Metrics

Request Rate Scaling

Scale based on requests per second:

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "10"

Latency-based Scaling

Scale when latency is high:

metrics:
- type: Pods
  pods:
    metric:
      name: request_latency_p95
    target:
      type: AverageValue
      averageValue: "500m"  # 500ms

Scaling Behavior

Scale-up Policy

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Percent
      value: 100
      periodSeconds: 15
    - type: Pods
      value: 4
      periodSeconds: 15
    selectPolicy: Max

Scale-down Policy

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 50
      periodSeconds: 60
    selectPolicy: Min

Metrics Server

HPA needs metrics. Install metrics server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Verify:

kubectl top nodes
kubectl top pods

Prometheus Adapter

For custom metrics, use Prometheus Adapter:

1. Install Prometheus
2. Install Prometheus Adapter
3. Configure custom metrics
4. HPA can query Prometheus

See 08_monitoring/ for Prometheus setup.

Complete HPA Example

See hpa.yaml for a complete example with:

• CPU scaling
• Memory scaling
• Custom metrics
• Scaling policies

Testing Autoscaling

Generate Load

# Use Locust or similar
locust -f locust_test.py --host=http://<service-url> \
  --users 50 --spawn-rate 5

Watch Scaling

# Watch HPA
kubectl get hpa -w

# Watch pods
kubectl get pods -w

# Watch deployment
kubectl get deployment llm-serving -w

Check Metrics

# CPU/Memory
kubectl top pods

# HPA events
kubectl describe hpa llm-serving-hpa

Best Practices

1. Set min replicas: Always have some pods running
2. Set max replicas: Prevent runaway scaling
3. Use stabilization windows: Avoid flapping
4. Monitor scaling events: Track scale-up/down
5. Test under load: Verify scaling works
6. Consider costs: More pods = more cost

Common Issues

HPA Not Scaling

• Check metrics server is running
• Verify metrics are available
• Check HPA status: kubectl describe hpa
• Ensure deployment has resource requests

Scaling Too Aggressively

• Increase stabilization window
• Adjust scaling policies
• Check for metric spikes

Scaling Too Slowly

• Decrease stabilization window
• Adjust scaling policies
• Check metric collection delay

Exercises

1. Basic HPA: Set up CPU-based autoscaling
2. Custom metrics: Scale based on request rate
3. Scaling policies: Configure scale-up/down behavior
4. Load test: Generate load and observe scaling
5. Monitor: Track scaling events and metrics

Next Steps

• Topic 7: Canary deployments for safe updates
• Topic 8: Monitor autoscaling with Grafana
• Topic 9: Version management with scaling

Further Reading

• Kubernetes HPA
• Custom Metrics
• Scaling Best Practices

Autoscaling LLM Inference

Scaling GPU serving with demand — without wrecking latency or blowing the budget.

Why This Matters

A web service scales cheaply: pods are small, start in seconds, and CPU utilization is a clean proxy for load. LLM inference breaks all three assumptions. A single replica pins one or more GPUs that cost more per hour than an entire fleet of CPU pods. A new replica must pull tens of gigabytes of weights and warm CUDA before it serves a single token, so “add capacity” is a multi-minute operation, not a multi-second one. And traffic is bursty — a Slack integration or a batch job can 10x your request rate in seconds.

Get autoscaling wrong and you fail in one of two expensive directions:

• Under-provision: the queue backs up, time-to-first-token (TTFT) climbs, requests time out, and by the time a new replica is ready the spike is over.
• Over-provision: you pay for idle H100s around the clock to hedge against a spike that comes twice a day.

This chapter is about threading that needle: scaling on the right signals, using the right mechanism (HPA, KEDA, Knative), and mitigating the cold-start tax that makes LLM autoscaling uniquely hard. For the metric definitions and the Prometheus/Grafana stack that feeds these controllers, see the Monitoring chapter.

Core Intuition: Why CPU-Based HPA Is Wrong for LLMs

The default Kubernetes HorizontalPodAutoscaler scales on CPU utilization. For a stateless web app that is a reasonable proxy: more requests → more CPU → scale up. For an LLM server it is actively misleading.

Consider what a vLLM or TGI process actually does. The heavy lifting happens on the GPU; the Python/host process spends most of its time waiting on CUDA kernels and shuffling tensors. So:

• A GPU that is 100% saturated — KV cache full, requests queuing — can show modest host CPU. HPA sees “plenty of headroom” and refuses to scale while your p99 latency melts.
• A freshly loaded replica warming its cache can spike CPU while serving nothing, tricking HPA into scaling up when it shouldn’t.

CPU utilization is decoupled from the thing you actually care about: can I admit another request and still hit my latency SLO? For LLMs the honest answer lives in queue depth, in-flight concurrency, KV-cache pressure, GPU utilization, and TTFT — never in host CPU.

The mental model: scale on the length of the line, and on how long people wait in it — not on how busy the cashier’s hands look.

The scaling control loop

Every mechanism below is the same loop with different parts swapped in. Keep it in your head:

  requests ──▶ [ vLLM replicas ] ──▶ metrics (queue depth, GPU util, TTFT)
                     ▲                          │
                     │                          ▼
              scale up / down          [ Prometheus / dcgm-exporter ]
                     ▲                          │
                     │                          ▼
             [ Deployment ] ◀── HPA / KEDA / KPA ◀── PromQL query vs target

The controller polls a metric, compares it to a target, and nudges the replica count. Everything interesting — which metric, which target, how fast to react, whether zero is allowed — is a knob on that loop.

The Right Scaling Signals

There is no single perfect signal. Each trades responsiveness against noise and against how directly it maps to user-visible latency. Good production setups combine a fast demand signal (queue depth / concurrency) with an SLO guardrail (TTFT or p95 latency) so a breach of either forces a scale-up.

Rule of thumb: lead with queue depth or concurrency, guard with a latency SLO, and treat GPU util as a sanity cross-check — not as the trigger.

Mechanism 1 — HPA with Custom / External Metrics

Kubernetes’ HPA can scale on more than CPU. Since autoscaling/v2 it supports three metric flavors:

• Resource — CPU/memory (the default; wrong for us).
• Pods — a custom per-pod metric averaged across pods (e.g. queue depth per replica).
• Object / External — a metric attached to another object or pulled from an external system (e.g. a Prometheus query).

To feed HPA a Prometheus metric you install the Prometheus Adapter (prometheus-adapter), which registers the custom.metrics.k8s.io / external.metrics.k8s.io APIs and translates HPA’s metric requests into PromQL. GPU utilization itself comes from NVIDIA’s dcgm-exporter (metric DCGM_FI_DEV_GPU_UTIL), scraped by Prometheus.

Wiring the metric pipeline

HPA does not speak PromQL. The Prometheus Adapter bridges the gap: you give it a rule that maps a Kubernetes metric name to a query. A minimal rule exposing vLLM’s queue depth as a per-pod custom metric looks like:

# prometheus-adapter values.yaml (rules.custom[])
rules:
  custom:
    - seriesQuery: 'vllm:num_requests_waiting{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "vllm:num_requests_waiting"
        as: "vllm_num_requests_waiting"     # HPA-friendly name (no colon)
      metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

Verify the metric is actually served before pointing HPA at it — a huge fraction of “HPA won’t scale” incidents are just a missing or misnamed metric:

kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/inference/pods/*/vllm_num_requests_waiting" | jq .

If that returns no metrics returned from custom metrics API, the problem is the pipeline (labels, series name, scrape) — not the HPA.

The HPA algorithm

HPA computes desired replicas with a simple ratio:

[ \text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right\rceil ]

With multiple metrics, HPA computes a target for each and takes the maximum — the metric demanding the most replicas wins. That is exactly why a fast demand signal plus a latency guardrail composes well: whichever is more stressed drives the decision.

Worked HPA manifest — scale on GPU utilization + queue depth

This assumes dcgm-exporter and prometheus-adapter are installed, and the adapter exposes DCGM_FI_DEV_GPU_UTIL as a Pods metric and vllm_num_requests_waiting as an External metric.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicas: 2            # provisioned floor — never cold-start the first request
  maxReplicas: 12           # capped by GPU quota (see pitfalls)
  metrics:
    # Primary demand signal: waiting requests per replica
    - type: Pods
      pods:
        metric:
          name: vllm_num_requests_waiting
        target:
          type: AverageValue
          averageValue: "5"        # aim to keep <5 queued per pod
    # Hardware cross-check: average GPU utilization
    - type: Pods
      pods:
        metric:
          name: DCGM_FI_DEV_GPU_UTIL
        target:
          type: AverageValue
          averageValue: "75"       # scale up past ~75% average util
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0     # react to spikes immediately
      policies:
        - type: Pods
          value: 4                      # add up to 4 pods...
          periodSeconds: 60             # ...per minute
        - type: Percent
          value: 100                    # ...or double, whichever is larger
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 min of calm before shrinking
      policies:
        - type: Pods
          value: 1                      # remove at most 1 pod...
          periodSeconds: 120            # ...every 2 min — GPUs are expensive to churn
      selectPolicy: Min

Tuning notes.

• scaleUp.stabilizationWindowSeconds: 0 — scale up on the freshest reading. Because cold starts are slow, hesitating on scale-up is the costliest mistake you can make.
• scaleDown.stabilizationWindowSeconds: 300 — HPA takes the highest recommendation over the window before shrinking, so a 5-minute window prevents a brief traffic lull from tearing down a replica you’ll re-pay a cold start to rebuild.
• Asymmetric policies — aggressive up, gentle down (one pod every two minutes). This is the opposite of a cost-first web-app config, and it’s deliberate: for GPUs, flapping is more expensive than a little idle.
• Target values are per-pod averages — averageValue: "5" means HPA aims for 5 waiting requests per replica, so the ratio math scales linearly with fleet size.

Mechanism 2 — KEDA for Event-Driven & Queue-Based Scaling

HPA + Prometheus Adapter works but is fiddly: you maintain adapter rules, and HPA alone cannot scale to zero. KEDA (Kubernetes Event-Driven Autoscaling) sits on top of HPA and fixes both. It ships 70+ scalers (Prometheus, Kafka, SQS, RabbitMQ, Redis, …) and, crucially, can scale a deployment from 0 → 1 and back to 0.

KEDA introduces two ideas HPA lacks:

• activationThreshold — the value that flips a workload from zero to one. This is separate from the scaling threshold (which governs 1→N). It exists precisely so a single stray request doesn’t wake a cold GPU, and so a trickle doesn’t keep one warm.
• minReplicaCount: 0 — legal in KEDA, impossible in raw HPA.

Worked KEDA ScaledObject — queue depth + latency guardrail

This mirrors the pattern AWS documents for vLLM on EKS: a primary queue-depth trigger and a p95-latency guardrail, scaling to satisfy whichever demands more replicas.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicaCount: 1          # warm floor; set 0 only if cold starts are acceptable
  maxReplicaCount: 12
  pollingInterval: 15         # query Prometheus every 15s
  cooldownPeriod: 300         # after last trigger, wait 5 min before scaling toward min
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:               # KEDA passes this straight through to the HPA it manages
        scaleDown:
          stabilizationWindowSeconds: 300
        scaleUp:
          stabilizationWindowSeconds: 0
  triggers:
    # Primary: queue depth (waiting requests), averaged per replica
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
        query: sum(vllm:num_requests_waiting) or vector(0)
        threshold: "5"
        activationThreshold: "1"     # first waiting request wakes the deployment
    # Guardrail: p95 end-to-end latency in seconds
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(vllm:e2e_request_latency_seconds_bucket[1m])) by (le)) or vector(0)
        threshold: "5"               # scale up if p95 latency exceeds 5s

Tuning notes.

• or vector(0) is not decoration — if the query returns no series (e.g. the deployment is scaled to zero and exporting nothing), KEDA would otherwise error. or vector(0) yields a clean 0, which is exactly the “no demand” reading you want.
• metricType: AverageValue divides the query result by the current replica count so the target is per-pod; use Value only when your query already returns a per-pod figure.
• activationThreshold: "1" vs threshold: "5" — one waiting request is enough to justify the first replica; you only add more replicas once the per-pod queue passes 5.
• cooldownPeriod governs the final ramp toward minReplicaCount (including the drop to zero); the HPA behavior block governs the 1→N steps.
• Prefer a dedicated queue metric (num_requests_waiting) over latency as the primary — latency lags, and by the time p95 breaches, the SLO is already violated. Latency is the seatbelt, not the accelerator.

Mechanism 3 — Scale-to-Zero, Cold Starts, and Serverless GPU

Scale-to-zero is the dream: pay nothing when idle. For LLMs it collides head-on with the cold-start problem.

Anatomy of an LLM cold start

When a scaled-to-zero deployment gets a request, the clock runs through:

1. Scheduling — Kubernetes finds a node with a free GPU (seconds → minutes if the cluster autoscaler must add a node).
2. Image pull — the container image is often 5–15 GB (CUDA, PyTorch, vLLM). Seconds to minutes if not cached on the node.
3. Weight load — read tens of GB of weights from disk/network into host RAM, then copy to VRAM. This dominates — often the largest single chunk.
4. CUDA / engine warm-up — initialize CUDA context, compile/capture CUDA graphs, allocate the KV cache.

For a mid-size model this is routinely 1–5 minutes, and can be far worse if the cluster autoscaler has to boot a fresh GPU node first. That is an eternity for an interactive request. So true scale-to-zero is only acceptable for latency-tolerant workloads (batch, internal tools, dev). For anything user-facing, you keep a warm floor.

Cold-start mitigation comparison

Knative / serverless GPU. Knative Serving offers request-driven autoscaling with native scale-to-zero. Its default KPA (Knative Pod Autoscaler) scales on concurrency or RPS rather than CPU — a much better fit for LLMs than raw HPA. Key pieces:

• containerConcurrency / target concurrency — the per-replica in-flight target KPA scales to maintain.
• The activator buffers requests while a scaled-to-zero service spins up, so requests aren’t dropped — they’re held (and pay the cold-start latency).
• Panic mode / target-burst-capacity — when traffic spikes sharply, KPA enters a short “panic” window and scales on a much shorter horizon to react fast, then relaxes.

Knative is elegant for bursty, latency-tolerant serving, but the activator’s request buffering doesn’t erase the cold start — it just prevents dropped requests. You still pay the minutes. Pair scale-to-zero with a snapshot/fast-load strategy, or keep minScale ≥ 1 for interactive paths.

Worked Knative Service — concurrency-driven with a warm floor

The same workload as a Knative Service, scaling on concurrency with KPA. Note minScale: 1 — a warm floor that dodges the cold start on the interactive path while still capping cost with maxScale.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: vllm-llama3-8b
  namespace: inference
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/class: "kpa.autoscaling.knative.dev"
        autoscaling.knative.dev/metric: "concurrency"
        autoscaling.knative.dev/target: "8"        # ~8 in-flight requests per replica
        autoscaling.knative.dev/target-utilization-percentage: "80"
        autoscaling.knative.dev/min-scale: "1"     # warm floor (set 0 for scale-to-zero)
        autoscaling.knative.dev/max-scale: "12"
        autoscaling.knative.dev/scale-down-delay: "5m"   # hold before shrinking
        autoscaling.knative.dev/target-burst-capacity: "200"  # activator buffers bursts
    spec:
      containerConcurrency: 16     # hard per-replica ceiling
      containers:
        - image: vllm/vllm-openai:latest
          resources:
            limits:
              nvidia.com/gpu: "1"

Tuning notes.

• target: 8 vs containerConcurrency: 16 — KPA aims to keep ~8 concurrent requests per replica (the soft target) while 16 is the hard cap. Keeping the soft target well below the hard cap leaves slack for the seconds before a new replica is ready.
• target-burst-capacity decides how much spike the activator absorbs (buffering requests) before KPA has scaled out; higher values route more traffic through the activator, trading a little steady-state latency for burst safety.
• scale-down-delay is Knative’s answer to flapping — the KPA equivalent of HPA’s scale-down stabilization window.
• Set min-scale: 0 only when the workload tolerates the cold start; the activator will hold the request but the user still waits out the load.

Which mechanism should I pick?

These aren’t mutually exclusive: a common production shape is KEDA for the demand-driven scale (including 0→1) plus a provisioned floor for the interactive tier, with all three fed by the same Prometheus pipeline.

The Cost vs Latency Tradeoff

Every autoscaling decision is a bet on this tradeoff. Warm replicas cost money every second they’re idle; cold replicas cost latency (and lost requests) every time you’re caught short. You can make the tradeoff explicit.

A worked calculation

Suppose one H100 replica costs about ( $3.50 ) per hour and serves a steady ( 20 ) requests/second at your latency SLO. Your traffic is ( 20 ) req/s for 8 business hours and near-zero the other 16.

Option A — always-on flat fleet. Provision for peak, run 24/7:

[ \text{Cost}_A = 1 \text{ replica} \times $3.50/\text{hr} \times 24 \text{ hr} = $84 \text{ per day} ]

Option B — autoscale with a warm floor. Keep minReplicas = 1 only during the 8 busy hours, scale to zero otherwise. Assume the 16 idle hours truly cost nothing:

[ \text{Cost}_B = 1 \times $3.50 \times 8 = $28 \text{ per day} ]

That’s a 67% saving — but it buys a cold start on the first request each morning and after any midday lull. If the SLO forbids a multi-minute first response, you instead keep a warm floor 24/7 and land back near Option A, or you spend engineering effort on snapshot/fast-load so scale-to-zero becomes safe.

Headroom sizing. To absorb bursts without waiting on a cold start, provision a buffer. If your scale-up (schedule + pull + load + warm) takes ( T_{cold} = 180 ) s and traffic can climb at ( 2 ) req/s², a replica in flight can’t help for 3 minutes. Size the warm buffer to cover demand growth over ( T_{cold} ):

[ \text{buffer replicas} = \left\lceil \frac{\Delta(\text{req/s over } T_{cold})}{\text{capacity per replica}} \right\rceil ]

The general shape: provisioned floor covers the baseline, headroom buffer covers what arrives during a cold start, and autoscaling handles the sustained ramp. Tune the floor and buffer to the cost of a missed SLO, not to a generic utilization target.

Failure Modes & Pitfalls

• Flapping (thrashing). Symmetric or twitchy thresholds scale up and down every few minutes, and each cycle pays a cold start. Fix: long scaleDown.stabilizationWindowSeconds (300s+), conservative scale-down policies, and a generous cooldownPeriod. For GPUs, err toward stickiness.
• Scaling on a lagging metric. If p95 latency or GPU utilization is your primary trigger, you scale after users are already hurting, and because cold starts are slow you stay behind the curve for minutes. Fix: lead with a leading signal (queue depth / concurrency); keep latency as a guardrail only.
• Thundering herd on cold model load. A big spike triggers many replicas at once; they simultaneously hammer the same weights bucket / registry, saturating network and disk, so all of them cold-start slower. Fix: cap scaleUp step size (Pods: value / periodSeconds), stagger with fast-load streaming, pre-warm images, and cache weights on nodes.
• GPU quota / capacity ceilings. maxReplicas is a wish; cloud GPU quota and actual availability are the reality. HPA will happily request 12 pods that sit Pending forever because there are no H100s to schedule them on. Fix: set maxReplicas to your real quota, alert on Pending GPU pods, and combine with cluster-autoscaler node pools sized to quota.
• Metric pipeline is a hidden dependency. If Prometheus, the adapter, or dcgm-exporter hiccups, HPA/KEDA see stale or missing metrics and may freeze or over-react. Fix: or vector(0) guards, ignoreNullValues, alert on the metrics pipeline itself, and a sane default replica count.
• Per-pod vs total metric confusion. Using Value where you meant AverageValue (or a raw sum() without dividing by replicas) makes the target scale wrong as the fleet grows — you either never scale or scale to the moon. Fix: be explicit about per-pod semantics and test with a load generator.
• Scale-down mid-generation. Terminating a pod that’s mid-stream kills in-flight long generations. Fix: graceful termination with a drain period longer than your max generation time, and terminationGracePeriodSeconds sized accordingly.

Production Checklist — What an Interviewer Probes

1. “Why not CPU-based HPA for an LLM?” — Host CPU is decoupled from GPU saturation; a full GPU can look idle to HPA. Name the real signals: queue depth, concurrency, KV-cache, GPU util, TTFT.
2. “What’s your primary scaling signal and why?” — A leading demand signal (vllm:num_requests_waiting / concurrency), with a lagging latency SLO as a guardrail, and the max-of-metrics behavior that composes them.
3. “How do you handle cold starts?” — Quantify the phases (schedule, pull, load, warm), then name concrete mitigations: warm floor, headroom buffer, fast weight streaming, snapshot/checkpoint-restore, image pre-pull.
4. “Scale to zero — yes or no?” — “It depends on the SLO.” Fine for batch/internal; dangerous for interactive unless snapshotting makes cold starts sub-second. Explain KEDA’s activationThreshold.
5. “How do you stop it flapping?” — Asymmetric behavior: aggressive scale-up (0s window), conservative scale-down (300s+ window, one pod at a time), cooldown.
6. “HPA vs KEDA vs Knative — when each?” — HPA+adapter for simple custom-metric scaling; KEDA for event/queue-driven and scale-to-zero on any metric; Knative/KPA for concurrency-driven serverless with request buffering.
7. “What breaks under a real traffic spike?” — Thundering herd on weight load, GPU quota ceilings leaving pods Pending, and lagging-metric lock-step. Have a mitigation for each.
8. “How do you pick the target value / threshold?” — Derive it from load tests (see the Load Testing chapter): find the per-replica concurrency/queue depth at which TTFT just meets SLO, then set the target below it.

Further Reading

• Kubernetes — Horizontal Pod Autoscaler (v2, behavior & algorithm)
• Kubernetes — HPA Walkthrough with custom metrics
• KEDA — Prometheus scaler and Scaling Deployments (activation vs threshold, scale-to-zero)
• AWS — Autoscale AI inference with HPA and KEDA on EKS (vLLM)
• vLLM — Autoscaling with KEDA (production-stack)
• Knative — Autoscaling (KPA, concurrency, scale-to-zero, panic mode)
• Prometheus Adapter — kubernetes-sigs/prometheus-adapter
• NVIDIA — dcgm-exporter (GPU metrics for Prometheus)
• NVIDIA — Dynamo Snapshot: fast startup for inference on Kubernetes
• Microsoft Azure — Eliminate LLM cold starts: load models up to 6x faster with Run:ai Model Streamer
• KServe — Autoscaler for generative inference

Related chapters: Load Testing to derive your target thresholds, vLLM Serving for the engine metrics, and Monitoring for the Prometheus/Grafana pipeline that feeds every controller above.

Topic 7: Canary Deployments

What You’ll Learn

This topic teaches you how to:

• Deploy new model versions safely
• Split traffic between versions
• Gradually roll out updates
• Rollback quickly if issues occur
• A/B test different models
• Monitor canary performance

Why Canary Deployments?

Benefits

• Risk reduction: Test new version on small traffic
• Gradual rollout: Increase confidence before full deployment
• Quick rollback: Revert if issues detected
• A/B testing: Compare model versions
• Zero downtime: No service interruption

When to Use

• Model updates: New model version
• Configuration changes: Different parameters
• A/B testing: Compare model performance
• Risk mitigation: Critical production systems

Canary Deployment Strategies

1. Traffic Splitting

Route X% of traffic to new version.

2. User-based

Route specific users to new version.

3. Header-based

Route based on HTTP headers.

4. Gradual Rollout

Start with 5%, increase to 100% over time.

Implementation Options

Option 1: Kubernetes with Service Mesh (Istio)

Most flexible, supports advanced routing.

Option 2: Kubernetes with Multiple Deployments

Simple, uses native K8s features.

Option 3: Application-level Routing

Handle in application code.

Simple Canary with K8s

Step 1: Deploy Stable Version

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: llm-serving
      version: stable

Step 2: Deploy Canary Version

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-serving
      version: canary

Step 3: Service Selector

Service routes to both versions:

apiVersion: v1
kind: Service
metadata:
  name: llm-serving
spec:
  selector:
    app: llm-serving  # Matches both stable and canary

Traffic splits 90/10 based on replica count.

Istio Canary (Advanced)

VirtualService

Control traffic routing:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: llm-serving
spec:
  hosts:
  - llm-serving
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: llm-serving
        subset: canary
      weight: 100
  - route:
    - destination:
        host: llm-serving
        subset: stable
      weight: 90
    - destination:
        host: llm-serving
        subset: canary
      weight: 10

DestinationRule

Define subsets:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: llm-serving
spec:
  host: llm-serving
  subsets:
  - name: stable
    labels:
      version: stable
  - name: canary
    labels:
      version: canary

Gradual Rollout

Phase 1: 5% Traffic (Day 1)

• Deploy canary with 1 replica
• Monitor for 24 hours
• Check metrics: latency, errors, throughput

Phase 2: 25% Traffic (Day 2)

• Scale canary to 3 replicas
• Continue monitoring

Phase 3: 50% Traffic (Day 3)

• Scale canary to 5 replicas
• Monitor closely

Phase 4: 100% Traffic (Day 4)

• Scale canary to 10 replicas
• Remove stable deployment
• Canary becomes new stable

Monitoring Canary

Key Metrics to Compare

• Latency: P50, P95, P99
• Error rate: Failed requests
• Throughput: Requests per second
• GPU utilization: Resource usage
• Model-specific: Tokens/second, quality metrics

Prometheus Queries

# Compare latency
histogram_quantile(0.95, 
  rate(llm_request_duration_seconds_bucket{version="canary"}[5m])
) / 
histogram_quantile(0.95, 
  rate(llm_request_duration_seconds_bucket{version="stable"}[5m])
)

# Compare error rates
rate(llm_requests_total{version="canary",status="500"}[5m]) /
rate(llm_requests_total{version="canary"}[5m])

Rollback Procedure

Automatic Rollback

Set up alerts that trigger rollback:

• Error rate > threshold
• Latency > threshold
• Custom metrics fail

Manual Rollback

# Scale canary to 0
kubectl scale deployment llm-serving-canary --replicas=0

# Or route all traffic to stable
# Update VirtualService weights

A/B Testing

Compare Models

Deploy two different models:

• Model A: Current production
• Model B: New candidate

Metrics to Track

• Performance: Latency, throughput
• Quality: User feedback, accuracy
• Cost: GPU hours, inference cost

Decision Criteria

• Better performance AND quality → Promote
• Worse performance OR quality → Reject
• Mixed results → Continue testing

Best Practices

1. Start small: 5-10% traffic initially
2. Monitor closely: Watch metrics during rollout
3. Have rollback plan: Know how to revert quickly
4. Test thoroughly: Test canary before production
5. Document changes: Track what changed
6. Communicate: Inform team of canary deployment

Common Issues

Canary Performing Worse

• Check if it’s expected (new model might be slower)
• Compare metrics carefully
• Consider rollback if critical

Traffic Not Splitting Correctly

• Verify service selectors
• Check replica counts
• Verify routing rules (if using Istio)

Canary Not Getting Traffic

• Check service endpoints
• Verify labels match
• Check routing configuration

Exercises

1. Basic canary: Deploy canary with 10% traffic
2. Gradual rollout: Increase from 10% to 100% over phases
3. Monitoring: Set up dashboards to compare versions
4. Rollback: Practice rolling back canary
5. A/B test: Compare two different models