How to Start Learning LLM Serving & Inference

🎯 Your Learning Journey Starts Here

This guide will walk you through exactly how to start learning LLM serving and inference, step by step.

📋 Prerequisites Check

Before you start, make sure you have:

Python 3.9 or higher (python --version)
pip installed
Basic understanding of Python
Basic understanding of REST APIs
(Optional) Docker installed
(Optional) Kubernetes cluster (minikube/kind)

Don’t worry if you don’t have Docker/K8s yet - you can learn the basics without them!

🚀 Step-by-Step Learning Path

Step 1: Understand the Fundamentals (30-60 minutes)

Read this first: docs/llm_inference_fundamentals.md

This explains:

How LLMs work internally
What happens during inference
Tokenization, attention, generation
Memory and computation requirements

Why this matters: You need to understand what’s happening under the hood before you can serve models effectively.

Action: Open the file and read through it. Don’t worry if you don’t understand everything - you’ll learn more as you build.

Step 2: Set Up Your Environment (10 minutes)

# Navigate to the project
cd /Users/faisal/Projects/mlops_serving

# Create a virtual environment (recommended)
python -m venv venv

# Activate it
source venv/bin/activate  # On Mac/Linux
# OR
venv\Scripts\activate  # On Windows

# Install dependencies for basic serving
cd 01_basic_serving
pip install -r requirements.txt

What this does: Sets up an isolated Python environment with all the libraries you need.

Step 3: Run Your First LLM Server (5 minutes)

# Make sure you're in 01_basic_serving directory
cd 01_basic_serving

# Start the server
python app.py

You should see:

INFO:     Started server process
INFO:     Waiting for application startup.
🚀 Starting up: Loading LLM model...
Loading model: gpt2 on device: cpu
✅ Model loaded successfully!
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000

What’s happening:

The server is loading a GPT-2 model (small, fast model for learning)
It’s starting a FastAPI web server
The model is now ready to serve requests

Step 4: Test the API (5 minutes)

Option A: Use the test script

# In a new terminal (keep server running)
cd 01_basic_serving
python test_api.py

Option B: Use curl

# Health check
curl http://localhost:8000/health

# Generate text
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_length": 50,
    "temperature": 0.7
  }'

Option C: Use the interactive docs Open http://localhost:8000/docs in your browser - FastAPI provides an interactive API explorer!

Step 5: Understand What Just Happened (15 minutes)

Read: 01_basic_serving/README.md

This explains:

What each file does
How the model loading works
How the API endpoints work
Key concepts you just used

Then explore the code:

model_loader.py - How models are loaded and used
app.py - How the API is structured

Try modifying:

Change the model name (try “distilgpt2” for a smaller model)
Change temperature (0.1 = more deterministic, 2.0 = more creative)
Add logging to see what’s happening

Step 6: Learn Each Topic in Order

Now that you’ve run your first server, work through each topic:

01_basic_serving ✅ (You just did this!)
02_docker - Containerize your app
03_kubernetes - Deploy to K8s
04_load_testing - Test performance
05_vllm_serving - High-performance serving
06_autoscaling - Scale automatically
07_canary_deployments - Safe rollouts
08_monitoring - Track metrics
09_model_versioning - Manage versions
10_drift_detection - Detect issues
11_triton - Multi-model serving

For each topic:

Read the README.md
Study the code
Run the examples
Modify and experiment
Move to the next topic

🎓 Learning Tips

1. Don’t Rush

Understanding is more important than speed. Take time to:

Read error messages carefully
Experiment with parameters
Break things and fix them

2. Experiment

After running each example:

Change parameters
Modify the code
See what breaks
Understand why

3. Use the Documentation

Each topic has:

README.md explaining concepts
Code comments explaining “why”
Examples you can run

4. Ask Questions

As you learn, ask yourself:

“Why does this work this way?”
“What happens if I change X?”
“How does this scale?”
“What could go wrong?”

🐛 Common Issues & Solutions

Issue: “Model not found” or download errors

Solution: Check your internet connection. Models are downloaded from HuggingFace on first use.

Issue: “Out of memory”

Solution:

Use a smaller model (distilgpt2 instead of gpt2)
Reduce max_length
Close other applications

Issue: “Port already in use”

Solution:

Stop the previous server (Ctrl+C)
Or change the port in app.py

Issue: “Import errors”

Solution:

Make sure virtual environment is activated
Run pip install -r requirements.txt again

📊 What You’ll Learn

By the end of this journey, you’ll understand:

Core Concepts

✅ How LLM inference works internally
✅ Tokenization and vocabulary
✅ Attention mechanisms
✅ Memory and computation requirements

Serving Skills

✅ Building serving APIs
✅ Containerization
✅ Kubernetes deployment
✅ Performance optimization

Production Skills

✅ Monitoring and observability
✅ Scaling strategies
✅ Safe deployments
✅ Drift detection

🎯 Next Steps

Right now: Complete Steps 1-5 above
Today: Read through 01_basic_serving/README.md and understand the code
This week: Work through topics 2-4 (Docker, K8s, Load Testing)
This month: Complete topics 5-8 (vLLM, Scaling, Monitoring)
Ongoing: Topics 9-11 (Advanced production topics)

❓ Questions?

If you get stuck:

Check the README.md in each topic
Read error messages carefully
Check the docs/ directory for detailed explanations
Experiment with simpler examples first

Remember: Learning by doing is the best way. Don’t just read - run the code, modify it, break it, fix it!

Ready? Let’s start! 🚀

Begin with Step 1: Read docs/llm_inference_fundamentals.md