Topic 3: Metrics and Benchmarks

What You’ll Learn

This topic teaches you how to:

• Measure agent performance with metrics
• Use standard benchmarks (AgentBench, WebArena)
• Calculate success rates, efficiency, cost
• Compare agents using metrics
• Track performance over time

Why We Need This

Business Need

• Performance tracking: Know if agents are improving
• Cost optimization: Measure cost per task
• SLA compliance: Meet performance requirements
• Competitive analysis: Compare with other agents

Technical Need

• Quantitative evaluation: Need numbers, not just “works/doesn’t work”
• Comparability: Compare agents objectively
• Tracking: Monitor performance over time
• Optimization: Identify what to improve

Industry Use Cases

1. Performance Monitoring

Company: All agent platforms Use Case: Track agent performance metrics over time

2. Benchmarking

Company: Research labs, companies Use Case: Compare agents on standard benchmarks

3. Cost Analysis

Company: Cost-sensitive deployments Use Case: Measure cost per successful task

Industry-Standard Boilerplate Code

Metrics Calculator

"""
Metrics Calculator
Industry standard metrics for agent evaluation
"""
from typing import List, Dict
from dataclasses import dataclass

@dataclass
class AgentMetrics:
    """Comprehensive agent metrics"""
    success_rate: float
    avg_tokens: float
    avg_time: float
    cost_per_task: float
    tasks_completed: int
    tasks_failed: int

class MetricsCalculator:
    """Calculate standard agent metrics"""
    
    @staticmethod
    def calculate_metrics(results: List[Dict]) -> AgentMetrics:
        """Calculate metrics from evaluation results"""
        total = len(results)
        successful = sum(1 for r in results if r['success'])
        
        return AgentMetrics(
            success_rate=successful / total if total > 0 else 0,
            avg_tokens=sum(r.get('tokens', 0) for r in results) / total if total > 0 else 0,
            avg_time=sum(r.get('time', 0) for r in results) / total if total > 0 else 0,
            cost_per_task=sum(r.get('cost', 0) for r in results) / total if total > 0 else 0,
            tasks_completed=successful,
            tasks_failed=total - successful
        )

Exercises

1. Calculate success rate
2. Measure efficiency metrics
3. Track cost metrics
4. Compare agents using metrics

Next Steps

• Topic 4: Evaluate tool usage
• Topic 5: Evaluate reasoning