🎯 Key Takeaway
Fair benchmarking requires controlled conditions: same repository, same commit, same task, same configuration. Measure 8 dimensions — completion, correctness, tokens, latency, context accuracy, iterations, test pass rate, and human corrections. Never fabricate numbers. The framework provides methodology; you provide the real data.
Fair benchmarking requires controlled conditions: same repository, same commit, same task, same configuration. Measure 8 dimensions — completion, correctness, tokens, latency, context accuracy, iterations, test pass rate, and human corrections. Never fabricate numbers. The framework provides methodology; you provide the real data.
"Agent X completed the task in 10 seconds with 95% accuracy."
That statement is meaningless without context. Which repository? Which task? Which hardware? How was "accuracy" measured? Was the task even the same?
Unfair benchmarks mislead developers into choosing the wrong tool. Fair benchmarks require controlled conditions, clear metrics, and honest reporting. This tutorial gives you the methodology to benchmark AI coding agents reproducibly — with no fabricated numbers.
This tutorial connects to AI Coding Agents Evolution, AI Coding Agents & Junior Developers, and Using AI Agents Safely.
1. Why Fair Benchmarking Matters
Unfair benchmarks are everywhere. Here's why they mislead:
| Unfair Claim | What's Missing |
|---|---|
| "Completed in 10 seconds" | Which repo? Which task? Which hardware? |
| "95% accuracy" | How was accuracy defined? What test set? |
| "Agent X is the best" | Best at what? For which tasks? Under which conditions? |
| "Faster than the competition" | Same starting point? Same network? Same model? |
Fair benchmark principle: "On repository R, commit C, task T, with config K, Agent X completed in 45s using 3500 tokens, passing 12/12 tests with 0 human edits." Every number is traceable to a specific, reproducible condition.
2. The 8 Metrics
Every benchmark should measure all 8 dimensions. No single metric tells the full story.
| # | Metric | What It Measures | Unit |
|---|---|---|---|
| 1 | Task Completion | Did the agent finish the task? | completed / partial / failed / timeout |
| 2 | Correctness | Does the solution actually work? | tests passed / total |
| 3 | Token Efficiency | How much context was consumed? | tokens, tokens/step |
| 4 | Latency | How long did it take? | wall_time_seconds |
| 5 | Context Accuracy | Did it read the right files? | precision, recall |
| 6 | Iterations | How many steps to complete? | steps, tool_calls |
| 7 | Test Pass Rate | Did tests pass before and after? | %, regressions |
| 8 | Human Corrections | How much did a human fix? | edits, lines, review_time |
3. Reproducibility: The 5 Controls
For results to be comparable, you must control these 5 variables:
# Benchmark Configuration (must be fixed)
config = {
"repository": "https://github.com/org/project",
"starting_commit": "a1b2c3d4e5f6", # ← Pin this!
"task_description": "Add email validation",
"model": "gpt-4", # ← Or your target model
"temperature": 0.0, # ← Deterministic for reproducibility
"timeout": 300,
"hardware": "M2 MacBook Pro 16GB",
}
config = {
"repository": "https://github.com/org/project",
"starting_commit": "a1b2c3d4e5f6", # ← Pin this!
"task_description": "Add email validation",
"model": "gpt-4", # ← Or your target model
"temperature": 0.0, # ← Deterministic for reproducibility
"timeout": 300,
"hardware": "M2 MacBook Pro 16GB",
}
| Control | Why It Matters |
|---|---|
| Same repository | Different repos have different complexity |
| Same starting commit | Code changes between commits |
| Same task description | Wording affects agent behavior |
| Same model | Different models have different capabilities |
| Same hardware | Affects local model latency |
4. Metric Deep-Dives
Metric 1: Task Completion
The most basic question: did the agent finish?
completed — Task fully addressed
partial — Some work done, needs human completion
failed — Agent couldn't complete
timeout — Exceeded time limit
partial — Some work done, needs human completion
failed — Agent couldn't complete
timeout — Exceeded time limit
Metric 2: Correctness
Completion alone is insufficient. The solution must actually work.
# Run the test suite
pytest --tb=short
# 12 passed, 0 failed ← 100% correctness
# 10 passed, 2 failed ← 83% correctness
pytest --tb=short
# 12 passed, 0 failed ← 100% correctness
# 10 passed, 2 failed ← 83% correctness
Metric 5: Context Accuracy
Did the agent read the right files, or waste tokens on irrelevant ones?
# Precision: of files read, how many were relevant?
precision = relevant_read / total_read
# Agent read 5 files, 2 were relevant → 40%
# Recall: of relevant files, how many were read?
recall = relevant_read / total_relevant
# 3 relevant files exist, agent found 2 → 67%
precision = relevant_read / total_read
# Agent read 5 files, 2 were relevant → 40%
# Recall: of relevant files, how many were read?
recall = relevant_read / total_relevant
# 3 relevant files exist, agent found 2 → 67%
Metric 8: Human Corrections
The most overlooked metric. Even a "successful" agent may require human fixes.
Why this matters: If the agent completes the task but a human spends 30 minutes fixing its output, the "efficiency" claim is misleading. Always measure the human overhead.
5. Task Design: Creating Fair Tasks
The task itself must be well-defined and verifiable.
| Good Task | Bad Task |
|---|---|
| "Add email validation to login.py. Test with valid/invalid emails." | "Make it better" |
| "Fix the bug in calculate_total where empty list returns 0 instead of raising ValueError" | "Fix the bug" |
| "Refactor the database module to use connection pooling. All existing tests must pass." | "Refactor everything" |
| "Add a /health endpoint that returns 200 OK with JSON status" | "Add an endpoint" |
A good benchmark task has:
- Specific deliverable — what exactly should be built/changed
- Verifiable outcome — tests that prove correctness
- Clear scope — which files/modules are involved
- Expected behavior — what the solution should do
- Difficulty label — easy / medium / hard
6. The Benchmark Workflow
Clone
→
Checkout
→
Run Agent
→
Collect
→
Test
→
Score
→
Compare
→
Report
- Clone the repository to a clean directory
- Checkout the pinned commit hash
- Run the agent with the task description
- Collect all metrics (tokens, time, files, steps)
- Test — run the test suite and compare before/after
- Score — compute all 8 metrics
- Compare — run the same task with different agents
- Report — publish results with full configuration details
7. Comparison Table Template
When comparing agents, use this format — every number must be traceable:
| Metric | Agent A | Agent B | Agent C |
|---|---|---|---|
| Completion | completed | completed | partial |
| Tests Passed | 12/12 | 10/12 | 8/12 |
| Tokens | 3,500 | 5,200 | 2,800 |
| Latency | 45s | 90s | 30s |
| Iterations | 4 | 8 | 3 |
| Human Edits | 0 | 8 lines | 20 lines |
| Regression | ✅ | ✅ | ❌ 1 broken |
Important: The numbers above are templates — not real benchmark results. Real benchmarks must be run on actual agents with actual repositories. Never fill in fabricated numbers.
8. Common Benchmarking Mistakes
| Mistake | Impact | Fix |
|---|---|---|
| Different tasks per agent | Comparison is meaningless | Same exact task description |
| Different starting commits | Different codebases | Pin the commit hash |
| Only measuring completion | Ignores quality and cost | Measure all 8 metrics |
| Not measuring human corrections | Overstates agent capability | Track human edit count |
| Running only 1 task | Statistical noise | Run 10-20+ tasks minimum |
| Not reporting configuration | Results can't be reproduced | Publish full config |
9. FAQ
Continue Learning
AI Coding Agents: Evolution
From autocomplete to autonomous agents AI Coding Agents & Junior Developers
Productivity vs programming skill Using AI Agents Safely
Pre-flight checklist and defensive practices How AI Coding Agents Work
The agent loop: plan, execute, observe RAG Evaluation Metrics
Precision, recall, faithfulness for RAG systems GitHub Actions CI/CD
Automate benchmark runs in CI
From autocomplete to autonomous agents AI Coding Agents & Junior Developers
Productivity vs programming skill Using AI Agents Safely
Pre-flight checklist and defensive practices How AI Coding Agents Work
The agent loop: plan, execute, observe RAG Evaluation Metrics
Precision, recall, faithfulness for RAG systems GitHub Actions CI/CD
Automate benchmark runs in CI