You deploy an AI coding agent. It runs overnight. In the morning, you see: "Task completed successfully."
But what did it actually do? How many model calls did it make? Which tools did it use? Did any fail and recover? How much did it cost? How long did each step take?
"Task completed" tells you nothing. Agent observability tells you everything.
Why Traditional Logs Are Not Enough
2026-08-29 10:00:01 INFO Agent completed task
→ Missing: tokens, cost, tool calls, errors, latency breakdown
Agent Trace:
trace_id: a1b2c3d4
goal: "Fix authentication bug"
spans: 6
model_calls: 4 (tokens: 1630, cost: $0.0049)
tool_calls: 5 (search: 1, read: 1, edit: 2, test: 2)
errors: 0
duration: 652ms
→ Complete visibility into what happened and why
Traditional application logs record events as flat text lines. They work for request/response systems where each request is independent. But agents are stateful, iterative, and multi-step. A single task might involve 10 model calls, 5 tool executions, 3 state changes, and 2 error recoveries. A flat log cannot capture this.
| Aspect | Traditional Logging | Agent Observability |
|---|---|---|
| Structure | Flat text lines | Hierarchical traces with spans |
| Cause/effect | Not tracked | Parent-child relationships |
| Token tracking | Not tracked | Per-call and aggregate |
| Cost tracking | Not tracked | Per-trace and per-session |
| Tool calls | Maybe logged | Full args, results, latency |
| Error context | Error message only | Full trace of what led to error |
| State | Not tracked | Snapshots at each step |
The Agent Observability Architecture
Agent ──→ Trace Collector ──→ Metrics Aggregator ──→ Dashboard
│ │
├─ model_call ──→ tokens, latency, cost ├─ token_usage
├─ tool_call ───→ tool, args, result, latency ├─ cost_per_task
├─ state_change → before, after ├─ error_rate
└─ error ───────→ error_type, context └─ latency_p95
The 8 Components of Agent Observability
1. Traces
A trace is a complete record of one agent task — from goal to completion. It contains ordered spans, each representing one step in the agent loop.
2. Tool Calls
Each tool call is recorded with its arguments, result, latency, and success status. This is the most granular level of observability.
| Field | Example |
|---|---|
| tool | file_search |
| args | {"pattern": "*.py"} |
| result | "Found 5 Python files" |
| latency_ms | 45 |
| success | true |
3. Tokens
Track tokens consumed per model call and in aggregate. This directly affects cost and latency.
4. Latency
Measure latency at multiple levels: per model call, per tool call, and per trace. Report percentiles, not just averages.
| Level | What to Measure | Typical Range |
|---|---|---|
| Model call | LLM inference time | 100-2000ms |
| Tool call | Tool execution time | 5-5000ms |
| Trace | Total task duration | 1-60s |
5. Errors
Record every error with full context: what tool was called, what arguments were used, what the error was, and what happened next.
6. State
Snapshots of agent state at key points: what file is being edited, what the current plan is, what tests have run. This helps reconstruct the agent's reasoning.
7. Cost
Calculate cost per trace based on token usage and pricing. Aggregate across sessions for budget tracking.
8. Model Calls
Count and characterize each model call: what was it for (planning, reasoning, analysis), how many tokens did it consume, what was the latency.
Building an Observability Collector
The Python demo implements a complete observability collector. Here is the core pattern:
demo.py and run python demo.py to see the full observability pipeline in action. Run python demo.py --test to verify all 15 test cases.
The Dashboard View
Aggregated metrics across all agent traces give you a production-level view:
┌──────────────────┬──────────────────┬──────────────────┐
│ Token Usage │ Cost │ Tool Calls │
│ 1,630 per trace │ $0.0049 per task │ 5 per trace avg │
│ 4 model calls │ $0.0147/session │ search, read, │
│ │ │ edit, test │
├──────────────────┼──────────────────┼──────────────────┤
│ Errors │ Latency │ Success Rate │
│ 0 per trace │ 652ms avg │ 95.2% │
│ 1 recovery │ 2.3s p95 │ 4.8% retry │
└──────────────────┴──────────────────┴──────────────────┘
What to Alert On
| Metric | Alert Threshold | Why |
|---|---|---|
| Error rate | > 10% of tool calls | Agent is failing repeatedly |
| Cost per trace | > 3× average | Runaway token usage |
| Trace duration | > 60 seconds | Agent may be stuck in loop |
| Token usage | > 10,000 per trace | Context overflow risk |
| Model calls | > 20 per trace | Agent not converging |
| Retry count | > 3 per tool | Persistent failure pattern |
Common Observability Mistakes
| Mistake | Problem | Fix |
|---|---|---|
| Logging only completion | No visibility into steps | Record every span in the trace |
| Not tracking tokens | Cannot estimate cost | Count tokens per model call |
| Averaging latency | Hides tail latency | Report p50, p95, p99 |
| Ignoring errors | Silent failures in production | Log every error with context |
| No cost tracking | Budget overruns | Calculate cost per trace |
| Flat logs only | Cannot reconstruct agent reasoning | Use structured traces with spans |
Practical Exercises
Exercise 1: Trace a Task
Run the demo and examine the trace for "Fix authentication bug." How many spans does it have? What is the most expensive step (by tokens)? What is the slowest step (by latency)?
Exercise 2: Design a Dashboard
If you were building a dashboard for agent observability, what 5 charts would you include? What time ranges would you use? What alerts would you set?
Exercise 3: Cost Analysis
If each agent task costs $0.005 and you run 1,000 tasks per day, what is the daily cost? Monthly? At what point would you need to optimize token usage?
Exercise 4: Error Investigation
Run the demo and examine the failed task trace. The agent encountered an error and recovered. How would you use the trace to understand what happened and whether the recovery was appropriate?
✅ Agent Observability Checklist
- ☐ Every agent task produces a trace with unique ID
- ☐ Every model call records tokens, latency, and cost
- ☐ Every tool call records args, result, latency, and success
- ☐ Errors are logged with full context (not just error message)
- ☐ State snapshots at key decision points
- ☐ Cost calculated per trace and per session
- ☐ Latency reported as p50/p95/p99, not just average
- ☐ Metrics aggregated for dashboard view
- ☐ Alerts configured for error rate, cost, and duration
- ☐ Traces retained for debugging (with privacy protections)
FAQ
Q: How much overhead does observability add?
A: Minimal. Recording a trace adds microseconds per step. The dominant cost is storage for trace retention, which can be managed with sampling and retention policies.
Q: Should I trace every agent task?
A: In development, yes. In production, use sampling (e.g., 10-20% of traces) for cost control, but trace 100% of tasks that encounter errors or exceed latency thresholds.
Q: What is the difference between observability and monitoring?
A: Monitoring tells you something is wrong. Observability tells you why it is wrong. Monitoring is dashboards and alerts. Observability is traces, logs, and metrics that let you reconstruct what happened.
Q: Can I use OpenTelemetry for agent traces?
A: Yes. OpenTelemetry is the standard for distributed tracing. Its span model maps naturally to agent steps. See the LLM Observability article for OpenTelemetry examples.
Q: How do I handle privacy in agent traces?
A: Traces may contain user data, code snippets, and file contents. Apply data classification, redact PII, set retention limits, and restrict access. Never log API keys or secrets.
Further Reading
- LLM Observability: Monitoring and Debugging AI Systems in Production
- AI Audit Trails Explained: What Should Developers Log?
- How AI Agent Loops Work
- AI Agent Memory Explained
- Multi-Agent AI Systems Explained
- How to Use AI Coding Agents Safely
Continue Learning: Explore LLM observability, understand audit trails, learn how agent loops work, and see multi-agent observability.
Discuss this topic on BestWordz Community.