Topic 14 · Production systems
Observability & performance
Seeing where the time, tokens and money actually go.
An agent run is a tree of model calls, tool calls and waits, spread across services. Distributed tracing records that tree as timed spans, so a slow answer can be pinned on a long prefill, a sluggish tool or a queue that backed up.
Averages hide most of the story. Users feel the slow tail, agents multiply it across every step, and GPU dashboards can say "busy" while the hardware does little useful work. Good observability measures percentiles, tokens and task outcomes, not just requests.
In this topic
33
terms, each defined below
One agent run, traced
A trace waterfall of a single agent run: each bar is a span, nested under its parent, positioned by when it started. Hover or tap a span for details. Below it, the latency of 400 simulated runs shows why p95 and p99 matter more than the mean.
tool.search
1.02 s
starts at 1.26 s · 24% of the run
The slowest single step: an external search API. A cache or timeout would help most here.
End-to-end latency of 400 simulated runs
- p50
- 3.9 s
- mean
- 4.5 s
- p95
- 7.7 s
- p99
- 17.0 s
The mean looks fine, but one run in twenty takes longer than the p95 line. If a task chains five such steps, almost a quarter of tasks (1 − 0.95⁵ ≈ 23%) hit at least one of them.
Glossary
33 terms
Observability
Being able to understand what a system is doing inside from what it emits: traces, logs and metrics.
Distributed tracing
Following one request across services as a tree of timed spans, showing where its time went.
Execution traces
Traces of an agent run: every model call, tool call and decision as a span with inputs, outputs and timing.
Trace correlation
Linking logs, metrics and spans through shared trace and span IDs, so one request can be followed everywhere.
Structured logging
Logging machine-readable records, usually JSON with consistent fields, instead of free-form text.
Metrics
Numeric time series such as counters, gauges and histograms that show how a system behaves over time.
GPU telemetry
Hardware metrics from GPUs, including utilisation, memory, temperature, power, clocks, NVLink traffic and errors, usually gathered through DCGM.
Performance profiling
Measuring where time and resources are spent in code, to find the bottlenecks.
Inference profiling
Profiling a serving stack specifically: prefill against decode time, kernel time, scheduler overhead and batch composition.
Latency
How long an operation takes. For LLMs, time to first token, time per output token and end-to-end time should be tracked separately.
End-to-end latency
The total time from a user's action to the complete result, including queues, every model call and every tool.
p95 latency
The value that 95 percent of requests come in under. Percentiles reveal what averages hide.
Tail latency
High-percentile latency (p99, p99.9). In multi-step agents one slow step slows the whole run, so tails compound.
Throughput
Work completed per unit of time: tokens per second, requests per second or tasks per hour.
Tokens per second
Generation speed, either per stream (what one user sees) or summed across streams (what the server delivers).
Requests per second
The rate of requests a system handles. For LLMs it means little without the token counts behind it.
Task completion rate
The share of agent tasks that finish successfully, often the metric users care about most.
Time to first token
TTFT as an observed metric, from request arrival to the first streamed token, ideally broken down into queueing and prefill.
Scheduling delay
The time between a request being ready and actually being scheduled, in the inference engine or the orchestration layer.
Queue depth
The number of items waiting in a queue, a leading indicator of overload.
GPU memory usage
Memory consumed on each GPU by weights, KV cache, activations and fragmentation.
GPU utilization
The share of time kernels are running on the GPU. Useful but coarse; pair it with multiprocessor occupancy and tensor-core activity.
KV-cache utilization
The share of KV-cache blocks in use. Close to 100 percent means preemption and queueing are about to start.
Cost tracking
Attributing token, GPU and tool costs to users, features and runs.
Token usage
Input, cached and output tokens per call, the main driver of LLM cost.
Load testing
Driving synthetic traffic at a system to measure its capacity and find its breaking points, for example with k6.
Benchmarking
Measuring performance or quality under fixed, repeatable conditions to compare systems or versions.
Performance regressions
Changes that make a system slower or more expensive than before, ideally caught by benchmarks in CI.
Bottleneck analysis
Finding the one resource that limits throughput, whether GPU compute, memory bandwidth, a lock or a queue.
Capacity planning
Forecasting demand and provisioning enough resources ahead of time, using load tests and growth data.
Agent evaluations
Systematic tests of agent behaviour on representative tasks, scored by checks, rubrics or model graders.
Behavioral correctness
Whether an agent does the right thing: the right tools in the right order, the right outcome, and its constraints respected.
Regression testing
Re-running a fixed test suite after every change to catch behaviour that broke, including evaluations after prompt or model updates.