# 14. Observability & performance

> Seeing where the time, tokens and money actually go.

Track: Production systems. 33 terms.

An agent run is a tree of model calls, tool calls and waits, spread across services. Distributed tracing records that tree as timed spans, so a slow answer can be pinned on a long prefill, a sluggish tool or a queue that backed up.

Averages hide most of the story. Users feel the slow tail, agents multiply it across every step, and GPU dashboards can say "busy" while the hardware does little useful work. Good observability measures percentiles, tokens and task outcomes, not just requests.

**Interactive figure on the web page: One agent run, traced.** A trace waterfall of a single agent run: each bar is a span, nested under its parent, positioned by when it started. Hover or tap a span for details. Below it, the latency of 400 simulated runs shows why p95 and p99 matter more than the mean.

## Terms

### Observability

Being able to understand what a system is doing inside from what it emits: traces, logs and metrics.

### Distributed tracing

Following one request across services as a tree of timed spans, showing where its time went.

### Execution traces

Traces of an agent run: every model call, tool call and decision as a span with inputs, outputs and timing.

### Trace correlation

Linking logs, metrics and spans through shared trace and span IDs, so one request can be followed everywhere.

### Structured logging

Logging machine-readable records, usually JSON with consistent fields, instead of free-form text.

### Metrics

Numeric time series such as counters, gauges and histograms that show how a system behaves over time.

### GPU telemetry

Hardware metrics from GPUs, including utilisation, memory, temperature, power, clocks, NVLink traffic and errors, usually gathered through DCGM.

### Performance profiling

Measuring where time and resources are spent in code, to find the bottlenecks.

### Inference profiling

Profiling a serving stack specifically: prefill against decode time, kernel time, scheduler overhead and batch composition.

See: [related](https://llm-atlas-7or.pages.dev/learn/inference/)

### Latency

How long an operation takes. For LLMs, time to first token, time per output token and end-to-end time should be tracked separately.

### End-to-end latency

The total time from a user's action to the complete result, including queues, every model call and every tool.

### p95 latency

The value that 95 percent of requests come in under. Percentiles reveal what averages hide.

### Tail latency

High-percentile latency (p99, p99.9). In multi-step agents one slow step slows the whole run, so tails compound.

See: [also in Batching & scheduling](https://llm-atlas-7or.pages.dev/learn/batching.md)

### Throughput

Work completed per unit of time: tokens per second, requests per second or tasks per hour.

### Tokens per second

Generation speed, either per stream (what one user sees) or summed across streams (what the server delivers).

### Requests per second

The rate of requests a system handles. For LLMs it means little without the token counts behind it.

### Task completion rate

The share of agent tasks that finish successfully, often the metric users care about most.

### Time to first token

TTFT as an observed metric, from request arrival to the first streamed token, ideally broken down into queueing and prefill.

See: [related](https://llm-atlas-7or.pages.dev/learn/inference/#time-to-first-token-ttft)

### Scheduling delay

The time between a request being ready and actually being scheduled, in the inference engine or the orchestration layer.

### Queue depth

The number of items waiting in a queue, a leading indicator of overload.

### GPU memory usage

Memory consumed on each GPU by weights, KV cache, activations and fragmentation.

### GPU utilization

The share of time kernels are running on the GPU. Useful but coarse; pair it with multiprocessor occupancy and tensor-core activity.

See: [also in Batching & scheduling](https://llm-atlas-7or.pages.dev/learn/batching.md)

### KV-cache utilization

The share of KV-cache blocks in use. Close to 100 percent means preemption and queueing are about to start.

### Cost tracking

Attributing token, GPU and tool costs to users, features and runs.

### Token usage

Input, cached and output tokens per call, the main driver of LLM cost.

### Load testing

Driving synthetic traffic at a system to measure its capacity and find its breaking points, for example with k6.

### Benchmarking

Measuring performance or quality under fixed, repeatable conditions to compare systems or versions.

### Performance regressions

Changes that make a system slower or more expensive than before, ideally caught by benchmarks in CI.

### Bottleneck analysis

Finding the one resource that limits throughput, whether GPU compute, memory bandwidth, a lock or a queue.

### Capacity planning

Forecasting demand and provisioning enough resources ahead of time, using load tests and growth data.

### Agent evaluations

Systematic tests of agent behaviour on representative tasks, scored by checks, rubrics or model graders.

### Behavioral correctness

Whether an agent does the right thing: the right tools in the right order, the right outcome, and its constraints respected.

### Regression testing

Re-running a fixed test suite after every change to catch behaviour that broke, including evaluations after prompt or model updates.

Web page: https://llm-atlas-7or.pages.dev/learn/observability/

_Definitions written for LLM Atlas._