489 terms · 18 topics

Glossary

Every term in the field guide, alphabetically. Select a term's topic to see it in context, next to its interactive figure.

489 terms

A

Accelerator clusters8. GPU & CUDA engineering →: Many accelerators networked together, thousands of GPUs or TPUs, for training and large-scale inference.
Accelerator placement18. Advanced systems research questions →: Matching workloads to accelerator types, such as H100s, consumer GPUs or TPUs, by performance per unit of cost.
Accelerator-aware scheduling6. Distributed LLM inference →: Placing workloads according to accelerator type, memory and interconnect, for example putting a large model only on nodes with eight NVLinked GPUs.
Activation patterns16. Weight-space programming & interpretability →: Which neurons or directions fire for which inputs. Recording them over many prompts reveals features.
Active parameters7. Mixture of Experts (MoE) →: The parameters actually used for one token: attention, shared parts and the chosen experts. They drive compute per token and are the "A22B" in names like 235B-A22B.
Active sequences5. Batching & scheduling →: Sequences in the running batch with their KV cache resident on the GPU. Their number is bounded by memory and the engine's maximum batch setting.
Adam optimizer15. Model training & post-training →: An optimiser that keeps running averages of each parameter's gradient and squared gradient, giving every parameter its own adaptive step size. AdamW, with decoupled weight decay, is the standard for LLMs.
Admission control6. Distributed LLM inference →: Rejecting or delaying requests at the edge when the system is saturated, so the requests that are accepted still meet their latency targets.
AG-UI protocol10. Tool calling & protocols →: The Agent-User Interaction protocol: an open, event-based standard started by CopilotKit for streaming agent events, such as messages, tool calls and state changes, to front-end apps.
Agent architecture9. Agent architecture & internals →: How an agent is put together: its model, instructions, tools, memory, control loop and any sub-agents or workflow graph around it.
Agent capabilities9. Agent architecture & internals →: What an agent can actually do. It is set by the tools and permissions it is given, not by the model alone.
Agent concurrency optimization18. Advanced systems research questions →: Running independent agent steps and tool calls in parallel, and tuning how many run at once to maximise throughput without overload.
Agent configuration9. Agent architecture & internals →: The settings that define an agent: model, sampling settings, instructions, enabled tools, limits and memory options.
Agent control loop9. Agent architecture & internals →: The central cycle: assemble context, call the model, run any requested tools and append their results, then repeat until the model answers or a limit is reached.
Agent coordination9. Agent architecture & internals →: The mechanisms that keep several agents consistent: shared state, locks, turn-taking or a supervising agent.
Agent delegation9. Agent architecture & internals →: Handing a task to another agent, including deciding what context to pass along and what result to expect back.
Agent evaluations14. Observability & performance →: Systematic tests of agent behaviour on representative tasks, scored by checks, rubrics or model graders.
Agent execution profiling18. Advanced systems research questions →: Profiling whole agent runs to see how time splits between model calls, tools, queueing and overhead.
Agent harness9. Agent architecture & internals →: The scaffolding that turns a model into an agent: prompt assembly, tool definitions, output parsing, the loop and its stop rules. The same model can perform very differently in different harnesses.
Agent identity9. Agent architecture & internals →: Who the agent is to other systems: its credentials, permissions and audit identity, kept separate from the user it acts for.
Agent instructions9. Agent architecture & internals →: The system prompt and guidelines that set the agent's role, goals, constraints and rules for using tools.
Agent lifecycle9. Agent architecture & internals →: The stages of one agent instance: created, configured, running, paused or waiting for input, completed or failed, and cleaned up.
Agent orchestration9. Agent architecture & internals →: Coordinating agents, tools and steps: what runs when, in which order or in parallel, and how results flow between them.
Agent runtime9. Agent architecture & internals →: The software that executes agents: it runs the loop, calls the model, dispatches tools, saves state and enforces limits.
Agent sessions9. Agent architecture & internals →: A bounded run or conversation with its own message history, state and resources.
Agent state9. Agent architecture & internals →: Everything needed to continue an agent from where it is: messages, intermediate results, its plan, counters and pending tool calls.
Agent state machines9. Agent architecture & internals →: Modelling an agent as explicit states and transitions, such as planning, acting, awaiting approval and done, which makes behaviour predictable and recovery easier.
Agent systems engineering9. Agent architecture & internals →: The discipline of making agents dependable in production: runtimes, state, tools, evaluation, security and cost control.
Agent termination9. Agent architecture & internals →: Ending the loop: the model gives a final answer, a step, cost or time limit is hit, an error cannot be recovered, or a person stops it.
Agent workload characterization18. Advanced systems research questions →: Measuring what agent workloads really look like: number of calls, prompt reuse, tool latency, context growth and branching.
Agent-aware inference serving18. Advanced systems research questions →: Inference engines that understand agent workloads: keeping an agent's KV cache warm between steps, prioritising calls on a task's critical path, anticipating the next call.
Agent-runtime operating-system analogy18. Advanced systems research questions →: Viewing the agent runtime as an operating system: the model as the CPU, the context window as RAM, tools as system calls, the scheduler deciding which agent runs, and context compaction as paging.
Agent-to-agent communication9. Agent architecture & internals →: Messages exchanged between agents, directly or through shared memory, increasingly over standard protocols such as A2A.
Agent-to-application communication10. Tool calling & protocols →: How an agent talks to the app around it: streamed tokens, tool-call events, state updates and requests for user input.
Agent-to-tool communication10. Tool calling & protocols →: How an agent reaches its tools: direct function calls, HTTP APIs or protocols such as MCP.
Agentic systems9. Agent architecture & internals →: Software in which LLM decisions steer part of the control flow, from a single tool-calling loop to large multi-agent workflows.
AI agents9. Agent architecture & internals →: Systems where an LLM decides which actions to take, calls tools, looks at the results and repeats until a goal is reached, instead of answering once.
All-to-all communication7. Mixture of Experts (MoE) →: A collective operation where every GPU sends a different piece of data to every other GPU. Used to dispatch tokens to experts and bring the results back.
API gateway12. Backend & distributed systems →: The front door for client requests, handling authentication, rate limits, routing and request shaping.
Argmax3. Decoding & sampling →: The operation that returns the position of the largest value. Greedy decoding is an argmax over the logits at every step.
Argument validation10. Tool calling & protocols →: Checking parsed arguments against the schema and business rules before running the tool, and returning a useful error to the model when they fail.
Arithmetic intensity5. Batching & scheduling →: Floating-point operations done per byte read from memory. Batch-1 decode does about one per byte; batching raises it roughly in proportion to batch size until the hardware becomes compute-bound.
ASP.NET Core17. Infrastructure & development tools →: Microsoft's cross-platform, high-performance framework for building web APIs and services in C#.
Asynchronous messaging12. Backend & distributed systems →: Sending a message without waiting for the receiver to process it, so services stay decoupled in time.
At-least-once execution12. Backend & distributed systems →: A guarantee that an operation will run, possibly more than once. Safe only for idempotent operations.
Attention computation4. LLM inference internals →: The QKᵀ, softmax and weighted-sum work inside each attention layer. It grows with context length; fused kernels like FlashAttention compute it without ever storing the full score matrix.
Attention layers2. Transformer architecture →: The attention sublayer of each block: Q, K and V projections, the attention computation and the output projection. Variants such as GQA, MLA and sliding-window attention change this layer to save memory or compute.
Attention masks2. Transformer architecture →: A matrix added to the attention scores that blocks certain pairs, usually by setting them to minus infinity before the softmax. Used for causality, padding and packing several documents into one sequence.
Audit logging13. Reliability, security & sandboxing →: Recording who did what, when and with which inputs, especially for actions with side effects.
Audit trails13. Reliability, security & sandboxing →: The tamper-evident sequence of audit records that lets an incident be reconstructed afterwards.
Authentication13. Reliability, security & sandboxing →: Verifying who or what is making a request, for users and for agents alike.
Authorization13. Reliability, security & sandboxing →: Deciding whether an authenticated identity may perform a given action on a given resource.
Automated neural network engineering16. Weight-space programming & interpretability →: Using tools, often AI agents, to inspect, modify and test networks automatically rather than by hand.
Autonomous task execution9. Agent architecture & internals →: An agent carrying a task through many steps without human input, inside guardrails and limits.
Autoregressive generation1. LLM fundamentals →: Producing text one token at a time, where each new token is appended to the input and fed back in to predict the next. It is why generation is sequential and why output length drives latency.
Autoscaling6. Distributed LLM inference →: Adding or removing replicas as load changes. Loading large weights takes minutes, so predictive scaling and warm pools matter.
AWS Trainium8. GPU & CUDA engineering →: Amazon's own AI accelerator chips for training and inference on AWS, programmed through the Neuron SDK.

B

Backend engineering12. Backend & distributed systems →: Building the server side of an AI product: APIs, data, queues, workers and integrations with model providers.
Backpressure12. Backend & distributed systems →: Signalling upstream to slow down when downstream is overloaded, instead of letting queues grow without limit.
Backpropagation15. Model training & post-training →: The algorithm that computes the gradient for every parameter efficiently, by applying the chain rule backwards through the network.
Batch scheduling5. Batching & scheduling →: Deciding at every step which requests run, how many prompt tokens to mix in with decode tokens, and who waits.
Batch size5. Batching & scheduling →: How many sequences, or tokens, are processed together in one step. Bigger batches raise throughput until compute saturates, but slow each request and need more KV-cache memory.
Batched forward pass5. Batching & scheduling →: One forward pass over the whole batch, with tokens from different requests stacked so each weight matrix is multiplied once for all of them.
Batching5. Batching & scheduling →: Processing several requests in the same forward pass. Weights are read from memory once and reused for every request in the batch, the main lever for serving throughput.
Behavioral correctness14. Observability & performance →: Whether an agent does the right thing: the right tools in the right order, the right outcome, and its constraints respected.
Behavioral directions16. Weight-space programming & interpretability →: Directions in activation space that line up with a behaviour, such as refusal, sentiment or truthfulness. Adding or removing them steers the model.
Benchmark evaluation15. Model training & post-training →: Scoring a model on standard benchmarks such as MMLU, GPQA, SWE-bench and MATH-500. Comparable across models, but contamination and overfitting limit what a score means.
Benchmarking14. Observability & performance →: Measuring performance or quality under fixed, repeatable conditions to compare systems or versions.
Bottleneck analysis14. Observability & performance →: Finding the one resource that limits throughput, whether GPU compute, memory bandwidth, a lock or a queue.
Browser isolation13. Reliability, security & sandboxing →: Running agent web browsing in a separate, disposable browser so pages cannot reach the user's real sessions or the host machine.
Bursty agent workloads18. Advanced systems research questions →: Agents produce sudden bursts of model and tool calls, from fan-out and parallel sub-agents, unlike the steadier traffic of chat.

C

C# / .NET17. Infrastructure & development tools →: Microsoft's statically typed language and the runtime and libraries around it, a common choice for enterprise backends.
Cache eviction4. LLM inference internals →: Removing cache blocks when memory runs out, usually the least recently used prefixes first. An evicted prefix has to be recomputed if it is needed again.
Cache preemption4. LLM inference internals →: Pausing a running sequence and freeing its cache, either by swapping it to CPU memory or by dropping it and recomputing later, so other requests can proceed when memory is full.
Cache-aware routing6. Distributed LLM inference →: Sending a request to the replica that already holds its prefix in KV cache, so it skips recomputing that prefill.
Cancellation13. Reliability, security & sandboxing →: Stopping in-flight work cleanly when a user or parent task cancels, and passing that on to sub-tasks and tool processes.
Capability security13. Reliability, security & sandboxing →: Granting access by handing out specific, unforgeable capabilities, such as a token for one folder, instead of relying on broad ambient identity.
Capacity planning14. Observability & performance →: Forecasting demand and provisioning enough resources ahead of time, using load tests and growth data.
Capacity-aware routing6. Distributed LLM inference →: Routing that uses live capacity signals such as free cache blocks, queue length or tokens in flight.
Causal attention2. Transformer architecture →: Attention where a token may only look at itself and earlier tokens, enforced with an upper-triangular mask. It lets the model learn from every position of a sequence in one pass without seeing the answer.
Chat templates10. Tool calling & protocols →: The model-specific recipe, often a Jinja template shipped with the tokenizer, that turns a list of messages and tools into the exact token sequence the model was trained on. A mismatched template quietly degrades quality.
Checkpointing11. Context, memory & persistence →: Saving the agent's state at defined points so it can resume from there instead of starting over.
Cluster scheduling6. Distributed LLM inference →: Allocating GPUs and nodes to models and jobs across a cluster, for example with Kubernetes and GPU-aware schedulers.
Compensating actions12. Backend & distributed systems →: Steps that undo or offset an earlier action when a later step fails, as in the saga pattern, since true distributed transactions are rarely available.
Compute resource contention18. Advanced systems research questions →: Workloads competing for the same GPUs, memory bandwidth or CPUs and slowing each other down.
Compute-bound workloads4. LLM inference internals →: Work whose speed is capped by arithmetic throughput rather than memory. Prefill and large-batch matrix multiplications are compute-bound.
Concurrency5. Batching & scheduling →: The number of requests in flight at once, running or waiting. Throughput rises with concurrency until the GPU or its memory saturates.
Concurrency limits12. Backend & distributed systems →: Caps on how many operations run at once per user, tenant, tool or dependency, protecting the systems downstream.
Concurrent inference5. Batching & scheduling →: Serving many independent requests on the same model instance at the same time through batching, rather than one after another.
Constrained decoding3. Decoding & sampling →: Limiting generation so the output always matches a formal rule, such as a regex, a grammar or a fixed list of options, by masking every token that would break it at each step.
Context assembly11. Context, memory & persistence →: Building the final prompt from its parts in a stable order: instructions, tool definitions, memory, retrieved documents and recent messages. A stable order also keeps prefix caching effective.
Context compaction11. Context, memory & persistence →: Shrinking the context while keeping what matters: summarising old turns, dropping stale tool output, holding on to key decisions.
Context length11. Context, memory & persistence →: The number of tokens actually in a given request. Cost and latency grow with it, and quality can slip well before the window is full.
Context management11. Context, memory & persistence →: Deciding what goes into the limited context on each model call: which instructions, history, tool results and retrieved knowledge.
Context summarization11. Context, memory & persistence →: Replacing a span of history with a model-written summary. It saves tokens but can lose details.
Context truncation11. Context, memory & persistence →: Dropping the oldest or least important content when the context exceeds its budget. Simple, but information disappears abruptly.
Context window11. Context, memory & persistence →: The maximum number of tokens a model can attend to at once, prompt and output together, such as 128K or 1M tokens.
Context-aware routing18. Advanced systems research questions →: Routing requests by what is in their context, such as cached prefixes, length or modality, to the best-suited backend.
Continuous batching5. Batching & scheduling →: Rebuilding the batch at every decode step: finished sequences leave immediately and waiting requests join. Also called in-flight or iteration-level batching; it is standard in vLLM, SGLang and TensorRT-LLM.
Control plane12. Backend & distributed systems →: The part of a system that decides and manages (configuration, scheduling, placement, policy) rather than doing the work itself.
Control tokens10. Tool calling & protocols →: Special tokens that steer format or behaviour: role switches, turning a thinking mode on or off, end of turn.
Conversation history11. Context, memory & persistence →: The sequence of past user, assistant and tool messages in a session.
CopilotKit17. Infrastructure & development tools →: CopilotKit is an open-source toolkit for putting agents inside React apps, with in-app chat, shared state and generative UI. Its team created the AG-UI protocol.
Cost tracking14. Observability & performance →: Attributing token, GPU and tool costs to users, features and runs.
Cost-aware model routing18. Advanced systems research questions →: Sending each request to the cheapest model likely to handle it well, and escalating to larger models only when needed.
CPU–GPU data transfers8. GPU & CUDA engineering →: Copies between host and device over PCIe or NVLink-C2C. They are slow compared with HBM, so engines keep them out of the hot loop; cache offloading uses them on purpose.
Crash recovery13. Reliability, security & sandboxing →: Restoring service and in-progress work after a process or node crash, usually from checkpoints and durable logs.
Credential encryption13. Reliability, security & sandboxing →: Encrypting stored credentials at rest and in transit, with tightly controlled keys.
CrewAI17. Infrastructure & development tools →: A Python framework for orchestrating teams of role-based agents.
Cross-layer optimization18. Advanced systems research questions →: Coordinated optimisations across the app, agent runtime, inference engine and hardware that no single layer can achieve alone.
CUDA8. GPU & CUDA engineering →: NVIDIA's platform and programming model for its GPUs: compiler, runtime, driver API and libraries such as cuBLAS and cuDNN. Most AI frameworks run on top of it.
CUDA17. Infrastructure & development tools →: NVIDIA's GPU programming platform, the foundation under nearly every training and inference stack.
CUDA C++8. GPU & CUDA engineering →: The C++ dialect for writing CUDA kernels, with extensions for launching grids of threads and managing GPU memory.

D

Data parallelism6. Distributed LLM inference →: Running full copies of the model on different devices, each handling different requests or training batches. It scales throughput, not the speed of any one request.
Data-parallel serving6. Distributed LLM inference →: Data parallelism at inference time: several replicas behind a load balancer. Large MoE deployments often run attention data-parallel and experts expert-parallel.
Decode phase4. LLM inference internals →: Generating output one token at a time. Each step is a forward pass over a single new token that reads the whole KV cache, which makes it memory-bandwidth-bound.
Decoder-only Transformer2. Transformer architecture →: The architecture of nearly every current LLM: a stack of transformer blocks with causal self-attention and no separate encoder. It reads a sequence and predicts the next token at every position.
Decoding3. Decoding & sampling →: The strategy that turns the model's probability distribution into an actual token at each step, repeated until a stopping condition is met.
DeepSeek17. Infrastructure & development tools →: The Chinese lab behind DeepSeek-V3 and R1, known for multi-head latent attention, fine-grained MoE and very efficient training.
Dense models7. Mixture of Experts (MoE) →: Models where every parameter is used for every token. Simpler to train and serve, but compute per token rises with total size.
Deterministic replay11. Context, memory & persistence →: Replay that follows the same path every time because nondeterministic results, such as model outputs, clock reads and tool responses, are read from the log instead of recomputed. Workflow engines like Temporal depend on it.
Detokenization1. LLM fundamentals →: Turning generated token IDs back into text. Byte-level tokenizers can split one character (an emoji, say) across tokens, so streaming output is buffered until the bytes form valid text.
Direct weight modification16. Weight-space programming & interpretability →: Changing parameter tensors by hand or by algorithm, without gradient training on a dataset.
Disaggregated serving6. Distributed LLM inference →: The general design of splitting inference stages (prefill, decode, sometimes attention and experts) onto hardware pools that scale independently.
Distributed agent benchmarking18. Advanced systems research questions →: Benchmarking agent systems at scale on realistic, concurrent, multi-tenant workloads instead of single runs.
Distributed coordination12. Backend & distributed systems →: Getting nodes to agree on leaders, locks, membership or configuration, often through a consensus system.
Distributed inference6. Distributed LLM inference →: Running one model across several accelerators, or many copies across a cluster, because it does not fit on one device or one device is too slow.
Distributed KV caching6. Distributed LLM inference →: Storing and sharing KV caches across machines, in CPU memory, SSDs or a dedicated cache tier, so a prefix computed once can be reused cluster-wide.
Distributed locks12. Backend & distributed systems →: Locks held across processes or machines so only one actor works on a resource at a time.
Distributed model serving6. Distributed LLM inference →: The whole system for serving models on a cluster: replicas, routers, autoscaling, caches and health checks.
Distributed orchestration12. Backend & distributed systems →: Coordinating multi-step work across machines, with retries, timeouts and durable state.
Distributed recovery12. Backend & distributed systems →: Bringing a distributed system back to a consistent state after a failure, for example by reassigning leases and replaying logs.
Distributed request routing6. Distributed LLM inference →: Choosing which replica or worker handles each incoming request.
Distributed systems12. Backend & distributed systems →: Systems of several machines cooperating over a network, which must cope with partial failure, latency and the lack of a shared clock.
Distributed tracing14. Observability & performance →: Following one request across services as a tree of timed spans, showing where its time went.
Docker17. Infrastructure & development tools →: Tooling to build and run containers, packaging an application together with its dependencies.
Docker Compose17. Infrastructure & development tools →: A tool for defining and running multi-container setups from one YAML file, mostly for local development.
Docker containers13. Reliability, security & sandboxing →: Lightweight isolated environments that share the host's kernel. Convenient for packaging tools, but the shared kernel makes them a weaker wall against hostile code.
Draft model4. LLM inference internals →: The cheap proposer in speculative decoding: a smaller model from the same family, extra prediction heads, a multi-token-prediction module, or even n-gram lookups from the prompt.
Durable state11. Context, memory & persistence →: State stored where it survives process and machine failures, such as a database or an event log.
Dynamic admission control18. Advanced systems research questions →: Adjusting admission thresholds in real time from load signals, such as accepting fewer new agent runs while KV-cache utilisation is high.
Dynamic batch membership5. Batching & scheduling →: The set of sequences in a batch changing from step to step. Continuous batching relies on it, and it requires per-sequence cache bookkeeping.
Dynamic batching5. Batching & scheduling →: Forming batches on the fly from whatever requests arrive within a short time window or up to a size limit, trading a little waiting for better utilisation.

E

Embedding dimension1. LLM fundamentals →: The length of each embedding vector, also called the hidden size or d_model, for example 4,096 or 7,168. It sets the width of the residual stream that runs through every layer.
End-of-sequence token3. Decoding & sampling →: A special token such as <|endoftext|> or <|im_end|> that the model learns to emit when its answer is complete. Generation stops when it is picked.
End-to-end latency14. Observability & performance →: The total time from a user's action to the complete result, including queues, every model call and every tool.
Entity Framework Core17. Infrastructure & development tools →: The .NET object-relational mapper for working with databases through C# classes and LINQ queries.
Error handling13. Reliability, security & sandboxing →: Catching, classifying and responding to failures: deciding whether to retry, tell the model, or abort.
Event messaging12. Backend & distributed systems →: Publishing and subscribing to events through a broker, with fan-out to many consumers.
Event-driven architecture12. Backend & distributed systems →: Services reacting to published events, such as "run completed" or "tool approved", instead of calling each other directly.
Exactly-once execution12. Backend & distributed systems →: The goal that each operation takes effect exactly once. Exactly-once delivery over a network is impossible, so systems combine at-least-once delivery with idempotency to get the same effect.
Execution history11. Context, memory & persistence →: A record of every step an agent took: model calls, tool calls, results and decisions.
Execution plane12. Backend & distributed systems →: Where agent work actually runs: the workers that execute agent loops and tools.
Execution replay11. Context, memory & persistence →: Re-running a recorded execution, reusing the recorded model and tool results, to debug it or to recover.
Execution sandbox13. Reliability, security & sandboxing →: A restricted environment where agent-generated code and commands run, with limited filesystem, network and system access.
Execution traces14. Observability & performance →: Traces of an agent run: every model call, tool call and decision as a span with inputs, outputs and timing.
Expert combine7. Mixture of Experts (MoE) →: Gathering expert outputs back into the original token order and summing them with the gating weights.
Expert dispatch7. Mixture of Experts (MoE) →: Sending tokens to their chosen experts: regrouping them by expert and, under expert parallelism, moving them to other GPUs.
Expert load balancing7. Mixture of Experts (MoE) →: Keeping experts evenly used, through auxiliary balancing losses, per-expert router biases (as in DeepSeek's auxiliary-loss-free method) or capacity limits.
Expert MLPs7. Mixture of Experts (MoE) →: The MLP each expert is, typically a SwiGLU FFN much narrower than the dense FFN it replaces. Many designs also add one or more shared experts that every token uses.
Expert parallelism6. Distributed LLM inference →: Placing different MoE experts on different GPUs, so each token is shipped with an all-to-all exchange to the GPUs that hold its chosen experts.
Expert parallelism7. Mixture of Experts (MoE) →: Spreading experts across GPUs: each holds a subset and receives the tokens routed to it through all-to-all. Large deployments pair it with data-parallel attention.
Expert placement7. Mixture of Experts (MoE) →: Deciding which experts live on which GPUs. Popular experts can be replicated to spread their load.
Experts7. Mixture of Experts (MoE) →: The individual FFNs inside an MoE layer. Nobody assigns them topics; any specialisation emerges in training and tends to follow token types and syntax more than subjects.
External side effects13. Reliability, security & sandboxing →: Effects outside your own system, such as sending email, charging a card or writing to a third-party API, that cannot simply be rolled back.

F

Fail-closed security13. Reliability, security & sandboxing →: When a check cannot be completed, because of an error, a timeout or an unknown action, deny by default instead of allowing.
Failure handling13. Reliability, security & sandboxing →: The overall plan for failures: detect, contain, retry, compensate, escalate.
Failure-mode localization16. Weight-space programming & interpretability →: Finding which layers, heads or directions cause a particular failure, so it can be fixed precisely.
Fault tolerance12. Backend & distributed systems →: Continuing to work correctly when components fail, through redundancy, retries and isolation.
Feed-forward network (FFN)2. Transformer architecture →: The per-token sublayer after attention: widen the vector (often 3 to 4 times), apply a nonlinearity, project back. In modern models it is a gated SwiGLU and holds most of the parameters.
Filesystem boundaries13. Reliability, security & sandboxing →: Limits on which paths an agent can read or write, enforced with mounts, permissions or path checks.
Final hidden state1. LLM fundamentals →: The hidden state at the last position after the final layer and final norm. It is the only vector the LM head needs to score the next token.
Final normalization2. Transformer architecture →: One last norm on the hidden states after the final block and before the LM head, which keeps the scale of the residual stream stable for the output layer.
Forced tool selection10. Tool calling & protocols →: Requiring the model to call a particular tool, or any tool, on this turn through the tool_choice setting.
Forward pass1. LLM fundamentals →: One run of inputs through the network, from embeddings to logits. Training pairs every forward pass with a backward pass; inference only does forward passes, one per generated token after the prompt.
Frequency penalty3. Decoding & sampling →: Subtracts an amount proportional to how often a token has already been generated, so heavily repeated tokens are pushed down more. The related presence penalty applies a flat amount once a token has appeared at all.
Function arguments10. Tool calling & protocols →: The values the model fills in for a tool's parameters, emitted as JSON.
Function calling10. Tool calling & protocols →: A model's ability to output a structured request to run a named function with arguments, instead of plain text.
Fused sampling8. GPU & CUDA engineering →: Doing temperature, top-k or top-p filtering and the random draw in one GPU kernel, avoiding a full-vocabulary sort and extra trips to the CPU.

G

Gating network7. Mixture of Experts (MoE) →: Another name for the router. "Gate" stresses that it also sets how much each chosen expert's output counts.
Gating weights7. Mixture of Experts (MoE) →: The normalised router scores of the selected experts, used to weight each expert's output when combining them.
GitHub Actions17. Infrastructure & development tools →: GitHub's CI/CD service for running builds, tests and deployments on every change.
Global routing6. Distributed LLM inference →: Routing across regions or clusters, weighing latency to the user, spare capacity and cost.
Google ADK17. Infrastructure & development tools →: Google's Agent Development Kit, an open-source framework for building, evaluating and deploying agents, integrated with Gemini and Vertex AI.
Google TPU8. GPU & CUDA engineering →: Google's Tensor Processing Units: systolic-array accelerators linked into large pods and programmed through JAX and XLA. Gemini and Gemma are trained on them.
GPU compute utilization8. GPU & CUDA engineering →: The share of peak FLOPs actually achieved. Model FLOPs utilisation of 40 to 60 percent is good for training; batch-1 decode reaches far less.
GPU computing8. GPU & CUDA engineering →: Using graphics processors for general parallel computing. Thousands of simple cores plus specialised tensor cores make them ideal for neural-network matrix maths.
GPU kernels8. GPU & CUDA engineering →: Functions that run on the GPU across many threads in parallel, such as a matrix multiplication, a softmax or a fused attention kernel.
GPU memory8. GPU & CUDA engineering →: Memory attached to the GPU. It must hold the weights, KV cache, activations and workspace, and running out of it is the most common serving limit.
GPU memory hierarchy8. GPU & CUDA engineering →: The levels from registers (fastest, tiny) through shared memory and L1 per streaming multiprocessor, the shared L2 cache and HBM, out to host RAM over PCIe. Fast kernels keep data as high in the hierarchy as they can.
GPU memory usage14. Observability & performance →: Memory consumed on each GPU by weights, KV cache, activations and fragmentation.
GPU profiling8. GPU & CUDA engineering →: Measuring where GPU time goes with tools such as Nsight Systems and the PyTorch profiler, to find stalls, idle gaps and slow kernels.
GPU telemetry14. Observability & performance →: Hardware metrics from GPUs, including utilisation, memory, temperature, power, clocks, NVLink traffic and errors, usually gathered through DCGM.
GPU utilization5. Batching & scheduling →: How busy the GPU is. A busy reading in nvidia-smi only means some kernel was running, not that the tensor cores were saturated; model FLOPs utilisation (MFU) is the stricter measure.
GPU utilization14. Observability & performance →: The share of time kernels are running on the GPU. Useful but coarse; pair it with multiprocessor occupancy and tensor-core activity.
GPU-aware orchestration18. Advanced systems research questions →: Orchestration that considers GPU locality, memory and load when placing agent work and model calls.
Gradient descent15. Model training & post-training →: The optimisation method behind all of it: compute how the loss changes with each parameter (the gradient) and move every parameter a small step in the opposite direction.
Grafana17. Infrastructure & development tools →: Dashboards and visualisation across metrics, logs and traces.
Grafana Loki17. Infrastructure & development tools →: Grafana's log aggregation system. It indexes labels rather than full text, which keeps it cheap to run.
Grafana Tempo17. Infrastructure & development tools →: Grafana's distributed tracing backend for storing and querying traces at scale.
Greedy decoding3. Decoding & sampling →: Always choose the single most likely token. Deterministic and cheap, but prone to repetition and bland text in open-ended writing.
Grouped matrix multiplication7. Mixture of Experts (MoE) →: One kernel that runs many independent matrix multiplications of different sizes, one per expert, instead of launching each separately. Essential for fast MoE layers.

H

Hallucinated tools10. Tool calling & protocols →: The model calling a tool that does not exist or inventing parameters. Validation catches it; clear definitions and constrained decoding make it rarer.
Hardware-specific optimization8. GPU & CUDA engineering →: Tuning kernels, number formats and parallelism for a particular chip, such as FP8 on Hopper, FP4 on Blackwell, or tile sizes matched to a multiprocessor's shared memory.
Hermes Agent17. Infrastructure & development tools →: An open-source agent from Nous Research, built around its Hermes models, with persistent memory, skills and tool use.
Hidden states1. LLM fundamentals →: The vectors carried between layers, one per token position. Each block reads them, adds an update and passes them on, so they gradually shift from "which token is this" to "what should come next".
High availability6. Distributed LLM inference →: Staying up through failures with redundant replicas, health checks, retries and failover across zones.
High-speed interconnects6. Distributed LLM inference →: Links like NVLink and NVSwitch inside a server and InfiniBand or RoCE between servers. Their bandwidth decides which parallelism schemes are practical.
Human approval13. Reliability, security & sandboxing →: Requiring a person to confirm sensitive actions, such as payments, deletions or outgoing messages, before they run.
Human-in-the-loop13. Reliability, security & sandboxing →: Designs where people review, correct or approve agent work at key points instead of granting full autonomy.

I

Idempotency12. Backend & distributed systems →: The property that doing an operation twice has the same effect as doing it once. Essential wherever retries can happen.
Idempotency keys12. Backend & distributed systems →: Unique IDs attached to requests so the server can recognise and skip duplicates of an operation it has already done.
Inference4. LLM inference internals →: Running a trained model to get outputs. For LLMs it splits into processing the prompt, then generating tokens one by one, and the two halves behave very differently.
Inference cost optimization18. Advanced systems research questions →: Lowering the cost per token or per task through batching, caching, quantization, routing and right-sized hardware.
Inference plane12. Backend & distributed systems →: The model-serving layer: GPU clusters, inference engines and routers that turn prompts into tokens.
Inference profiling14. Observability & performance →: Profiling a serving stack specifically: prefill against decode time, kernel time, scheduler overhead and batch composition.
Inference quality of service (QoS)6. Distributed LLM inference →: Latency and throughput guarantees or priorities per tenant or request class, such as interactive traffic ahead of batch jobs.
Inference scheduler5. Batching & scheduling →: The engine component that makes batching, admission and preemption decisions every iteration. Its policy shapes throughput, fairness and tail latency.
Inference-aware agent scheduling18. Advanced systems research questions →: Scheduling agent steps with knowledge of the inference layer, for example grouping calls that share a prefix, or holding back low-priority agents when GPUs are saturated.
Inter-GPU communication7. Mixture of Experts (MoE) →: Data exchanged between GPUs during a forward pass: all-reduce for tensor parallelism, all-to-all for expert parallelism. It often caps MoE throughput.
Inter-token latency4. LLM inference internals →: The gap between two consecutive streamed tokens. TPOT is its average; its spikes, often caused by other requests' prefills sharing the GPU, are what make streaming stutter.

J

Joint agent/inference scheduling18. Advanced systems research questions →: Optimising agent orchestration and model serving together instead of treating the inference API as a black box.
JSON parsing10. Tool calling & protocols →: Turning the model's emitted text into data. Robust harnesses cope with partial streams, trailing text and malformed JSON.
JSON Schema10. Tool calling & protocols →: A standard vocabulary for describing the shape of JSON data. It is the common language for tool parameters and structured outputs.
JSON-constrained decoding3. Decoding & sampling →: Constrained decoding whose grammar is compiled from a JSON Schema, so the output is always parseable JSON with the required fields. Used for tool calls and data extraction.
JSONB17. Infrastructure & development tools →: PostgreSQL's binary JSON column type, which can be indexed and queried. Handy for flexible agent state and tool payloads.

K

k617. Infrastructure & development tools →: A load-testing tool whose scenarios are written in JavaScript, good for measuring API capacity and latency percentiles.
Kernel fusion8. GPU & CUDA engineering →: Combining several operations into one kernel so intermediate results stay in registers or shared memory instead of round-tripping through main memory. FlashAttention is the best-known example.
Kernel launches8. GPU & CUDA engineering →: The CPU asking the GPU to start a kernel. Each launch costs a few microseconds, which adds up over thousands of small kernels per token; CUDA graphs record and replay them to cut the cost.
Kernel profiling8. GPU & CUDA engineering →: Inspecting a single kernel in depth with Nsight Compute: achieved bandwidth, occupancy, tensor-core use and the reasons it stalls.
Key (K)2. Transformer architecture →: A vector each token produces that advertises "what I contain". During generation the keys of past tokens are stored in the KV cache.
Kubernetes17. Infrastructure & development tools →: The standard container orchestrator for deploying, scaling and healing services across a cluster, GPU workloads included.
KV cache4. LLM inference internals →: The stored keys and values of every earlier token in every layer, so each decode step only computes K and V for the newest token. It trades memory for compute and often fills most of a serving GPU.
KV-cache allocation4. LLM inference internals →: How an engine hands out GPU memory for caches. Paged allocation, as in vLLM's PagedAttention, splits it into fixed-size blocks so little memory is reserved but left unused.
KV-cache capacity4. LLM inference internals →: How many tokens of cache fit in GPU memory once the weights are loaded. It caps concurrent sequences times their length; compare the per-token cache figures in the gallery.
KV-cache reuse4. LLM inference internals →: Using keys and values computed for one request in another that starts with the same tokens, which skips that part of prefill entirely.
KV-cache utilization14. Observability & performance →: The share of KV-cache blocks in use. Close to 100 percent means preemption and queueing are about to start.

L

LangGraph17. Infrastructure & development tools →: A library from the LangChain team for building agents as stateful graphs, with persistence, checkpoints and human-in-the-loop steps.
Latency14. Observability & performance →: How long an operation takes. For LLMs, time to first token, time per output token and end-to-end time should be tracked separately.
Layer normalization2. Transformer architecture →: Normalises each token's vector to zero mean and unit variance, then applies a learned scale and shift. Used in the original Transformer and GPT-2; most new models use RMSNorm instead.
Learning from demonstrations15. Model training & post-training →: Training a model to imitate example behaviour shown by people or stronger models (imitation learning). SFT is a form of it.
Lease-based ownership12. Backend & distributed systems →: A worker owns a task only while it keeps renewing the lease. If it crashes, the lease lapses and another worker takes over.
Leases12. Backend & distributed systems →: Time-limited locks or ownership grants that expire on their own if the holder dies, so work never stays stuck.
Linear layers2. Transformer architecture →: Layers that multiply their input by a weight matrix, optionally adding a bias. Q, K, V and O projections, FFN projections, MoE routers and the LM head are all linear layers; modern LLMs mostly drop the biases.
Llama17. Infrastructure & development tools →: Meta's open-weight model family, whose 2023 release kick-started the open-weight ecosystem.
llama.cpp17. Infrastructure & development tools →: A C/C++ inference engine for running quantized models (GGUF files) on CPUs, Apple Silicon and consumer GPUs.
llm-d17. Infrastructure & development tools →: An open-source, Kubernetes-native distributed inference stack built around vLLM, adding cache-aware routing and prefill/decode disaggregation.
LM head1. LLM fundamentals →: The final linear layer that maps the final hidden state to one score per vocabulary token. Some models tie it to the embedding matrix to save parameters.
Load balancing6. Distributed LLM inference →: Spreading requests across replicas to avoid hot spots. LLM balancers should look at queue depth and cache usage, not just request counts, because requests vary so much in size.
Load testing14. Observability & performance →: Driving synthetic traffic at a system to measure its capacity and find its breaking points, for example with k6.
Logit masking3. Decoding & sampling →: Setting some logits to minus infinity so those tokens can never be chosen. It is the mechanism behind constrained decoding and banned tokens.
Logit processing3. Decoding & sampling →: Any step that changes the logits between the LM head and the final pick: temperature, penalties, logit bias, masks. Engines run them as a chain of logit processors.
Logits1. LLM fundamentals →: The raw, unnormalised scores the LM head produces, one per vocabulary token. Higher means more likely, and they can be any real number.
Long-term memory11. Context, memory & persistence →: Information kept across sessions in external storage, such as files, databases or vector stores, and brought back when relevant.
Loss functions15. Model training & post-training →: Functions that measure how wrong a prediction is. LLM pretraining uses cross-entropy on the next token.

M

Mastra17. Infrastructure & development tools →: A TypeScript framework for building agents, workflows and retrieval pipelines.
MATH-500 benchmark16. Weight-space programming & interpretability →: A 500-problem subset of the MATH competition dataset, widely used to measure mathematical reasoning and a handy check that an edit did not hurt capability.
Matrix multiplication1. LLM fundamentals →: The core operation of every layer (GEMM): activations times weights. GPUs are built around doing it fast, and how quickly they can do it and feed it with data decides model speed.
Matrix properties16. Weight-space programming & interpretability →: Features of a weight matrix such as rank, norm, condition number and singular-value spectrum, which hint at its role and stability.
Matrix-matrix multiplication8. GPU & CUDA engineering →: Multiplying two matrices (GEMM). Batching and prefill turn GEMVs into GEMMs, which keep tensor cores busy.
Matrix-vector multiplication8. GPU & CUDA engineering →: Multiplying a matrix by a single vector (GEMV). Batch-1 decode is effectively a chain of these, which is why it is bandwidth-bound.
Mechanistic interpretability16. Weight-space programming & interpretability →: Reverse-engineering the algorithms a network implements, down to individual features and circuits.
Megatron-LM17. Infrastructure & development tools →: NVIDIA's library for training very large transformers with tensor, pipeline, sequence and expert parallelism.
Memory bandwidth8. GPU & CUDA engineering →: How many bytes per second can move between HBM and the compute units, about 3.35 TB/s on an H100 SXM. It bounds decode speed: tokens per second is roughly bandwidth divided by the bytes read per token.
Memory locality8. GPU & CUDA engineering →: Accessing data that is nearby or recently used, so caches hit and loads coalesce. Poor locality wastes bandwidth.
Memory retrieval11. Context, memory & persistence →: Selecting the stored memories relevant to the current step, by recency, keywords or embedding similarity.
Memory traffic8. GPU & CUDA engineering →: The total bytes a kernel reads and writes. Cutting it through fusion, quantization or smaller KV caches is often the most effective optimisation.
Memory utilization8. GPU & CUDA engineering →: Either the share of memory capacity in use or the share of peak bandwidth achieved. Serving engines deliberately reserve most free memory for KV cache, so high capacity use is normal.
Memory-bandwidth-bound workloads4. LLM inference internals →: Work capped by how fast data can stream from memory. Small-batch decode is the classic case: each token needs every weight and the whole cache to be read for very little maths.
Message boundary tokens10. Tool calling & protocols →: Tokens marking where each message starts and ends, such as <|im_start|> and <|im_end|> in ChatML-style templates.
Message brokers12. Backend & distributed systems →: Servers that route and store messages between producers and consumers, such as NATS, RabbitMQ or Kafka.
Message history11. Context, memory & persistence →: The raw list of messages sent to the model on each turn. Because the model is stateless, the whole history is sent again on every call, ideally with caching.
Message queues12. Backend & distributed systems →: Infrastructure that holds messages until consumers process them, smoothing bursts between producers and consumers.
Metrics14. Observability & performance →: Numeric time series such as counters, gauges and histograms that show how a system behaves over time.
Microservices12. Backend & distributed systems →: Many small, independently deployed services talking over the network. Independent scaling and teams, at the cost of distributed-systems complexity.
Microsoft Agent Framework17. Infrastructure & development tools →: Microsoft's open-source SDK for building agents and multi-agent workflows in .NET and Python, the successor to Semantic Kernel and AutoGen.
MicroVMs13. Reliability, security & sandboxing →: Minimal virtual machines, such as Firecracker, that boot in milliseconds and give each sandbox its own kernel, a stronger boundary than containers.
Mixture of Experts (MoE)7. Mixture of Experts (MoE) →: An architecture where the FFN in each block is replaced by many expert FFNs plus a router that sends each token to only a few of them. Total parameters grow without a matching rise in compute per token.
MLP weight editing16. Weight-space programming & interpretability →: Editing FFN weights, which partly act as key-value memories, to change stored associations, as in methods like ROME and MEMIT.
Model architecture2. Transformer architecture →: The full specification of a network: layer count, hidden size, heads, attention type, FFN or MoE layout, norms, positions, vocabulary and context length. The gallery compares these choices across 100+ releases.
Model behavior15. Model training & post-training →: How a model actually responds, including style, helpfulness, refusals and tool use. It is shaped mainly in post-training.
Model capabilities15. Model training & post-training →: What a model can do, such as reasoning, coding, languages, long context and tool use. Mostly built in pretraining and drawn out in post-training.
Model Context Protocol (MCP)10. Tool calling & protocols →: An open protocol, introduced by Anthropic, for connecting models to tools and data. MCP servers expose tools, resources and prompts over a standard JSON-RPC interface that any client can use.
Model editing16. Weight-space programming & interpretability →: Changing specific facts or behaviours of a trained model with as few side effects as possible.
Model evaluation15. Model training & post-training →: Measuring a model's quality on held-out tasks, safety tests and real-world use.
Model execution overhead4. LLM inference internals →: Time spent outside the core maths: scheduler logic, kernel launches, sampling and CPU-GPU synchronisation. At small batch sizes it can eat a large share of each step; CUDA graphs reduce it.
Model gateway12. Backend & distributed systems →: A proxy in front of one or more model providers that unifies their APIs and handles keys, fallbacks, routing, caching and cost tracking.
Model parallelism6. Distributed LLM inference →: Any scheme that splits a single model across devices: by tensor, by layer or by expert.
Model parameters1. LLM fundamentals →: All the learned numbers in a network: embedding tables, attention and FFN weight matrices, norm scales. A "70B" model has roughly seventy billion of them.
Model placement18. Advanced systems research questions →: Deciding which models run on which GPUs, and how many replicas each gets, given demand and memory.
Model regression evaluation16. Weight-space programming & interpretability →: Checking that an edited or updated model has not lost performance on tasks it used to handle.
Model replication6. Distributed LLM inference →: Keeping several identical copies of a model to serve more traffic or to survive failures.
Model sharding6. Distributed LLM inference →: Physically splitting the weights into pieces (shards) stored on different devices, whatever parallelism scheme decides the split.
Model weights1. LLM fundamentals →: The stored values of those parameters, usually as files in bf16, fp8 or a quantized format. "Open weights" means these files can be downloaded, even if the training data and code cannot.
Modular monolith12. Backend & distributed systems →: One deployable application split into strongly separated modules. It gets clear boundaries without the operational cost of microservices.
MoE routing7. Mixture of Experts (MoE) →: Deciding which experts process each token at each MoE layer.
Multi-agent collaboration9. Agent architecture & internals →: Agents working toward a shared goal by dividing the work, sharing results and checking each other.
Multi-agent systems9. Agent architecture & internals →: Systems of several agents with distinct roles that cooperate, such as a planner with workers or a writer with a reviewer.
Multi-GPU inference6. Distributed LLM inference →: One model instance spanning several GPUs in a single server, usually with tensor parallelism over NVLink.
Multi-head attention2. Transformer architecture →: Several attention operations (heads) run in parallel, each with its own Q, K and V projections over a slice of the hidden size, then their results are concatenated. Different heads learn different patterns.
Multi-layer perceptron (MLP)2. Transformer architecture →: Another name for the FFN sublayer: a small fully connected network applied to each token independently. In MoE models every expert is one MLP.
Multi-node inference6. Distributed LLM inference →: One model instance spanning several servers, needed for the very largest models. It depends on InfiniBand or RoCE networking and usually combines pipeline or expert parallelism across nodes.
Multi-tenant agent infrastructure18. Advanced systems research questions →: Shared platforms running many organisations' agents with isolation, fairness, quotas and per-tenant costs.
Multi-tenant inference6. Distributed LLM inference →: Serving many customers or applications on shared infrastructure, with isolation, quotas and fairness between them.

N

NATS JetStream17. Infrastructure & development tools →: The persistence layer of the NATS messaging system, adding durable streams, replay and acknowledged consumers.
Neural network parameter inspection16. Weight-space programming & interpretability →: Examining parameter tensors: their statistics, norms, spectra and how they connect components.
Next-token generation1. LLM fundamentals →: Using a trained model to produce the next token at inference time: run a forward pass, read the distribution, pick a token with a decoding strategy, append it.
Next-token prediction1. LLM fundamentals →: The training objective of LLMs: given all previous tokens, predict the next one. Repeated over trillions of tokens, this one task is enough to learn grammar, facts and patterns of reasoning.
Next.js17. Infrastructure & development tools →: A React framework for web apps with server rendering, static export and file-based routing. This site is built with it.
Nondeterministic execution11. Context, memory & persistence →: Steps that can give a different result each time: sampled model output, network calls, the current time. They must be recorded to make replay safe.
Numerical weight edits16. Weight-space programming & interpretability →: Precise arithmetic changes to weights, such as adding a rank-one update or scaling one direction.
NVIDIA DCGM Exporter17. Infrastructure & development tools →: Exposes NVIDIA Data Center GPU Manager metrics, such as utilisation, memory, temperature and errors, to Prometheus.
NVIDIA GPU8. GPU & CUDA engineering →: The dominant accelerator family for AI (A100, H100, H200, B200 and successors), combining tensor cores, high-bandwidth memory and NVLink with the CUDA software ecosystem.
NVIDIA H1008. GPU & CUDA engineering →: NVIDIA's Hopper-generation data-centre GPU from 2022: 80 GB of HBM3, about 3.35 TB/s of bandwidth, roughly 1 PFLOP of dense BF16 and FP8 support. The reference chip for most LLM work of 2023 to 2025.

O

Observability14. Observability & performance →: Being able to understand what a system is doing inside from what it emits: traces, logs and metrics.
OpenAI-compatible API10. Tool calling & protocols →: The de facto HTTP interface, /v1/chat/completions with messages, tools and tool_calls, that most inference servers and gateways implement so clients can switch providers.
OpenClaw17. Infrastructure & development tools →: An open-source, self-hosted personal AI agent that runs on your own machine and is driven through chat apps, extended with tools and skills.
OpenTelemetry17. Infrastructure & development tools →: The vendor-neutral standard and SDKs for producing traces, metrics and logs, with emerging conventions for LLM and agent spans.
Optimizers15. Model training & post-training →: Algorithms that turn gradients into parameter updates, adding momentum, adaptive step sizes and weight decay, such as SGD, AdamW and Muon.
Output projection1. LLM fundamentals →: A linear layer that maps a sublayer's result back to the hidden size, such as attention's W_O after the heads are concatenated. The LM head is sometimes called the output projection of the whole model.

P

p95 latency14. Observability & performance →: The value that 95 percent of requests come in under. Percentiles reveal what averages hide.
Parallel execution8. GPU & CUDA engineering →: Running work simultaneously at every level: threads in a warp, warps on a multiprocessor, multiprocessors on a GPU, GPUs in a cluster, and communication overlapped with compute.
Parameter-efficient modifications16. Weight-space programming & interpretability →: Changing a model by touching only a few parameters, such as LoRA adapters, rank-one edits or bias tweaks.
Performance profiling14. Observability & performance →: Measuring where time and resources are spent in code, to find the bottlenecks.
Performance regressions14. Observability & performance →: Changes that make a system slower or more expensive than before, ideally caught by benchmarks in CI.
Permission enforcement13. Reliability, security & sandboxing →: Checking permissions at the moment of action, in code the model cannot bypass, not just in the prompt.
Persistent agent state11. Context, memory & persistence →: Agent state saved to durable storage so a run can pause, survive crashes and resume.
Persistent memory11. Context, memory & persistence →: Memory that survives restarts and sessions, such as user preferences or project facts written to durable storage.
Persistent workflows11. Context, memory & persistence →: Long-running workflows whose progress is recorded durably, so they can run for hours or days and survive restarts.
Pipeline parallelism6. Distributed LLM inference →: Putting consecutive groups of layers on different devices and passing activations between them. Communication is light, but stages need a steady flow of micro-batches to avoid idle "bubbles".
Policy gateway13. Reliability, security & sandboxing →: A central checkpoint that every tool call passes through to be judged against policy before it runs.
Positional embeddings1. LLM fundamentals →: Information about where each token sits in the sequence. Attention by itself ignores order, so models either add learned position vectors to the embeddings or, more commonly today, rotate queries and keys by position (RoPE).
Post-training15. Model training & post-training →: Everything after pretraining that shapes behaviour: SFT, preference optimisation (RLHF, DPO), reinforcement learning with verifiable rewards and safety training.
PostgreSQL17. Infrastructure & development tools →: A widely used open-source relational database known for correctness, extensions and JSON support. With the pgvector extension it also stores embeddings.
Prefill phase4. LLM inference internals →: Processing all prompt tokens in one batched forward pass while computing and caching their keys and values. It is compute-bound, and its length mostly sets the time to first token.
Prefill/decode disaggregation6. Distributed LLM inference →: Running prefill and decode on separate GPU pools, because one is compute-bound and the other bandwidth-bound, and transferring the KV cache between them. It stops long prompts from stalling streams already in progress.
Prefix caching4. LLM inference internals →: Keeping cache blocks for common prompt beginnings (system prompts, few-shot examples, earlier turns) and matching new requests against them automatically.
Prefix-aware routing6. Distributed LLM inference →: Cache-aware routing that hashes or matches prompt prefixes, such as the system prompt and conversation so far, to choose a replica.
Pretraining15. Model training & post-training →: The first and largest training stage: next-token prediction on trillions of tokens of web text, code and books. It gives the model its knowledge and language ability.
Probability distribution1. LLM fundamentals →: The softmax output over the whole vocabulary: non-negative values that add up to 1. Decoding strategies decide how to pick one token from it.
Process isolation13. Reliability, security & sandboxing →: Separating work into OS processes with their own memory and privileges, so one crash or exploit stays contained.
Prometheus17. Infrastructure & development tools →: A time-series database and monitoring system that scrapes metrics and evaluates alert rules.
Prompt caching4. LLM inference internals →: The API feature built on prefix caching: a repeated prompt prefix is billed at a discount and answered faster. The cached part must usually be identical and at the start of the prompt.
Prompt processing4. LLM inference internals →: Another name for prefill, sometimes counting tokenization and chat-template formatting too. With long prompts it becomes the main cost of a request.
Pydantic AI17. Infrastructure & development tools →: A Python agent framework from the Pydantic team, built around type-safe tool definitions and validated structured outputs.
Python17. Infrastructure & development tools →: The main language of machine-learning research, training and most inference tooling.

Q

Query (Q)2. Transformer architecture →: A vector each token produces that asks "what am I looking for?". Its dot product with every key decides how much that token attends to each position.
Queue depth14. Observability & performance →: The number of items waiting in a queue, a leading indicator of overload.
Queueing delay5. Batching & scheduling →: Time a request waits before its prefill starts. Under load it can be far larger than the compute time, and it is often the hidden part of TTFT.
Qwen17. Infrastructure & development tools →: Alibaba's family of open-weight models, dense and MoE, covering text, code and vision. Many of its releases are in the gallery.

R

Rate limiting13. Reliability, security & sandboxing →: Restricting how many requests or tokens a client may use per time window, protecting capacity and budgets.
Reasoning loop9. Agent architecture & internals →: The model's think, act, observe pattern inside the control loop, popularised as ReAct, where intermediate reasoning guides the next action.
Redis17. Infrastructure & development tools →: An in-memory data store used for caching, rate limiting, locks, queues and pub/sub.
Regression testing14. Observability & performance →: Re-running a fixed test suite after every change to catch behaviour that broke, including evaluations after prompt or model updates.
Repetition penalty3. Decoding & sampling →: Lowers the scores of tokens that already appear in the context, for example by dividing their positive logits by a factor, to discourage loops.
Repetitive reasoning loops16. Weight-space programming & interpretability →: A failure mode where a reasoning model keeps repeating the same thoughts or tokens without finishing. It can be traced to specific directions and reduced by editing them or by changing decoding settings.
Replay safety13. Reliability, security & sandboxing →: Making sure that re-running steps during recovery does not repeat side effects, using idempotency keys and recorded results.
Request admission5. Batching & scheduling →: Deciding whether a newly arrived request can start now, given free cache memory and the batch budget, or must wait in the queue.
Request preemption5. Batching & scheduling →: Interrupting a running request so a waiting or higher-priority one can run, usually by evicting its cache to be swapped back or recomputed later.
Request queue5. Batching & scheduling →: Requests that have arrived but are not yet running. Its depth is an early warning of overload.
Requests per second14. Observability & performance →: The rate of requests a system handles. For LLMs it means little without the token counts behind it.
Residual addition2. Transformer architecture →: The add operation x + f(x) at the end of each sublayer. It is cheap, and it is the reason every layer must keep the same hidden size.
Residual connections2. Transformer architecture →: Skip paths that add each sublayer's input to its output, so layers write updates onto a shared residual stream instead of replacing it. They keep signals and gradients flowing through very deep stacks.
Resource allocation12. Backend & distributed systems →: Assigning CPU, memory, GPUs and sandbox slots to tasks according to their needs and the policy.
Resource limits13. Reliability, security & sandboxing →: Caps on CPU, memory, disk, processes, network and run time for sandboxes and agents.
Resource-aware scheduling18. Advanced systems research questions →: Scheduling with explicit knowledge of what each task needs: memory, KV-cache size, GPU type and sandbox capacity.
Retries13. Reliability, security & sandboxing →: Repeating a failed operation with exponential backoff and jitter, only for transient failures and idempotent operations.
Retrieval systems11. Context, memory & persistence →: Search infrastructure, such as BM25, vector search, hybrid search and rerankers, that finds documents to place into context, as in retrieval-augmented generation (RAG).
Retry amplification13. Reliability, security & sandboxing →: When retries at several layers multiply (three layers of three retries is 27 attempts) and turn a small outage into a self-inflicted overload.
RMSNorm2. Transformer architecture →: A normalisation that divides each vector by its root-mean-square and multiplies by a learned scale. It is cheaper than LayerNorm (no mean, no bias) and is the default in modern LLMs.
Rotary Positional Embeddings (RoPE)2. Transformer architecture →: Encodes position by rotating pairs of query and key dimensions by an angle that grows with the token's position. The attention score then depends on relative distance, and the rotation frequencies can be rescaled to extend the context.
Router network7. Mixture of Experts (MoE) →: A small linear layer that maps each token's hidden state to one score per expert.
Routing imbalance7. Mixture of Experts (MoE) →: When the router sends far more tokens to some experts than others, creating stragglers, dropped tokens or idle GPUs.
Routing logits7. Mixture of Experts (MoE) →: The raw per-expert scores from the router, before softmax or sigmoid. Top-k is taken over them.

S

Sampling3. Decoding & sampling →: Picking the next token at random in proportion to its probability, instead of always taking the most likely one. It gives varied, more natural text at the cost of determinism.
Sampling settings3. Decoding & sampling →: The knobs exposed by an API or engine: temperature, top-k, top-p, min-p, penalties, seed and maximum tokens. Code and data extraction usually want little randomness; brainstorming wants more.
Sandboxed execution13. Reliability, security & sandboxing →: Running untrusted code inside a sandbox. It should be the default for any code an LLM writes.
Scheduling delay14. Observability & performance →: The time between a request being ready and actually being scheduled, in the inference engine or the orchestration layer.
Scheduling fairness18. Advanced systems research questions →: Making sure tenants and agents get fair shares of capacity, so one heavy user's bursts cannot starve everyone else.
Scheduling policy5. Batching & scheduling →: The rule for ordering work: first come first served, shortest job first, priority classes, or fair shares across tenants.
Secrets management13. Reliability, security & sandboxing →: Storing and injecting API keys and credentials securely, through vaults and short-lived tokens, so they never end up in prompts or logs.
Security policies13. Reliability, security & sandboxing →: Rules that define what agents may do: allowed tools, domains and data, and which actions need approval.
Self-attention2. Transformer architecture →: The operation that lets each token gather information from other tokens in the same sequence. Every position builds a query, compares it with every key, and takes a weighted average of the values.
Sequence batching5. Batching & scheduling →: Grouping multiple token sequences into one batch, by padding them to a common length or packing them end to end.
Sequence completion5. Batching & scheduling →: A sequence reaching a stopping condition. With continuous batching its slot and cache blocks are handed to the next request right away.
Service boundaries12. Backend & distributed systems →: The lines between services: what each one owns, its API and its data.
SGLang17. Infrastructure & development tools →: An open-source serving engine with RadixAttention for automatic prefix sharing, fast structured outputs and strong support for large MoE models.
shadcn/ui17. Infrastructure & development tools →: A collection of copy-in React components built on headless primitives and Tailwind CSS. This site uses it on top of Base UI.
Short-term memory11. Context, memory & persistence →: Information kept within the current context or session, mainly the recent conversation and working notes.
Singular value decomposition (SVD)16. Weight-space programming & interpretability →: Factoring a matrix as W = UΣVᵀ: sets of orthogonal directions and the strength of each. Keeping only the top components gives the best possible low-rank approximation.
Singular vectors16. Weight-space programming & interpretability →: The input and output directions found by SVD. A matrix maps each right singular vector onto its left singular vector, scaled by the singular value.
Softmax1. LLM fundamentals →: The function that turns scores into probabilities: exponentiate each score, then divide by the sum. It is used on the logits at the output and on attention scores inside every attention layer.
Sparse computation7. Mixture of Experts (MoE) →: Doing only a fraction of the possible work for each input. MoE does it for the FFN; sparse attention applies the same idea to attention.
Sparse MoE models7. Mixture of Experts (MoE) →: MoE models where only a small subset of experts runs per token, for example 8 of 256. Most of the largest open-weight models are now sparse MoEs.
Special tokens10. Tool calling & protocols →: Reserved vocabulary entries with structural meaning, such as begin and end of text, role markers and tool-call markers. Ordinary text never produces them.
Speculative decoding4. LLM inference internals →: A fast drafter proposes several tokens, and the large model checks them all in one forward pass, keeping the longest accepted run. The output distribution is unchanged, but each expensive pass can yield several tokens.
State recovery11. Context, memory & persistence →: Restoring an agent from its last checkpoint after a failure.
State snapshots11. Context, memory & persistence →: Point-in-time copies of the full state, useful for debugging, branching and rolling back.
Static batching5. Batching & scheduling →: Collect a fixed batch, run it to completion, then start the next. Simple, but short requests wait for the longest one and finished slots sit empty.
Stopping conditions3. Decoding & sampling →: The rules that end generation: an end-of-sequence token, a stop string, the maximum token count, or the end of a tool call.
Structured generation3. Decoding & sampling →: Producing output in a guaranteed shape, such as JSON, XML or a function call, rather than asking nicely. Usually built on constrained decoding driven by a grammar or schema.
Structured logging14. Observability & performance →: Logging machine-readable records, usually JSON with consistent fields, instead of free-form text.
Structured model outputs10. Tool calling & protocols →: Responses guaranteed to match a schema, through constrained decoding or through validation and retry.
Subagents9. Agent architecture & internals →: Agents started by another agent for a sub-task. Each gets its own context window and returns only a summary, which keeps the parent's context small.
Supervised fine-tuning (SFT)15. Model training & post-training →: Training on curated prompt and response pairs so a base model learns to follow instructions and a chat format.

T

Tail latency5. Batching & scheduling →: The latency of the slowest requests, measured at p95 or p99. Batching choices that improve the average can make the tail worse.
Tail latency14. Observability & performance →: High-percentile latency (p99, p99.9). In multi-step agents one slow step slows the whole run, so tails compound.
Targeted model intervention16. Weight-space programming & interpretability →: Changing activations or weights at one specific location, to test a causal hypothesis or to fix a behaviour.
Task completion rate14. Observability & performance →: The share of agent tasks that finish successfully, often the metric users care about most.
Task decomposition9. Agent architecture & internals →: Breaking a goal into smaller steps or sub-tasks that can be run, delegated or parallelised.
Task queues12. Backend & distributed systems →: Queues of units of work waiting for workers, which decouple accepting a request from executing it.
Task scheduling12. Backend & distributed systems →: Deciding when and where tasks run, including delays, priorities and recurring jobs.
Task-throughput optimization18. Advanced systems research questions →: Optimising completed tasks per unit of time or money, rather than tokens per second, as the real objective.
Temperature3. Decoding & sampling →: A number the logits are divided by before the softmax. Below 1 sharpens the distribution toward the top token, above 1 flattens it, and 0 is treated as greedy decoding.
Temperature scaling3. Decoding & sampling →: The operation logits ÷ T itself. It changes how far apart the token probabilities are without changing their order.
Temporal17. Infrastructure & development tools →: A durable execution platform. Workflows are written as ordinary code, their progress is recorded, and they survive crashes by resuming through deterministic replay.
Tensor inspection16. Weight-space programming & interpretability →: Looking at the values, shapes and distributions of weights or activations, often visually.
Tensor operations8. GPU & CUDA engineering →: Operations on multi-dimensional arrays: matrix multiplications, elementwise maths, reductions and reshapes. Frameworks build models from them and compilers fuse them.
Tensor parallelism6. Distributed LLM inference →: Splitting each weight matrix across GPUs so every layer runs on all of them at once, with an all-reduce after attention and the FFN. Low latency, but it needs very fast links such as NVLink, so it usually stays inside one server.
TensorRT-LLM17. Infrastructure & development tools →: NVIDIA's library for heavily optimised LLM inference on NVIDIA GPUs, with compiled kernels, FP8 and FP4, and in-flight batching.
Terraform17. Infrastructure & development tools →: An infrastructure-as-code tool that declares cloud resources in configuration files and applies changes reproducibly.
Testcontainers17. Infrastructure & development tools →: A library that starts real dependencies, such as databases and brokers, in throwaway containers during tests.
Throughput14. Observability & performance →: Work completed per unit of time: tokens per second, requests per second or tasks per hour.
Throughput-versus-latency trade-off5. Batching & scheduling →: Bigger batches serve more tokens per second in total but slow every individual stream. Every deployment picks a point on this curve.
Time per output token (TPOT)4. LLM inference internals →: The average time between output tokens after the first. Its inverse is the streaming speed of one request in tokens per second.
Time to first token14. Observability & performance →: TTFT as an observed metric, from request arrival to the first streamed token, ideally broken down into queueing and prefill.
Time to first token (TTFT)4. LLM inference internals →: The delay from sending a request to receiving the first output token: queueing plus prefill, whose final position already yields the first token. It is what users feel as responsiveness.
Timeouts13. Reliability, security & sandboxing →: Upper limits on how long an operation may run before it is abandoned, applied to model calls, tools and whole runs.
Token acceptance4. LLM inference internals →: The verification step of speculative decoding, and the rate at which drafted tokens are accepted. Higher acceptance means more tokens per expensive forward pass.
Token budgets11. Context, memory & persistence →: Allocations of the window to each part (instructions, tools, history, retrieval, output), enforced when the context is assembled.
Token embeddings1. LLM fundamentals →: Learned vectors, one per vocabulary entry. Looking up a token ID returns its row of the embedding matrix, which turns a discrete symbol into numbers the network can compute with.
Token IDs1. LLM fundamentals →: The integer index of each token in the vocabulary. A word like "Hello" might become 9906. The network only ever sees these numbers, never characters.
Token permutation7. Mixture of Experts (MoE) →: Reordering a batch so that tokens for the same expert sit next to each other, which allows efficient grouped matrix multiplications; it is reversed after the combine.
Token selection3. Decoding & sampling →: The last step of each decoding iteration: choose one token ID from the processed distribution, by argmax or by sampling.
Token usage14. Observability & performance →: Input, cached and output tokens per call, the main driver of LLM cost.
Tokenization1. LLM fundamentals →: Splitting raw text into tokens before the model sees it. Most LLMs use subword schemes such as byte-pair encoding (BPE), which keep frequent words whole and break rare words into reusable pieces.
Tokenizer1. LLM fundamentals →: The component that converts text to tokens and back. It is built before the model is trained and ships with the weights; swapping it breaks the model, because every token ID would suddenly mean something else.
Tokens1. LLM fundamentals →: The units a language model reads and writes. A token is usually a word, a piece of a word, a punctuation mark or a byte sequence. In English one token is on average about three quarters of a word.
Tokens per second14. Observability & performance →: Generation speed, either per stream (what one user sees) or summed across streams (what the server delivers).
Tool authorization10. Tool calling & protocols →: Deciding whether this agent, acting for this user, may call this tool with these arguments right now.
Tool choice10. Tool calling & protocols →: The API parameter that controls tool use: auto (the model decides), none, required, or one specific named tool.
Tool definitions10. Tool calling & protocols →: Declarations of the tools the model may use, passed with the request: name, description and parameter schema.
Tool descriptions10. Tool calling & protocols →: Plain-language text saying what a tool does and when to use it. The model chooses tools mostly from these.
Tool dispatch9. Agent architecture & internals →: Routing a tool call to its implementation, whether a local function, an MCP server, an HTTP API or a sandboxed process, possibly several in parallel.
Tool execution10. Tool calling & protocols →: Running the tool's implementation, ideally in a sandbox with timeouts and resource limits.
Tool invocation9. Agent architecture & internals →: Actually calling the chosen tool with the arguments the model produced.
Tool isolation13. Reliability, security & sandboxing →: Running each tool so that a failure or compromise cannot affect the agent runtime, other tools or other users.
Tool permissions10. Tool calling & protocols →: The policy of which tools and scopes an agent has been granted, ideally the least it needs.
Tool result handling9. Agent architecture & internals →: Formatting a tool's output, or its error, back into the context, often trimming or summarising large results first.
Tool routing tokens10. Tool calling & protocols →: Tokens or markers that say which tool or output channel the following content belongs to, so the parser can route it.
Tool schemas10. Tool calling & protocols →: The JSON Schema for a tool's parameters: types, required fields, allowed values and field descriptions. Good schemas prevent bad calls.
Tool selection9. Agent architecture & internals →: The model choosing which tool to call, or whether to call one at all, from the tool descriptions and the current context.
Tool-call errors10. Tool calling & protocols →: Failures from bad arguments, missing tools, timeouts or exceptions. Reporting them clearly lets the model correct itself on the next step.
Tool-call finish reasons10. Tool calling & protocols →: The field in a response that says why generation stopped, such as "tool_calls" (run the tools and call me back), "stop" or "length".
Tool-call tokens10. Tool calling & protocols →: Special tokens a model is trained to put around tool calls, which the engine uses to detect and parse them.
Tool-calling loop9. Agent architecture & internals →: The control loop built on function calling: the model emits tool calls, the runtime runs them and returns the results as messages.
Tool-use training15. Model training & post-training →: Fine-tuning on trajectories that contain tool calls and results, and reinforcement learning in tool environments, so models learn when and how to call tools.
Top-2 routing7. Mixture of Experts (MoE) →: Top-k with k = 2, used by Mixtral and many early MoEs. Each token's output is the weighted sum of two experts.
Top-k expert selection7. Mixture of Experts (MoE) →: Choosing the k experts with the highest scores for each token. k is small, often between 1 and 8, however many experts the layer has.
Top-k sampling3. Decoding & sampling →: Keep only the k most likely tokens, renormalise and sample among them. It cuts off the long tail of unlikely tokens, but uses the same k whether the model is confident or unsure.
Top-p / nucleus sampling3. Decoding & sampling →: Keep the smallest set of top tokens whose probabilities add up to p (say 0.9), then sample. The set shrinks when the model is confident and grows when it is unsure.
Total parameters7. Mixture of Experts (MoE) →: All parameters, every expert included. They drive how much memory is needed to hold the model, even though most sit idle for any one token.
Trace correlation14. Observability & performance →: Linking logs, metrics and spans through shared trace and span IDs, so one request can be followed everywhere.
Training datasets15. Model training & post-training →: The text, code and trajectories used for training. Filtering, deduplication and mixing ratios shape the result as much as size does.
Training examples15. Model training & post-training →: The individual inputs and targets, or whole trajectories, a model learns from. In post-training, quality and diversity matter more than sheer count.
Training feedback15. Model training & post-training →: The signals used to improve a model: human preferences, reward models, verifiers or the results of running code.
Transactional outbox12. Backend & distributed systems →: Writing a database change and the event announcing it in the same transaction, then publishing from that outbox table, so the data and the events can never disagree.
Transformer blocks2. Transformer architecture →: The repeated unit of the network, often just called layers. A modern pre-norm block runs RMSNorm, attention, a residual add, RMSNorm, FFN and another residual add. Models stack anywhere from about 24 to over 100 of them.
Triton17. Infrastructure & development tools →: The Python-like GPU kernel language and compiler used for custom fused kernels in PyTorch and inference engines (not to be confused with NVIDIA's Triton Inference Server).
Triton language8. GPU & CUDA engineering →: OpenAI's Python-based language and compiler for writing GPU kernels at the level of tiles. It handles much of the low-level scheduling, so custom fused kernels take far less code than CUDA C++.
TypeScript17. Infrastructure & development tools →: JavaScript with static types, the default for modern web front ends and many agent frameworks.

V

Value (V)2. Transformer architecture →: The vector a token hands over when it is attended to: the information that actually moves. The attention output is a weighted sum of values.
vLLM17. Infrastructure & development tools →: A popular open-source LLM inference engine that introduced PagedAttention, with continuous batching, prefix caching and an OpenAI-compatible server.
Vocabulary1. LLM fundamentals →: The fixed list of every token the tokenizer can produce, including special tokens. Each entry has exactly one ID and one row in the embedding table.
Vocabulary size1. LLM fundamentals →: How many entries the vocabulary has, typically 32K to 256K in current models. A larger vocabulary packs text into fewer tokens, especially code and non-English text, but grows the embedding table and the LM head.
VRAM8. GPU & CUDA engineering →: The common name for GPU memory capacity, such as 80 GB on an H100 SXM or 141 GB on an H200.

W

Weight matrices1. LLM fundamentals →: The 2D arrays that hold most parameters: query, key, value and output projections, and the FFN up, gate and down projections. Nearly all compute in an LLM is multiplying activations by them.
Weight matrices16. Weight-space programming & interpretability →: The 2D parameter arrays of each layer. Interpretability treats them as maps that read from and write to directions in the residual stream.
Weight reuse5. Batching & scheduling →: Using each weight loaded from memory for many tokens. That reuse is exactly what batching buys.
Weight-space editing16. Weight-space programming & interpretability →: Targeted modification of specific weights, ideally changing one behaviour and leaving everything else intact.
Weight-space programming16. Weight-space programming & interpretability →: Changing a model's behaviour by writing to its weights directly, as an alternative to prompting or to fine-tuning on data.
Weight-space reading16. Weight-space programming & interpretability →: Analysing weights directly, without running inputs, to find structure such as dominant directions or what a neuron reads and writes.
Weight-space writing16. Weight-space programming & interpretability →: Editing parameter values to install, remove or change a behaviour.
Work distribution12. Backend & distributed systems →: Spreading tasks across workers fairly and efficiently.
Work stealing12. Backend & distributed systems →: Idle workers taking queued tasks from busy ones to even out the load.
Worker pools12. Backend & distributed systems →: Groups of workers sized to demand, sometimes specialised by resource such as CPU, GPU or sandbox slots.
Worker processes12. Backend & distributed systems →: Processes that pull work from a queue and execute it, scaled separately from the API.
Workspace isolation13. Reliability, security & sandboxing →: Giving each agent session its own working directory or volume so tasks cannot read or damage each other's files.

X

xUnit17. Infrastructure & development tools →: A popular unit-testing framework for .NET.

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

T

V

W

X

Search the atlas