Topic 8 · Serving at scale
GPU & CUDA engineering
The hardware realities that decide how fast a model can run.
A GPU can do roughly a thousand trillion half-precision operations per second, but it can only read a few trillion bytes per second from its main memory. Whether a workload is limited by the maths or by the memory depends on how many operations it does per byte it reads.
That ratio, arithmetic intensity, explains why batch-1 decode leaves most of the chip idle and why batching, fusion and quantization help so much. Kernel engineering is largely the craft of moving fewer bytes and keeping data close to the compute.
In this topic
29
terms, each defined below
The roofline
Attainable speed for an H100-class GPU (about 990 dense BF16 TFLOPS, 3.35 TB/s of memory bandwidth; rounded figures). Drag the batch size and watch decode climb the sloped, memory-bound part of the roof until it reaches the flat compute ceiling. Below: the memory hierarchy, from tiny and fast to huge and slow.
- Intensity
- ≈ 1 FLOP/B
- Regime
- memory-bound
- Attainable
- 3.4 TFLOPS
- Of peak
- 0.34%
- 70B model, per stream
- 24 tok/s
- 70B model, total
- 23.9 tok/s
Idealised: bf16 weights read once per step, KV-cache traffic and overheads ignored. Real engines land below the roof, but the shape holds: batching is nearly free until the ridge.
GPU memory hierarchy (H100-class, approximate)
- ~256 KB per SMfastest, per threadRegisters
- up to ~228 KB per SMon-chip SRAMShared memory / L1
- ~50 MBon-chip, sharedL2 cache
- 80 GB~3.35 TB/sHBM (VRAM)
- hundreds of GB+~64 GB/sHost RAM over PCIe
Faster and smaller at the top; bigger and slower at the bottom.
Glossary
29 terms
GPU computing
Using graphics processors for general parallel computing. Thousands of simple cores plus specialised tensor cores make them ideal for neural-network matrix maths.
NVIDIA GPU
The dominant accelerator family for AI (A100, H100, H200, B200 and successors), combining tensor cores, high-bandwidth memory and NVLink with the CUDA software ecosystem.
CUDA
NVIDIA's platform and programming model for its GPUs: compiler, runtime, driver API and libraries such as cuBLAS and cuDNN. Most AI frameworks run on top of it.
CUDA C++
The C++ dialect for writing CUDA kernels, with extensions for launching grids of threads and managing GPU memory.
Triton language
OpenAI's Python-based language and compiler for writing GPU kernels at the level of tiles. It handles much of the low-level scheduling, so custom fused kernels take far less code than CUDA C++.
GPU kernels
Functions that run on the GPU across many threads in parallel, such as a matrix multiplication, a softmax or a fused attention kernel.
Kernel launches
The CPU asking the GPU to start a kernel. Each launch costs a few microseconds, which adds up over thousands of small kernels per token; CUDA graphs record and replay them to cut the cost.
Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or shared memory instead of round-tripping through main memory. FlashAttention is the best-known example.
Fused sampling
Doing temperature, top-k or top-p filtering and the random draw in one GPU kernel, avoiding a full-vocabulary sort and extra trips to the CPU.
GPU memory
Memory attached to the GPU. It must hold the weights, KV cache, activations and workspace, and running out of it is the most common serving limit.
VRAM
The common name for GPU memory capacity, such as 80 GB on an H100 SXM or 141 GB on an H200.
GPU memory hierarchy
The levels from registers (fastest, tiny) through shared memory and L1 per streaming multiprocessor, the shared L2 cache and HBM, out to host RAM over PCIe. Fast kernels keep data as high in the hierarchy as they can.
Memory bandwidth
How many bytes per second can move between HBM and the compute units, about 3.35 TB/s on an H100 SXM. It bounds decode speed: tokens per second is roughly bandwidth divided by the bytes read per token.
Memory traffic
The total bytes a kernel reads and writes. Cutting it through fusion, quantization or smaller KV caches is often the most effective optimisation.
Memory locality
Accessing data that is nearby or recently used, so caches hit and loads coalesce. Poor locality wastes bandwidth.
GPU compute utilization
The share of peak FLOPs actually achieved. Model FLOPs utilisation of 40 to 60 percent is good for training; batch-1 decode reaches far less.
Memory utilization
Either the share of memory capacity in use or the share of peak bandwidth achieved. Serving engines deliberately reserve most free memory for KV cache, so high capacity use is normal.
CPU–GPU data transfers
Copies between host and device over PCIe or NVLink-C2C. They are slow compared with HBM, so engines keep them out of the hot loop; cache offloading uses them on purpose.
Tensor operations
Operations on multi-dimensional arrays: matrix multiplications, elementwise maths, reductions and reshapes. Frameworks build models from them and compilers fuse them.
Matrix-vector multiplication
Multiplying a matrix by a single vector (GEMV). Batch-1 decode is effectively a chain of these, which is why it is bandwidth-bound.
Matrix-matrix multiplication
Multiplying two matrices (GEMM). Batching and prefill turn GEMVs into GEMMs, which keep tensor cores busy.
Parallel execution
Running work simultaneously at every level: threads in a warp, warps on a multiprocessor, multiprocessors on a GPU, GPUs in a cluster, and communication overlapped with compute.
GPU profiling
Measuring where GPU time goes with tools such as Nsight Systems and the PyTorch profiler, to find stalls, idle gaps and slow kernels.
Kernel profiling
Inspecting a single kernel in depth with Nsight Compute: achieved bandwidth, occupancy, tensor-core use and the reasons it stalls.
Accelerator clusters
Many accelerators networked together, thousands of GPUs or TPUs, for training and large-scale inference.
NVIDIA H100
NVIDIA's Hopper-generation data-centre GPU from 2022: 80 GB of HBM3, about 3.35 TB/s of bandwidth, roughly 1 PFLOP of dense BF16 and FP8 support. The reference chip for most LLM work of 2023 to 2025.
AWS Trainium
Amazon's own AI accelerator chips for training and inference on AWS, programmed through the Neuron SDK.
Google TPU
Google's Tensor Processing Units: systolic-array accelerators linked into large pods and programmed through JAX and XLA. Gemini and Gemma are trained on them.
Hardware-specific optimization
Tuning kernels, number formats and parallelism for a particular chip, such as FP8 on Hopper, FP4 on Blackwell, or tile sizes matched to a multiprocessor's shared memory.