# 8. GPU & CUDA engineering

> The hardware realities that decide how fast a model can run.

Track: Serving at scale. 29 terms.

A GPU can do roughly a thousand trillion half-precision operations per second, but it can only read a few trillion bytes per second from its main memory. Whether a workload is limited by the maths or by the memory depends on how many operations it does per byte it reads.

That ratio, arithmetic intensity, explains why batch-1 decode leaves most of the chip idle and why batching, fusion and quantization help so much. Kernel engineering is largely the craft of moving fewer bytes and keeping data close to the compute.

**Interactive figure on the web page: The roofline.** Attainable speed for an H100-class GPU (about 990 dense BF16 TFLOPS, 3.35 TB/s of memory bandwidth; rounded figures). Drag the batch size and watch decode climb the sloped, memory-bound part of the roof until it reaches the flat compute ceiling. Below: the memory hierarchy, from tiny and fast to huge and slow.

## Terms

### GPU computing

Using graphics processors for general parallel computing. Thousands of simple cores plus specialised tensor cores make them ideal for neural-network matrix maths.

### NVIDIA GPU

The dominant accelerator family for AI (A100, H100, H200, B200 and successors), combining tensor cores, high-bandwidth memory and NVLink with the CUDA software ecosystem.

### CUDA

NVIDIA's platform and programming model for its GPUs: compiler, runtime, driver API and libraries such as cuBLAS and cuDNN. Most AI frameworks run on top of it.

See: [also in Infrastructure & development tools](https://llm-atlas-7or.pages.dev/learn/stack.md)

### CUDA C++

The C++ dialect for writing CUDA kernels, with extensions for launching grids of threads and managing GPU memory.

### Triton language

OpenAI's Python-based language and compiler for writing GPU kernels at the level of tiles. It handles much of the low-level scheduling, so custom fused kernels take far less code than CUDA C++.

### GPU kernels

Functions that run on the GPU across many threads in parallel, such as a matrix multiplication, a softmax or a fused attention kernel.

### Kernel launches

The CPU asking the GPU to start a kernel. Each launch costs a few microseconds, which adds up over thousands of small kernels per token; CUDA graphs record and replay them to cut the cost.

### Kernel fusion

Combining several operations into one kernel so intermediate results stay in registers or shared memory instead of round-tripping through main memory. FlashAttention is the best-known example.

### Fused sampling

Doing temperature, top-k or top-p filtering and the random draw in one GPU kernel, avoiding a full-vocabulary sort and extra trips to the CPU.

### GPU memory

Memory attached to the GPU. It must hold the weights, KV cache, activations and workspace, and running out of it is the most common serving limit.

### VRAM

The common name for GPU memory capacity, such as 80 GB on an H100 SXM or 141 GB on an H200.

### GPU memory hierarchy

The levels from registers (fastest, tiny) through shared memory and L1 per streaming multiprocessor, the shared L2 cache and HBM, out to host RAM over PCIe. Fast kernels keep data as high in the hierarchy as they can.

### Memory bandwidth

How many bytes per second can move between HBM and the compute units, about 3.35 TB/s on an H100 SXM. It bounds decode speed: tokens per second is roughly bandwidth divided by the bytes read per token.

### Memory traffic

The total bytes a kernel reads and writes. Cutting it through fusion, quantization or smaller KV caches is often the most effective optimisation.

### Memory locality

Accessing data that is nearby or recently used, so caches hit and loads coalesce. Poor locality wastes bandwidth.

### GPU compute utilization

The share of peak FLOPs actually achieved. Model FLOPs utilisation of 40 to 60 percent is good for training; batch-1 decode reaches far less.

### Memory utilization

Either the share of memory capacity in use or the share of peak bandwidth achieved. Serving engines deliberately reserve most free memory for KV cache, so high capacity use is normal.

### CPU–GPU data transfers

Copies between host and device over PCIe or NVLink-C2C. They are slow compared with HBM, so engines keep them out of the hot loop; cache offloading uses them on purpose.

### Tensor operations

Operations on multi-dimensional arrays: matrix multiplications, elementwise maths, reductions and reshapes. Frameworks build models from them and compilers fuse them.

### Matrix-vector multiplication

Multiplying a matrix by a single vector (GEMV). Batch-1 decode is effectively a chain of these, which is why it is bandwidth-bound.

### Matrix-matrix multiplication

Multiplying two matrices (GEMM). Batching and prefill turn GEMVs into GEMMs, which keep tensor cores busy.

### Parallel execution

Running work simultaneously at every level: threads in a warp, warps on a multiprocessor, multiprocessors on a GPU, GPUs in a cluster, and communication overlapped with compute.

### GPU profiling

Measuring where GPU time goes with tools such as Nsight Systems and the PyTorch profiler, to find stalls, idle gaps and slow kernels.

### Kernel profiling

Inspecting a single kernel in depth with Nsight Compute: achieved bandwidth, occupancy, tensor-core use and the reasons it stalls.

### Accelerator clusters

Many accelerators networked together, thousands of GPUs or TPUs, for training and large-scale inference.

### NVIDIA H100

NVIDIA's Hopper-generation data-centre GPU from 2022: 80 GB of HBM3, about 3.35 TB/s of bandwidth, roughly 1 PFLOP of dense BF16 and FP8 support. The reference chip for most LLM work of 2023 to 2025.

### AWS Trainium

Amazon's own AI accelerator chips for training and inference on AWS, programmed through the Neuron SDK.

### Google TPU

Google's Tensor Processing Units: systolic-array accelerators linked into large pods and programmed through JAX and XLA. Gemini and Gemma are trained on them.

### Hardware-specific optimization

Tuning kernels, number formats and parallelism for a particular chip, such as FP8 on Hopper, FP4 on Blackwell, or tile sizes matched to a multiprocessor's shared memory.

Web page: https://llm-atlas-7or.pages.dev/learn/gpu/

_Definitions written for LLM Atlas._