Topic 16 · Training, tools & research
Weight-space programming & interpretability
Reading and editing what a network has learned, directly in its weights.
A weight matrix is not an opaque blob. It can be decomposed into directions: things a layer reads from the residual stream and things it writes back, each with a strength. Singular value decomposition makes that structure explicit, and a handful of directions often carry most of a matrix's effect.
Once directions can be found, they can be changed. Weight-space programming edits parameters directly to install, remove or soften a behaviour, such as a repetitive reasoning loop, and then checks with regression evaluations that nothing else broke.
In this topic
24
terms, each defined below
A matrix, one direction at a time
A 16×16 weight matrix rebuilt from its singular components. The rank slider keeps only the strongest k directions; most of the structure comes back with very few. Click a singular value to ablate that direction and see exactly what it contributed, the basic move behind weight editing.
- Energy kept
- 88.0%
- Relative error
- 34.6%
- Numbers stored
- 99 instead of 256 (directions × (16 + 16 + 1))
Singular values σ₁…σ₁₆ (click one to ablate that direction)
Each σ is the strength of one rank-one piece σ·u·vᵀ: a direction the layer reads (v) and the direction it writes (u). The first few hold most of the energy. Removing one exactly subtracts that piece, which is what a targeted weight edit does.
Glossary
24 terms
Weight-space programming
Changing a model's behaviour by writing to its weights directly, as an alternative to prompting or to fine-tuning on data.
Weight-space reading
Analysing weights directly, without running inputs, to find structure such as dominant directions or what a neuron reads and writes.
Weight-space writing
Editing parameter values to install, remove or change a behaviour.
Weight-space editing
Targeted modification of specific weights, ideally changing one behaviour and leaving everything else intact.
Direct weight modification
Changing parameter tensors by hand or by algorithm, without gradient training on a dataset.
Neural network parameter inspection
Examining parameter tensors: their statistics, norms, spectra and how they connect components.
Numerical weight edits
Precise arithmetic changes to weights, such as adding a rank-one update or scaling one direction.
Weight matrices
The 2D parameter arrays of each layer. Interpretability treats them as maps that read from and write to directions in the residual stream.
Tensor inspection
Looking at the values, shapes and distributions of weights or activations, often visually.
Matrix properties
Features of a weight matrix such as rank, norm, condition number and singular-value spectrum, which hint at its role and stability.
Activation patterns
Which neurons or directions fire for which inputs. Recording them over many prompts reveals features.
Singular vectors
The input and output directions found by SVD. A matrix maps each right singular vector onto its left singular vector, scaled by the singular value.
Singular value decomposition (SVD)
Factoring a matrix as W = UΣVᵀ: sets of orthogonal directions and the strength of each. Keeping only the top components gives the best possible low-rank approximation.
Behavioral directions
Directions in activation space that line up with a behaviour, such as refusal, sentiment or truthfulness. Adding or removing them steers the model.
MLP weight editing
Editing FFN weights, which partly act as key-value memories, to change stored associations, as in methods like ROME and MEMIT.
Mechanistic interpretability
Reverse-engineering the algorithms a network implements, down to individual features and circuits.
Model editing
Changing specific facts or behaviours of a trained model with as few side effects as possible.
Targeted model intervention
Changing activations or weights at one specific location, to test a causal hypothesis or to fix a behaviour.
Parameter-efficient modifications
Changing a model by touching only a few parameters, such as LoRA adapters, rank-one edits or bias tweaks.
Failure-mode localization
Finding which layers, heads or directions cause a particular failure, so it can be fixed precisely.
Repetitive reasoning loops
A failure mode where a reasoning model keeps repeating the same thoughts or tokens without finishing. It can be traced to specific directions and reduced by editing them or by changing decoding settings.
Automated neural network engineering
Using tools, often AI agents, to inspect, modify and test networks automatically rather than by hand.
MATH-500 benchmark
A 500-problem subset of the MATH competition dataset, widely used to measure mathematical reasoning and a handy check that an edit did not hurt capability.
Model regression evaluation
Checking that an edited or updated model has not lost performance on tasks it used to handle.