# 16. Weight-space programming & interpretability

> Reading and editing what a network has learned, directly in its weights.

Track: Training, tools & research. 24 terms.

A weight matrix is not an opaque blob. It can be decomposed into directions: things a layer reads from the residual stream and things it writes back, each with a strength. Singular value decomposition makes that structure explicit, and a handful of directions often carry most of a matrix's effect.

Once directions can be found, they can be changed. Weight-space programming edits parameters directly to install, remove or soften a behaviour, such as a repetitive reasoning loop, and then checks with regression evaluations that nothing else broke.

**Interactive figure on the web page: A matrix, one direction at a time.** A 16×16 weight matrix rebuilt from its singular components. The rank slider keeps only the strongest k directions; most of the structure comes back with very few. Click a singular value to ablate that direction and see exactly what it contributed, the basic move behind weight editing.

## Terms

### Weight-space programming

Changing a model's behaviour by writing to its weights directly, as an alternative to prompting or to fine-tuning on data.

### Weight-space reading

Analysing weights directly, without running inputs, to find structure such as dominant directions or what a neuron reads and writes.

### Weight-space writing

Editing parameter values to install, remove or change a behaviour.

### Weight-space editing

Targeted modification of specific weights, ideally changing one behaviour and leaving everything else intact.

### Direct weight modification

Changing parameter tensors by hand or by algorithm, without gradient training on a dataset.

### Neural network parameter inspection

Examining parameter tensors: their statistics, norms, spectra and how they connect components.

### Numerical weight edits

Precise arithmetic changes to weights, such as adding a rank-one update or scaling one direction.

### Weight matrices

The 2D parameter arrays of each layer. Interpretability treats them as maps that read from and write to directions in the residual stream.

See: [related](https://llm-atlas-7or.pages.dev/learn/fundamentals/#weight-matrices), [also in LLM fundamentals](https://llm-atlas-7or.pages.dev/learn/fundamentals.md)

### Tensor inspection

Looking at the values, shapes and distributions of weights or activations, often visually.

### Matrix properties

Features of a weight matrix such as rank, norm, condition number and singular-value spectrum, which hint at its role and stability.

### Activation patterns

Which neurons or directions fire for which inputs. Recording them over many prompts reveals features.

### Singular vectors

The input and output directions found by SVD. A matrix maps each right singular vector onto its left singular vector, scaled by the singular value.

### Singular value decomposition (SVD)

Factoring a matrix as W = UΣVᵀ: sets of orthogonal directions and the strength of each. Keeping only the top components gives the best possible low-rank approximation.

### Behavioral directions

Directions in activation space that line up with a behaviour, such as refusal, sentiment or truthfulness. Adding or removing them steers the model.

### MLP weight editing

Editing FFN weights, which partly act as key-value memories, to change stored associations, as in methods like ROME and MEMIT.

### Mechanistic interpretability

Reverse-engineering the algorithms a network implements, down to individual features and circuits.

### Model editing

Changing specific facts or behaviours of a trained model with as few side effects as possible.

### Targeted model intervention

Changing activations or weights at one specific location, to test a causal hypothesis or to fix a behaviour.

### Parameter-efficient modifications

Changing a model by touching only a few parameters, such as LoRA adapters, rank-one edits or bias tweaks.

### Failure-mode localization

Finding which layers, heads or directions cause a particular failure, so it can be fixed precisely.

### Repetitive reasoning loops

A failure mode where a reasoning model keeps repeating the same thoughts or tokens without finishing. It can be traced to specific directions and reduced by editing them or by changing decoding settings.

See: [related](https://llm-atlas-7or.pages.dev/learn/decoding/#repetition-penalty)

### Automated neural network engineering

Using tools, often AI agents, to inspect, modify and test networks automatically rather than by hand.

### MATH-500 benchmark

A 500-problem subset of the MATH competition dataset, widely used to measure mathematical reasoning and a handy check that an edit did not hurt capability.

### Model regression evaluation

Checking that an edited or updated model has not lost performance on tasks it used to handle.

Web page: https://llm-atlas-7or.pages.dev/learn/interpretability/

_Definitions written for LLM Atlas._