Mechanistic Interpretability

A working neural network is legible to no one. We can train it, deploy it, and measure its outputs — but the hundreds of billions of weights inside it are an alien artifact, a frozen pattern of numbers that performs computations we cannot read. Mechanistic interpretability is the research programme that takes this seriously: rather than treating the model as a black box to be probed only from the outside by its behaviour, it tries to reverse-engineer the trained network into human-understandable parts — the features it represents and the circuits that combine them into computation. The goal is not just to predict what a model will do, but to reverse-engineer the algorithm it has learned and understand its internal workings the way one might understand a piece of human-written software — line by line, mechanism by mechanism, until the behaviour stops being a surprise.

The early wins were in vision. Chris Olah and collaborators showed that early CNN layers learn edge detectors, middle layers learn textures and parts, deep layers learn object detectors — and that these features are connected by interpretable circuits that compute recognizable algorithms (curve detection, dog-head detection). Crucially, a circuit can be tested: ablate or amplify its parts and the behaviour shifts as the story predicts, which turns a plausible narrative into a causal claim rather than a just-so explanation. The frontier then moved to transformers, where researchers found small reusable mechanisms such as induction heads — paired attention components that notice a token has appeared before and copy what followed it, a simple in-context copying rule that underlies much of a model's startling ability to learn from its own prompt. But neuron-by-neuron reading kept failing, because most neurons are polysemantic: a single unit fires for many unrelated concepts at once. The deep cause is superposition: a network packs more features than it has dimensions by storing them as overlapping linear combinations, trading a little interference for far greater representational capacity. The response has been sparse dictionary learning — training sparse autoencoders that decompose tangled activations into a large vocabulary of cleaner, monosemantic features, each one ideally meaning a single thing. Using them, work like Mapping the Mind of a Large Language Model extracted millions of human-interpretable features from a frontier model — features for landmarks, for code bugs, for sycophancy, for deception — and showed they could be turned up or down to steer behaviour. The toolkit (probing, ablation, activation patching, dictionary learning) improves rapidly; current understanding is best described as islands of clarity in a large unknown.