I gave a tiny language model a lobotomy: neuron ablation in the browser

By Lucas Espindola · November 9, 2026 · 9 min read

Mechanistic interpretability · neuron ablation · side project

In the last layer of an eight-million-parameter storyteller there is a neuron that waits for a girl to walk on stage. It lights up on “She”, “she”, “Lily” and “mom”. Cut out the 512 neurons of the model's first layer instead, and it starts a sentence with “a a a the his little sad very” and then writes “good” until the page runs out. This is Neuron Dissector: a small language model opened up in your browser, every neuron on screen as it writes, and a scalpel for the basic moves of mechanistic interpretability: ablation, activation steering and the logit lens.

Open it here, or type app dissect in the terminal on this site's desktop. Interpretability work usually happens in Python notebooks, with libraries like TransformerLens; this is the same kind of poking around in a browser tab, with nothing to install. Everything below happened in it, and you can do it again.

The specimen

The model is TinyStories (Eldan & Li, 2023), a family of tiny GPT-Neo models trained on short, synthetic children's stories that only use words a three- or four-year-old would know. A model this small can still write a coherent story because its world is small. The default specimen, TinyStories-3M, has 8.3 million parameters, and 6.4 million of them are just its vocabulary of 50,257 tokens. The rest is eight layers, each with 16 attention heads and 512 MLP neurons: 4,096 neurons and 128 heads in all, few enough to fit on one screen, each drawn as a cell that glows when it fires.

A neuron that waits for a girl

Neuron Dissector with the prompt 'Once upon a time, there was a little girl named Lily. She'. The scan shows eight columns of glowing neurons, one per layer, with 16 attention heads as dots above each. Neuron 117 of layer 7 is selected: its activation along the text spikes on each 'She', 'she', 'Lily' and 'mom', which are highlighted in the story below. The next-token panel after 'She' shows took 11%, decided 9%, promised 8%.

Above: the scan at the token “ She”. Each column is a layer; the dots on top are its attention heads, the grid is its MLP neurons. The selected neuron, layer 7 #117, lights the words it fires on in the story.

Start the model on “Once upon a time, there was a little girl named Lily. She” and stop the playhead on the last word. The most active neuron in the whole network at that token is #117 in the last layer. Select it and the story lights up wherever it fires: “She”, “she”, the “ily” of Lily, “my”, “mom”. It's the closest thing in this model to a “the girl is the subject” detector, at least in this story. After “She”, the model's top guesses for the next word are “took” (11%), “decided” (9%) and “promised” (8%).

A neuron can take on several roles, a property called polysemanticity. The words that light up #117 here give me a lead to investigate across more stories; this one story shows only part of what it responds to.

A lobotomy

The first layer's MLP is cut: its whole column is covered in red crosses. The original story continues normally, while the dissected one reads 'She a a a the his little sad very good good good good', with 'good' repeated to the end. The next-token odds after 'sad' put 'sad' at 38%.

The scalpel cuts a neuron by multiplying its activation by zero on every token. In interpretability terms that's zero-ablation, the simplest way to ask whether a part of a model matters: take it out and see what breaks. Ablate all 512 neurons in layer 0, the first MLP the words pass through, and the model loses the ability to write. Same prompt, same random numbers, and the dissected story goes: “She a a a the his little sad very good good good good good…” with “good” all the way to the end, while the original next to it loves to play in her garden.

A random lesion is gentler. In one run, cutting a random 10% of all neurons gave a Lily who “was a very anxious child. She was so shy and she would run and she ran, and so he ran and ran around trees. He ran, and he ran…”: still English, with the plot gone and a loop where it used to be.

Hunting the dog

Now a different story: “Tom and his dog Max went to the park. Max saw a”. Left alone, the model finishes it sweetly: “big tree and Max started to play. Max won the game and Max was happy. Max was not sad anymore. They played and had fun together.”

The inspector can search for a word: it ranks every neuron by how much more it fires on that word's tokens than on the rest of the text. For “dog”, the top three are layer 5 #390, layer 3 #78 and layer 3 #5, and all of them also fire on “Max”, who is the dog. That ranking measures correlation within this story. Cutting the top ten with the scalpel, one click each, leaves the story with a Max who doesn't quite know what he is: Max “saw a big tree and went up to it. He used his long, thin legs to hold the tree from the tree.” A few lines later a girl called Lily, who was never in the prompt, “picked up the tree and looked at the big tree. She saw a tiny, cozy nest.”

Ten neurons that fire on 'dog' are cut, marked with red crosses across layers 0, 2, 3, 5 and 7. The dissected story diverges at token 16: Max goes up to the tree, uses his long, thin legs to hold it, and later Lily picks up the tree and finds a nest.

The palace neuron

Layer 3 neuron #5 is selected and its dial turned to ×8. It fires most on dog, Max, tree and park; reading its weights, it pushes the next token toward court, axe, church, tending, temple, hunt, guard and palace. The dissected story reads: Max saw a lot of junk that their and Timmy and he was sad. They were very dirty, but they were very dirty… and they played in the mud, the mud and the mud.

Every neuron also gets a reading of what it writes: its output weights, projected straight through the model's unembedding onto its vocabulary, give the tokens it pushes up and the ones it pushes down. It's a per-neuron logit lens, or direct logit attribution. Layer 3 #5, one of the dog neurons, pushes toward “court”, “axe”, “church”, “temple”, “hunt”, “guard” and “palace”. A neuron that fires on dogs and talks about guards and palaces sounded like a story waiting to happen, so I turned its dial to ×8, a crude kind of activation steering.

There were no palaces. Max “saw a lot of junk”, then everyone “were very dirty, but they were very dirty, but they were very dirty”, and the story ended with Tom, Max and his friends playing “in the mud, the mud and the mud and the mud.”

That's the most useful lesson the tool has taught me. What a middle-layer neuron writes straight to the output, its direct effect, is a small part of what it does; most of its effect goes through the four layers after it, which read what it wrote into the residual stream and do something else with it. The palace reading was real, and it was the wrong question.

How it works

No runtime will hand you the scalpel

Running a model in the browser is solved: Doorman, on this same desktop, runs its model with transformers.js and ONNX Runtime. But those run a compiled graph, tokens in and logits out, and there is nowhere to stand between GELU and the next matrix multiply. To change neuron 117 mid-run I needed the forward pass in my own hands, so I wrote one: token and position embeddings, LayerNorm, attention with a key-value cache, the GELU MLP and the unembedding, in plain JavaScript in a Web Worker so the page stays smooth. It writes about 60 tokens a second on a laptop, edits included.

GPT-Neo has two quirks you only find by getting them wrong: it doesn't scale attention scores by 1/√d like most transformers, and its layers alternate between global attention and local attention over the last 256 tokens.

Reading a PyTorch checkpoint in a browser

The weights come straight from Hugging Face, as the pytorch_model.bin the authors uploaded. Since PyTorch 1.6 that file is a zip archive: one raw file per tensor storage, plus data.pkl, a Python pickle that says which storage holds which tensor, at what offset and in what shape. The archive is stored without compression, so reading it is a walk through its central directory. The pickle needs a small interpreter: this checkpoint uses 20 of pickle's opcodes, and the unpickler that runs them is 66 lines.

One surprise: half of TinyStories-3M's 67 MB download isn't weights. Each of the eight layers saved its causal attention mask with the model, a 2,048 × 2,048 grid of booleans, 4 MB apiece. The worker copies out the 33 MB of real weights and lets the masks go, and the browser keeps the download, so the next visit opens in seconds.

Where the scalpel goes

Diagram of one of the eight layers. The residual stream runs along the bottom. The attention block reads from it; each of its 16 heads' outputs is multiplied by a gain before being added back, and one head is shown cut, another doubled. The MLP block reads from the stream too: c_fc, GELU, activations a1 to a512, c_proj. Each activation right after GELU can be cut (a = 0), boosted (a × 4) or clamped to a fixed value (a = c), on every token including the prompt. Then seven more layers, ln_f, the unembedding and the next-token odds.

There are two kinds of edit. An MLP neuron's activation, right after GELU, can be multiplied by any gain from −4 to 8 (0 cuts it, negative inverts it) or clamped to a fixed value. An attention head's output can be multiplied by a gain before it's mixed back into the stream. Every edit applies on every token, the prompt included, so the dissected model reads the prompt differently too, not only what it writes next.

The same dice

The original and the dissected model run side by side, one token at a time, and both draw from the same seeded random number generator, one draw per token. While their odds agree they pick the same word, so the two stories only part where an edit shifts the odds enough to tip a draw. The timeline under the scan marks that point with a red line. Set the temperature to 0 and both models always take their top choice.

The ripple

Ripple view of the lobotomy at token 20: every neuron in layers 1 to 7 is lit pink by how far its activation moved compared with the original model; layer 0, the cut layer, is covered in red crosses. The timeline below shows pink bars rising after the point where the text diverges.

Above: the ripple view, lit by how far each neuron moved: the lobotomy, twenty tokens in.

Switch the scan from activity to ripple and each neuron glows by how far your edit moved it from the original model at that token. It's the same clean-versus-intervened comparison that causal methods like activation patching are built on. Before the stories part, the ripple is the edit's own effect spreading through the layers after it. After they part it's everything, because the two models are now reading different words.

Drawing 4,096 neurons

The scan is a 2D canvas redrawn every frame. Neurons are grouped into 64 brightness levels, plus 64 for the few that dip below zero, so a layer takes at most 128 fills instead of 512. The brightest neurons get a faint halo: they're drawn again into a canvas a quarter the size, which is stretched back over the scan with additive blending. When a new token arrives, a thin line sweeps from the first layer to the last, and each column switches to the new token's activations as it passes.

I turned up a neuron that pointed toward “palace” and got a story about mud. That’s the experiment I keep coming back to. The output weights gave me a plausible prediction; running the intervention gave me something else.

There are 4,096 neurons here to try. Pick a story, change one, and see where it goes.