Neuron Dissector: watch a language model’s neurons fire, then cut them

Neuron Dissector opens up a small story-writing language model (TinyStories, GPT-Neo), or any GPT-Neo or GPT-2 model you pick from Hugging Face, in your browser. It shows every MLP neuron and attention head firing token by token, traces a neuron’s circuit (the earlier neurons, heads and tokens whose outputs add up to its activation), and lets you cut, boost, invert or clamp them and compare the edited model’s story with the original’s.

How it works

  1. The model writes a story from your prompt while the scan shows every layer: its attention heads and MLP neurons, lit by how hard they fire on each token.
  2. Click a neuron to see where it fires along the text and the words it pushes toward, then cut, boost, invert or clamp it, or drag a scalpel across the scan.
  3. Trace what drives it: a graph of the earlier neurons, the attention heads (with the token each copied from) and the token itself whose outputs add up to the neuron’s activation at that token, each line its share. Cut a unit from the graph and the circuit is traced again in the edited story.
  4. The model writes the story again beside the original, from the same random numbers, so only your edits make the two differ; the places where they part are marked.

Privacy and limits

  • Everything runs in your browser: prompts and edits stay on your device unless you share a recipe link.
  • The weights download once from Hugging Face and your browser keeps them.
  • The forward pass is written from scratch in JavaScript, with a WebAssembly SIMD kernel for its matrix multiplies, so any neuron can be read and rewritten; transformers.js supplies the tokenizer.
  • A tiny model: its stories are simple, and one neuron seldom means one thing.
  • A circuit is exact along direct paths: with each LayerNorm’s scale and every attention pattern held as they were, its parts add up to the neuron’s input. It doesn’t show a unit changing where a later head looks.

Models

TinyStories (Eldan & Li 2023)

More

How Neuron Dissector works

Neuron Dissector runs on this site’s desktop, which needs JavaScript: with it on, this address opens the app. In the desktop’s terminal it’s app dissect. This page as Markdown.

Finder