- AI, But Simple
- Posts
- Sparse Autoencoders (SAEs), Simply Explained
Sparse Autoencoders (SAEs), Simply Explained
AI, But Simple Issue #112
Edwin Dong & Lalit Julapalli
July 27, 2026
Sparse Autoencoders (SAEs), Simply Explained
AI, But Simple Issue #112
Blackbox. That’s a term that’s floated around AI research for the last few years.
From the moment you submit your query to the LLM, researchers have been investigating the journey your words take through the neural network in order to understand why a certain output comes out the other side.
Our current architectures are defined by matrix operations that correspond to weights, biases, and probabilities but are semantically hard to interpret.
Take a picture of a dog, for example: with a simple CNN, we’ll be able to determine whether an image depicts a dog or not, but it’s not immediately apparent from the output logits that a dog is what’s identified.
Over the years, research has moved from trying to analyze the outputs to inside the network’s layers. This research area is called Mechanistic Interpretability.
Researchers try to reverse engineer a way to get model weights to try and see what happens to models at the neuron level.
However, they faced a huge issue: superposition, which is when a network tries to identify more features than it has neurons, causing them to overlap in what their activation weights correspond to.
The attribute of when neurons represent multiple things at the same time is polysemanticity, which complicates mechanistic interpretability further.
Trying to decipher the tangled neuron activations is the job of the Sparse Autoencoder (SAE).
We’ll provide a visual breakdown of what Sparse Autoencoders are and how they’re used in mechanistic interpretability research.
What You'll Learn
The SAE architecture
Polysemanticity and superposition, and why forcing sparsity is the solution.
The training process behind the SAE
Current SAE research and open problems
What's Helpful to Know
Cosine Similarity
A alignment metric between two vectors, normalized for length, between [-1, 1].
Permutation Matrix
A matrix that reorders positions of entries in a vector without modifying their values.
SAE’s at a Glance
The Sparse Autoencoder is a special neural network architecture that relies on two core mechanisms: expansion and flattening.
Let’s say a neural network outputs a 512-dimension vector of activations.
Each of these vector entries may represent a feature wholly and/or may contribute to another feature partially, corresponding to the principle of polysemanticity.
The 512-dimension vector is then expanded to a greater number of dimensions through matrix multiplication with an Encoder Matrix (for instance, with size 512 x 16384), holding the possible features we can observe.
Continuing the ReLU activation function, we add a bias and bring resultant negative vector entries to 0.
The “sparseness” of the vector is now revealed, as many entries lay at zero while the core feature activations have significant magnitudes (imagine 20 nonzero entries).
The encoder matrix acts as a template for the activation vector to relate to, with the strongest ones surviving.
From here, the encoded, sparse vector is given to a decoder (16384 x 512) matrix, which essentially tries to reconstruct/flatten the original activation vector.
Put simply, it reads the 16384-dim vector and the “activated” features and decides what the 512-dim vector could potentially look like through matrix multiplication.
At first, this seems counterintuitive. When we use our encoder matrix, wasn’t the point to receive an interpretable vector of features we could point to?
Why would we ever want the decoder matrix to turn it back to the messy 512-dim vector?
The answer is that researchers only ever look at the intermediate, “expanded” vector.
The decoder’s purpose is to test whether the encoder matrix was actually laying out features that correspond to the original activation vector.
If the decoder fails to reach a permissible reconstruction, both matrices are updated by a composite loss.
The two parts of this function are the reconstruction loss, typically calculated by taking the Mean Squared Error (MSE) between the reconstructed vector and the original vector (both 512-d).
The penalized loss represents a regularization loss, where the model adds up all the entries in the feature vector.
The SAE wants as many values to be 0 as possible, enhancing our understanding of the features present.
The penalized loss essentially acts as a punishment for the model trying to present too many things (directly proportional to composite loss). The lambda is just a slider which enhances or diminishes the punishment.
Therefore, depending on this loss, both the encoder and decoder are updated to find the admissible threshold for the reconstructions.
It’s a cooperative game, much like coaching a game of catch. If one drops the ball, the coach (our loss function) instructs them both to make a change.
This process of training is the conventional way of training what we know as the standard SAE.
If you enjoy our content and custom visuals, consider sharing this newsletter with others or upgrading so we can keep doing what we do.
Get full access with the PRO newsletter tier:
AI Bootcamp in Your Inbox (Hands-on Code tutorials, Practice Examples + Solutions)
Interview Questions and Cases (Tracked from 100+ Companies)
Full Article Access: More Visuals, More Explanations
Ad-Free Reading
The Interpretability Angle
The reality is that the expanded feature vector is simply only numbers, remaining abstract to the naked eye.
It’s the job of the researchers to identify what particular features are actually being represented.
They typically examine the inputs that most strongly activate specific features.
For example, take the activation vector to be one of an LLM. It’ll produce its output sentence, “Pluto is not officially a planet in our solar system,” making a particular set of entries “light up.”
We can then adjust our prompt for the LLM to then output “Mars is an official planet in our solar system,” which may cause the feature representing “Pluto” to go to zero, while the feature that captures “Mars” activates.
Of course, this is a simplification, but under the hood, SAEs do present substantive opportunities for researchers to uncover truths in blackbox model activations.
With SAEs, we’ve made progress on the polysemanticity challenge and even achieved a level of monosemanticity, where the sparse vector contains one-to-one feature relations that we can test.
Current Research
Let’s re-examine the process of training your SAE. When you are developing your encoder and decoder matrices, they start out as randomly initialized matrices, similar to attention matrices for transformers.
If you’re familiar with this process, you’ll know that PyTorch and NumPy libraries generate randomness using a “seed.”
This seed is meant to make your randomness be reproducible on other machines, which allows our code tutorials (in PRO) to be shareable to you all and get the same results.
The question now is how this random seed determines your encoder, decoder, and feature matrices/vectors.
A study in January 2025 (Paulo et al.) showed that often, they would be different.
That’s not a favorable result at all for interpretability research since we don’t even know what features correspond to our first time seeing the vector entries.
If these training runs produce inconsistent results, based on how they’re initialized, we don’t have as much control as we once thought.
A more recent study (Song et al., May 2025) explored this problem to propose a way of benchmarking this consistency.
The important observation is that as long as we have the same activation vector, which is invariant when we interpret the LLM/model, it remains frozen throughout the training process.
This means that the SAE will always have the same input vector to encode and decode, even if the encode and decode matrices are different.
This leads to one of the main points of the paper: the same features remain, but in a different spot.
Because the activation vector that the SAE sees is the same, different initializations of the encode and decode will eventually still result in the same features being represented.
The claim is that the columns and rows of the decoder matrix may end up being permuted as it turns features back to the activation.
Here, P is a permutation matrix and D is a diagonal scaling matrix.
The paper shows that finding the best possible pairing of these matrices comes down to “feature matching,” then after, something called the Mean Correlation Coefficient (MCC) between features.
The A and B represent the two decoder matrices, with n being the minimum dictionary size (number of features) between the two matrices.
Here, a and b are the individual column and row vectors. The ratio that is summed represents the cosine similarity.
Building with Voice AI?
The Voice Layer is a weekly newsletter for engineering and product leaders covering LLMs, voice agents, and lessons from real-world deployments. Stay current on what’s working and what’s coming next.
So how do researchers actually get more consistent features across training runs? One answer lies in how the SAE decides which features get to "activate" in the first place.
Remember our Standard SAE from earlier: after the encoder matrix expands our vector, we apply ReLU and a loss that discourages too many nonzero entries, nudging the model towards sparsity.
It's like a soft rule telling a kid to "try not to eat too much candy" and hoping self-restraint kicks in.
TopK SAEs take a stricter approach. Instead of penalizing the model for having too many active features, TopK just forces the issue directly.
After the encoder produces its expanded vector, only the k largest values are kept, and every other entry is zeroed out automatically, leading to no penalty loss.
You decide this k ahead of time (say, the top 20), and every single input, no matter what it is, gets reconstructed from only those k active features.
This rigidity turns out to matter a lot for the consistency problem we just described.
Because TopK guarantees a fixed, exact number of active features every time, the SAE has less wiggle room to add or delete features.
The Standard SAE's softer penalty allows the model to settle into a different combination of features each time it's retrained, which is exactly the kind of freedom that hurts reproducibility.
This change brings TopK better results than Standard SAE methods, as shown by their results:
That said, TopK’s rigidity can introduce its own issues, like features being overshadowed or becoming overly sensitive to the choice of k.
It’s worth keeping in mind that no architecture here is a strict upgrade with zero tradeoffs. The added consistency is useful if we need the SAE for extended periods of time, but for one-time interpretations Standard SAE is viable.
The Bottom Line
Sparse Autoencoders gave mechanistic interpretability researchers something new: a way to turn the polysemantic activation vectors into human-readable feature maps.
But "a way to extract features" and a reliable way to extract them turned out to be two different promises: train the same SAE twice with only changing where the starting line was, and you probably won’t get the same features back.
If your explanation of a model changes on a coin flip, how much are you really explaining?
To reduce the impact of this problem, we keep an exact, fixed number of active features (TopK, or now BatchTopK) instead of just discouraging extras with the penalty loss.
SAEs let us peek inside the black box, and the ongoing consistency research is what tells us whether we can believe what we see.
Here's a special thanks to our biggest supporters:
If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!
Want to reach 10000+ AI researchers? Advertising Inquiries:
Feedback or inquiries? Send us an email at [email protected].
That's it for this week's issue of AI, but simple. See you next week!
—AI, but simple team