Diffusion Language Models (DLMs), Simply Explained

Diffusion Language Models (DLMs), Simply Explained

AI, But Simple Issue #109

Many language models you use today work autoregressively, generating tokens left to right given the context of what came before.

LLMs like GPT, LLaMA, and Claude all mostly follow this structure.

This idea is so deeply baked into how we build these models that most researchers also treat it as a given.

A team from Renmin University and Ant Group decided to ask a simple question: Does it always have to be this way?

Can language models instead be guided by the stochastic process of diffusion for generating text, much like for generating high-fidelity images and video?

Their answer is in the form of a model called LLaDA (Nie et al., 2025), and it is a new interesting direction.

What You’ll Learn

1. The reversal curse (the consequence of autoregressive reasoning)

2. How the LLaDA masked diffusion model works

3. Why LLaDA opts for bidirectional attention

4. LLaDA results and benchmarks and what they show

5. DiffusionGemma, Google’s new diffusion language model

6. Flow matching and embedding diffusion

What's Helpful to Know

• Autoregressive models (ARM)

  • Models that generate tokens one at a time (left to right), where each token is conditioned on all previous tokens. This is how most current LLMs work.

  • More precisely, they factorize a joint distribution into a product of conditional distributions.

    

• Diffusion models

  • Originally popularized for images, they are generative models that learn to reverse a noising process. Data is iteratively corrupted, then a diffusion model learns to reverse this corruption.

  • For images, this means adding Gaussian noise and learning to remove it. For text, they use a discrete corruption process over tokens.

• Masked diffusion model (MDM)

  • A diffusion model that corrupts text by randomly replacing tokens with [MASK] tokens, then trains a neural network to predict what the masked tokens originally were.

When Autoregressive Models Break Down

Before getting into LLaDA, it helps to understand with a simple example the one specific place where the left-to-right approach in autoregressive models can break down.

When you ask GPT-4 to complete the next line of a poem, it does fine, but if you ask it to complete the previous line, performance drops sharply.

This is not because the model lacks knowledge of poetry, but instead because the model was trained to predict what comes after and not what came before.

Essentially, it has no mechanism to sharply reason in the other direction.

This failure mode has a term in the AI research community called the reversal curse.

This is considered a structural consequence of the left-to-right design. The model is great at "A comes before B" and worse at "B comes after A," even when those are logically the same fact.

LLaDA addresses this directly with striking results. On a reversal poem completion task, it outperformed GPT-4o by a significant margin, LLaDA scoring 45.6 while GPT-4o scored 34.3.

This all comes from a model with a fraction of the data and compute.

How Does LLaDA Actually Work?

The elegant core idea is that, instead of generating tokens one at a time from left to right, LLaDA starts with a completely masked sequence and gradually works on unmasking that full sequence.

Think of it as starting with a page of blacked-out text and progressively revealing words until the full response appears. You'll realize that this is very similar to diffusion tasks such as image generation.