- AI, But Simple
- Posts
- Foundation Models, Simply Explained
Foundation Models, Simply Explained
AI, But Simple Issue #111
A couple years ago, most models were experts at one use case. A machine translation model translated. A spam filter filtered spam. Each one was either trained or fine-tuned on its own dataset for its own narrow task.
Foundation models were a key directional change that breaks the “one use case” pattern. The term was defined in a 2021 Stanford report with over 100 researchers in collaboration.
A foundation model is a model trained on broad data at scale that can be adapted to a wide range of downstream tasks.
Train one huge model once on a massive corpus of mostly unlabeled data, then adapt that model to translation, spam detection, captioning, and hundreds of other use cases it was never directly trained on.
Examples of foundation models we know and love today include:
LLMs (language or vision-language) such as GPT, Llama, Claude, Gemini
Vision models such as DINOv3, SAM 3
Image generation models like Stable Diffusion 3, FLUX.1, Imagen 4
Video generation models such as Sora, Veo 3, Cosmos Transfer
World models like NVIDIA Cosmos World Foundation Models, Genie 2
And much, much more.
Today, we’ll break foundation models down and explore the research problems that decide how performant, how fast, and how cheap these models actually are.
What You'll Learn
What actually makes a model a "foundation model”
Large language foundation models and how they work
Pure vision and multimodal foundation models
What scaling laws tell us about model size and reasoning
What's Helpful to Know
Pretraining objective
The function a model seeks to minimize during a large (self-supervised) training run.
Contrastive objective
An objective function where similar pairs of inputs are close together in representation space, and dissimilar pairs get pushed apart.
Mixture of experts (MoE)
A model architecture where data gets routed to certain specialized sub-networks, called experts, which take up only a fraction of the model parameters.
This increases model sparsity, keeping inference compute relatively low and maintaining decent performance given a parameter size.
What Actually Makes a Foundation Model
A foundation model can process many different types of input:
Text
Pixels (images/video)
Audio
Multiple forms at once
What actually makes a foundation model “foundational” is how the data has to be broad and mostly unlabeled and how the scale has to be large enough that the model generalizes past its training corpus.
The model is then able to be reused across tasks it wasn’t optimized for, becoming a generalist for many tasks.
Why can this happen? The downstream applicability only works because of how the model is trained in the first place.
Most foundation models are pretrained with self-supervised learning (or weakly supervised learning), requires no (or little) explicit labels.
Two types of objectives are common here.
Predictive objectives make the model guess a hidden or future piece of its own input.
Contrastive objectives have the model place matching pairs of inputs together and push mismatched pairs apart in a shared latent space.
Large language models (LLMs) lean almost entirely on the first family. Vision and multimodal models (VLM/MLLM) often use both approaches. Full vision models might only use a contrastive objective (e.g., CLIP).
Large Language Foundation Models
The dominant architecture here is the autoregressive decoder-only transformer.
The pretraining objective is very standard and well known, formulated from the negative log-likelihood loss of the next token (under the model's inferred distribution, summed over the whole sequence):
We seek to predict the next token conditioned on the tokens that came before it. You can learn more about this autoregressive objective and factorization in our JEPA article.
The loss penalizes the model in proportion to “how low” the probability it assigns the next ground-truth token. Minimizing this loss, over trillions of tokens, makes up the pretraining process.
Here's what happens inside one pass over a batch of text (from input to output).
Raw text first gets tokenized, split into sub-word pieces from a fixed vocabulary of tens of thousands of entries.
Each token then gets embedded into a dense vector, with a position vector added so the model knows where in the sequence it sits.
From there, the sequence passes through a stack of transformer blocks. Inside each block, self-attention lets every token look at every earlier token, determining how much weight each one obtains for predicting what comes next.
Each block also runs the result through a small feedforward network, applied identically at every position.
After the final block, a linear layer projects the last hidden state back up to vocabulary size (back into an understandable vocabulary space), producing a probability distribution over every possible next token.
Cross-entropy loss compares that distribution against the actual next token, and the gradient backpropagates through every layer to update the weights.
This whole loop repeats continuously over the full training corpus, often for multiple passes, until compute runs out or the loss flattens.
For a more detailed walkthrough of the attention and feedforward steps, we covered it in our past transformer issues.
The original GPT-3 (Brown et al., 2020) trained 175 billion parameters this way.
Modern LLMs like DeepSeek, Gemini, Grok, Claude, etc. also all use the same core recipe. Anthropic's newest tier, Mythos and Fable, now sits above Opus, adding extra safety layers for biology, cybersecurity, and LLM research.
Vision models borrow this exact machinery. What changes is what they're being asked to predict.
Vision and Multimodal Foundation Models
Vision foundation models branch into two categories, depending on whether they deal with one modality or whether they align two modalities with each other.
The first family adapts the transformer by turning an image into a sequence of tokens.
This is the Vision Transformer (Dosovitskiy et al., 2020). Modern examples include DINOv3 and most consumer models (which are vision-language) like Gemini, Claude, ChatGPT, etc.
A large-sized image gets separated into square chunks or downsampled, then sliced into many square patches (typically 16×16 or 32×32).
A small image gets directly sliced into these patches.
Each patch gets flattened into a vector, linearly projected into an embedding, then summed with a learned position embedding.
From there, the rest of the model is an unmodified transformer encoder.
It lets the model learn spatial relationships from the entire image, removing the local pixel assumption used in convolutional networks.
What’s amazing about this approach is how there's no image-specific transformation beyond patching, yet the model is still able to obtain high visual performance.
Recently, models like Gemini have been attempting to push for internal (native) multimodal reasoning, but for the majority of vision foundation models, a vision encoder is all that is required.
The second family aligns images with corresponding text instead of predicting pixels/words directly.
This described the CLIP family of models (Radford et al., 2021). CLIP trains two separate encoders, one for images and one for text, on roughly 400 million image-caption pairs scraped from the web.
Both encoders produce normalized embeddings for a batch of pairs.
The model gets trained so each image's embedding sits close to its own caption's embedding and far from every other caption's embedding in that batch.
That's the contrastive objective applied in both directions and averaged. The result is a shared embedding space where an image and its caption end up close together.
That shared space is what makes CLIP usable for zero-shot classification. Score an image against a set of candidate captions, and the closest one wins.
What's changed recently is how these two families combine and what they get used for.
Earlier multimodal systems, like LLaVA, bolted a frozen CLIP-style encoder onto a language model after both were pretrained separately. Llama 4 instead uses early fusion.
Text and image tokens combine into one sequence from the very first layer, so the whole model gets pretrained jointly on mixed text, image, and video data from the start.
Generation has moved the same direction. Instead of bolting a separate diffusion model onto a text model, the newest systems generate images natively, inside the same architecture that understands language.
Google's Gemini Image models, nicknamed Nano Banana, and OpenAI's GPT Image models both work this way, and video systems like Sora and Veo are pushing toward a single-model approach.
CLIP or VLM, we still have to decide how large to make the model and how much data to feed it before spending the compute.
That's what scaling laws are for.
If you enjoy our content and custom visuals, consider sharing this newsletter with others or upgrading so we can keep doing what we do.
Get full access with the PRO newsletter tier:
AI Bootcamp in Your Inbox (Hands-on Code tutorials, Practice Examples + Solutions)
Interview Questions and Cases (Tracked from 100+ Companies)
Full Article Access: More Visuals, More Explanations
Ad-Free Reading
Scaling Laws
Training a frontier model costs tens of millions of dollars in compute.
Kaplan et al. (2020) and the OpenAI team found that language model loss follows a power law in model size, holding data effectively unlimited.
Increasing the parameters cuts the loss by a predictable, shrinking amount every time. A separate fit in dataset size showed the same shape.
Because larger models turned out to be more sample-efficient per token, Kaplan's paper concluded you should mostly grow the model under a fixed compute budget and only modestly grow the dataset.
Hoffmann et al. (2022) at DeepMind trained over 400 models to re-run this analysis, reaching quite a different answer.
They deduced that model size and training data should grow together, roughly 20 training tokens for every parameter.
Their Chinchilla model, with 70 billion parameters trained on 1.4 trillion tokens, outperformed the 280-billion-parameter Gopher at the same training compute, a model trained on only around 300 billion tokens.
Most large models built under the earlier prescription turned out to be undertrained for their size. This finding reshaped how every major lab has planned pretraining runs since.
Want to learn more about scaling laws? We wrote about it in a previous issue, found here.
What's changed most recently is beyond model size and training data, but rather the compute spent at inference time (test-time compute).
Snell et al. (2024) showed that allowing a model spend more compute during query time (with reasoning chains or repeated sampling), can improve performance more than compute spent on parameters for pretraining.
That finding underlies the current wave of reasoning models, including OpenAI's o1 and DeepSeek-R1.
Compute-optimal training is no longer just about picking a model size and a data size. It also has to account for how much inference compute the deployed model will actually use.
Knowing the right size to train doesn't make training cheap, though. That's a separate problem, solved by a different set of tricks entirely.
Conclusion
A foundation model is defined by its training more than its architecture. A vast corpus of unlabeled data with a self-supervised objective. Downstream applicability across tasks it wasn't directly trained for.
Language models and vision models differ mainly in what objective is optimized, both objectives relying on the transformer backbone.
Scaling laws now cover three separate levers instead of two: model size, data size, and inference-time compute.
What's still unsettled is whether the same assumptions will hold as models keep growing larger.
Here's a special thanks to our biggest supporters:
If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!
Want to reach 10000+ AI researchers? Advertising Inquiries:
Feedback or inquiries? Send us an email at [email protected].
That's it for this week's issue of AI, but simple. See you next week!
—AI, but simple team