Decision Models, World Action Models, Egocentric Video (PRO)

Decision Models, World Action Models, Egocentric Video (PRO)

AI, But Simple Issue #124

Welcome back to AI, but simple, your AI research beacon with visual explanations.

In this issue, we’ll unpack and visually explain the three directions that each gained a cluster of new papers over the past few weeks.

1. Decision models (e.g., Jev) have become their own class of model, and the open question is now whether those probabilities can be trusted.

2. World action models are now considering how much of the future the model actually needs to simulate.

3. New pipelines turn first-person (egocentric) video into 3D records of human activity that models can learn from.

From Jev to General Decision Models

Decision models have become one of the most active topics in LLM research over the past month.

They’re bringing back an older interface: the classifier.

TypeSafe AI's Jev started the trend in September (we covered it here), and open replicas and a wave of evaluation papers followed within weeks.

Jev's weights are closed, but Laya [model card] is a useful open model to look inside to understand the general mechanism behind decision-based models.

Here’s how it works.

Laya is built on ModernBERT-large, an efficient 2024 update of BERT, paired with a small decision head trained from scratch.

Let's go through how the data passes through the network.

The Laya Architecture

First, Laya builds an input sequence from:

• The state, the input being decided on, like an email or JSON

• A question, what you’re asking it about, such as “Is this a refund request?”

• The option space you define, which is the possible answers with a [MASK] token in front of it.

So, an example could be:

Why take this form? In regular BERT, [MASK] replaces a real word, and the model is trained to predict it.

Here the [MASK] is just a marker slot, a spot in the sequence that belongs to one option.

Since Laya gives a slot per option, you can swap the options out without retraining.

Then, the sequence goes through ModernBERT's encoder. Its attention is bidirectional, so each option's [MASK] position can attend to the state, the question, and every other option.

ModernBERT-large uses 28 transformer blocks, each with the usual layout of attention and a feed-forward layer and a residual connection, with a few modern changes from the original BERT:

• Alternating attention, where every 3rd block uses global attention (every token attends to every other token). The local attention attends to tokens inside a 128-token sliding window, which keeps long inputs cheap.

• Rotary position embeddings (RoPE), a modern position encoding that rotates the query and key vectors inside attention. It's common now since it helps with relative distances between tokens (handling longer inputs better).

• GeGLU feed-forward layers, a gated version of the GeLU activation that the original BERT used.

• Pre-norm with fewer biases, where LayerNorm is applied before each sublayer instead of after it, and most linear layers drop their bias terms.

Since a local layer only sees a small window, information from distant parts of the input reaches each [MASK] mainly through the global layers.

The final layer's hidden states then go to the decision head.

It has two more transformer layers, followed by an option-marker scorer that turns the hidden state at each [MASK] into a logit.

A softmax over that question's options gives the output probability distribution.

And for the output formats:

• If you had a choice question, you get a probability per option.

• If you had a score question, it returns the expected level on a rubric scale (e.g., 2/4).

• If you had a noul question, it returns the probability that the state is true.

Laya and Calibration

Another thing to note is that Laya calibrates the outputs for higher reliability with RLCD (Reinforcement Learning for Calibrated Decisions).

The reward is a strictly proper scoring rule, a score that is highest only when the model reports its true belief.

Formally, if the true answer follows a distribution q, every other prediction p scores lower in expectation:

Laya combines two such rules, the log score and the spherical score. Both punish a confident wrong answer and reward probabilities that match how often the model is actually right.

They also add a temperature scaling step to correct the remaining overconfidence.

If you want to go deeper, we've compiled a full reading list for decision models, including Releasing the Inner Jev, the open recipe for LLM-based decision models, LEV, and DecisionPivot, an eval-based paper.

Do Robots Need to Actually Imagine the Future?

World action models (WAMs) became one of the main directions in robot learning this year.

Prior to WAMs, Vision-language-action (VLA) models were popular, with models like π-0 starting from a backbone pretrained mostly on images and text.

A WAM instead is based on world models, and it starts from a pretrained video generator and trains it to predict future frames together with an action chunk (a short sequence of robot commands).

The idea is that a model trained to predict how scenes change has already learned something useful about motion.

For example, say a robot is told to put a cup in a bowl.

A WAM looks at the current camera frame and predicts two things at once.

• What the next few frames should look like as the arm reaches for the cup

• The motor commands that would make that happen.

We separate these functionalities into a large video expert, initialized from the video generator, with a smaller action expert.

Both are trained with flow matching, a training objective used in diffusion models where the model learns the velocity field of a straight line path.

Let's go through how it works.

Start with a clean sample x, like the latents of the next few video frames, and some random Gaussian noise ε.

If you mix the two, you get a point on the straight line between them:

At τ = 0 you have the clean frames, at τ = 1 you have pure noise, and at τ = 0.5 you have a half-and-half blend.

The network is shown a blended point along with τ, and it's trained to predict the velocity, which is the direction you'd move along that line.

For a straight line, that's just ε − x.

So, the training loss is the squared error between the predicted and true velocity:

At inference, the model starts from pure noise at τ = 1 and takes a few small steps against the predicted velocity, walking back toward clean frames at τ = 0.

Each of those steps is one denoising step.

The action expert does the same thing for the action chunk, starting from noisy actions and denoising them into motor commands.

This is where the cost problem shows up.

Video tokens far outnumber action tokens, so denoising the future video at every control step dominates the compute cost.

The fixes for this problem split into 2 general directions:

Explicit WAMs denoise the future video alongside every action chunk, and the action expert reads it.

Latent WAMs, such as Fast-WAM [arxiv], keep video prediction as a training loss but drop the future video tokens at inference. That is much faster and scores about the same on standard benchmarks.

Simple-WAM and Dropping The Future

Researchers at Tsinghua asked whether latent WAMs lose anything by skipping the future.

In What Makes World Action Models Generalize? [arxiv], they tested 2 WAMs on the same backbone, data, and training budget.

The only difference is that during training, one model masked the future video tokens, and one didn’t.

As expected, there was a gap in performance, and seeing the future did matter during inference.

So, the authors tried something more interesting.

They took a trained explicit WAM and only let the video expert run for the first few denoising steps (out of 10), then froze the data, which was passed to the action expert.

This recovered almost all of the gain, which is surprising, since at the first step the future video tokens are still pure noise. Nothing about the future has actually been generated yet.

But why would noise help?

During pretraining, the video model saw exactly this kind of input over and over: a clean frame followed by noisy future frames (the diffusion noising process).

So the model is already well equipped to deal with noise, so the benefit comes from preparing the future, not generating it.

This is the basis Simple-WAM is built around:

• At inference, the future video tokens are set to pure noise.

• The video expert runs a forward pass over the current frame and the noise tokens.

• Then the action expert reuses the features at its usual denoising steps.

If you want to go deeper for more recent WAM releases, we recommend Fast-WAM, Long-WAM, and more, and the full reading list is below.

Learning from First-Person (Egocentric) Video

Robot learning has long looked to human video as a source of demonstrations.

Egocentric (first-person) video is the closest match to what a robot sees, with the camera near the eyes and the hands in view.

Large datasets now exist, from Meta's Ego4D to Apple's EgoDex, but Raw video isn't enough, though.

To learn how people handle objects, a model needs the camera's motion, the scene's 3D shape, the hands, and the objects, all in one shared 3D coordinate frame at real-world (metric) scale.

That's hard with egocentric video because the camera moves with the person's head.

Ropedia's EgoXperience [arxiv] works on both the training data and the model.

The data collection is interesting, but the model is more intriguing. They release a model called EgoX, which extends VGGT, a transformer that predicts camera poses and depth for a set of images in one forward pass.

EgoX adds a few extra tokens to each frame: one for the camera, two for the hands (left and right), and one for each object.

These tokens attend within each frame and across frames, so each one can use information from earlier and later moments in the clip.

Given a clip and a first-frame mask of each object, one forward pass predicts the cameras, depth, both hands, the object poses, and one metric scale for the whole clip.

Everything is placed in the coordinate frame of the first camera. The key piece is one shared mapping that takes any pixel and puts it into that frame.

So why does this matter?

Scene points, wrist positions, and object centers all go through this same mapping.

So during training, the loss compares hands and objects after they've been placed in the shared frame.

If a hand lands in the wrong spot, the error flows back into the camera estimate as well as the hand estimate, so both get corrected together.

If you want to read deeper, we recommend Holi-World, Ropedia's companion paper, and EgoDex, Apple's large dexterous-manipulation dataset. The full reading list is below.

Wrapping Up

Decision models are now finding their place in research, and Laya shows that a fine-tuned encoder with a small decision head can produce it in a single pass.

World action models return actions, and Simple-WAM suggests that one pass over a noisy future gets most of the benefit of imagining it.

Egocentric pipelines return human activity in a shared 3D frame, which is what world models and robots will need to learn from people at scale.

Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan

Want to reach 10000+ AI engineers, researchers, and technical founders? Advertising Inquiries:

Feedback or inquiries? Send us an email at [email protected]