Multi-Agent RL, Simply Explained

Multi-Agent RL, Simply Explained

AI, But Simple Issue #110

A single reinforcement learning (RL) agent learns in a closed feedback loop. It observes an environment, picks an action, and updates its policy based on the reward it receives.

The environment may be complex, but from the agent's point of view, it behaves consistently enough to learn useful patterns from.

In MARL, agents act in parallel in a single environment, trained on rewards that depend on what the other agents do.

Picture a self-driving car navigating an intersection, a robot-controlled warehouse, or traffic signals coordinating roads. These are all examples of MARL problems.

Most of MARL research (old and new) is about routing around a dynamic, complex environment without giving up what makes RL useful in the first place.

What You’ll Learn

1. Single-agent RL vs. Multi-agent RL

2. Formalizing the multi-agent problem (Dec-POMDPs and Markov games)

3. Multi-agent training and execution

4. The centralized training, decentralized execution (CTDE) loop

5. Recent MARL methods and research

What's Helpful to Know

Policy (π)

• A function mapping an agent's observation to a probability distribution over possible actions.

Markov Decision Process (MDP)

• The formalism for sequential decision-making under uncertainty. An MDP has 4 components: a state (s), an action (a), a transition function (that depends only on current state and action), and a reward (R).

Partial observability

• When an agent only sees a local observation of the environment, not the full state. POMDPs are partially observable MDPs, which we discussed in the previous world models article.

Non-stationarity

• When the environment's dynamics (from an agent's point of view) change over time due to other agents that are learning.

Advantage function (A)

• How much “better” an action is than the policy's average action in a state, defined as A(s,a) = Q(s,a) - V(s)

The Problem With Non-Stationarity

A standard MDP demonstrates the Markov property, indicating that the transition function P(s′ | s, a) depends only on the current state and action, staying fixed for the entire training process.

This fixed quality is what lets popular RL algorithms like Q-learning prove convergence.

Add more agents into the same environment, and the transition becomes:

Here, aⁱ is the action from the i-th agent.

The next state now depends on the joint action of every agent, not just one.

From agent i's perspective, every other agent's action is effectively part of the environment.

Since those other agents are also updating their own policies during training, the effective transition agent i experiences keeps shifting underneath it, update after update.

Subscribe to PRO to read the rest.