Mixture of Experts (MoE)

Question 1

What is the main benefit of Mixture of Experts over dense models?

Accepted Answer

Massive scale with efficient compute — MoE allows models to have trillions of parameters while only activating a small subset (e.g., 2 of 8 experts) per token, making compute costs manageable.

Question 2

What does the gating network determine?

Accepted Answer

Which experts process each token and with what weight — The gating network (router) computes probabilities for each expert and selects the top-K experts to process each token, determining how to combine their outputs.

Question 3

A Mixture of Experts (MoE) layer consists of…

Accepted Answer

Many expert sub-networks plus a router — MoE replaces a dense layer with multiple experts; a learned router decides which experts process each input.

Question 4

The ROUTER (gating network) in MoE…

Accepted Answer

Decides which experts process each incoming token — The gating network scores experts per input and routes the token to the top-scoring one(s), enabling conditional computation.

Question 5

MoE enables 'conditional computation', meaning…

Accepted Answer

Only a subset of experts is activated per token — Sparse activation means each input uses only a few experts, so compute per token stays low even as total parameters grow huge.

Question 6

The key scaling advantage of MoE is…

Accepted Answer

More total parameters without increasing per-token compute — Because only a few experts fire per token, you can add many experts (parameters) while keeping per-token FLOPs roughly constant.

Question 7

'Top-k routing' means each token is processed by…

Accepted Answer

The k highest-scoring experts based on gate probabilities — Top-k (often top-1 or top-2) selects the few best experts per token, combining their outputs weighted by gate scores.

Question 8

Over training, experts tend to…

Accepted Answer

Specialize in handling different types of input patterns — Routing encourages experts to develop distinct competencies, so different inputs are handled by the most suitable experts.

Question 9

A 'load balancing' loss is added in MoE to…

Accepted Answer

Prevent the router from collapsing onto a few heavily used experts — Without it, the router may overuse a few experts; an auxiliary loss encourages balanced token assignment across all experts.

Question 10

Famous large-scale MoE models include…

Accepted Answer

Switch Transformer, GShard, and Mixtral — These models use MoE layers to reach very large parameter counts while keeping per-token compute manageable.

Animation

Architecture

Statistics

Total or active parameters?

Test your knowledge

What is the main benefit of Mixture of Experts over dense models?

Learn these first

Interview Questions

References & Further Reading

Love this visualization?