Expert routing
i
Expert routing
A Mixture-of-Experts model contains several specialized sub-networks. A router chooses only a few of them for each token.
See a sparse Mixture-of-Experts layer choose two specialized networks for each token.
Route a token
Total experts8
Activated2
Compute used25%
The router evaluates every expert but sends the token through only the highest-scoring two, saving compute.
Tokencalculaterouter softmax → top 2
Syntax8%
Facts18%
Code3%
Math14%
Style24%
Planning9%
Translation20%
Commonsense5%