Jan 11, 2021
A mixture-of-experts architecture routes each token to a subset of model parameters.
Sparse activation offers a way to increase model capacity without activating every parameter for every token.
Explore on Humanity Unfolds
