An MoE model can activate a subset of its expert weights for each token. To judge the benefit, separate total storage, active computation, routing balance, and measured runtime.
What an expert actually is
In a sparse mixture-of-experts layer, a learned router selects a small subset of expert networks for each token. In many Transformer designs, these experts replace a feed-forward sublayer while attention and other components remain shared. The original sparsely gated MoE work established conditional computation as a way to increase capacity without using every expert for every input. [S1]
An expert is not necessarily a human-readable specialist such as a legal adviser or a Python tutor. Those labels can be tempting descriptions, but a routing decision alone does not establish a stable semantic role. Describe what the router does before assigning a story to its experts.
Total parameters and active parameters answer different questions
Consider a toy layer with four equally sized experts, where each expert has one million parameters and the router selects two experts per token. The expert pool contains four million parameters, while a token uses two million expert parameters in that layer. Shared components and routing weights would add to both relevant totals.
This arithmetic is a bookkeeping example. It is not a FLOP count, a GPU-memory estimate, or a prediction of response time. The unused experts still need to be available in the serving system; changing how they are stored or moved creates additional engineering decisions.
Swipe or scroll sideways to see every column.
| Quantity | What it describes | What it does not establish |
|---|---|---|
| Total parameters | The complete learned parameter set | Work performed for one token |
| Active parameters | A defined estimate of weights used per token | Exact operations or wall-clock time |
| Expert traffic | Which tokens are sent to each expert | Whether their answers are correct |
| End-to-end latency | Time measured for a stated request | Performance under every workload |
Why balance matters
Suppose eight tokens are routed to four experts with counts of six, one, one, and zero. Equal total expert capacity does not make the load equal. One expert can become busy while another has little to do.
Switch Transformers simplifies routing to one expert per token and discusses training stability, communication costs, and balancing. [S2] That makes the engineering issue concrete: a router must learn useful assignments while the system handles the resulting traffic.
In a hypothetical implementation that gives every expert capacity for only two tokens in a batch, the six-token expert has four overflow tokens. A system must specify how it handles them. A different implementation may process all assigned tokens dynamically. Read that behavior from the implementation; it cannot be inferred from the label MoE.
Why sparse is not automatically faster
Dispatching tokens, gathering outputs, launching smaller operations, and communicating between devices can all contribute to runtime. Saving work in expert matrix multiplications is only one part of the request. The Mixtral paper provides a concrete example of a Transformer using top-two expert routing; its design and reported results belong to that model and setup. [S3]
For a small workload, routing overhead might outweigh the work saved. For a larger workload, a different bottleneck might dominate. These are possibilities to test, not universal claims about which architecture wins.
A comparison worth reading
My preferred report would state total parameters, the definition of active parameters, top-k routing, expert-load statistics, context length, batch size, hardware, and latency. It would include a quality comparison under a clearly described training budget.
I would also ask whether a dense baseline is matched for total capacity, active work, or training cost. Those are different experiments. A useful result identifies the tradeoff it measured rather than turning one attractive parameter count into a general speed claim.