1. Sparse Mixture-of-Experts (MoE) Gating
Dynamic top-k expert routing that activates specialized sub-networks per token without increasing dense FLOPS.
- Dynamic routing layer routes each token through 2 of 16 specialized expert feed-forward networks (FFNs).
- Total parameter efficiency: scales model capacity by 4x while maintaining sub-300ms inference latency.
- Auxiliary load-balancing loss prevents expert starvation and ensures uniform memory bandwidth utilization across clusters.