Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers.

Tianlong Chen, Zhenyu Zhang, Ajay Kumar Jaiswal, Shiwei Liu, Zhangyang Wang

Browse the full ICLR paper archive.