A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training.

Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele

Browse the full ICS paper archive.