Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints.

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby

Browse the full ICLR paper archive.