Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization.

Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, Jun Suzuki

Browse the full ICLR paper archive.