Stabilizing Transformer Training by Preventing Attention Entropy Collapse.

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, Joshua M. Susskind

Browse the full ICML paper archive.