Linear attention is (maybe) all you need (to understand Transformer optimization).

Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, Suvrit Sra

Browse the full ICLR paper archive.