Attention is not all you need: pure attention loses rank doubly exponentially with depth.

Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas

Browse the full ICML paper archive.