Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers.

Thiziri Nait Saada, Alireza Naderi, Jared Tanner

Browse the full ICML paper archive.