Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems.

David T. Hoffmann, Simon Schrodi, Jelena Bratulic, Nadine Behrmann, Volker Fischer, Thomas Brox

Browse the full ICML paper archive.