Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be.

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, Mark Schmidt

Browse the full ICLR paper archive.