Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training.

Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, Tengyu Ma

Browse the full ICLR paper archive.