Functional Interpolation for Relative Positions improves Long Context Transformers.

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontan, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli

Browse the full ICLR paper archive.