Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models.

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, Yulia Tsvetkov

Browse the full EMNLP paper archive.