Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance.

Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, Reut Tsarfaty

Browse the full ACL paper archive.