The Foundations of Tokenization: Statistical and Computational Concerns.

Juan Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim Vieira, Ryan Cotterell

Browse the full ICLR paper archive.