An Information-Theoretic Perspective on LLM Tokenizers.

Mete Erdogan, Abhiram Rao Gorle, Shubham Chandak, Mert Pilanci, Tsachy Weissman

Browse the full ISIT paper archive.