Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models.

Sawsan Alqahtani, Mir Tafseer Nayeem, Md. Tahmid Rahman Laskar, Tasnim Mohiuddin, M. Saiful Bari

Browse the full EACL paper archive.