Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs.

Sachin Pawar, Manoj Apte, Kshitij Jadhav, Girish Keshav Palshikar, Nitin Ramrakhiyani

Browse the full IJCNLP paper archive.