You should evaluate your language model on marginal likelihood over tokenisations.

Kris Cao, Laura Rimell

Browse the full EMNLP paper archive.