TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling.

Junyi Chen, Chuheng Du, Renyuan Liu, Shuochao Yao, Dingtian Yan, Jiang Liao, Shengzhong Liu, Fan Wu, Guihai Chen

Browse the full EuroSys paper archive.