FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference.

Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin

Browse the full NSDI paper archive.