A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length.

Yuqing Yang, Lei Jiao, Yuedong Xu

Browse the full WiOpt paper archive.