Noah’s Ark Lab, Huawei; Department of Computer Science, National University of Singapore
Zangwei Zheng, Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, Yang You
large language models;inference optimization;batch processing
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline