Harvard University; National Yang Ming Chiao Tung University
Yunho Jin, Chun-Feng Wu, David Brooks, Gu-Yeon Wei
Throughput;GPU utilization;Sequence length prediction
$S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput