University of Illinois Urbana-Champaign; Microsoft
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao
Large Language Model;Efficient Inference;Generative Inference;Key-Value Cache
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs