Department of Computer Science, Stevens Institute of Technology; Department of Computer Science, Rice University & ThirdAI Corp.; Department of Computer Science, Rice University
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava
Large language model; KV Cache Compression
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time