File size: 2,712 Bytes
d6bdae9 47db08c d6bdae9 47db08c e3cdc97 d6bdae9 fc4bcdd d6bdae9 8fcf369 d6bdae9 47db08c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
---
license: apache-2.0
language:
- en
- zh
tags:
- context compression
- sentence selection
- probing classifier
- attention probing
- RAG
- LongBench
pipeline_tag: text-classification
---
# Sentinel Probing Classifier (Logistic Regression)
This repository contains the sentence-level classifier used in **Sentinel**, a lightweight context compression framework introduced in our paper:
> **Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective**
> Yong Zhang, Yanwen Huang, Ning Cheng, Yang Guo, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
> 📄 [Paper (Arxiv 2025)](https://arxiv.org/abs/2505.23277) | 💻 [Code on GitHub](https://github.com/yzhangchuck/Sentinel)
---
## 🧠 What is Sentinel?
**Sentinel** reframes LLM context compression as a lightweight attention-based *understanding* task. Instead of fine-tuning a full compression model, it:
- Extracts **decoder attention** from a small proxy LLM (e.g., Qwen-2.5-0.5B)
- Computes **sentence-level attention features**
- Applies a **logistic regression (LR) classifier** to select relevant sentences
This approach is efficient, model-agnostic, and highly interpretable.
---
## 📦 Files Included
| File | Description |
|-------------------------|----------------------------------------------|
| `sentinel_lr_model.pkl` | Trained logistic regression classifier |
| `sentinel_config.json` | Feature extraction configuration |
---
## 🚀 Usage
Use this classifier on attention-derived feature vectors to predict sentence-level relevance scores:
🛠 Feature extraction code and full pipeline available at:
👉 https://github.com/yzhangchuck/Sentinel
## 📈 Benchmark Results
<p align="center">
<img src="longbench_gpt35.png" alt="LongBench GPT-3.5 Results" width="750"/>
</p>
<p align="center">
<img src="longbench_qwen7b.png" alt="LongBench Qwen Results" width="750"/>
</p>
## 📄 Citation
Please cite us if you use this model:
@misc{zhang2025sentinelattentionprobingproxy,
title={Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective},
author={Yong Zhang and Yanwen Huang and Ning Cheng and Yang Guo and Yun Zhu and Yanmeng Wang and Shaojun Wang and Jing Xiao},
year={2025},
eprint={2505.23277},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.23277},
}
## 📬 Contact
• 📧 [email protected]
• 🔗 Project: https://github.com/yzhangchuck/Sentinel
## 🔒 License
Apache License 2.0 — Free for research and commercial use with attribution. |