README.md

---
license: apache-2.0
language:
- en
- zh
tags:
- context compression
- sentence selection
- probing classifier
- attention probing
- RAG
- LongBench
pipeline_tag: text-classification
---

# Sentinel Probing Classifier (Logistic Regression)

This repository contains the sentence-level classifier used in **Sentinel**, a lightweight context compression framework introduced in our paper:

> **Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective**  
> Yong Zhang, Yanwen Huang, Ning Cheng, Yang Guo, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao  
> 📄 [Paper (Arxiv 2025)](https://arxiv.org/abs/2505.23277) | 💻 [Code on GitHub](https://github.com/yzhangchuck/Sentinel)

---

## 🧠 What is Sentinel?

**Sentinel** reframes LLM context compression as a lightweight attention-based *understanding* task. Instead of fine-tuning a full compression model, it:

- Extracts **decoder attention** from a small proxy LLM (e.g., Qwen-2.5-0.5B)
- Computes **sentence-level attention features**
- Applies a **logistic regression (LR) classifier** to select relevant sentences

This approach is efficient, model-agnostic, and highly interpretable.

---

## 📦 Files Included

| File                    | Description                                  |
|-------------------------|----------------------------------------------|
| `sentinel_lr_model.pkl` | Trained logistic regression classifier       |
| `sentinel_config.json`  | Feature extraction configuration             |

---

## 🚀 Usage

Use this classifier on attention-derived feature vectors to predict sentence-level relevance scores:

🛠 Feature extraction code and full pipeline available at:
👉 https://github.com/yzhangchuck/Sentinel

## 📈 Benchmark Results
<p align="center">
  <img src="longbench_gpt35.png" alt="LongBench GPT-3.5 Results" width="750"/>
</p>


<p align="center">
  <img src="longbench_qwen7b.png" alt="LongBench Qwen Results" width="750"/>
</p>


## 📄 Citation
Please cite us if you use this model:

@misc{zhang2025sentinelattentionprobingproxy,
      title={Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective}, 
      author={Yong Zhang and Yanwen Huang and Ning Cheng and Yang Guo and Yun Zhu and Yanmeng Wang and Shaojun Wang and Jing Xiao},
      year={2025},
      eprint={2505.23277},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.23277}, 
}

## 📬 Contact
	•	📧 [email protected]
	•	🔗 Project: https://github.com/yzhangchuck/Sentinel


## 🔒 License

Apache License 2.0 — Free for research and commercial use with attribution.