arxiv:2508.13650

CRISP: Persistent Concept Unlearning via Sparse Autoencoders

Published on Aug 19

· Submitted by

Tomertech on Aug 25

Upvote

Authors:

Tomer Ashuach ,

Abstract

CRISP is a parameter-efficient method using sparse autoencoders to permanently remove unwanted knowledge from large language models while preserving their utility.

AI-generated summary

As large language models (LLMs) are increasingly deployed in real-world applications, the need to selectively remove unwanted knowledge while preserving model utility has become paramount. Recent work has explored sparse autoencoders (SAEs) to perform precise interventions on monosemantic features. However, most SAE-based methods operate at inference time, which does not create persistent changes in the model's parameters. Such interventions can be bypassed or reversed by malicious actors with parameter access. We introduce CRISP, a parameter-efficient method for persistent concept unlearning using SAEs. CRISP automatically identifies salient SAE features across multiple layers and suppresses their activations. We experiment with two LLMs and show that our method outperforms prior approaches on safety-critical unlearning tasks from the WMDP benchmark, successfully removing harmful knowledge while preserving general and in-domain capabilities. Feature-level analysis reveals that CRISP achieves semantically coherent separation between target and benign concepts, allowing precise suppression of the target features.

View arXiv page View PDF Add to collection

Community

Tomertech

Paper author Paper submitter 9 days ago

•

edited 9 days ago

CRISP: Persistent Concept Unlearning via Sparse Autoencoders

Tomer Ashuach (Technion), Dana Arad (Technion), Aaron Mueller (Boston University), Martin Tutek (University of Zagreb), Yonatan Belinkov (Technion)

📄 Abstract

As large language models (LLMs) are increasingly deployed in real-world applications, the ability to remove unwanted or harmful knowledge while preserving model utility has become essential. Existing unlearning methods often operate at inference time or make broad parameter edits, which either fail to persist or damage benign capabilities.

We introduce CRISP (Concept Removal via Interpretable Sparse Projections), a parameter-efficient method for persistent concept unlearning using sparse autoencoders (SAEs). CRISP automatically identifies salient SAE features linked to harmful concepts and fine-tunes the model to suppress them, while preserving related benign features and maintaining fluent generation.

Across two open-weight models (Llama-3.1-8B, Gemma-2-2B) and safety-critical domains from the WMDP benchmark (biosecurity, cybersecurity), CRISP outperforms prior approaches, achieving up to 34-point gains in overall score while preserving benign knowledge and fluency. Feature-level analysis shows CRISP discovers semantically coherent features, enabling precise and interpretable knowledge removal.

🚀 Highlights

✅ Persistent unlearning (not just inference-time steering)
✅ Automated SAE feature selection via contrastive activation analysis
✅ Parameter-efficient fine-tuning with unlearning, retention, and coherence losses
✅ Outperforms SOTA baselines (RMU and ELM) on WMDP Bio & Cyber benchmarks
✅ Produces fluent, coherent generations while suppressing harmful knowledge
Paper on arXiv

#Unlearning #Interpretability #SparseAutoencoders #AI Safety #Knowledge Editing #LLMs