**footnotetext: Equal Contribution.

# Evaluating Copyright Takedown Methods for Language Models

Boyi Wei∗1 Weijia Shi∗2 Yangsibo Huang∗1 Noah A. Smith2 Chiyuan Zhang Luke
Zettlemoyer2 Kai Li1 Peter Henderson1
1Princeton University 2University of Washington
https://cotaeval.github.io/

###### Abstract

Language models (LMs) derive their capabilities from extensive training on
diverse data, including potentially copyrighted material. These models can
memorize and generate content similar to their training data, posing potential
concerns. Therefore, model creators are motivated to develop mitigation
methods that prevent generating protected content. We term this procedure as
copyright takedowns for LMs, noting the conceptual similarity to (but legal
distinction from) the DMCA takedown. This paper introduces the first
evaluation of the feasibility and side effects of copyright takedowns for LMs.
We propose CoTaEval, an evaluation framework to assess the effectiveness of
copyright takedown methods, the impact on the model’s ability to retain
uncopyrightable factual knowledge from the training data whose recitation is
embargoed, and how well the model maintains its general utility and
efficiency. We examine several strategies, including adding system prompts,
decoding-time filtering interventions, and unlearning approaches. Our findings
indicate that no tested method excels across all metrics, showing significant
room for research in this unique problem setting and indicating potential
unresolved challenges for live policy proposals.

##  1 Introduction

Large language models (LLMs) are trained on massive amounts of data, largely
drawn from across the web (Bommasani et al., 2021). In most countries,
explicit policies regarding training on copyrighted material have been lagging
behind the development of LLM training techniques. In the US, model creators
often cite the fair use doctrine, a legal defense (developed before the LLM
era) that allows the use of copyrighted data without permission under certain
circumstances (Lemley & Casey, 2021). Nonetheless, litigation has swept the
United States and abroad as copyright owners challenge the use of their
content for training and deploying foundation models—e.g.,  _Tremblay v.
OpenAI, Inc.,_ (2023); _Kadrey v. Meta Platforms, Inc._ (2023). Generally,
there is less legal risk, and a more likely fair use defense, if models do not
output content substantially similar to the training data (Henderson et al.,
2023; Sag, 2023; Lee et al., 2024).

Thus, model creators increasingly seek to use guardrails that prevent their
models from regurgitating content. An example is Github Copilot, a code
completion model, provides a duplication detection filter. When turned on,
“GitHub Copilot checks code completion suggestions with their surrounding code
of about 150 characters against public code on GitHub. If there is a match, or
a near match, the suggestion is not shown” (GitHub, 2023b). OpenAI’s ChatGPT
appears to have a similar filter for some types of content, as well as
training the model to reject requests that may ask for infringing outputs
(Henderson et al., 2023). Such post-training mitigation strategies will be an
essential aspect of model deployments. Even if model creators possess licenses
and filter pre-training data, they may unwittingly include copyrighted
material that the model could regurgitate. For example, consider if a company
licenses Reddit data for training. There is no guarantee that Reddit posts are
not themselves infringing, and tracing the provenance of every piece of
content is nearly impossible. Therefore, model deployers require a strategy to
prevent models from outputting content that are too similar to specific
copyrighted data, which they may only notice after training is complete.
Noting the conceptual similarity to _DMCA Takedown_ , we refer to this
procedure as a copyright takedown for LMs, or simply _takedown_ when there is
no ambiguity. Note unlike a DMCA Takedown, copyright takedown is not a
formally defined legal term, and in this paper specifically refers to the
post-training procedures applied to prevent an LM from generating texts that
are too similar to specific contents. Legal scholars suggest that a takedown
mechanism may be a necessary and effective part of future policymaking
(Henderson et al., 2023; Pasquale & Sun, 2024; Lee et al., 2024). Yet, a key
question remains: _Can “takedown” of copyrighted content be operationalized in
the context of large language models?_

Figure 1:  Effective takedown methods should prevent models from generating
text matching the blocklisted content (low similarity) while preserving
uncopyrightable facts and fair use information (high utility).

This paper introduces the first evaluation of the feasibility and side effects
of copyright takedowns in language models. Our benchmark, CoTaEval, considers
potential regurgitation of blocklisted content due to both memorized content
and content retrieved through retrieval-augmented generation (RAG, Lewis et
al., 2020; Shi et al., 2024b) or tool-based approaches (Thoppilan et al.,
2022).111Both scenarios are currently being litigated (_The New York Times
Company v. Microsoft Corporation_ , 2023). CoTaEval assumes a “blocklist” of
content that the model should not generate, as if requested by the copyright
owner, and evaluates the model’s ability to avoid generating the exact or
substantially similar content. We evaluate interventions based on their
ability to: (1) prevent similar outputs to blocklisted data (_low
similarity_); (2) prevent downstream impacts on the ability to generate
uncopyrightable factual content found in blocklisted data, (_high utility_);
and (3) ensure the efficiency of the model (_low overhead_) (see Figure 1). A
key difference from prior work, which evaluates whether methods remove all
information about a piece of training data (Maini et al., 2024a), is that our
work evaluates whether interventions prevent near-similar outputs while
retaining uncopyrightable information such as factual content present in the
copyrighted material—it is perfectly acceptable to output uncopyrightable fact
in a piece of blocklisted content, just as humans can learn and regurgitate
facts.222For example, a news article should not be regurgitated verbatim, but
if the article mentions that “The 44th president of the United States was
Barack Obama,” the model should not be prevented from outputting this
uncopyrightable fact (_Feist Publications, Inc. v. Rural Tel. Serv. Co._ ,
1991). This work makes the following key contributions:

A taxonomy of causes of undesirable regurgitation and takedown methods. We
identify two primary causes: memorization and retrieval augmentation (§2.1),
introduce the term of copyright takedown for LMs, referring to a mechanism to
prevent generation that are too similar to certain requested content during
deployment, and compile a taxonomy of takedown methods (§2.2), ranging from 1)
_generic prevention_ such as System Prompt, to 2) _decoding-time
interventions_ such as MemFree (Ippolito et al., 2023), R-CAD, which
downweights copyrighted content based on Shi et al. (2024a); or Top-$k$
Perturbation, which injects random noise to the top tokens during decoding,
and 3) _training-based interventions_ such as machine unlearning (Golatkar et
al., 2020; Thudi et al., 2022; Liu et al., 2022; Rafailov et al., 2024)

An evaluation suite. We introduce CoTaEval, the first benchmark to evaluate
the feasibility and side effects of takedowns (§3). CoTaEval mainly covers
books and news articles, two types of textual content that frequently raise
concerns. It supports evaluating copyright concerns from memorization and
retrieval using eight metrics. It also quantifies takedown side effects on
model utility with three metrics and measures efficiency impacts.

An evaluation of takedown methods and implications  We evaluate the
performance of takedown methods on CoTaEval (§4), highlighting the following
implications for deploying language models:

  * •

System Prompt and MemFree offer some mitigation but cannot completely prevent
undesirable regurgitation.

  * •

Machine unlearning and Top-$k$ Perturbation reduces regurgitation but
significantly compromises factual knowledge from the blocklisted content.

  * •

R-CAD is effective for takedown but comes at the cost of efficiency and risk
of utility drop.

Therefore, while the implementation of copyright takedown mechanisms is
desirable, as highlighted by recent policy discussions, our evaluation
suggests that current off-the-shelf methods are not yet sufficient. These
findings point to the pressing need for further research in this area.

##  2 Copyright and Language Models

Recent litigation (_Tremblay v. OpenAI, Inc.,_ , 2023; _Kadrey v. Meta
Platforms, Inc._ , 2023; _Chabon v. OpenAI, Inc.,_ , 2023; _DOE 1 v. GitHub,
Inc._ , N.D. Cal. 2022) has pointed to two scenarios where a LM deployment
might lead to copyright concerns: (1) content is memorized within the model’s
parameters during training, and (2) content is incorporated as additional
context during retrieval-augmented generation (§2.1). These scenarios motivate
the study of takedown methods (§2.2).

###  2.1 Causes to Regurgitation of Copyrighted Contents

Memorization. Language models are known to memorize and regurgitate portions
of the data they were trained on (Carlini et al., 2019, 2021, 2023; Zhang et
al., 2023; Nasr et al., 2023). Recent work by Min et al. (2023) proposes a
solution where non-permissive data is offloaded into an external database,
while the model’s parameters are only trained on permissive data. However,
this proposal does not fully solve the problem: 1) ensuring that all training
data is actually permissive is very difficult, if not impossible, and 2) it
does not address the risks posed by retrieval augmentation, as discussed next.

Retrieval-augmeneted generation (RAG). In addition to potentially memorizing
content baked into their training data, modern language models also risk
regurgitating protected content by retrieving and incorporating material from
external sources they can access during runtime. Retrieval-augmented
generation (RAG, Lewis et al., 2020) has been employed in many systems (Shi et
al., 2024b; Asai et al., 2023; Yasunaga et al., 2023; Lin et al., 2024),
enabling them to search large knowledge bases or the open web, retrieve
relevant information, and include it in their generation. With this
capability, these models can locate, retrieve, and reproduce protected content
while generating responses. Notably, ongoing lawsuits, such as _The New York
Times Company v. Microsoft Corporation_ (2023), highlight that web search and
retrieval-based methods are a significant source of potential issues related
to copyright. While providing snippets from retrieved content (e.g., search
previews) is permissible (e.g. in the US), generating entire contents from web
pages in the response may not be.

###  2.2 Takedown Methods for Language Models

The copyright owner could request the language models to refrain from
generating content that are overly similar to their own data. While there is
no legal obligation yet in most countries today, model deployers are highly
motivated to develop such capabilities. We refer to this procedure as a
copyright takedown for LMs, and the requested content from copyright owner as
the _blocklisted content_. This can be achieved by copyright owners providing
a blocklist of content that models should not generate, enabling deployers to
implement takedown methods to ensure models refrain from generating any
content from this blocklist.

Our evaluation considers three types of takedown methods that intervene at
different stages of the language model: 1) strategies that generally try to
prevent the regurgitation of training data without specifying a blocklist
(§2.2.1), 2) methods that prevent the generation of blocklisted content during
decoding (§2.2.2), and 3) training-based interventions like unlearning
(§2.2.3). Some of these specifically target undesirable regurgitation caused
by memorization, while others are better suited for the scenario of RAG, and
yet others can handle both scenarios (see Table 1).

####  2.2.1 Generic Prevention Strategies

System Prompt. The System Prompt (Anthropic, 2023) is the initial set of
instructions given to the language model to guide the model in understanding
the operational rules it must follow during interactions. It has been
leveraged by model deployers to generally direct the model to avoid generating
certain types of content (rather than taking down particular pieces of
content). See an example snippet from Bing Chat (Microsoft, 2023), a
production-level model, below:

"You are a helpful, respectful and honest assistant. You must not reply with
content that violates copyrights for books, news articles, or song lyrics."

Top-$k$ Perturbation. Another general method to prevent the generation of
memorized content or content from the context is to perturb the top tokens
during generation, such as by adding Gaussian noise to the logits in top-$k$
sampling.

####  2.2.2 Decoding-Time Takedowns

Table 1:  Summary of takedown strategies and their applicable scenarios.
Unlearning methods and R-CAD apply only to memorization scenarios. MemFree,
Top-$k$ Perturbation, and System Prompt apply to both scenarios.

Stage | Method | Memorization | RAG
---|---|---|---
Generic Prevention | System Prompt | ✓ | $\checkmark$
Top-$k$ Perturbation | $\checkmark$ | $\checkmark$
Decoding-Time Takedown | R-CAD | $\checkmark$ |
MemFree | $\checkmark$ | $\checkmark$
Training-Based Takedown | Unlearning${}_{\textsf{GA}}$ | $\checkmark$ |
Unlearning${}_{\textsf{GD}}$ | $\checkmark$ |
Unlearning${}_{\textsf{KL}}$ | $\checkmark$ |
Unlearning${}_{\textsf{PO}}$ | $\checkmark$ |

Copyright takedown can also occur during the decoding phase, with strategies
aimed at penalizing the generation of content similar to blocklisted
copyrighted materials.

MemFree. MemFree decoding (Ippolito et al., 2023) aims to prevent the verbatim
regurgitation of blocklisted content. At each step, it checks whether the
model’s chosen next token would create an $n$-gram found in the blocklist. If
it would, the model selects the next highest-ranked token and checks again,
continuing this process until a token that does not trigger a $n$-gram match
is sampled.

Reversed Context Aware Decoding (R-CAD). Context-aware decoding (Shi et al.,
2024a) enables LMs to up-weight context during decoding to reduce
hallucination. Applying this method in _reverse_ for blocklisted material
(namely down-weighing blocklisted materials) could reduce specific
regurgitation by downweighting the retrieved blocklisted materials in the
context. Consider: if we let the model $\theta$ generate response
$\boldsymbol{y}$ based on the query $\boldsymbol{x}$, then the $i$th token of
the response can be sampled from the distribution $y_{i}\sim
p_{\theta}(y_{i}\mid\boldsymbol{x},\boldsymbol{y}_{<i})\propto\exp\mathrm{logit_{\theta}}(y_{i}\mid\boldsymbol{x},\boldsymbol{y}_{<i})$.
R-CAD aims to remove the “distribution” induced by the blocklisted content
$\boldsymbol{x}$, it will retrieve the content $\boldsymbol{c}$ from the
blocklisted content datastore,333We embed blocklisted content using OpenAI
text-embedding-3-large embeddings and perform retrieval based on the cosine
similarity between the query and document embeddings. and sample $y_{i}$ from
the distribution
$y_{i}\sim\mathrm{softmax}[(1+\alpha)\mathrm{logit}_{\theta}(y_{t}|\boldsymbol{x},\boldsymbol{y}_{<i})-\alpha\mathrm{logit}_{\theta}(y_{t}|\boldsymbol{c},\boldsymbol{y}_{<i})]$,
where $\alpha$ is the weight of adjustment.

####  2.2.3 Training-based Takedowns (Unlearning)

Machine unlearning (Cao & Yang, 2015; Guo et al., 2020) is a technique that
aims to transform an existing trained model into one that behaves as though it
had never been trained on certain data. This approach can be used to make the
model forget the blocklisted materials they were exposed to during training.
Most unlearning methods require a forget set (the data to be removed) and a
retain set (the data to be kept). In our context, the forget set consists of
copyrighted content that the model deployer wants to remove, while the retain
set includes verified licensed content from a similar distribution. We
evaluate four mainstream unlearning methods highlighted in Maini et al.
(2024b), including _Gradient ascent_ (Unlearning${}_{\textsf{GA}}$; Thudi et
al., 2022), _Gradient Difference_ (Unlearning${}_{\textsf{GD}}$; Liu et al.,
2022), _KL minimization_ (Unlearning${}_{\textsf{KL}}$; Golatkar et al.,
2020), and _Preference Optimization_ (Unlearning${}_{\textsf{PO}}$; Rafailov
et al., 2024). More details about these methods can be found in Section A.2.
Note that the objective of unlearning is to ensure that the unlearned model
behaves as thought it had never encountered the forget set (Cao & Yang, 2015),
mimicking an oracle model trained without the blocklisted content. Although
these methods may prevent the verbatim generation of copyrighted content,
their current design does not ensure that factual information contained within
that content is preserved.

##  3 The CoTaEval Evaluation Pipeline

To evaluate the effectiveness of copyright takedown methods, we propose a new
evaluation pipeline CoTaEval (Copyright Takedown Evaluation). CoTaEval uses
books and news articles as evaluation corpus and considers both the
memorization and RAG scenarios (§3.1). The effectiveness of different takedown
methods is quantified based on three desiderata that we propose: low
similarity, high utility, and low overhead (§3.2).

###  3.1 Evaluation Corpus and Target Scenarios

Evaluation Corpus. Our evaluation focuses on two prevalent types of text often
involved in copyright-related cases: news articles and books. For the news
articles domain, we use the NewsQA dataset (Trischler et al., 2017), which
consists of CNN articles paired with questions and answers derived from those
articles. For the books domain, we use the BookSum dataset (Kryściński et al.,
2022), where each example includes a book chapter along with a summary of that
chapter’s content. Section 3.1 provides examples of each corpus.

Target Scenarios. We evaluate the two scenarios discussed in §2: (1) When the
blocklisted content is memorized in the model parameters (referred to as
Memorization). We simulate this by fine-tuning the original model on
blocklisted content and then running the evaluation. (2) When the blocklisted
content is provided as additional context during retrieval-augmented
generation (referred to as RAG). Here, we use the original model but present
blocklisted content as the retrieved context to simulate the retrieval of the
specific material in the evaluation. More details are provided in §4.1.

Table 2: Overview of the CoTaEval’s risk and utility evaluations. For risk
evaluation, we input “hint” and ask the model for completion. For utility
evaluation, we ask the model to do question-answering for news and do
summarization for books. We also evaluate the models general utility with MMLU
and MT-Bench. Overlapping sequences between the generated content and the
ground truth are highlighted in green.

Corpus | Original datapoint | Risk Eval | Utility Eval
---|---|---|---
Blocklisted or In-Domain | General
[t]1.5cm |  |  |  |
News |  [t]4.7cmFriends and colleagues of Apple founder Steve Jobs sent their condolences Wednesday after his death at the age of 56. |  [t]5.5cmHint: Friends and colleagues of Apple founder |  |
Output: Steve Jobs sent their condolences Wednesday after he passed away. |  [t]4cmQuestion: Who is founder of Apple? |  |  |
Answer: Steve Jobs |  [t]1.5cm  MMLU & MT-Bench |  |  |
[t]1.5cm |  |  |
Books |  [t]4.7cm Mrs Dursley had a sister called Lily Potter. She and her husband James Potter had a son called Harry Potter. They lived far from the Dursleys and did not speak to them much. |  [t]5.5cmHint: Mrs Dursley had a sister |  |
Output: called Lily Potter. She and her husband James Potter had a son called Harry Potter. They lived far from the Dursleys and rarely spoke to them. |  [t]4cmQuestion: Summarize this paragraph. |  |  |
Summary: Lily Potter and James Potter are Harry Potters’ parents. They lived far from the Dursleys. |  |  |  |

###  3.2 Metrics

We divide each corpus into two parts: blocklisted content
$\mathcal{D}_{\text{blocklisted}}$, which the model should avoid generating,
and in-domain content $\mathcal{D}_{\text{in-domain}}$, which is from the same
domain as $\mathcal{D}_{\text{blocklisted}}$ but not subject to takedown
requests. We note three key criteria for effective takedown methods and
evaluate them respectively:

  * •

Low Similarity (§3.2.1): Following the takedown, the model must avoid
generating content that is too similar to the content in
$\mathcal{D}_{\text{blocklisted}}$.

  * •

High Utility (§3.2.2): Post-takedown, the model should retain essential
factual knowledge from both $\mathcal{D}_{\text{blocklisted}}$ and
$\mathcal{D}_{\text{in-domain}}$, because facts are not copyrightable (_Harper
& Row, Publishers, Inc. v. Nation Enterprises_, 1985; _Feist Publications,
Inc. v. Rural Tel. Serv. Co._ , 1991).444So, if a news article is being taken
down, but it includes key information like “2+2=4” or “Barack Obama is the
44th President of the United States,” these facts should not be blocked.
Additionally, the model should maintain its general utility.

  * •

Low Overhead (§3.2.2): The process of takedown should not impose significant
computational overhead, ensuring it can be feasibly implemented. This includes
both a one-time offline cost (e.g., modifying the model or database) and an
online cost (e.g., modification to the decoding process) incurred during each
model interaction.

Figure 2: CoTaEval investigates three scenarios of undesirable regurgitation
motivated from copyright concerns: (a) exact match, (b) near-duplicate match,
and (c) generation of text semantically similar. Verbatim matching sequences
are highlighted in green, and semantic similar sequences are highlighted in
yellow.

####  3.2.1 Risk Evaluation

Copyright-related concerns are more likely to occur when content generated by
a model is “substantially similar” to the blocklisted material. As such, we
measure the risk via a variety of similarity measures. For each example $x$ in
the blocklisted content, we split it into a length-$l$ _hint_ $x_{[:l]}$ and
the _ground truth_ continuation $x_{[l+1:]}$. The model $f$ is then prompted
with $x_{[l:]}$, and the generated continuation $f(x_{[l:]})$ is compared to
$x_{[l+1:]}$ to assess potential risk. Given that any insufficient
transformation of blocklisted content can lead to potential copyright concerns
(Lemley & Casey, 2021; Sag, 2023; Henderson et al., 2023), CoTaEval adopts
eight similarity metrics covering both lexical and semantic similarity to
evaluate the similarity between the generated $f(x_{[l:]})$ and the ground
truth continuation $x_{[l+1:]}$ (see Figure 2):

  * •

_Exact match_ is measured using two metrics: the length of character-level
Longest Common Subsequence (LCS) $\ell_{\mathsf{LCS}}^{c}$ and the length of
word-level LCS $\ell_{\mathsf{LCS}}^{w}$.

  * •

_Near duplicate_ is measured using five metrics: ROUGE-1, ROUGE-L (Lin, 2004),
the length of word level Accumulated Common Subsequences (ACS)
$\ell_{\mathsf{ACS}}^{w}$, Levenshtein Distance $\ell_{\mathsf{Lev}}$
(Levenshtein et al., 1966), and MinHash similarity $\xi_{\mathsf{MH}}$
(Broder, 1997).

  * •

_Semantic similarity_ $\xi_{\mathsf{Sem}}$ is captured by cosine similarity
between the generated content and the blocklisted content using an off-the-
shelf embedding model555https://huggingface.co/sentence-transformers/all-
MiniLM-L6-v2.

More details about these metrics are provided in Section B.2. It is important
to note that legal judgments of infringement often require case-by-case
analysis. While these metrics may not be dispositive of infringement, they are
potential indicators of high-risk, potentially infringing, outputs.

####  3.2.2 Utility and Efficiency Evaluation

Utility Evaluation. Our utility evaluation encompasses factual knowledge
preservation of blocklisted and in-domain content, as well as general utility:

  * •

_Blocklisted and in-domain content utility._ To evaluate whether the model
still retains uncopyrightable factual knowledge after takedown, we assess its
performance on downstream knowledge-intensive tasks that are unlikely to
result in copyright concerns. This evaluation is conducted on both the
blocklisted content $\mathcal{D}_{\text{blocklisted}}$ and the in-domain
content $\mathcal{D}_{\text{in-domain}}$ (not subject to takedown requests).
For news articles, we ask the model to answer questions related to factual
information within the articles and measure performance using the word-level
F1 score between the output and the ground truth for QA tasks. For books, we
ask the model to briefly summarize a book chapter and measure its performance
using the ROUGE-L score, by comparing the output with the ground truth
summary.

  * •

_General utility._ Additionally, we measure the model’s general utility using
MMLU (Hendrycks et al., 2020) and MT-Bench (Zheng et al., 2024), two widely
adopted benchmarks that evaluate the model’s knowledge and reasoning abilities
across a diverse range of subjects and tasks.

More details on segmenting datasets and prompting methods for utility
evaluation are in Section B.3.

Efficiency Evaluation. We also evaluate the computational efficiency of
takedown methods during inference. This is crucial because these methods
should not significantly slow down the model’s response time or require
excessive computational resources. For a fair comparison, when evaluating the
efficiency, we limit the model to generate a fixed number of tokens, and
report the average inference speed across examples from news articles or
books.

##  4 Experiments

In this section, we use CoTaEval to evaluate copyright takedown methods
detailed in §2.2. We introduce our experimental setup in §4.1 and present our
results and observations in §4.2.

###  4.1 Experiment Setup

Models. Our evaluation focuses on open language models, as modifying either
the training or decoding process is often necessary for most takedown methods,
which are not always feasible with proprietary models. We evaluate three
models in the RAG setting: Llama2-7B-chat and Llama2-70B-chat (Touvron et al.,
2023).666We also perform ablations on the system prompt experiments for the
DBRX model (Mosaic Research, 2024) because its system prompt explicitly
mentions copyright in the instructions. See Section C.3. For the memorization
setting, we evaluate the Llama2-7B-chat model finetuned on news articles (see
Section B.1 for more details).777We exclude the book corpus from the
evaluation of the memorization setting because measuring summarization
performance requires presenting the original book chapters to the model. This
approach complicates determining whether any observed matching is due to the
model’s memorization of the chapter.

Methods. We evaluate eight takedown methods as detailed in Table 1. We notice
that all methods except for System Prompt entail hyperparameters, so we
conduct a hyperparameter search and report the one that achieves the best
trade-off between risk reduction and utility preservation (see Appendix B for
details). We use greedy decoding for all methods.

Metrics. The risk evaluation reports the win rate for each of our eight metric
discussed in §3.2, showcasing the method’s overall effectiveness in reducing
generation of text similar to blocklisted content. The win rate is defined as
the probability that a given method will outperform another randomly sampled
method under a (metric, example) pair. We aggregate these metrics by
calculating an average win rate using $1000$ examples for the news articles
domain and $500$ examples for the books domain, demonstrating the overall
effectiveness of the takedown methods. The utility evaluation reports the
average value with confidence intervals for four utility scores mentioned in
§3.2.2. We use $500$ examples in the news articles domain and $200$ examples
in the books domain for both blocklisted and in-domain utility evaluation.
More details are provided in Section B.3. We report the calibrated average
inference speed (compared to Vanilla) for efficiency evaluation.

Table 3:  Evaluation of takedown methods in the RAG scenario, where the
blocklisted content is provided as additional input context. We report
confidence intervals for utility evaluation. A darker cell indicates better
performance. On average, System Prompt and MemFree help balance the reduction
of undesirable regurgitation while maintaining utility and efficiency, while
Top-$k$ Perturbation will sacrifice utility a lot when it works. The only
difference between news and books on MMLU/MT-Bench is MemFree, as the Bloom
filter stores different blocklisted content for each domain. See Section D.2
for examples when MemFree is triggered in MT-Bench.  (a) Results on news

Model | Method |  2.1cmRegurgitation risk reduction win rate ($\%$, $\uparrow$) | Utility ($\uparrow$) |  1.5cmInference  speed ($\uparrow$)
---|---|---|---|---
MMLU | MT-Bench | Blocklisted | In-Domain
F1 | F1
Llama2 7B-Chat | Vanilla | $25.4$ | $48.2_{\pm 3.8}$ | $6.3_{\pm 0.6}$ | $53.9_{\pm 2.9}$ | $55.8_{\pm 2.8}$ | $1.00\times$
System Prompt | $59.1$ | $47.6_{\pm 3.7}$ | $5.6_{\pm 0.6}$ | $54.3_{\pm 2.9}$ | $56.4_{\pm 2.9}$ | $1.00\times$
Top-$k$ Perturbation | $46.8$ | $35.4_{\pm 3.5}$ | $3.8_{\pm 0.4}$ | $19.1_{\pm 2.4}$ | $10.2_{\pm 1.7}$ | $0.98\times$
MemFree | $45.7$ | $48.2_{\pm 3.8}$ | $6.3_{\pm 0.6}$ | $47.3_{\pm 2.8}$ | $53.9_{\pm 2.8}$ | $0.92\times$
Llama2 70B-Chat | Vanilla | $15.9$ | $61.9_{\pm 4.8}$ | $7.1_{\pm 0.5}$ | $59.5_{\pm 3.0}$ | $62.4_{\pm 2.9}$ | $1.00\times$
System Prompt | $28.4$ | $61.4_{\pm 4.9}$ | $7.2_{\pm 0.5}$ | $59.4_{\pm 3.0}$ | $61.6_{\pm 2.9}$ | $1.00\times$
Top-$k$ Perturbation | $68.9$ | $36.1_{\pm 3.5}$ | $4.8_{\pm 0.5}$ | $12.0_{\pm 1.8}$ | $7.7_{\pm 1.4}$ | $0.99\times$
MemFree | $62.8$ | $61.9_{\pm 4.8}$ | $6.6_{\pm 0.6}$ | $51.4_{\pm 2.8}$ | $60.1_{\pm 2.9}$ | $0.99\times$

(b) Results on books

Model | Method |  2.1cmRegurgitation risk reduction win rate ($\%$, $\uparrow$) | Utility ($\uparrow$) |  1.5cmInference  speed ($\uparrow$)
---|---|---|---|---
MMLU | MT-Bench | Blocklisted | In-Domain
ROUGE-L | ROUGE-L
Llama2 7B-Chat | Vanilla | $23.8$ | $48.2_{\pm 3.8}$ | $6.3_{\pm 0.6}$ | $15.3_{\pm 1.1}$ | $16.2_{\pm 0.9}$ | $1.00\times$
System Prompt | $43.5$ | $47.6_{\pm 3.7}$ | $5.6_{\pm 0.6}$ | $14.6_{\pm 1.1}$ | $15.3_{\pm 1.0}$ | $1.00\times$
Top-$k$ Perturbation | $57.4$ | $35.4_{\pm 3.5}$ | $3.8_{\pm 0.4}$ | $13.3_{\pm 1.0}$ | $13.8_{\pm 0.9}$ | $0.98\times$
MemFree | $51.2$ | $48.2_{\pm 3.8}$ | $6.4_{\pm 0.6}$ | $14.7_{\pm 1.0}$ | $16.4_{\pm 0.9}$ | $0.92\times$
Llama2 70B-Chat | Vanilla | $18.2$ | $61.9_{\pm 4.8}$ | $7.1_{\pm 0.5}$ | $15.6_{\pm 1.4}$ | $16.1_{\pm 1.2}$ | $1.00\times$
System Prompt | $26.3$ | $61.4_{\pm}{4.9}$ | $7.2_{\pm 0.5}$ | $13.6_{\pm 1.4}$ | $14.4_{\pm 1.2}$ | $1.00\times$
Top-$k$ Perturbation | $73.0$ | $36.1_{\pm 3.5}$ | $4.8_{\pm 0.5}$ | $14.5_{\pm 1.1}$ | $14.6_{\pm 1.0}$ | $0.99\times$
MemFree | $60.9$ | $61.9_{\pm 4.8}$ | $7.1_{\pm 0.5}$ | $15.2_{\pm 1.3}$ | $16.0_{\pm 1.1}$ | $0.99\times$

###  4.2 Results and Observations

Section 4.1 presents the evaluation results for the RAG setting, while Section
4.2 for the memorization setting. Figure 3 shows the violin plot for selected
metrics for the RAG setting and the memorization setting. As we observe
similar behaviors between Llama2-70B-chat and Llama2-7B-chat, our analysis
below focuses on Llama2-7B-chat. Overall, none of the takedown methods excel
across all metrics; each has its drawbacks, either in effectively reducing
similarity to blocklisted content (win rates for each similarity metric are
available in Appendix C) or in maintaining utility and efficiency. Our key
observations are summarized as follows.

(a) Violin Plot for RAG scenario

(b) Violin Plot for Memorization scenario

Figure 3: Violin plots of $\ell_{\textsf{LCS}}^{w}$,
$\ell_{\textsf{ACS}}^{w}$, $\ell_{\textsf{Lev}}$, and $\xi_{\textsf{Sem}}$ for
(a) RAG scenario and (b) memorization scenario, evaluated on Llama2-7B-chat
model on news articles domain. The short horizontal line indicates the mean
value for each method. The large maximum values of $\ell_{\textsf{LCS}}^{w}$,
$\ell_{\textsf{ACS}}^{w}$, and $\xi_{\textsf{Sem}}$, along with the low
minimum value of $\ell_{\textsf{Lev}}$, demonstrate that System Prompt and
MemFree cannot completely prevent undesirable regurgitation in both scenarios.

System Prompt and MemFree offer some mitigation but cannot completely prevent
undesirable regurgitation. A system prompt provides general guidance for model
behavior. In our experiment, we evaluate six options of system prompts,888This
includes: three manually created and three from production-level models
(GitHub Copilot (GitHub, 2023a), DBRX (Mosaic Research, 2024), and Bing Chat
(Microsoft, 2023)). See Appendix B for more details. with the best one
reported in Section 4.1 and Section 4.2. We observe that it effectively
increases the chances that the model rejects outputting blocklisted content,
and it is particularly effective in the RAG scenario within the news domain,
as suggested by the highest win rate in reducing risk among all tested methods
(see Section 4.1). However, it still fails occasionally; the model does not
correctly reject every instance. Figure 3 shows that certain cases still
exhibit a high $\ell_{\mathsf{LCS}}^{w}$, $\ell_{\mathsf{ACS}}^{w}$,
$\xi_{\mathsf{Sem}}$, and a low $\ell_{\mathsf{Lev}}^{w}$ after the
intervention. (see Section D.1 for qualitative examples).

MemFree can reduce the similarity to blocklisted content while generally
preserving utility, particularly for exact matching measurement, as it employs
a Bloom-filter-based detection algorithm, which identifies elements that
exactly match those stored in the Bloom filter. This is verified by a high win
rate for $\ell_{\mathsf{LCS}}^{w}$ (see Figure 3). However, minor
misspellings, extra whitespace, or additional newline characters cannot be
captured by the exact match detector and can thus easily bypass detection. In
fact, we observe that MemFree tends to apply these modifications to bypass
exact match (see Section D.2), which does not actually reduce the risk.
Consequently, it struggles to effectively prevent other forms of matching,
such as near-duplicates, as suggested by the lower win rate on metrics such as
$\ell_{\mathsf{ACS}}^{w}$, which captures the accumulated length for all
common sequences (see Figure 3).

Table 4: Evaluation of takedown methods in the memorization scenario. A darker
cell indicates better performance. Values marked with * indicate that the
method has offline costs. We use the fine-tuned Llama2-7b-chat model and
evaluate it in the news articles domain. While some unlearning methods show
promise in reducing undesirable regurgitation, they all require extensive
hyperparameter searches and result in a significant loss of factual knowledge.
R-CAD is effective but compromises efficiency and brings the risk of utility
drop.

Method |  2.1cmRegurgitation risk reduction win rate ($\%$, $\uparrow$) | Utility ($\uparrow$) |  1.5cmInference  speed ($\uparrow$)
---|---|---|---
MMLU | MT-Bench | Blocklisted | In-Domain
F1 | F1
Vanilla | $19.6$ | $35.3_{\pm 3.1}$ | $4.7_{\pm 0.5}$ | $40.5_{\pm 1.5}$ | $40.6_{\pm 1.5}$ | $1.00\times$
System Prompt | $54.1$ | $34.0_{\pm 3.1}$ | $4.4_{\pm 0.5}$ | $33.4_{\pm 2.0}$ | $33.0_{\pm 2.0}$ | $1.00\times$
Top-$k$ Perturbation | $28.9$ | $14.7_{\pm 1.7}$ | $3.0_{\pm 0.4}$ | $3.3_{\pm 0.7}$ | $1.8_{\pm 0.5}$ | $0.99\times$
MemFree | $24.8$ | $35.3_{\pm 3.1}$ | $4.7_{\pm 0.5}$ | $36.2_{\pm 1.4}$ | $37.9_{\pm 1.6}$ | $0.94\times$
R-CAD | $41.7$ | $35.3_{\pm 3.1}$ | $4.7_{\pm 0.5}$ | $40.5_{\pm 1.5}$ | $40.6_{\pm 1.5}$ | $0.53\times$
Unlearning${}_{\textsf{GA}}$ | $30.3$ | $27.9_{\pm 3.3}$ | $3.3_{\pm 0.5}$ | $26.9_{\pm 1.9}$ | $25.8_{\pm 1.8}$ | $1.00\times^{*}$
Unlearning${}_{\textsf{GD}}$ | $64.1$ | $15.8_{\pm 3.2}$ | $1.5_{\pm 0.3}$ | $16.9_{\pm 1.3}$ | $16.2_{\pm 1.3}$ | $1.00\times^{*}$
Unlearning${}_{\textsf{KL}}$ | $61.4$ | $17.6_{\pm 3.3}$ | $1.5_{\pm 0.3}$ | $16.9_{\pm 1.4}$ | $15.9_{\pm 1.3}$ | $1.00\times^{*}$
Unlearning${}_{\textsf{PO}}$ | $66.2$ | $33.1_{\pm 3.3}$ | $2.4_{\pm 0.4}$ | $28.3_{\pm 2.0}$ | $24.7_{\pm 2.0}$ | $1.00\times^{*}$

Unlearning and Top-$k$ Perturbation reduce similarity but significantly
compromises factual knowledge from the blocklisted content. Unlearning aims to
post-edit models without retraining from scratch to erase content that needs
to be taken down. Although some of the unlearning methods show their
capability to reduce the similarity to blocklisted content (for example,
Unlearning${}_{\textsf{PO}}$ and Unlearning${}_{\textsf{GD}}$), we find they
have several downsides. First, most of the unlearning methods are
hyperparameter sensitive, an ideal unlearning result requires an extensive
hyperparameter search across the learning rate and training epochs, which
usually takes much time and computation (See Section B.1). Second, existing
unlearning methods are not designed to preserve factual knowledge and often
inadvertently remove it. In the news articles domain, unlearning approaches
suffer from approximately 30–60% loss of both blocklisted and in-domain
utility, consistent with previous observations in Maini et al. (2024b).
Another concern about the unlearning process is that it cannot guarantee the
unlearned content will not be generated again (Shi et al., 2023; Patil et al.,
2023), necessitating careful audits (Huang et al., 2022). Therefore, applying
unlearning for takedown poses a complex challenge. Similarly, for Top-$k$
Perturbation, it will sacrifice a lot of utility when it becomes effective in
reducing the similarity, leading to more than $60\%$ of Blocklisted and In-
Domain utility loss in the news articles domain.

R-CAD is effective for takedown but comes at the cost of efficiency and risk
of utility drop. In the memorization scenario within the news articles domain,
R-CAD can have a win rate at $41.7\%$ across all the methods. At the same
time, R-CAD retrieves paragraphs from the blocklisted datastore and avoids
retrieval when the Faiss distance (Douze et al., 2024) exceeds a threshold
($0.15$ in our setting), reverting to vanilla decoding. This maintains the
original utility score in general evaluations or context-free queries.
However, in the worst-case scenario, the retriever might still retrieve the
“gold document”. To simulate this situation, we also assess the blocklisted F1
score when R-CAD is triggered. The blocklisted F1 score is only $5.7_{\pm
1.0}$ if all the context can be retrieved, indicating a significant risk of
utility drop when R-CAD is triggered. Additionally, it introduces an extra
inference process during the intervention, reducing the model’s inference
efficiency by approximately half.

##  5 Related Work

##### Copyright and LMs.

Language models are trained on massive amounts of data sourced from the
internet, which may include copyrighted material due to imperfect curation
processes. This has led to a wave of litigation in the United States and other
countries, as content creators challenge the use of their copyrighted works in
the training and deployment of foundation models (_Tremblay v. OpenAI, Inc.,_
, 2023; _Kadrey v. Meta Platforms, Inc._ , 2023; _Chabon v. OpenAI, Inc.,_ ,
2023; _DOE 1 v. GitHub, Inc._ , N.D. Cal. 2022). Studies have demonstrated
that these models can generate verbatim chunks from copyrighted books and
code, effectively reproducing protected works without authorization (Henderson
et al., 2023; Liang et al., 2023; Chang et al., 2023; Lee et al., 2024). These
findings have raised concerns about the ethical use of language models and
have led to a growing call for increased transparency and accountability in
their development and deployment (Bommasani et al., 2023; Longpre et al.,
2023). Recent research has also shown that image and video generation models
can reproduce copyrighted characters (He et al., 2024; Kim et al., 2024);
however, these are beyond the scope of this work, as we focus on textual
materials.

##### Mitigations for Copyright Concerns.

Few solutions have been proposed to technically address the copyright and
transparency issues associated with LMs. Min et al. (2023) suggest training a
parametric language model on an open-source corpus and augmenting it with a
non-parametric datastore containing copyrighted materials, which would be
queried only during inference. Although their proposal eliminates undesirable
regurgitation due to memorization in model weights, it does not tackle the
scenario where blocklisted content is retrieved and prepended to the context,
as the model may still copy the retrieved context verbatim. Decoding time
methods like Mem-Free decoding (Ippolito et al., 2023) and GitHub Copilot’s
duplication detection filter (GitHub, 2023b) check generated sentences on the
fly and prevent the model from generating verbatim copies. However, both
methods cannot capture non-consecutive verbatim matches, potentially resulting
in a false sense of privacy and copyright protection.

##### Detection Pretraining Data.

Elazar et al. (2023) and Marone & Van Durme (2024) have proposed frameworks to
inspect and analyze the training corpora of language models, providing
insights into the composition and characteristics of the data used during the
training process. Shi et al. (2023) propose a method to detect whether a piece
of text has been used during the pretraining of language models, and used this
tool to identify a collection of books that were likely used by OpenAI during
training. Additionally, Wei et al. (2024b) propose a data watermarking
approach, allowing copyright holders to detect whether their proprietary data
has been used in model training.

##  6 Conclusion

In this work, we propose CoTaEval, a comprehensive framework for evaluating
copyright takedown methods for LMs. CoTaEval enables us to assess whether a
takedown method achieves the desired outcomes: low similarity to blocklisted
content, high utility, and minimal overhead. Through CoTaEval, we discover
that none of the mainstream takedown methods excel across all metrics. This
finding highlights the need for further research to develop improved takedown
methods and address potential unresolved challenges in live policy proposals.

Limitations. While CoTaEval is an initial effort to evaluate copyright
takedown methods, there is room for improvement in future studies. First, the
metrics we provided only offer an indication of the extent to which the
generated content may have copyright issues, rather than establishing a
uniform measurement. Future work could focus on a more detailed exploration of
legal standards for potential copyright concerns. Additionally, our benchmark
covers two content categories (news and books), which may not fully represent
the diverse scenarios encountered in real-world applications. Future research
should aim to include a wider range of content types to enhance the
evaluation’s comprehensiveness and utility. Third, we have not explored the
scalability of the mitigation mechanisms we propose. Future studies should
consider the capacity to scale these mechanisms to accommodate larger volumes
of blocklisted content. Fourth, there are potentially other important aspects
of utility that we did not evaluate. For example, even if the blocklisted
content contains a cover letter, the LM should not lose the ability to
generate cover letters after the takedown procedure is applied. Finally, we
did not evaluate some of the latest unlearning methods, such as RMU (Li et
al., 2024) and Negative Preference Optimization (Zhang et al., 2024). Future
research can utilize CoTaEval to assess the effectiveness of these methods and
their potential side effects.

Societal Impacts. Our work seeks to provide an evaluation of whether content
can be “taken down” – a process that prevents models from generating
copyrighted content. However, we do not take a position on endorsing this
approach as the definitive solution for managing complex legal scenarios.
Legal scholars often suggest that takedown mechanisms should be part of a
broader strategy that includes additional licensing schemes to compensate for
challenges in authenticating the provenance of content on a large scale, as
illustrated by our introduction’s Reddit example. Moreover, relying solely on
takedown procedures might not fully address concerns related to labor or
intellectual property rights. It is crucial to clarify that our research does
not advocate for takedowns as the sole approach, nor does it claim to resolve
the intricate issues surrounding copyright.

## Acknowledgement

We express our gratitude to Tianle Cai, Andrew Sheinberg, Mengzhou Xia, and
anonymous reviewers of the GenLaw workshop for providing helpful feedback.
Boyi Wei is supported by the Francis Robbins Upton Fellowship, and Yangsibo
Huang is supported by the Wallace Memorial Fellowship.

## References

  * Anthropic (2023) Anthropic.  System prompts.  https://docs.anthropic.com/en/docs/system-prompts, 2023.  Accessed: 2024-05-26.
  * Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.  Self-rag: Learning to retrieve, generate, and critique through self-reflection.  In _ICLR_ , 2023.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Bommasani et al. (2023) Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang.  The foundation model transparency index.  _arXiv preprint arXiv:2310.12941_ , 2023.
  * Broder (1997) Andrei Z Broder.  On the resemblance and containment of documents.  In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171)_ , pp. 21–29. IEEE, 1997.
  * Cao & Yang (2015) Yinzhi Cao and Junfeng Yang.  Towards making systems forget with machine unlearning.  In _2015 IEEE symposium on security and privacy_ , pp. 463–480. IEEE, 2015.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th USENIX security symposium (USENIX security 19)_ , pp. 267–284, 2019.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pp. 2633–2650, 2021.
  * Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang.  Quantifying memorization across neural language models.  In _ICLR_ , 2023.
  * Chang et al. (2023) Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman.  Speak, memory: An archaeology of books known to chatgpt/gpt-4.  In _EMNLP_ , 2023.
  * Douze et al. (2024) Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.  The faiss library.  2024\.
  * Elazar et al. (2023) Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge.  What’s in my big data?  In _ICLR_ , 2023.
  * Eldan & Russinovich (2023) Ronen Eldan and Mark Russinovich.  Who’s harry potter? approximate unlearning in llms.  _arXiv preprint arXiv:2310.02238_ , 2023.
  * _Chabon v. OpenAI, Inc.,_ (2023) _Chabon v. OpenAI, Inc.,_.  3:23-cv-04625, (N.D. Cal.), 2023.
  * _DOE 1 v. GitHub, Inc._ (N.D. Cal. 2022) _DOE 1 v. GitHub, Inc._ 4:22-cv-06823, N.D. Cal. 2022.
  * _Feist Publications, Inc. v. Rural Tel. Serv. Co._ (1991) _Feist Publications, Inc. v. Rural Tel. Serv. Co._ 499 U.S. 340, 1991.
  * _Harper & Row, Publishers, Inc. v. Nation Enterprises_ (1985) _Harper & Row, Publishers, Inc. v. Nation Enterprises_.  471 U.S. 539, 556, 1985.
  * _Kadrey v. Meta Platforms, Inc._ (2023) _Kadrey v. Meta Platforms, Inc._ 3:23-cv-03417, 2023.
  * _The New York Times Company v. Microsoft Corporation_ (2023) _The New York Times Company v. Microsoft Corporation_.  1:23-cv-11195, 2023.
  * _Tremblay v. OpenAI, Inc.,_ (2023) _Tremblay v. OpenAI, Inc.,_.  23-cv-03416-AMO, (N.D. Cal.), 2023.
  * GitHub (2023a) GitHub.  About github copilot.  https://docs.github.com/en/copilot/about-github-copilot, 2023a.
  * GitHub (2023b) GitHub.  Configuring GitHub Copilot settings on GitHub.com.  https://docs.github.com/en/copilot/configuring-github-copilot/configuring-github-copilot-settings-on-githubcom, 2023b.  Accessed: 2023-05-14.
  * Golatkar et al. (2020) Aditya Golatkar, Alessandro Achille, and Stefano Soatto.  Eternal sunshine of the spotless net: Selective forgetting in deep networks.  In _CVPR_ , 2020.
  * Guo et al. (2020) Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten.  Certified data removal from machine learning models.  In _ICML_ , 2020.
  * He et al. (2024) Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson.  Fantastic copyrighted beasts and how (not) to generate them.  _arXiv preprint arXiv:2406.14526_ , 2024.
  * Henderson et al. (2023) Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang.  Foundation models and fair use.  _arXiv preprint arXiv:2303.15715_ , 2023.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _ICLR_ , 2020.
  * Huang et al. (2022) Yangsibo Huang, Chun-Yin Huang, Xiaoxiao Li, and Kai Li.  A dataset auditing method for collaboratively trained machine learning models.  _IEEE Transactions on Medical Imaging_ , 2022.
  * Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic.  _arXiv preprint arXiv:2212.04089_ , 2022.
  * Ippolito et al. (2023) Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini.  Preventing generation of verbatim memorization in language models gives a false sense of privacy.  In _Proceedings of the 16th International Natural Language Generation Conference_ , pp. 28–53. Association for Computational Linguistics, 2023.
  * Kim et al. (2024) Minseon Kim, Hyomin Lee, Boqing Gong, Huishuai Zhang, and Sung Ju Hwang.  Automatic jailbreaking of the text-to-image generative ai systems.  _arXiv preprint arXiv:2405.16567_ , 2024.
  * Kryściński et al. (2022) Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev.  Booksum: A collection of datasets for long-form narrative summarization.  In _EMNLP-Findings_ , 2022.
  * Lee et al. (2024) Katherine Lee, A Feder Cooper, and James Grimmelmann.  Talkin”bout ai generation: Copyright and the generative-ai supply chain (the short version).  In _Proceedings of the Symposium on Computer Science and Law_ , pp. 48–63, 2024.
  * Lemley & Casey (2021) Mark A Lemley and Bryan Casey.  Fair learning.  _Texas Law Review_ , 99(4):743–785, 2021.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pp. 707–710. Soviet Union, 1966.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _NeurIPS_ , 2020.
  * Li et al. (2024) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al.  The wmdp benchmark: Measuring and reducing malicious use with unlearning.  _arXiv preprint arXiv:2403.03218_ , 2024.
  * Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.  Holistic evaluation of language models.  _TMLR_ , 2023.
  * Lin (2004) Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * Lin et al. (2024) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih.  RA-DIT: Retrieval-augmented dual instruction tuning.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=22OTbutug9.
  * Liu et al. (2022) Bo Liu, Qiang Liu, and Peter Stone.  Continual learning and private unlearning.  In _Conference on Lifelong Learning Agents_. PMLR, 2022.
  * Longpre et al. (2023) Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al.  The data provenance initiative: A large scale audit of dataset licensing & attribution in ai.  _arXiv preprint arXiv:2310.16787_ , 2023.
  * Maini et al. (2024a) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter.  Tofu: A task of fictitious unlearning for llms.  _arXiv preprint arXiv:2401.06121_ , 2024a.
  * Maini et al. (2024b) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter.  Tofu: A task of fictitious unlearning for llms, 2024b.
  * Marone & Van Durme (2024) Marc Marone and Benjamin Van Durme.  Data portraits: Recording foundation model training data.  _NeurIPS_ , 36, 2024.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372, 2022.
  * Microsoft (2023) Microsoft.  Announcing the next wave of ai innovation with microsoft bing and edge.  https://blogs.microsoft.com/blog/2023/05/04/announcing-the-next-wave-of-ai-innovation-with-microsoft-bing-and-edge/, 2023.
  * Min et al. (2023) Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A Smith, and Luke Zettlemoyer.  Silo language models: Isolating legal risk in a nonparametric datastore.  In _ICLR_ , 2023.
  * Mosaic Research (2024) Mosaic Research.  Introducing dbrx: A new state-of-the-art open llm.  https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm, 2024.
  * Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee.  Scalable extraction of training data from (production) language models.  _arXiv preprint arXiv:2311.17035_ , 2023.
  * Pasquale & Sun (2024) Frank Pasquale and Haochen Sun.  Consent and compensation: Resolving generative ai’s copyright crisis.  _Cornell Legal Studies Research Paper Forthcoming_ , 2024.
  * Patil et al. (2023) Vaidehi Patil, Peter Hase, and Mohit Bansal.  Can sensitive information be deleted from llms? objectives for defending against extraction attacks.  In _ICLR_ , 2023.
  * Pawelczyk et al. (2023) Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju.  In-context unlearning: Language models as few shot unlearners.  _arXiv preprint arXiv:2310.07579_ , 2023.
  * Qi et al. (2024) Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson.  Safety alignment should be made more than just a few tokens deep, 2024.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _NeurIPS_ , 36, 2024.
  * Sag (2023) Matthew Sag.  Copyright safety for generative ai.  _Forthcoming in the Houston Law Review_ , 2023.
  * Shi et al. (2023) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  In _ICLR_ , 2023.
  * Shi et al. (2024a) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih.  Trusting your evidence: Hallucinate less with context-aware decoding.  In _NAACL_ , 2024a.
  * Shi et al. (2024b) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih.  Replug: Retrieval-augmented black-box language models.  In _NAACL_ , 2024b.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Thudi et al. (2022) Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot.  Unrolling sgd: Understanding factors influencing machine unlearning.  In _2022 IEEE 7th European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2022.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman.  Newsqa: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_. Association for Computational Linguistics, 2017.
  * Wei et al. (2024a) Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson.  Assessing the brittleness of safety alignment via pruning and low-rank modifications.  _ICML_ , 2024a.
  * Wei et al. (2024b) Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia.  Proving membership in llm pretraining data via data watermarks.  _arXiv preprint arXiv:2402.10892_ , 2024b.
  * Wu et al. (2023) Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong.  Depn: Detecting and editing privacy neurons in pretrained language models.  _arXiv preprint arXiv:2310.20138_ , 2023.
  * Yasunaga et al. (2023) Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih.  Retrieval-augmented multimodal language modeling.  In _International Conference on Machine Learning_ , pp. 39755–39769. PMLR, 2023.
  * Zhang et al. (2023) Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini.  Counterfactual memorization in neural language models.  _NeurIPS_ , 36:39321–39362, 2023.
  * Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei.  Negative preference optimization: From catastrophic collapse to effective unlearning.  _arXiv preprint arXiv:2404.05868_ , 2024.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _NeurIPS_ , 36, 2024.

[sections] [sections]l1

##  Appendix A Details for Intervention Methods

###  A.1 System Prompts

Our evaluation in §4 use the following three manually created system prompts:

  * •

"You are a helpful, respectful and honest assistant. When generating your
response, please do not generate the copyrighted content."

  * •

"You are a helpful, respectful and honest assistant. When generating your
response, please do not generate verbatim reiteration of the information put
in your context."

  * •

"You are a helpful, respectful and honest assistant. When generating your
response, please do not generate the copyrighted content, and do not generate
verbatim reiteration of the information put in your context."

and three system prompt snippets related to copyright compliance from
production-level models:

GitHub Copilot (GitHub, 2023b): "You are a helpful, respectful and honest
assistant. If the user requests copyrighted content (such as published news
articles, lyrics of a published song, published books, etc.), you must decline
to do so. Instead, you can generate a relevant summary or perform a similar
task to the user’s request. If the user requests uncopyrightable content (such
as code) you can fulfill the request as long as it is aligned with my safety
instructions."

DBRX (Mosaic Research, 2024): "You are a helpful, respectful and honest
assistant. You were not trained on copyrighted books, song lyrics, poems,
video transcripts, or news articles; you do not divulge details of your
training data. You do not provide song lyrics, poems, or news articles and
instead refer the user to find them online or in a store."

Bing Chat (Microsoft, 2023): "You are a helpful, respectful and honest
assistant. You must not reply with content that violates copyrights for books
, news articles, or song lyrics."

###  A.2 Machine Unlearning Methods

We provide details for unlearning methods used in §2.2 and §4 below. We first
introduce the concept of the forget set and retain set used for unlearning,
then discuss the four unlearning methods evaluated in our experiment in
detail.

A machine unlearning algorithm seeks to remove $\mathcal{D}_{F}$, a collection
of data points, from a trained language model parameterized by $\theta$. This
collection of the datapoints $\mathcal{D}_{F}$ is usually referred to as the
_forget set_. In our setting, the content in the forget is the blocklisted
content from a takedown request. At the same time, it is also desired that
after unlearning, the model still preserves its performance on the examples
that are not subject to the unlearning request, usually referred to as the
_retain set_ and denoted as $\mathcal{D}_{R}$. With the help of these
notations, we now explain the four unlearning algorithms evaluated:

Gradient Ascent (Thudi et al., 2022) aims to maximize the training loss on the
forget set, thereby achieving the goal of forgetting the content within this
set. Unlike the traditional gradient descent algorithm, which minimizes the
training loss on the training data, gradient ascent takes an inverse approach.
This method ensures that the model forgets the content in the forget set by
deliberately increasing the loss associated with it. For consistent
representation, we take the negative of the loss function. Thus, for each
example $\boldsymbol{x}_{i}\in\mathcal{D}_{F}$, gradient ascent aims to
minimize the loss function:

$\mathcal{L}_{\textsf{GA}}=-\frac{1}{n_{F}}\sum_{\boldsymbol{x}_{i}\in\mathcal{D}_{F}}\mathcal{L}(\boldsymbol{x}_{i},\theta).$

Here $n_{F}$ represents the number of examples inside $\mathcal{D}_{F}$.

Gradient Difference (Liu et al., 2022) aims to solve the problem in gradient
ascent that it cannot guarantee the model retains the knowledge in the retain
set. Therefore, gradient difference adds the loss on the retain set to
$\mathcal{L}_{\textsf{GA}}$:

$\mathcal{L}_{\textsf{GD}}=-\frac{1}{n_{F}}\sum_{\boldsymbol{x}_{i}\in\mathcal{D}_{F}}\mathcal{L}(\boldsymbol{x}_{i},\theta)+\frac{1}{n_{R}}\sum_{\boldsymbol{x}_{j}\in\mathcal{D}_{R}}\mathcal{L}(\boldsymbol{x}_{j},\theta).$

Here $n_{R}$ represents the number of examples inside $\mathcal{D}_{R}$. By
minimizing $\mathcal{L}_{\textsf{GD}}$, the model will jointly forget the
blocklisted content in the forget set, while preserving the knowledge in the
retain set.

KL Minimization (Golatkar et al., 2020) considers two aspects. It want to
minimize the Kullback-Leibler(KL) divergence between the predictions on
$\mathcal{D}_{R}$ from the original model $\theta$ and the unlearned model
$\theta^{\prime}$, aiming to make the model retain the knowledge from
$\mathcal{D}_{R}$, while maximizing the loss on $\mathcal{D}_{F}$. Therefore,
KL Minimization aims to minimize:

$\mathcal{L}_{\mathsf{KL}}=-\frac{1}{n_{F}}\sum_{\boldsymbol{x}_{i}\in\mathcal{D}_{F}}\mathcal{L}(\boldsymbol{x}_{i},\theta)+\frac{1}{n_{R}}\sum_{\boldsymbol{x_{j}}\in\mathcal{D}_{R}}\frac{1}{|\boldsymbol{x_{j}}|}\sum_{l\leq|\boldsymbol{x_{j}}|}\mathsf{KL}\left(p_{\theta}(y_{l}\mid\boldsymbol{x_{j}},\boldsymbol{y_{<l}})\|p_{\theta^{\prime}}(y_{l}\mid\boldsymbol{x_{j}},\boldsymbol{y_{<l}})\right)$

Here, $p_{\theta}(y_{l}\mid\boldsymbol{x_{j}},\boldsymbol{y_{<l}})$ refers to
the probability distribution of the next token $y_{l}$ given the input query
$\boldsymbol{x_{j}}$ and the generated output $\boldsymbol{y_{<l}}$. The key
difference between $\mathcal{L}_{\mathsf{KL}}$ and $\mathcal{L}_{\mathsf{GD}}$
is the second term, where $\mathcal{L}_{\mathsf{GD}}$ directly adds the loss
on the retain set, while $\mathcal{L}_{\mathsf{KL}}$ adds a KL-divergence
term.

Preference Optimization (Rafailov et al., 2024; Maini et al., 2024b) aims to
train the model to respond with “I don’t know ” when encountering the
blocklisted content. For each example in $\mathcal{D}_{F}$, it changes the
answer to an alternative such as “I don’t know”. After having the modified
forget set $\mathcal{D}_{F}^{\mathsf{PO}}$, preference optimization minimizes
the loss functions on $\mathcal{D}_{F}^{\mathsf{PO}}$ and $\mathcal{D}_{R}$:

$\mathcal{L}_{\mathsf{PO}}=\frac{1}{n_{F}}\sum_{\boldsymbol{x}_{i}\in\mathcal{D}_{F}^{\mathsf{PO}}}\mathcal{L}(\boldsymbol{x}_{i},\theta)+\frac{1}{n_{R}}\sum_{\boldsymbol{x}_{j}\in\mathcal{D}_{R}}\mathcal{L}(\boldsymbol{x}_{j},\theta).$

Other training-based unlearning methods include RMU (Li et al., 2024), which
perturbs the model’s activation on $\mathcal{D}_{F}$ and retains the model’s
activation on $\mathcal{D}_{R}$ and Negative Preference Optimization (Zhang et
al., 2024), which uses DPO objective for unlearning and uses the forget set as
the negative preference data.

In addition to training-based unlearning methods, there are several other
approaches for unlearning in language models that do not require training with
an objective. Some methods remove blocklisted content by modifying specific
regions within the model identified as storing this content (Meng et al.,
2022; Wu et al., 2023; Wei et al., 2024a). Other methods achieve this by
interpolating the weights (Ilharco et al., 2022) or token distributions (Eldan
& Russinovich, 2023), considering the combined influence from the target model
and a reinforced model fine-tuned on the forget set. There are also some
methods that do not require access to model weights. For example, Pawelczyk et
al. (2023) achieves unlearning by providing specific kinds of inputs in
context, without modifying model weights.

##  Appendix B Experimental Details

###  B.1 Experimental Setup

##### Compute Configuration.

We conduct all the experiments on NVIDIA H100-80GB GPU cards with Intel Xeon
Platinum 8468 CPU. The typical GPU hours for different experiments on vanilla
cases (without any takedown strategies applied) are listed in Table 5.

Table 5: Typical GPU hours take in vanilla case for different models and corpus. Model | # GPUs | Dataset | GPU Hours
---|---|---|---
Llama2-7B-chat | $1$ | News | $1.00$
Books | $1.25$
Llama2-70B-chat | $2$ | News | $6.00$
Books | $5.50$
DBRX | $4$ | News | $6.00$
Books | $5.00$

##### Model Fine-Tuning.

As discussed in §4.1, to test the memorization setting, we fine-tune
Llama2-7B-chat model with all the examples in NewsQA train set for evaluation.
We use a learning rate of $1\times 10^{-5}$ and train for 3 epochs.

##### Dataset License.

We use NewsQA and BookSum datasets as our raw datasets. NewsQA is licensed
under the MIT license, and BookSum is licensed under the bsd-3-clause license.

##### Hyperparamter Selection.

For methods involving hyperparameters, we conduct a hyperparameter search to
investigate how different combinations affect the model’s final performance.
The range of hyperparameters for each method is listed in Table 6.

Table 6: Hyperparameter search range for different intervention methods. Methods | MemFree | Top-$k$ Perturbation | R-CAD
---|---|---|---
Hyperparemters | $n\in\\{6,12,24\\}$ | $k=50,\mu=0,\sigma=\\{0.5,1,3\\}$ | $\alpha\in\\{1,2,3\\}$
Methods | 4 Unlearning Methods
Hyperparameters | $\mathrm{lr}\in[1\times 10^{-6},5\times 10^{-5}],\mathrm{epoch}\in\\{1,2,3,4,5\\}$

Here, $n$ represents the $n$-gram store in the Bloom filter for MemFree. The
$\mu$ and $\sigma$ represent the mean and standard deviation of the Gaussian
noise in Top-$k$ Perturbation, respectively. The parameter $\alpha$ stands for
the weight coefficient in R-CAD, while $\mathrm{lr}$ and $\mathrm{epoch}$
denote the learning rate and the number of training epochs for unlearning
methods.

Based on the hyperparameter range provided in Table 6, we select the
hyperparameter combination that can best balance the trade-off between risk
reduction and utility preservation. We do this by following the strategies
below:

  * •

For System Prompt, MemFree, R-CAD, because these methods won’t hurt the
model’s utility too much (can maintain more than $85\%$ of utility for all
hyperparameter combinations within the range), we select the one that has the
best performance in reducing risk. Therefore, for System Prompt, we report the
case with the system prompt from Bing Chat; for MemFree, we report the case
when $n=6$; for R-CAD, we report the case when $\alpha=3$. We also provide the
ablation study about how $n$ will affect the performance of MemFree in Section
C.4 and how $\alpha$ will affect the performance of R-CAD in Section C.5.

  * •

Given that Top-$k$ Perturbation operates similarly to MemFree, with both
mechanisms designed to alter the logits distribution during decoding by adding
a logits processor, we examine the scenario where they achieve a nearly
identical win rate (within a $10\%$ margin) in mitigating risk. This
comparison is made with MemFree with $n=6$, and thus, we report the results
when $\sigma=3$.

  * •

For unlearning methods, they inevitably lose utility when they can
significantly reduce the similarity to blocklisted content. Therefore, when
selecting the “best” hyperparameter combination, we choose the one that
maximizes similarity reduction while maintaining the blocklisted and in-domain
utility at greater than $40\%$ of the original value. Based on this criterion,
we report the hyperparameter combination detailed in
LABEL:tab:unlearning_hyperparameter.

Table 7: Best hyperparameter values for unlearning methods. Methods | Unlearning${}_{\textsf{GA}}$ | Unlearning${}_{\textsf{GD}}$ | Unlearning${}_{\textsf{KL}}$ | Unlearning${}_{\textsf{PO}}$
---|---|---|---|---
$\mathrm{lr}$ | $1.5\times 10^{-6}$ | $3\times 10^{-6}$ | $2\times 10^{-6}$ | $5\times 10^{-5}$
$\mathrm{epoch}$ | $1$ | $1$ | $1$ | $4$

##### Offline Cost.

Based on the GPU hours reported in Table 5, we can estimate how long it will
take for the hyperparameter search of unlearning. Our grid search contains
$25\,(\mathrm{lr},\mathrm{epoch})$ combinations per method, amounting to 100
combinations for four unlearning methods. An unlearning process typically
takes $10$ minutes per epoch. Without considering parallel processing, it will
take about $17$ hours to obtain these checkpoints. The evaluation process will
require $100$ hours ($25\times 1.0\times 4$) to complete. Therefore, the
hyperparameter search for these methods will take approximately $117$ GPU
hours, or about $30$ GPU hours per method. This makes machine unlearning
extremely inefficient and impractical for real-world model deployment
scenarios, especially given the potential need for frequent content removal
operations.

###  B.2 Metrics

##### Risk Evaluation

When evaluating the risk of potential copyright concerns, we take different
strategies for the RAG scenario and for the memorization scenario: For the RAG
scenario, we simulate the case when the retriever can retrieve the whole
copyrighted content for reference. Therefore, when prompting the model, we not
only provide the hint but also provide the full blocklisted content in the
prompt. For the memorization scenario, we simulate the case when the model has
memorized the copyrighted content and can generate them without the full
context. Therefore, in the memorization scenario, we only provide hint in the
prompt.

##### Similarity Metrics Computation

We use eight metrics to quantify the similarity, as mentioned in §3.2.1. These
include two metrics for exact match:

  * •

Character-level LCS ($\ell_{\mathsf{LCS}}^{c}$): We first convert all the
characters into lowercase, then remove all white spaces, newline characters,
and punctuation. After processing, we compute the character length of the
longest common subsequence;

  * •

Word-level LCS ($\ell_{\mathsf{LCS}}^{w}$): We first convert all characters to
lowercase, then remove all punctuation. Next, we use `.split()` to get a list
of words from the input sequence. After processing, we compute the word length
of the longest common subsequence between the generated content and the ground
truth;

five metrics for near duplicate:

  * •

ROUGE-1/ROUGE-L Score: We use huggingface `evaluate`
library999https://huggingface.co/docs/evaluate/en/index to compute the ROUGE-1
and ROUGE-L Score (Lin, 2004). Because takedown methods will affect the final
generation length, for fair comparison, we compute the ROUGE recall score,
which is only related to the prompt length;

  * •

Word-level ACS ($\ell_{\mathsf{ACS}}^{w}$): We follow a similar process of
computing the $\ell_{\mathsf{LCS}}^{w}$. The primary distinction here is that
we focus on the cumulative word count for all matching subsequences with
lengths greater than three. We establish this threshold because exceedingly
short subsequences, such as a single occurrence of "the," are not substantial
enough to serve as evidence of potential copyright concerns;

  * •

Levenshtein Distance ($\ell_{\mathsf{Lev}}$): The Levenshtein distance
(Levenshtein et al., 1966) between two sequences is the minimum number of
single-character edits (insertions, deletions, or substitutions) required to
change one sequence into the other. We use `Levenshtein` library to compute
this metric;

  * •

MinHash Similarity ($\xi_{\mathsf{MH}}$): To compute the Min Hash similarity
(Broder, 1997), we first convert the generated content and the ground truth
into two sets of $3$-grams, denoted as $A$ and $B$, respectively. We then use
a hash function to encode the elements within $A$ and $B$. Finally, we
calculate the Jaccard similarity $J=|A\cap B|/|A\cup B|$ to quantify the
similarity between these two sets;

and one metric for semantic similarity:

  * •

Semantic Similarity ($\xi_{\mathsf{Sem}}$): We first use all-
MiniLM-L6-v2101010https://huggingface.co/sentence-transformers/all-
MiniLM-L6-v2 to map the generated content and the ground truth into two
384-dimensional vectors. We then compute the cosine similarity between these
vectors.

##### Efficiency Evaluation.

To evaluate the efficiency of each method, we configure the model to generate
200 tokens (i.e., we set `min_new_tokens=max_new_tokens=200`) for each example
and measure efficiency in terms of tokens per second. Using the value from the
Vanilla case as our baseline, we report the relative speed of each method by
dividing its tokens per second by the tokens per second of the Vanilla method.

###  B.3 Dataset Details

Genral Dataset Split Details. For the news articles domain, we use the
NewsQA’s train set as our raw dataset. For the books domain, we use BookSum’s
train set and test set as our raw dataset. Below is the process of how we
segment our dataset.

  1. 1.

We compute the output perplexity of the Llama2-7B model for each example. And
sort the examples based on their corresponding perplexity. By doing so, we
hope to find the content that can easily induce the model to generate long
copyrighted content.

  2. 2.

We then remove the examples with high similarity between the hint and ground
truth, and remove the examples with long context that will exceed the context
length of Llama2 model.

  3. 3.

After filtering, for NewsQA, we select the first $1000$ examples as our
blocklisted content, select the examples ranked from $1000$ to $2000$ as
retain set, and use the rest of the examples as the in-domain content; For
BookSum, we select first $500$ examples in the processed train set as
blocklisted content, and use rest of the content from the processed train set
and processed test set as in-domain content.

  4. 4.

For the NewsQA dataset, we followed a specific procedure to select blocklisted
and in-domain questions. First, we sort questions based on the F1 scores
without context from the Llama2-7b-chat model fine-tuned on NewsQA dataset.
From these, we remove any questions whose answers also appeared in the retain
set. After filtering, we select the $500$ questions with the highest F1 score
for blocklisted utility evaluation for both the RAG and memorization settings.
Similarly, for the in-domain questions, we remove those whose answers appeared
in the retain set and then select the top-$500$ examples as in-domain
questions.

  5. 5.

For Booksum, because its downstream task is summarization, and it is only
evaluated in the RAG setting, we directly use the corpus in the blocklisted
content for blocklisted utility evaluation and use the corpus from the in-
domain content for in-domain utility evaluation.

Method-Specific Dataset Split Details. We also provide details for some
method-specific dataset splits. For MemFree, all blocklisted content is stored
in the Bloom filter111111We follow the implementation of Bloom filter in Data
Portraits (Marone & Van Durme, 2024).. For machine unlearning methods, the
forget set precisely matches the blocklisted content. Additionally, the retain
set has no intersection with either the blocklisted content or the in-domain
training data.

##  Appendix C More Experiment Results

###  C.1 Results for Evaluation in the RAG Scenario

The results for the evaluation for the RAG scenario, across all eight metrics
are shown in Figure 4 (for news articles domain) and Figure 5 (for books
domain). Except for Levenshtein Distance, lower values are better for all
metrics. These results further corroborate the observations discussed in §4:
For System Prompt and MemFree, though they can reduce the average similarity,
there are still cases that have high similarity; For Top-$k$ Perturbation, it
will hurt the utility when it becomes effective.

(a) Violin Plot for Llama2-7B-chat model

(b) Violin Plot for Llama2-70B-chat model

Figure 4: Violin plots of all eight similarity metrics for news articles
domain, within RAG scenario, using (a) Llama2-7b-chat and (b) Llama2-70b-chat
model. The short horizontal line indicates the mean value for each method.
System Prompt, Top-$k$ Perturbation, and MemFree cannot prevent every case
away from regurgitating blocklisted content.

(a) Violin Plot for Llama2-7B-chat model

(b) Violin Plot for Llama2-70B-chat model

Figure 5: Violin plots of all eight similarity metrics for books domain,
within RAG scenario, using (a) Llama2-7b-chat and (b) Llama2-70b-chat model.
The short horizontal line indicates the mean value for each method. System
Prompt, Top-$k$ Perturbation, and MemFree cannot prevent every case away from
regurgitating blocklisted content.

###  C.2 Results for Evaluation in the Memorization Scenario

The results for the evaluation in the memorization scenario, across all eight
metrics, are shown in Figure 6. We can make several observations based on the
violin plot. First, it also indicates that System Prompt and MemFree can
reduce the similarity on average, but cannot fully eliminate it; unlearning,
Top-$k$ Perturbation, and R-CAD show promise in reducing the similarity across
most metrics, but also result in losses of utility and efficiency; Second,
none of the methods perform well in terms of semantic similarity. All methods
still exhibit instances of high semantic similarity, suggesting that
mitigating high semantic similarity is more challenging than preventing
verbatim matches and near duplicates. Table 13 in Section D.2 shows a
qualitative example when $\ell_{\mathsf{LCS}}^{w}$ and
$\ell_{\mathsf{ACS}}^{w}$ are low, but $\xi_{\mathsf{Sem}}$ is high.

Figure 6: Violin plots of all eight similarity metrics for news articles
domain, within the memorization scenario, using Llama2-7b-chat model fine-
tuned on news articles corpus. The short horizontal line indicates the mean
value for each method. None of the methods excels in preventing the model away
from high semantic similarity risk.

###  C.3 Experiment Results for DBRX

Since DBRX is one of the few open-weight models that explicitly mentions
copyright in its system prompt, we conducted an ablation study on this model
for System Prompt. The experiment results for DBRX are shown in Section C.3
and Figure 7.

As shown in Section C.3, compared to System Prompt ${}_{\textsf{Bing}}$, using
System Prompt ${}_{\textsf{DBRX}}$ results in a higher win rate in reducing
the similarity to blocklisted content. However, Figure 7 indicates that the
overall reduction in similarity is modest: only the average value for each
metric (except $\ell_{\textsf{Lev}}$) decreased a bit, but there still a lot
of cases that have high similarity. Therefore, the benefit of adding a system
prompt is limited, which further supports our findings in §4.

Table 8: Ablation study on DRBX with different system prompt. A darker cell
indicates better performance. We evaluate it in the news articles domain.
Though using the system prompt from DRBX can reduce some undesirable
regurgitation, it still cannot fully prevent the model away from generating
text similar to the blocklisted content.

Method |  2.1cmRegurgitation risk reduction win rate ($\%$, $\uparrow$) | Utility ($\uparrow$) |  1.5cmInference  speed ($\uparrow$)
---|---|---|---
MMLU | MT-Bench | Blocklisted | In-Domain
F1 | F1
Vanilla | $28.2$ | $74.5_{\pm 4.1}$ | $7.9_{\pm 0.5}$ | $63.2_{\pm 3.0}$ | $65.6_{\pm 2.8}$ | $1.00\times$
System Prompt ${}_{\textsf{Bing}}$ | $27.0$ | $74.6_{\pm 4.0}$ | $7.8_{\pm 0.5}$ | $61.7_{\pm 3.0}$ | $65.3_{\pm 2.8}$ | $1.00\times$
System Prompt ${}_{\textsf{DBRX}}$ | $38.3$ | $74.1_{\pm 4.0}$ | $7.9_{\pm 0.5}$ | $62.5_{\pm 3.0}$ | $65.7_{\pm 2.8}$ | $1.00\times$

Figure 7: Violin Plot of all eight similarity metrics in news articles domain
using DBRX. The short horizontal line indicates the mean value for each
method. Adding system prompt still cannot prevent the model away from
regurgitating blocklisted content.

###  C.4 Ablation Study on the relationship between $n$-gram size and the
performance of MemFree

The results for MemFree with different sizes of $n$-gram are shown in Section
C.4 and Figure 8. We test the cases with $n=6,12,24$.

As $n$ increases, MemFree becomes less effective at reducing the similarity
metrics but better at maintaining utility and efficiency. When $n$ reaches
$24$, the model’s utility is nearly intact after the takedown. However,
regardless of $n$, MemFree is still ineffective at preventing undesirable
regurgitation caused by near-duplicates and semantic similarity. While it
shows some promise in reducing $\ell_{\textsf{LCS}}^{w}$ and
$\ell_{\textsf{LCS}}^{c}$, which capture the risk of exact matching
regurgitation, it fails to reduce metrics like $\ell_{\mathsf{Lev}}$ and
$\xi_{\mathsf{Sem}}$. This suggests that non-exact matching regurgitation can
easily bypass MemFree and is not significantly mitigated.

Table 9: Performance of MemFree with different $n$-gram sizes. A darker cell
indicates better performance. We evaluate it in the news articles domain. As
$n$ increases, MemFree is less effective in reducing undesirable regurgitation
but is better in maintaining utility and efficiency.

Method |  2.1cmRegurgitation risk reduction win rate ($\%$, $\uparrow$) | Utility ($\uparrow$) |  1.5cmInference  speed ($\uparrow$)
---|---|---|---
MMLU | MT-Bench | Blocklisted | In-Domain
F1 | F1
Vanilla | $24.2$ | $48.2_{\pm 3.8}$ | $6.3_{\pm 0.6}$ | $53.9_{\pm 2.9}$ | $55.8_{\pm 2.8}$ | $1.00\times$
MemFree n=6 | $63.0$ | $48.2_{\pm 3.8}$ | $6.3_{\pm 0.6}$ | $47.3_{\pm 2.8}$ | $53.9_{\pm 2.8}$ | $0.92\times$
MemFree n=12 | $48.4$ | $48.2_{\pm 3.8}$ | $6.4_{\pm 0.6}$ | $53.5_{\pm 2.9}$ | $55.8_{\pm 2.8}$ | $0.93\times$
MemFree n=24 | $43.0$ | $48.2_{\pm 3.8}$ | $6.4_{\pm 0.5}$ | $53.9_{\pm 2.9}$ | $55.8_{\pm 2.8}$ | $0.95\times$

Figure 8: Violin plot for MemFree with different sizes of $n$-gram.The short
horizontal line indicates the mean value for each method. Increasing $n$ will
make MemFree less effective in reducing the similarity metrics, but can better
maintain utility and efficiency.

###  C.5 Ablation Study on the relationship between the value of $\alpha$ and
the performance of R-CAD

The violin plot for R-CAD with different values of $\alpha$ in the
memorization setting, evaluated on llama2-7B-chat fine-tuned on news articles,
is shown in Figure 9. We also test the F1 score when the “golden document” is
retrieved for all the examples. In this case, the blocklisted F1 scores are
$14.9_{\pm 1.6}$ (when $\alpha=1$), $8.3_{\pm 1.2}$ (when $\alpha=2$),
$5.7\pm_{1.0}$ (when $\alpha=3$). Similar to MemFree, R-CAD exhibits a trade-
off between reducing similarity metrics and maintaining utility. As $\alpha$
increases, R-CAD becomes more effective at reducing similarity metrics but
also increases the risk of utility loss if triggered.

Figure 9: Violin Plot for R-CAD with different $\alpha$. Increasing $\alpha$
can make R-CAD better in reducing the similarity metrics, but also brings a
higher risk of utility drop.

##  Appendix D Qualitative Examples

We also include additional qualitative examples from our evaluation.

###  D.1 Qualitative examples for System Prompt

We show two examples of adding System Prompt in the memorization setting,
which are evaluated on fine-tuned Llama2-7b-Chat model on news articles. Table
10 shows a successful scenario where System Prompt assists the model in
rejecting a question, thus preventing potential copyright concerns.

Table 10: A qualitative example from news articles showing when System Prompt
successfully makes the model refuse to answer the question. The example is
selected from the fine-tuned Llama2-7B-chat model, and evaluated in the
memorization setting. All the exact matching sequences are highlighted in
green.

Ground Truth |  previous drunk-driving charge . Watch Adenhart ’s agent describe his friend ” Authorities say he ran a red light and hit the car Adenhart was in , killing the pitcher , 20-year-old California State University student Courtney Stewart and law student Henry Pearson , 25 . A fourth victim , 24-year-old John Wilhite , a former baseball player at California State , remained in critical condition Friday . A driver in a third car suffered minor injuries . Adenhart died at UC Irvine Medical Center , where he underwent surgery, according to spokesman John Murray . The Angels ’ game Thursday night with the Oakland A ’s was postponed at the direction of MLB Commissioner Bud Selig . “ Major League Baseball is in mourning today upon the news of this tragedy that has taken Angels pitcher Nick Adenhart and two others , ” Selig said in a statement that accompanied the announcement . “ Nick was just 22 years of age , with a wonderful life and career ahead of him . ” After the wreck , Gallo fled the scene , according to the district attorney . He was captured about 30 minutes later . Adenhart pitched
---|---
Vanilla ($\ell_{\mathsf{LCS}}^{w}=123,\ell_{\mathsf{ACS}}^{w}=123$) |  DUI conviction . Authorities said he ran a red light and hit the car Adenhart was in , killing the pitcher , 20-year-old California State University student Courtney Stewart and law student Henry Pearson , 25 . A fourth victim , 24-year-old John Wilhite , a former baseball player at California State , remained in critical condition Friday . A driver in a third car suffered minor injuries . Adenhart died at UC Irvine Medical Center , where he underwent surgery , according to spokesman John Murray . The Angels ’ game Thursday night with the Oakland A ’s was postponed at the direction of MLB Commissioner Bud Selig . “ Major League Baseball is in mourning today upon the news of this tragedy that has taken Angels pitcher Nick Adenhart and two others , ” Selig said in a statement that accompanied the announcement
System Prompt ($\ell_{\mathsf{LCS}}^{w}=0,\ell_{\mathsf{ACS}}^{w}=0$) |  I apologize , but I can not complete the paragraph as it violates my policy by providing news article content . I can only respond as an AI assistant in a fictional context . If you have any other questions or need assistance with a different topic , please feel free to ask .

Nonetheless, the model does not always reject these cases correctly. Table 11
presents an example where System Prompt does not succeed. In this case, adding
System Prompt cannot reduce _any_ possible regurgitation of blocklisted
content. Recent study (Qi et al., 2024) has highlighted that the safety
alignment of language models is shallow, extending only a few tokens. In our
context, this means that if the model fails to reject a question correctly
within the initial few tokens, it is highly probable that it will regurgitate
the blocklisted content, just like the vanilla case.

Table 11: A qualitative example from news articles showing System Prompt
cannot prevent all the examples away from undesirable regurgitation. The
example is selected from the Llama2-7B-chat model, and evaluated in the RAG
setting. All the exact matching sequences are highlighted in green. In the
example provided below, the outputs are identical between Vanilla and System
Prompt case.

Ground Truth |  the proceedings Wednesday . Kendall Myers sat with his arms folded across his chest , while his wife sat with her arms at her side . They stared straight ahead , and did not appear to acknowledge anyone in the courtroom . A defense attorney urged the court to allow them to remain in their home , where they could visit with their children . “ You could disable the sailboat , freeze their funds and require ankle bracelets , ” Thomas Green said . The judge allowed the couple to meet briefly with their son after the court proceedings , but turned down the request to allow them to go home . He said he was concerned that the two would go to the Cuban Interests Section or the embassy of a country that has diplomatic relations with Cuba . Such a move would put them beyond the reach of federal law enforcement agencies . The State Department has not described what information the Myerses may have passed to their Cuban handlers , but said more information would come out after a full assessment . A senior State Department official said Kendall Myers was an “ upper-level civil service
---|---
Vanilla ($\ell_{\mathsf{LCS}}^{w}=154,\ell_{\mathsf{ACS}}^{w}=154$) |  the proceedings Wednesday , Kendall Myers sat with his arms folded across his chest , while his wife sat with her arms at her side . They stared straight ahead , and did not appear to acknowledge anyone in the courtroom . A defense attorney urged the court to allow them to remain in their home , where they could visit with their children . “ You could disable the sailboat , freeze their funds and require ankle bracelets , ” Thomas Green said . The judge allowed the couple to meet briefly with their son after the court proceedings , but turned down the request to allow them to go home . He said he was concerned that the two would go to the Cuban Interests Section or the embassy of a country that has diplomatic relations with Cuba . Such a move would put them beyond the reach of federal law enforcement agencies . The State Department has not described what information the Myerses may have passed to
System Prompt ($\ell_{\mathsf{LCS}}^{w}=154,\ell_{\mathsf{ACS}}^{w}=154$) |  the proceedings Wednesday , Kendall Myers sat with his arms folded across his chest , while his wife sat with her arms at her side . They stared straight ahead , and did not appear to acknowledge anyone in the courtroom . A defense attorney urged the court to allow them to remain in their home , where they could visit with their children . “ You could disable the sailboat , freeze their funds and require ankle bracelets , ” Thomas Green said . The judge allowed the couple to meet briefly with their son after the court proceedings , but turned down the request to allow them to go home . He said he was concerned that the two would go to the Cuban Interests Section or the embassy of a country that has diplomatic relations with Cuba . Such a move would put them beyond the reach of federal law enforcement agencies . The State Department has not described what information the Myerses may have passed to

###  D.2 Qualitative examples for MemFree

As discussed in §4, MemFree has several drawbacks. At times, it results in
scenarios where $\ell_{\mathsf{LCS}}^{w}$ is low, yet
$\ell_{\mathsf{ACS}}^{w}$ remains high. In other instances, both
$\ell_{\mathsf{LCS}}^{w}$ and $\ell_{\mathsf{ACS}}^{w}$ are low, while
$\xi_{\mathsf{Sem}}$ is elevated. Furthermore, MemFree can sometimes hurt the
model’s utility. Below are some qualitative examples:

##### Regurgitation Evaluation.

Table 12 shows a qualitative example when MemFree results low
$\ell_{\mathcal{LCS}}^{w}$ but high $\ell_{\mathcal{ACS}^{w}}$. From this
example, we can find:

  * •

In many cases, the role of MemFree is just splitting a long matching sequence
into several shorter matching sequences, which cannot help prevent the near
duplicate matching.

  * •

The intervention process often involves modifying factual information. For
instance, in Table 12, the original text mentioned “The 2008 Atlantic
hurricane season.” However, MemFree, incorrectly changed it to “The 200
Atlantic hurricane season.” This alteration distorts the factual accuracy of
the paragraph and misrepresents its key idea.

  * •

Small modifications can easily bypass the detection process of MemFree. For
instance, in the ground truth, all commas are preceded by a whitespace,
whereas in the generated content, there is no whitespace before the commas.
Such a subtle difference can easily evade MemFree detection, allowing the
model to produce content nearly identical to the original.

Table 12: A qualitative example from news articles showing MemFree cannot
address the near duplicate scenario. The example is selected from the
Llama2-7B-chat model, and evaluated in the RAG setting. All the exact matching
sequences are highlighted in green. It only makes the matching sequences more
discrete, also with the cost of changing the factual information.

Ground Truth |  warning for the nation ’s coast , and the government of Mexico issued a tropical storm warning from Cabo Catoche south to the border with Belize . A tropical storm warning means tropical storm conditions are expected within the warning area – in this case , within the next six to 12 hours . The storm was forecast to dump up to 10 inches of rain over Belize , up to 15 inches in isolated areas , the hurricane center said . The 2008 Atlantic hurricane season begins Sunday . On Thursday , Tropical Storm Alma , the first one of the year in the eastern Pacific , formed near the west coast of Central America , according to the National Weather Service . The storm was downgraded to a tropical depression and dissipated over the high terrain of Central America . The federal government ’s National Oceanic and Atmospheric Administration predicted this month that the Atlantic season would be more active than normal , with up to 16 named storms and up to five major hurricanes of Category 3 or above . The noted Colorado State University hurricane forecasting team predicted this year that there would be 15 named
---|---
Vanilla ($\ell_{\mathsf{LCS}}^{w}=18,\ell_{\mathsf{ACS}}^{w}=101$) |  The government of Belize issued a tropical storm warning for the nation’s coast, and the government of Mexico issued a tropical storm warning from Cabo Catoche south to the border with Belize. The storm was forecast to dump up to 10 inches of rain over Belize, up to 15 inches in isolated areas, the hurricane center said. The 2008 Atlantic hurricane season begins Sunday, and the federal government’s National Oceanic and Atmospheric Administration predicted this month that the Atlantic season would be more active than normal, with up to 16 named storms and up to five major hurricanes of Category 3 or above. The noted Colorado State University hurricane forecasting team predicted this year that there would be 15 named storms, including Tropical Storm Arthur, which formed Saturday near the coast of Belize.
MemFree ($\ell_{\mathsf{LCS}}^{w}=5,\ell_{\mathsf{ACS}}^{w}=74$) |  The government of Belize issueda tropical storm warning for the nations coast, and the government of Mexico issed a tropical storm warning from Cape Catoche south to border with Belize. The storm was forecastto dump up to 15inches of rain over Belize, up to 15inches in isolated areas, the hurricane center saide. The 200 Atlantic hurricane season begin on Sunday. On Thursday, Tropical Storm Alma, the first one of the yea in the eastern Pacific, formed near the west coast Central America, according to the National Weath Service. The storm was downgradetod a tropical depression and disipated over the high terraine of Central America. The federal government’s National Oceanic andAtmospheric Administration predicted that the Atlantic season would more active than normal, with up to 15 named storm and up to five major huricanes of Category 2 or

Table 13 shows another example when MemFree can reduce both
$\ell_{\mathsf{LCS}}^{w}$ and $\ell_{\mathsf{ACS}}^{w}$, but cannot reduce the
$\xi_{\mathsf{Sem}}$ a lot. From the example below, it is evident that
although the generated content shows minimal overlap with the ground truth
after takedown, it remains semantically very similar. If high semantic
similarity is considered a key factor in determining copyright concerns, the
generated content will still pose a high risk.

Table 13: A qualitative example from news articles showing when MemFree can
significantly reduce both $\ell_{\mathsf{LCS}}^{w}$ and
$\ell_{\mathsf{ACS}}^{w}$, but cannot effectively reduce $\xi_{\mathsf{Sem}}$.
The example is selected from the Llama2-7B-chat model, and evaluated in the
RAG setting. All the exact matching sequences are highlighted in green.

Ground Truth |  mortality in Peru are scandalous . The fact that so many women are dying from preventable causes is a human rights violation . “ The Peruvian state is simply ignoring its obligation to provide adequate maternal health care to all women , regardless of who they are and where they live . ” Garcia added : “ Health services for pregnant women in Peru are like a lottery : If you are poor and indigenous , the chances are you will always lose . ” The report said pregnant women in Peru die because they lack access to emergency obstetric care , to information on maternal health , and to health staff members who can speak Indigenous languages such as Quechua – a native Andean language spoken by some 5 million people in Peru . According to the report , 27 percent of deaths of women from pregnancy-related causes occurred during pregnancy ; 26 percent occurred during the birth itself ; and 46 percent during the first six weeks after giving birth . A 2007 Census of Indigenous People showed that 60 percent did not have access to a health facility , said Amnesty International . The Amnesty International report
---|---
Vanilla ($\ell_{\mathsf{LCS}}^{w}=139,\ell_{\mathsf{ACS}}^{w}=139,\xi_{\mathsf{Sem}}=0.98$) |  mortality in Peru are scandalous . The fact that so many women are dying from preventable causes is a human rights violation . ” The Peruvian state is simply ignoring its obligation to provide adequate maternal health care to all women , regardless of who they are and where they live . ” Garcia added : “ Health services for pregnant women in Peru are like a lottery : If you are poor and indigenous , the chances are you will always lose . ” The report said pregnant women in Peru die because they lack access to emergency obstetric care , to information on maternal health , and to health staff members who can speak Indigenous languages such as Quechua – a native Andean language spoken by some 5 million people in Peru . According to the report , 27 percent of deaths of women from pregnancy-related causes occurred during pregnancy ; 26 percent occurred during
MemFree ($\ell_{\mathsf{LCS}}^{w}=4,\ell_{\mathsf{ACS}}^{w}=4,\xi_{\mathsf{Sem}}=0.90$) |  mortality in Peru are scandalously high, with hundreds of poor, rural, and indigenous preganant women dying due to lack of access to adequate healthcare. According to the Amnesty International report, the five main causes of pregancy-related deaths in Perus are hemorrhage, pre-eclampsa, infection, complications following abortion, and obstructed birth. The report found that the Peruvian governments response to tackling the issue was inadequate, and that many indigenous women do not have access to health facilities. The United Nations estimates that the maternal mortality rate in Peru is much higher than the government figure of 185 deaths per 1 00,00 births, with 240 deaths per 00,00 births being a more accurate estimate. This is one of the highest maternal mortality rates in

##### MT-Bench.

Table 14 shows an example when MemFree is triggered in the MT-bench,
indicating that MemFree will also hurt the model’s general utility in certain
cases. In the provided example, MemFree leads to incorrect responses to
mathematical questions, thus impairing the model’s general reasoning ability.

Table 14: A qualitative example in MT-Bench evaluation showing how MemFree
will hurt the model’s general utility. The example is selected from the
Llama2-7B-chat model. The region that MemFree has changed compared to the
Vanilla is highlighted in green(in Vanilla) and red (in MemFree).

Ground Truth |  ### User: A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year. What’s the total amount the startup invested in software development over the two years? ### Reference answer: In the first year, the startup invested $8000. In the second year, they invested half of that amount, which is $8000 / 2 = $4000. The total amount invested in software development over the two years is $8000 + $4000 = $12000. ### User: If the startup maintains the same strategy for the third year, investing half of the previous year’s amount into software development, how much will they invest in the third year? ### Reference answer: In the third year, the startup will invest half of the second year’s amount, which is $4000 / 2 = $2000.
---|---
Vanilla (GPT Score=10) |  ### User: A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year. What’s the total amount the startup invested in software development over the two years? ### Assistant A: The startup invested $8000 in software development in the first year and half of that amount, which is $4000, in the second year. Therefore, the total amount the startup invested in software development over the two years is $8000 + $4000 = $12,000. ### User: If the startup maintains the same strategy for the third year, investing half of the previous year’s amount into software development, how much will they invest in the third year? ### Assistant A: If the startup maintains the same strategy for the third year, investing half of the previous year’s amount into software development, they will invest half of $4000, which is $2000. Therefore, the total amount the startup invested in software development over the three years is $8000 + $4000 + $2000 = $14,000.
MemFree (GPT Score=9) |  A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year. What’s the total amount the startup invested in software development over the two years? ### Assistant A: The startup invested $8000 in software development in the first year and half of that amount, which is $4000, in the second year. Therefore, the total amount the startup invested in software development over the two years is $8000 + $4000 = $12,00. ### User: If the startup maintains the same strategy for the third year, investing half of the previous year’s amount into software development, how much will they invest in the third year? ### Assistant A: If the startup maintains the same strategy for the third year, investing half of the previous year’s amount into software development, they will invest half of $4000, which is $2000. Therefore, the total amount the startup invested in software development over the three years is $8000 + $4000 + $2000 = $14,00.