# Copyright Protection in Generative AI: A Technical Perspective

###### Abstract

Generative AI has witnessed rapid advancement in recent years, expanding their
capabilities to create synthesized content such as text, images, audio, and
code. The high fidelity and authenticity of contents generated by these Deep
Generative Models (DGMs) have sparked significant copyright concerns. There
have been various legal debates on how to effectively safeguard copyrights in
DGMs. This work delves into this issue by providing a comprehensive overview
of copyright protection from a technical perspective. We examine from two
distinct viewpoints: the copyrights pertaining to the source data held by the
data owners and those of the generative models maintained by the model
builders. For data copyright, we delve into methods data owners can protect
their content and DGMs can be utilized without infringing upon these rights.
For model copyright, our discussion extends to strategies for preventing model
theft and identifying outputs generated by specific models. Finally, we
highlight the limitations of existing techniques and identify areas that
remain unexplored. Furthermore, we discuss prospective directions for the
future of copyright protection, underscoring its importance for the
sustainable and ethical development of Generative AI.

Machine Learning, ICML

Jie Ren1 Han Xu1 Pengfei He1 Yingqian Cui1 Shenglai Zeng1 Jiankun Zhang2,3,4
Hongzhi Wen1 Jiayuan Ding1 Hui Liu1 Yi Chang2,3,4 Jiliang Tang1
1Department of Computer Science & Engineering, Michigan State University
2School of Artificial Intelligence, Jilin University
3International Center of Future Science, Jilin University
4Engineering Research Center of Knowledge-Driven Human-Machine Intelligence,
MOE, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Recently, generative AI models have been extensively developed to produce a
wide range of synthesized content, including text, images, audio, and code,
among others. For example, advanced image generative models, such as Diffusion
Models (DMs) (Ho et al., 2020), can produce highly realistic and detailed
photographs and paintings. Similarly, large language models (LLMs) like
ChatGPT (Achiam et al., 2023) can be leveraged to compose coherent and
creative text articles with arbitrary genres and storylines. We refer to these
advanced models as “Deep Generative Models” (DGMs). However, because of the
remarkable fidelity and authenticity of the generated contents from DGMs,
concerns have been raised regarding the associated copyright issues. For
example, the New York Times sued OpenAI and Microsoft for using copyrighted
work for training chatGPT
111https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-
microsoft-lawsuit.html, https://www.nytimes.com/2024/01/08/technology/openai-
new-york-times-lawsuit.html. Midjourney was accused to output images copied
from commercial films222https://spectrum.ieee.org/midjourney-copyright. These
copyright issues may pertain to various parties involved in the generation
process. In specifics:

  * (1)

Source Data Owners. To generate high-quality contents, DGMs require training
on a large amount of data, collected from various resources such as the
Internet, even without the permission of the original data owner. As
demonstrated in recent studies (Carlini et al., 2022, 2023), it is likely that
both DMs and LLMs can produce contents with a high coincidence to parts of the
training data samples. Besides, DGMs can also be utilized to directly edit the
contents or imitate the artistic styles from source images and texts. These
facts raise concerns for the data owners, as DGMs can generate data that
closely resembles or replicates their original data without authorization.

  * (2)

DGM Users. DGMs are also frequently utilized by DGM users and assist DGM users
for creative composing. However, whether the DGMs users should receive
copyright for their generated contents is still a complex and evolving legal
and ethical issue. For example, in 2023333Registration of “Zarya of the Dawn”,
US Copyright Office refused to register a graphic novel for an artist
facilitated by Midjourney444https://www.midjourney.com/ (a popular AI image
generation model). However, another giant generative-AI company, OpenAI,
claims that the model users own the created data via the models from OpenAI,
including the right to reprint, sell, and merchandise – regardless of whether
an image was generated through a free or paid
credit555https://openai.com/policies/terms-of-use.

  * (3)

DGM Providers. The DGM providers also contribute great efforts on the training
of a DGM. It includes collecting and processing the large amount of data,
engineering on the training and tuning for optimized model performance.
Therefore, the DGM providers also have reasons to demand the copyright of the
generative contents.

Until the end of 2023, it has been under active discussion among government
officials, lawmakers, and the general public, about who should claim the
copyright and how to protect it. For instance, in early 2023, the U.S.
Copyright Office initiated a process to gather feedback on copyright-related
concerns related to generative AI. This included the discussions on the scope
of copyright for works created via using AI tools and the use of copyrighted
materials in AI training. Additionally, legal perspectives on these matters
have also been explored in various opinions (Zirpoli, 2023; Franceschelli &
Musolesi, 2022; Samuelson, 2023). However, in this article, we approach the
topic from a different angle, where we provide an overview on existing
computational methodologies, which have been proposed for copyright protection
from a technical perspective. These are potential viable solutions for either
AI model creator or model users, for copyright protection. In general, these
computational techniques can be categorized according to the receiver of the
copyright:

  * •

For source data owners, the protection on their original works could be
achieved by: (a) “Crafting unrecognizeable examples”, which refers to the
process that the data owners manipulate their owned data to cause DGMs hard to
extract useful information during training; (b) “Watermark techniques” can be
used by the data owner to trace and distinguish whether a generated work is
produced based on their original creation; (c) “Machine Unlearning” means that
the data owner can request a deletion of their data from the model or its
output, once they identify the copyright infringement; (d) “Dataset De-
duplication” removes duplicated data to mitigate the memorization effect to
prevent the training data from being generated; (e) “Alignment” which uses a
reward signal to reduce the memorization in LLMs; and (f) others including
improved training and generation algorithms for better behaviors of LLMs.

  * •

For DGM users who create new works assisted by DGMs, the protected object is
the generated contents from DGMs. Thus, the technical methodology for this
type of copyright is barely related to the generation process of DGMs, as
traditional copyright protection strategies can be also applied for protection
of the DGM generated contents.

  * •

For DGM providers, there are representative “watermarking strategies” to
inject the watermarks into the generated content or model parameters, such
that we can track the ownership of the model.

Given the diversity in protection objectives, as well as DGM applications, we
are motivated to have an overview on existing computational methods in this
direction. Essentially, in Section 2, we will majorly discuss the copyright
protection techniques for DGMs in the image domain. In Section 3, we discuss
the strategies for text generation. Finally, we discuss the related problems
in other domains, such as graphs, codes and audio generation in Section 4. In
each section, we will introduce the background knowledge of existing DGMs, as
well as the existing methodologies for data protection under different
scenarios.

##  2 Copyright in Image Generation

In this section, we first introduce background knowledge on existing popular
image generation models. Then, we define the problems related to the copyright
issues for these image generation models, and introduce different strategies
which can be utilized for data and model copyright protection.

###  2.1 DGMs for Image Generation

In the era of deep learning, there are various types of image generation
models that have been extensively studied. Most of them follow a pipeline to
first collect a set of $n$ training images
$\mathcal{X}=\\{x_{i}\\}^{n}_{i=1}$, whose samples’ distribution can be
denoted as $p(x)$, and a model is trained to explicitly or implicitly estimate
this data distribution. We denote the learned distribution as $p^{\prime}(x)$.
During the image generation stage, the new images are generated by sampling
from the learned distribution $p^{\prime}(x)$. For example, some popular
models (Goodfellow et al., 2014a; Ho et al., 2020) can be written as the form
of $f(\cdot)$, which takes random noise $z$ (following distributions $p_{z}$
such as Gaussian Distribution) as input and generates samples $f(z)$. Besides,
there are more advanced generative models, which can generate new samples
based on user’s particular demands. For example, there are conditional
generative models (Mirza & Osindero, 2014) to generate samples $f(z,c)$ which
belong to a specific sub-distribution of $p(x)$, where $c$ can denote the
given condition. In text-to-image models (Rombach et al., 2022), the condition
$c$ can also be formatted as a language prompt for the desired generated
samples. Furthermore, there are also advanced models which can directly take
inputs from a few source samples $x$, and then edit or modify them to obtain
new samples $f(x)$. In the following, we provide a brief overview on the
mechanisms of several popular image generative models.

  * •

Autoencoders (Vincent et al., 2010; Kingma & Welling, 2013) refer to the
generative models, which consist of an encoder module and a decoder module for
image reconstruction (see Figure 1). In detail, the encoder projects a real
image $x$ into a latent vector $z$ in the latent space. Then, the decoder
takes the input of the latent vectors $z$ to generate new samples
$x^{\prime}$. The decoder is trained to reconstruct the original sample $x$
from the its latent $z$ produced by the encoder. In the work (Kingma &
Welling, 2013), the latent vectors are further regulated to follow a standard
Gaussian distribution. During the generation process, new latent vectors are
sampled from the regulated distribution directly without the encoder and then
decoded to obtain the generated images.

Figure 1: An Illustrative Framework of Autoencoder. (Image Credit to Kingma &
Welling (2013))

  * •

Generative Adversarial Networks (GAN) (Goodfellow et al., 2014a) are proposed
to synthesize images via solving a min-max game. In GANs, there are two
adversarial players, a generator $G(\cdot)$ and a discriminator $D(\cdot)$.
The discriminator aims to determining whether a sample is real or synthesized
by the generator $G(\cdot)$, whereas the generator aims to generate images
that are highly realistic, to the extent that they can deceive the
discriminator. After being alternatively optimized and competing with each
other, the generator and the discriminator will eventually converge to a point
at which the discriminator is well trained to distinguish the real and fake
images, but the generator can generate convincing images that are still
possible to confuse the discriminator. Following this intuition, GAN is
formulated as to solve an adversarial min-max problem:

$\displaystyle\min_{G}\max_{D}\mathbb{E}_{{x}\sim p({x})}\log
D({x})+\mathbb{E}_{{z}\sim p_{{z}}}\log(1-D(G({z})))$ (1)

In detail, given the latent vector $z\sim p_{z}$ ($p_{z}$ is the predefined
distribution like an isotropic Gaussian noise), the generator $G(\cdot)$ first
generates new images $G(z)$. $G(\cdot)$ is trained to generate images that can
maximize the loss of Eq. (1), while $D(\cdot)$ is trained to minimize it for
distinguishing the real samples $x$, and synthesized samples $G(z)$. It means
that the goal of $G(\cdot)$ is to be trained to generate more authentic and
realistic samples to mislead $D(\cdot)$.

Based on this general and initial idea of GAN, more variants have been
proposed for conditional generation or image editing. For example, Conditional
GAN (Mirza & Osindero, 2014) generates samples in specific classes of the
training data distribution. GAN-based Deepfake techniques (Westerlund, 2019)
can be utilized for direct image editing, such as swapping the face from
source images to a target person (Lu., 2018).

  * •

Diffusion Models, like DDPM (Ho et al., 2020), are a type of generative models
which generate images via two opposing procedures: a forward process and a
reverse process. As shown in Figure 2, the forward process gradually adds
noise to an input image until it becomes an isotropic Gaussian distribution.
Then, in the reverse process, a denoising neural network is trained to learn
how to recover an image from the Gaussian distribution. In detail, given an
input image $x_{0}$, the forward process injects increasing level of noises
through $T$ steps and gets the Gaussian sample $x_{T}$:

${x}_{t}=\sqrt{1-\beta_{t}}{x}_{t-1}+\beta_{t}\epsilon~{}~{}~{}\text{where}~{}~{}1\leq
t\leq T$

where $\epsilon\sim\mathcal{N}(0,I)$ and $\beta_{t}$ controls the variance of
injected noise. Then, the reverse process learns to denoise from noisy
variable $x_{t+1}$ to less-noisy variable $x_{t}$. The denoising step is
approximately equivalent to estimating the injected noise $\epsilon_{t}$ with
a parametric neural network $\epsilon_{\theta}(x_{t+1},t)$ in practice. The
network is trained to minimize $l_{2}$ distance between estimated noise and
true noise:

$\begin{aligned}
\mathcal{L}_{\text{dm}}=\mathbb{E}_{t,{x}_{0},{\epsilon}\sim\mathcal{N}(0,I)}\left[\left\|{\epsilon}-{\epsilon}_{\theta}(x_{t+1},t))\right\|_{2}^{2}\right]\end{aligned}.$
(2)

Refer to Figure 2, once the training of denoising network
${\epsilon}_{\theta}$ is complete, a new image can be generated starting from
a random noise by solely employing the reverse process.

Figure 2: An Illustration of the DDPM Framework. (Image Credit to Ho et al.
(2020))

Based on DDPM, Latent Diffusion Model (LDM) (Rombach et al., 2022) is a
specific class of diffusion models, which apply the forward process in the
latent space instead of the pixel (image) space. In detail, to train an LDM,
an input image $x_{0}$ is first mapped to its latent representation
${z}_{0}=\mathcal{E}({x}_{0})$, where $\mathcal{E}(\cdot)$ is a given image
encoder. Then the forward process continues by repeatedly adding noise to the
latent representation $z_{0}$ for $T$ steps, while the reverse process
generates data by the denoising network $\epsilon_{\theta}$ trained with the
following loss:

$\displaystyle\mathcal{L}_{\text{LDM}}=\mathbb{E}_{t,z_{0},{\epsilon}\sim\mathcal{N}(0,I)}\left[\left\|{\epsilon}-{\epsilon}_{\theta}\left({z}_{t+1},c,t\right)\right\|_{2}^{2}\right]$
(3)

Once the latent representation ${z}^{\prime}$ is generated by the reverse
process, it is then decoded by a decoder model $\mathcal{D}(\cdot)$ to get the
final generated image ${x}^{\prime}$. The introduction of Latent Diffusion
Model not only improves the generation efficiency (given that the dimension of
$z_{0}$ is significantly smaller than $x_{0}$), but also enables diffusion
models to combine desired condition $c$ in Eq. (3) to enrich the generation
tasks. One of the most popular “text-to-image” diffusion models, Stable
Diffusion is constructed based on LDM when $c$ is text prompt.

  * •

Text-to-image Diffusion Models, such as Stable Diffusion (Rombach et al.,
2022), MidJourney, and DALL$\cdot$E2, allow the model users to generate images
based on language descriptions, which are also called “prompt-guided”
diffusion models. Stable Diffusion is achieved by combining LDMs with CLIP
(Radford et al., 2021), a powerful model that learns the connection between
the concepts of human language and images. Briefly speaking, given one of a
few sentences as a prompt, a new image is produced following the basic
pipeline of LDM, but conditioning on the (embedding of) language information
of the given prompt. As a result, the generated contents can contain the
desired semantic features and patterns by the prompt.

Recently there are more advanced techniques such as Textual Inversion (Gal et
al., 2022) and DreamBooth (Ruiz et al., 2023) which further upgrade text-to-
image diffusion models for customized image editing and modifying. For
example, refer to Figure 3, one can fine-tune Stable Diffusion on a few “input
images” via the algorithm of DreamBooth, to make the model learn new objects
from the input images, and then generate new images for the targeted object in
different scenes.

Figure 3: An Illustration of DreamBooth. (Image Credit to Ruiz et al. (2023))

###  2.2 Copyright Issues in Image Generation

The development of Deep Generative Models (DGMs) marks a noteworthy
advancement in image generation. Nevertheless, the impressive quality and
authenticity of the generated images, as well as the efficiency in producing
new ones, give rise to legitimate concerns regarding copyright matters within
the realm of DGMs.

Data copyright protection. For the source data owner, which refers to the
party or individual who owns the originality of image works, their data can be
intentionally or unintentionally collected by model trainers as training
samples to construct DGMs as introduced above. For example, recent studies
(Carlini et al., 2023; Vyas et al., 2023) have demonstrated that popular DGMs
are highly possible to completely replicate their training data samples (see
Figure 4), or part of the training data samples (Shan et al., 2023; Van Le et
al., 2023). The possibility of data replication may severely offend the
ownership of the original data samples. Moreover, the development of fine-
tuning strategies such as DreamBooth can greatly improve the efficiency for
unauthorized parties to directly edit or modify the source data to obtain new
samples, which also severely infringes the copyright of the original works.

Figure 4: An example of data replication. (Image Credit to Carlini et al.
(2023))

Model copyright protection. To obtain DGMs with advanced generation
performance, it is always necessary for the model trainers to invest a
significant amount of funds and labor. It grants them intellectual property
rights over the trained model. However, recent works also identify the
possibility to steal others models (Tramèr et al., 2016).

###  2.3 Source Data Owner’s Copyright Protection

In this subsection, we review the protection techniques for the source data
owners. In general, we can categorize these techniques into four major types.

  * •

Unrecognizable Examples, which aim to prevent models from learning important
features of protected images. This often results in the generation of either
low-quality images or completely incorrect ones;

  * •

Watermark, which involves inserting unnoticeable watermarks into protected
images. Data owners can detect these watermarks if their data is used for
training;

  * •

Machine Unlearning, which aims to ablate the contribution of copyrighted data
on the DGM to prevent the model from generating based on the information of
the protected images;

  * •

Dataset De-duplication, which mitigates the memorization problem by removing
the duplicated data from the training data.

In the four categories, unrecognizable examples and watermarks are employed
from the side of source data owners. They modify their data before releasing
to the public to protect the copyright. As for machine unlearning and dataset
de-duplication, they are proposed for the model builders who aim to provide
DGM services legally without infringement on the copyrighted data. Next, we
will introduce these strategies in detail.

####  2.3.1 Unrecognizable Examples

From the perspective of the source data owner, one major strategy for data
protection is to make their data “unrecognizable” to potential DGMs. In
general, a data owner can consider injecting imperceptible perturbations into
the protected images, such that the DGMs cannot effectively exploit useful
features from them, and hence hardly generate qualified new samples. These
works of “unrecognizable examples” can be categorized based on how the DGM is
utilized to extract information from the source image: (1) “inference-stage
protection” counteracts DGMs which operate without the need for fine-tuning on
the source image $x$. These models are capable of extracting image features
directly from the source image $x$ in an inference mode; (2) “training-stage
protection” is against DGMs that is fine-tuned on the source images to extract
the desired information and generate new samples based on the extracted
information. Targeting on these two types of image editing manners of DGM, a
variety of (the copyright of) data protection strategies are devised.

1) Inference stage protection. Well-trained DGMs, such as GAN models and
diffusion models, can be directly used for various image generation tasks,
such as image-to-image synthesis and image editing. Thus, it is crucial to
address inference-stage protection which aims at preventing models from
extracting important information.

Using deep learning models for image modifying or editing has been conducted
on GAN-based methods. For example, GAN Inversion (Zhu et al., 2016, 2020) is a
target image manipulation technique which can facilitate GAN models to modify
the attributes in an image, e.g., changing a person’s age or hair color. Refer
to Figure 5, this technique inverts a victim picture $x_{v}$ to obtain a
latent representation $z_{v}$ of the GAN model. By inputting $z_{v}$ back to
the generator, $G(z_{v})$ can recreate images closely resembling $x_{v}$.
Utilizing the “latent code manipulation” (Abdal et al., 2019), this latent
representation $z_{v}$ is altered in the desired direction $n$ to produce
$z_{v}+n$, thereby generating new images. This method, commonly used for
altering photographs or artistic creations, poses a risk of infringing on the
copyrights of the original photographers and artists. Therefore, as a
countermeasure against GAN Inversion, UnGANable (Li et al., 2022b) is the
first defense system against GAN inversion to significantly reduce its
performance which naturally leads to the failure of GAN-Inversion-based face
manipulation. The basic idea resembles to the method of adversarial examples
(Goodfellow et al., 2014b): given an input (victim) image $x_{v}$, it searches
for the adversary $x^{adv}$ that maximizes the discrepancy between
${x}^{adv}_{v}$ and original ${x}_{v}$ in the latent space of the generator
$G(\cdot)$ and the feature space of a feature extractor pre-trained by a
third-party. By maximizing the discrepancy of $x_{v}$ and $x^{adv}_{v}$ in the
feature space, the GAN Inversion process is misguided to another latent
representation, which is distant from the correct latent representation
$z_{v}$. As a result, the generation process can hardly generate new images
with features from the original image $x_{v}$.

Figure 5: An overview of GAN Inversion. (Image Credit to Ruiz et al. (2020))

Similar ideas of using adversarial examples are also shown to be applicable to
image protection in other GAN-based models. Ruiz et al. (2020) aimed at data
protection against Image-Translation GAN (Liu et al., 2017), which is a
variant of conditional GAN that directly generates new images by inputting a
source image $x$ instead of a random latent vector. It can generate an image
which is manipulated from the source image $x$. The work (Ruiz et al., 2020)
was proposed to add human-imperceptible perturbations, $\delta$, onto the
input images via maximizing the distortion in the generated sample:

$\displaystyle\max\limits_{{||\delta}||\leq\delta}||(G({x}+{\delta})-G({x})||_{p}~{}~{}~{}p=2~{}\text{or}~{}\infty$
(4)

where ${||\delta}||\leq\delta$ constraints that the added term $\delta$ is
imperceptible to human eyes. By maximizing the distance between the generated
image based on the original image and the perturbed image, the generated image
is much different from $x$ and has no threat on it.

There are more works studying how to leverage the idea of adversarial examples
(Goodfellow et al., 2014b) to protect images against various GAN-based models.
Yeh et al. (2020) aimed to protect users’ images from DeepNude (Sigal Samuel,
2019), a deep generative software based on the image-to-image translation
algorithm. This method also borrows the idea of adversarial examples. It
defines adversarial loss for Nullifying Attack which maximizes the distance of
features extracted by the generator $G(\cdot)$ between perturbed images and
the original image, as well as adversarial loss for Distorting Attack which is
the same as Eq. (4). Besides, Huang et al. (2021) expanded adversarial
examples to more practical “grey-box” and “black-box” settings, which refer to
the case that the data owner is unaware of the specific model that might be
used for potential copyright infringement. This method adopts a surrogate
model to approximate the manipulation model and update surrogate parameters
and adversarial examples in an alternative manner. These examples demonstrate
that the framework to generate adversarial examples can be adapted to
different image manipulation models by properly designing the loss function
for different specific tasks.

Beyond GAN-based models, recently diffusion models have also been exposing a
significant risk for copyright infringement. Similar to the GAN-based methods,
diffusion models have also been exploited for various types of modifying or
editing tasks. For example, Textual Inversion (Gal et al., 2022) is an image
modifying technique by Stable Diffusion, without any training or fine-tuning
process. Refer to the illustration in Figure 6, given a few source samples
$x$, Textual Inversion aims to extract the knowledge from $x$ by linking the
sample $x$ to a specific text string such as $S_{*}$. This process is achieved
by adjusting the text embedding of $S_{*}$ in Stable Diffusion to embed the
information of $x$ into it. The model user can utilize $S_{*}$ to compose new
prompts, and generate new images with the information embedded in $S_{*}$,
such as the original object or style from the source image, which might
infringe the copyright of the source images.

Figure 6: An overview of Textual Inversion. (Image Credit to Gal et al.
(2022))

Targeted on Textual Inversion, the work (Liang et al., 2023) aims to find an
adversarial example $x^{adv}$ to protect the source images $x$. $x^{adv}$ lies
out of the distribution of the generated data samples of the diffusion models.
Thus, the inversion process cannot find proper language tokens $S_{*}$
corresponding to the adversarial image. In detail, following the general
definition of the diffusion loss in Eq. (2), we define the loss function on a
specific sample $x$:
$\mathcal{L}_{\text{dm}}(x)=\mathbb{E}_{t,{\epsilon}\sim\mathcal{N}(0,I)}\left[\left\|{\epsilon}-{\epsilon}_{\theta}(x_{t+1},t))\right\|_{2}^{2}\right]$,
the work (Liang et al., 2023) aims to find a perturbation $\delta$ to maximize
the loss value of the diffusion model on $x+\delta$:

$\displaystyle\max_{{\delta}}\mathcal{L}_{\text{dm}}(x+\delta)~{}~{}\text{s.t.}~{}~{}||\delta||\leq\sigma$

Because the diffusion model has a maximized loss on the perturbed image
$x+\delta$, the sample $x+\delta$ can be seen as a natural outlier from the
distribution of the generated samples of the model. As a result, the Textual
Inversion process cannot effectively find reasonable token $S_{*}$, and thus
protect the original information.

Similar to the idea of (Liang et al., 2023), the work (Salman et al., 2023)
directly leverages the adversarial examples for data protection against a
image editing model based on LDM. In the work of (Salman et al., 2023), two
attack strategies are proposed: (a) Encoder Attack: Considering a Latent
Diffusion Model is employed for image editing (refer to Eq. (3)), given the
encoder model $\mathcal{E}(\cdot)$ which maps a source image to the latent
representation $\mathcal{E}(x)$, the encoder attack searches for an
imperceptible perturbation $\delta$ satisfying:

$\displaystyle\min\limits_{\|\delta\|_{\infty}\leq\epsilon}\|\mathcal{E}(x+\delta)-z_{target}\|^{2}_{2},$

where $z_{target}$ is a target latent representation. In this way, the latent
representation of $x+\delta$ is close to the target representation which is
pre-specified to be distinct from the original latent representation
$\mathcal{E}(x)$, which severely disrupts the LDM process. (b) Diffusion
Attack: The full image editing process can be simply denoted as a model
$f(\cdot)$, and the generated image $f(x)$ can fulfill the generation goal of
the model user, the diffusion attack directly finds a perturbation $\delta$ to
maximize the discrepancy of the generated sample $f(x+\delta)$ and $f(x)$:

$\displaystyle\min\limits_{\|\delta\|_{\infty}\leq\epsilon}\|f(x+\delta)-x_{target}\|^{2}_{2}$

As a result, the newly generated sample $f(x+\delta)$ is close to the target
image $x_{target}$ thus very different from $f(x)$, which breaks the original
image editing goal. However, addressing this optimization problem is
computationally costly due to the extensive number of parameters involved and
the multiple steps during the diffusion process. Consequently, they suggested
calculating the loss based on only a limited number of steps instead of the
entire diffusion process.

According to the discussion in (Salman et al., 2023), currently existing data
protection strategies based on adversarial examples still face major drawbacks
that can hinder their feasibility and reliability in practice:

  * •

Lack of robustness to transformations. The protected images may also be
subject to image transformations and noise purification techniques, such as
cropping the image, adding filters to it, or applying a rotation. However, the
authors also mention that this problem can be addressed by leveraging “robust”
adversarial perturbations, as discussed in (Athalye et al., 2018; Kurakin et
al., 2018).

  * •

Generalization on different models. The protection techniques that are
designed for one generative model may not be guaranteed to be effective
against future versions of these models or other types of generative models.
The authors mentioned that one could hope to improve on the “transferability’
of adversarial perturbations (Ren et al., 2022; He et al., 2023; Li et al.,
2024). However, these “transferable” perturbations will not always be
transferable for all circumstances.

2) Training stage protection Different from directly employing DGMs to
generate new images, DGMs are also usually trained or fine-tuned on some
source images $x$ to effectively learn useful information from $x$ for future
generations. Therefore, “training-stage protection” aims to add imperceptible
noise to the copyrighted images, to break the training process of potential
DGMs for data copyright protection.

Figure 7: An illustration of transformation-aware image protection against
Deepfake. (Image Credit to Yang et al. (2021))

This type of method is first explored on GAN-based models. For example, GAN-
based Deepfake models (Lu., 2018) are representative tools that can be
leveraged to swap the faces from the source images to the faces of a target
person, which severely abuses the copyrights of both source image holders and
target people. Similar tools like EditGAN (Ling et al., 2021) and
Introspective Adversarial Network (Brock et al., 2016) are developed to edit
images that pose more threats to the copyrights of creative and artistic
works. To handle this problem, Yang et al. (2021) proposed to utilize the idea
of adversarial examples (Goodfellow et al., 2014b), to break the balance in
the min-max game in GAN-based DeepFake models. Specifically, they focus on
Deepfake models which are trained on the target person’s face images $x$ to
generate other face images belonging to the target person. To protect the
targeted face $x$ being exploited by Deepfake, they directly adopt the fast
gradient sign method (FGSM) method to generate adversarial examples for GAN-
based models,

$\displaystyle{x}^{\text{adv}}={x}+\alpha\text{sign}(\nabla_{x}\mathcal{L}_{D}(\text{Tr}({x}),y_{\text{real}})),$
(5)

where $\mathcal{L}_{D}$ refers to the loss of the discriminator in GAN. During
the generation process, the discriminator $D(\cdot)$ will have a large loss
value on the target samples ${x}_{\text{target}}$, which consequently breaks
the balance of the min-max game during the training of GAN. As a result, the
generated images based on the protected target images will have a degraded
quality (see Figure. 7). Notably, in Eq.(5), a transformation operator
$\text{Tr}(\cdot)$ is introduced to improve the robustness of the perturbation
under various image transformations, including resizing, affine transformation
and image remapping.

Wang et al. (2022) pointed out that previous adversarial perturbations via
gradient-based strategies (e.g, FGSM) could be easily removed or destroyed by
image reconstructions such as MagDR (Chen et al., 2021b) and proposed Anti-
Forgery targeted on GAN-based Deepfake attacks. Anti-Forgery generates
perturbations that are robust to input transformation, natural to human eyes,
and applicable to black-box settings. They observed that adversarial
perturbations on the LAB color space are robust to input reconstruction.
Therefore, they converted the input from RGB space to the LAB color space and
added perceptual-aware adversarial perturbations to the color channel to
maintain robustness against input transformations including image
reconstruction (Chen et al., 2021b) and image compression (Dziugaite et al.,
2016).

Regarding diffusion models (Rombach et al., 2022; Ruiz et al., 2023), which
can be easily deployed to mimic the style of specific artists, via advanced
fine-tuning techniques, the special structure of diffusion models (see Section
2.1) poses unique challenges, especially the denoising procedure during the
reverse process can eliminate noises added to the original image (Nie et al.,
2022).

Figure 8: A high-level overview of GLAZE. (Image Credit to Shan et al. (2023))

GLAZE (Shan et al., 2023) is a representative copyright protection method
focusing on text-to-image LDM and aiming at protecting artists from style
mimicry. As shown in Figure 8, the core idea of GLAZE is to guide the
diffusion model to learn an alternative target style $S_{T}$ that is totally
different from the style of protected images. In detail, the method consists
of three steps: target style choosing, style transfer, and cloak perturbation
computation. GLAZE first chooses a target style $S_{T}$ which is sufficiently
different from the protected style. A pre-trained style-transfer model
$\Omega$ is utilized to transfer the protected artworks $x$ into the target
style $\Omega(x,S_{T})$ for optimization. Then, GLAZE computes the cloak
perturbation $\delta_{x}$ by:

$\min\limits_{{\delta}}\|\mathcal{E}(x+\delta)-\mathcal{E}(\Omega(x,S_{T}))\|^{2}_{2}+\alpha\max(\text{LPIPS}({\delta}_{x})-p,0),$
(6)

where $\mathcal{E}(\cdot)$ is the feature extractor of the LDM. This objective
minimizes the distance between features of perturbed images and target-style
transferred images, and LPIPS (Zhang et al., 2018) constraints the
perturbation to be imperceptible. When fine-tuning diffusion models on
protected arts, the generated images will mimic the target style rather than
the arts’ true style.

MIST (Liang & Wu, 2023) emphasizes that existing methods generate
perturbations relying on some strong assumptions on a specific model which are
hard to generalize to other scenarios. For example, perturbations generated
for image-to-image DGMs usually fail for Textual Inversion (Salman et al.,
2023). Therefore, they propose to generate perturbations that can work for
various DGMs simultaneously, including DreamBooth (training-stage) (Ruiz et
al., 2023), Textual Inversion (inference-stage) (Gal et al., 2022) and image-
to-image generation (Rombach et al., 2022). To achieve this goal, they combine
semantic loss from (Liang et al., 2023) and textual loss from (Salman et al.,
2023). They empirically show that the maximization of semantic loss leads to
chaotic contents in the generated image, and the maximization of textual loss
leads to a mimic of the pre-specified target image. Their empirical results
reveal that perturbations from the combination of two losses can protect
images under different scenarios.

Anti-DreamBooth (Van Le et al., 2023) specifically targets a powerful and
popular finetuning technique, DreamBooth (Ruiz et al., 2023), which is
proposed to personalize text-to-image diffusion models onto given source
images. DreamBooth has a similar target as Textual Inversion but requires
fine-tuning of diffusion models. To avoid malicious usage of DreamBooth on
users’ owned images, Anti-DreamBooth aims to attack the training process of
DreamBooth, following the idea of data poisoning attacks. In detail, it
formulates the protection problem as a bi-level optimization problem, to find
perturbation $\delta$ satisfying:

$\displaystyle\delta^{*}$
$\displaystyle=\max\limits_{\delta}\mathcal{L}_{\text{cond}}({\theta^{*}},{x})$
$\displaystyle\text{s.t.}~{}\theta^{*}$
$\displaystyle=\arg\min\limits_{\theta}\mathcal{L}_{\text{db}}(\theta,{x}+{\delta})~{}\text{and}~{}\|{\delta}\|\leq\sigma$

where $\mathcal{L}_{\text{db}}$ denotes the training loss for DreamBooth, and
$\mathcal{L}_{\text{cond}}(\theta^{*},x)$ refers to the conditional loss of
sample $x$ in prompt-guided diffusion models. By solving this problem, it
searches for the perturbation, such that the fine-tuned diffusion models will
disconnect the image $x$ with its corresponding language concept because of
the high conditional loss. In this way, during the fine-tuning process,
DreamBooth will overfit the adversarial images and experience worse
performance in synthesizing images with high quality. Later, the work ADAF (Wu
et al., 2023) also focuses on text-to-image models but pays more attention to
the text part. It points out two drawbacks of existing methods: existing
methods ignore the combination of the text encoder and the image encoder;
existing methods are not robust to the perturbation of prompts. Consequently,
ADAF implements multi-level text-related augmentations to enhance defense
stability against a range of attacker prompts.

####  2.3.2 Watermarks

Another approach for protection on source data copyright is to track or detect
whether a suspect piece of artwork is generated by a model trained on the
copyrighted data. Various AI-generated image detection methods (Epstein et
al., 2023; Dogoulis et al., 2023) can be applied to distinguish whether a
sample is generated by certain models, which partially fulfill the objective.
However, these methods are still not applicable to identify the source of the
generated contents. Therefore, the “watermarking” strategy is alternatively
studied. This technique involves encoding sophisticated “identifiable
information” into the copyrighted source data, such that this information also
exists in the generated samples which are trained on the watermarked images.
Subsequently, a detector is leveraged to assess whether a suspect image
contains this encoded information, to trace and verify the ownership of
copyright.

Figure 9: An example of watermark in DiffusionShield. (Image Credit to Cui et
al. (2023b))

Before DGMs, there exist various watermarking methods(Baluja, 2017; Hayes &
Danezis, 2017; Vukotić et al., 2018; Zhu et al., 2018; Zhang et al., 2019;
Tancik et al., 2020; Luo et al., 2020) hiding data like a message or even an
image behind imperceptible perturbations. These techniques primarily
concentrate on hiding information in specific images, without being
specifically applied to DGMs. But the objective of protection copyright
against malicious DGMs is to identify hidden messages within generated images.
Focusing on DDPM (Nichol & Dhariwal, 2021), Cui et al. (2023b) evaluated
whether the injected watermarks via previous methods for traditional image
watermarks (Navas et al., 2008; Zhu et al., 2018; Yu et al., 2021) can still
be preserved in the generated samples. The empirical results show that these
methods are either partially preserved in generated images or requires large
perturbation budgets. Therefore, they proposed DiffusionShield (Cui et al.,
2023b), a watermarking method designed for diffusion models. To elaborate,
blockwise watermarks, as exemplified in Figure 9, are engineered to convey a
greater amount of information, allowing distinct copyright information to be
more readily decoded. Then, a joint optimization strategy is leveraged to
optimize both the pixel values of watermark patches, as well as a decoding
model, which is utilized to detect and decode the encoded information from the
generated images.

Fine-tuning text-to-image diffusion models, like Stable Diffusion,
demonstrates significant potential in personalizing image synthesis and
editing. Consequently, watermarking techniques are increasingly applied as a
means of copyright protection during the fine-tuning phase. GenWatermark (Ma
et al., 2023c) is the first to propose a novel watermark system that is based
on the joint learning of a watermark generator and a detector. In particular,
it adopts a GAN-like structure, where a GAN generator $G$ serves as the
watermark generator and a detector $D$ is trained to distinguish between clean
and watermarked images. Wang et al. (2023) aimed at a method that is
independent of the choice of text-to-image diffusion models so that the
perturbation can effectively protect the images from various models. In
detail, they add specific stealthy transformations on the protected images as
well as injecting a corresponding trigger into the caption of those images.
Since they use the image warping function as the watermark generator, this
method can work without a surrogate model and thus can work on different
diffusion models. Cui et al. (2023a) considered a practical scenario where
protectors can not control the fine-tuning process and emphasize that previous
methods require many fine-tuning steps to learn the embedded watermarks. In
order to make the watermark easily recognized by the model, they proposed FT-
Shield which adds imperceptible perturbations that can be learned prior to the
original image features, like styles and objects, by the text-to-image
diffusion model. In the detection stage, a binary classifier is trained to
distinguish the watermarked images and clean images. In particular, the
perturbations minimize the loss of a diffusion model trained on these
perturbed samples as shown in the training objective:

$\displaystyle\min\limits_{\delta}\min\limits_{\theta_{1}}\mathcal{L}_{dm}([\theta_{1},\theta_{2}],x+\delta,c)$
(7)

where $\theta_{1}$ denotes the parameters of the UNet (Ronneberger et al.,
2015) which is the denoise network within the text-to-image model structure,
while $\theta_{2}$ denotes parameters of the other parts; $x$ and $c$ denote
the protected image and the corresponding caption, respectively. In other
words, perturbation in Eq. (7) leads to a rapid decrease in the training loss
and thus serves as a ’shortcut’ feature that can be quickly learned and
emphasized by the diffusion model.

####  2.3.3 Machine Unlearning

In addition to the data owners, the protection of the data copyright is also
considered by model builders to ensure the legal provision of DGM services.
Companies try to filter out the copyrighted data from the training data. For
example, Stability AI cooperated with an AI startup Spawning to build tools
for data owners to claim their copyright and remove the data from the training
set of Stable Diffusion, which has removed 80 million images from the training
data of Stable Diffusion 3 666https://the-decoder.com/artists-
remove-80-million-images-from-stable-diffusion-3-training-data/. OpenAI also
provides solution to the data owners to report the violation of data copyright
777https://adguard.com/en/blog/ai-personal-data-privacy.html. Besides the data
filtering, the model builder also takes other strategies like machine
unlearning and dataset de-duplication (Section 2.3.4). Depending on whether
the intent behind their implementation is motivated by the model builders’
need to legitimize the generation process or the source data owners’
requirements for the model builders, we attribute to machine unlearning
(Bourtoule et al., 2021; Nguyen et al., 2022; Zhang et al., 2023a; Kumari et
al., 2023) as the passive method and dataset de-duplication (Webster et al.,
2023; Somepalli et al., 2023b) as the active method. For the passive method,
after the DGM is trained, the model builder provides an interface for data
owners to claim their copyright and ablate the influence of copyrighted data
from the DGM. The passive method is executed when the source data owners
request. In contrast, for the active method, the model builder considers the
copyright in the stage of model training. The active method is usually
implemented by the model builder without the request of source data owners. In
this subsection, we focus on the discussion of machine unlearning in data
copyright protection.

Especially, “Machine Unlearning” (Bourtoule et al., 2021) refers to the
protocol to make a trained model forget a specific subset of training data, by
editing the model parameter to follow the distribution identical to that of a
model trained without the forgotten subset. With different subsets of
undesirable concepts, machine unlearning can achieve not only copyright
protection, but also preserving privacy against membership inference attack
and the removal of biased, NSFW and harmful concepts.

Refer to Definition III.1 in (Bourtoule et al., 2021) and Section 3.1 in
(Nguyen et al., 2022), assuming that the collected training dataset is
$\mathcal{X}$, we notate the model obtained by training with the vanilla
learning algorithm $\mathcal{A}$ as $\mathcal{A}(\mathcal{X})$. Assuming the
undesirable (copyrighted) subset is $\mathcal{X}_{u}$, and the unlearning
mechanism is $\mathcal{U}$, the model obtained by $\mathcal{A}$ with
unlearning of $\mathcal{X}_{u}$ can be represented as
$\mathcal{A}_{\mathcal{U}}(\mathcal{X},\mathcal{X}_{u})$. The perfect
unlearning should have the distribution of parameters of unlearned model,
$\mathcal{A}_{\mathcal{U}}(\mathcal{X},\mathcal{X}_{u})$, to be identical to
the distribution of model parameter trained on the dataset stripped of the
undesirable $\mathcal{X}_{u}$, i.e.,

$\displaystyle\mathbb{D}(\mathcal{A}_{\mathcal{U}}(\mathcal{X},\mathcal{X}_{u}))=\mathbb{D}(\mathcal{A}(\mathcal{X}\backslash\mathcal{X}_{u})),$
(8)

where $\mathbb{D}(\cdot)$ denotes the distribution of a random variable. For
the problem of DGM where we consider the distribution of generated samples,
and adapt Eq. (8) into:

$\displaystyle\mathbb{D}(\mathcal{G}_{\mathcal{A}_{\mathcal{U}}(\mathcal{X},\mathcal{X}_{u})})=\mathbb{D}(\mathcal{G}_{\mathcal{A}(\mathcal{X}\backslash\mathcal{X}_{u})}),$
(9)

where $\mathcal{G}_{m}$ is the generated samples from model $m$. By choosing
different sub-dataset as $\mathcal{X}_{u}$, machine unlearning can fulfill
various objectives. When $\mathcal{X}_{u}$ is set as the data owned and
copyrighted by individuals, the DGM builder utilizes machine unlearning to
safeguard the copyright of this data. Zhang et al. (2023a) pointed out four
goals of unlearning for DGM: performance (successfully remove target data from
the model), integrity (at best keep other data of the model), generality (can
be applied to a wide range of data that covers all aspects of human
perceptions) and flexibility (can be applied to various models of different
tasks and domains). In the following, we discuss different unlearning methods
for GANs and Diffusion Models in the image domain.

Fine-tuning with a modified objective is usually used to achieve Eq. (9) for
DGMs efficiently. Compared with data filtering and re-training from scratch
(Nichol et al., 2022; Ramesh et al., 2022), it is more efficient in time and
energy. For example, a representative method (Kong & Chaudhuri, 2023a)
modifies the min-max adversarial objective of GAN by mixing the undesirable
data with generated data as negative (fake) samples of the discriminator.
Thus, the fine-tuned discriminator will not consider the undesirable data as
true samples and the generator will be fine-tuned to avoid generating it.
Another unlearning method considers how to change the guidance of conditions
in conditional DGMs, like text-to-image models. The copyrighted images can be
transformed as concepts like styles, branches, person names, and so on. By re-
directing the conditional representations of these concepts to conditions
unrelated to the copyrights, the generated images can avoid infringement on
the copyrighted images. Kong & Chaudhuri (2023b) represent the undesirable
concepts as $\mathcal{C}_{u}$. They fine-tune the DGM to push the condition
representation of concepts belonging to $\mathcal{C}_{u}$ towards a concept
$\hat{c}$ which is not in the undesirable set. Meanwhile, to keep the other
benign concepts unchanged, the fine-tuning objective also includes a term
maintaining the representation of them as follows

$\displaystyle\begin{aligned}
\min_{H^{\prime}}L\left(H^{\prime};\lambda\right)=&\mathbb{E}_{c\in\mathcal{C}\backslash\mathcal{C}_{u}}\left\|H^{\prime}(c)-H(c)\right\|+\\\
&\lambda\cdot\mathbb{E}_{c\in\mathcal{C}_{u}}\left\|H^{\prime}(c)-H(\hat{c})\right\|,\end{aligned}$
(10)

where $H^{\prime}(\cdot)$ is the fine-tuned condition representation function,
while $H(\cdot)$ is the original condition function. The condition
representation of undesired $\mathcal{C}_{u}$ is re-directed towards the
benign concept $\hat{c}$ and can avoid to guide the model to produce the
samples of concepts from $\mathcal{C}_{u}$. Kong & Chaudhuri (2023b) applied
this general framework on different DGMs like class-conditional GAN (Mirza &
Osindero, 2014), and GAN-based text-to-image models (Zhu et al., 2019).
Besides, Kong et al. (2021) also extended this method to diffusion-based text-
to-speech models.

More unlearning methods by fine-tuning are tailored for Generative Diffusion
Models, especially the text-to-image Stable Diffusion (SD) (Rombach et al.,
2022) (refer to Section 2.1). Specifically, Kumari et al. (2023) proposed to
unlearn SD by matching the conditional distribution of undesirable concepts to
anchor concepts in every diffusion step, which means the output of the
denoising network ${\epsilon}_{\theta}$ is flipped to anchor concepts that is
unrelated to the undesirable ones. The anchor concepts can be random noise,
${\epsilon}$, or the conditional distribution of benign concepts $\hat{c}$.
With the anchor concept of random noise, the fine-tuning objective is

$\displaystyle\arg\min_{{\epsilon}_{\theta}^{\prime}}\mathbb{E}_{{\epsilon},z,c\in\mathcal{C}_{u},t}\left[\left\|{\epsilon}-{\epsilon}_{\theta}^{\prime}\left({z}_{t},c,t\right)\right\|^{2}\right].$
(11)

This objective tries to induce the new diffusion network
${\epsilon}_{\theta}^{\prime}$ to generate random noise if the input condition
is from the undesirable concepts from $\mathcal{C}_{u}$. If the model is
unlearned by benign concepts, the fine-tuning objective is

$\displaystyle\arg\min_{{\epsilon}_{\theta}^{\prime}}\mathbb{E}_{z,c\in\mathcal{C}_{u},\hat{c},t}\left[\left\|{\epsilon}_{\theta}\left({z}_{t},\hat{c},t\right)-{\epsilon}_{\theta}^{\prime}\left({z}_{t},c,t\right)\right\|^{2}\right].$
(12)

${\epsilon}_{\theta}\left({z}_{t},\hat{c},t\right)$ is the output under benign
conditions. By optimizing this objective, the output under undesirable
concepts will be re-directed towards the benign concepts. This method has a
similar intuition to the second term in Eq. (10), but Eq. (10) fine-tunes the
component for conditional representation, while Eq. (11) and Eq. (12) fine-
tune the whole diffusion network of SD. In addition, a similar regularization
term to keep the other desirable concepts unchanged is also combined with Eq.
(11) and Eq. (12).

Figure 10: The Attention Re-steering in Forget-Me-Not. (Image Credit to Zhang
et al. (2023a))

Zhang et al. (2023a) proposed the method called Forget-Me-Not to resteer the
cross-attention layers in SD by minimizing the values of attention maps, which
disrupts the guidance of text condition in diffusion process. In Stable
Diffusion (Rombach et al., 2022; Zhang et al., 2023a), the cross-attention of
UNet transfers information from conditional text to hidden features through
dot product and softmax. As shown in Figure 10, the attention maps are
calculated based on hidden features and embeddings of condition texts. The
attention maps will show the information of the undesirable concepts. Thus,
distrupting the attention map will lead to a broken diffusion process. The
fine-tuning objective of Forget-Me-Not minimizes the values in the attention
map to disrupt the guidance of condition texts. In this way, Forget-Me-Not can
mislead and remove the undesirable concepts from the final generated data.

Different from the above two unlearning methods for diffusion models,
Gandikota et al. (2023) targeted on the classifier-free diffusion generation
(Ho & Salimans, 2022). The classifier-free diffusion guidance does not reply
on a classifier for conditional generation, which is opposite to the
classifier-based diffusion guidance. The classifier-based diffusion guidance
modifies the denoising step to follow the feature captured by an auxiliary
classifier $p_{\theta}(c|z_{t})$:

$\displaystyle\tilde{{{\epsilon}}}_{\theta}\left({z}_{t},{c},t\right)=$
$\displaystyle{\epsilon}_{\theta}\left({z}_{t},{c},t\right)-w\nabla_{z_{t}}p_{\theta}(c|z_{t}),$

where $-\nabla_{z_{t}}p_{\theta}(c|z_{t})$ provides the information of what
features $z_{t}$ should have if $z_{t}$ wants to be classified as $c$. This
guidance can shape $z_{t}$ into $c$ step by step, and $w$ is the parameter to
control the strength of the classifier guidance (Dhariwal & Nichol, 2021). In
contrast, the classifier-free diffusion guidance uses the difference between
the outputs of conditional denoising diffusion model and unconditional
denoising diffusion model as the guidance to boost the influence of
conditions:

$\displaystyle\tilde{{{\epsilon}}}_{\theta}\left({z}_{t},{c},t\right)=$
$\displaystyle{\epsilon}_{\theta}\left({z}_{t},{c},t\right)+w({\epsilon}_{\theta}\left({z}_{t},{c},t\right)-{\epsilon}_{\theta}\left({z}_{t},t\right)),$

where
${\epsilon}_{\theta}\left({z}_{t},{c},t\right)-{\epsilon}_{\theta}\left({z}_{t},t\right)$
is different between conditional and unconditional model, i.e. the influence
of condition. It can also provide the information of the features that
${z}_{t}$ should possess to likely generate it into class $c$. The classifier-
free diffusion model makes use of this information to guide the generation
process. Gandikota et al. (2023) proposed ESD which uses the opposite of the
guidance as the unlearning objective to fine-tune SD:

$\displaystyle\arg\min_{{\epsilon}_{\theta}^{\prime}}\mathbb{E}_{\mathcal{E}(\mathbf{x}),c\in\mathcal{C}_{u},t}$
$\displaystyle\left[\left\|\Big{(}{\epsilon}_{\theta^{*}}\left({z}_{t},c,t\right)-w({\epsilon}_{\theta^{*}}\left({z}_{t},c,t\right)\right.\right.$
$\displaystyle-\left.\left.{\epsilon}_{\theta^{*}}\left({z}_{t},t\right))\Big{)}-{{\epsilon}}_{\theta}^{\prime}\left({z}_{t},c,t\right)\right\|^{2}\right],$

The formulation uses several instances of diffusion models,
${\epsilon}_{\theta^{*}}$ and ${\epsilon}_{\theta}^{\prime}$.
${\epsilon}_{\theta^{*}}$ is fixed for quantifying the opposite of guidance,
while the ${\epsilon}_{\theta}^{\prime}$ is the unlearned network to solve.
ESD has two variants, ESD-x and ESD-u. They have the same fine-tuning
objective, but choose different subsets of parameters to fine-tune. ESD-x
fine-tunes the cross-attention layers, while ESD-u fine-tunes non-cross-
attention modules. Gandikota et al. (2023) found that the ESD-x can control
the unlearned data according to the specific prompt, such as a named artistic
style, and has little influence on other concepts. ESD-u can unlearn concepts
from SD no matter whether a specific prompt is used, which is more suitable
for global unlearning like NSFW.

In this subsection, we have introduced the methods of unlearning for different
models. For conditional generative models, the unlearning methods usually try
to re-directing the conditional representations of the concepts to conditions
unrelated to the copyrights. Instead of directly forgetting the copyrighted
images from the training data, they try to prevent the concepts from being
generated, such that the influence of copyrighted data like the style of a
specific artist can be removed from the model.

####  2.3.4 Dataset De-duplication

Memorization is another problem that threatens the copyright of source data.
It refers to the problem that the generation model might produce images that
are the same as the training data (van den Burg & Williams, 2021). It is found
by many works that duplicated examples in the training data are more possible
to be generated (Webster, 2023; Somepalli et al., 2023a; Carlini et al.,
2023). Somepalli et al. (2023a) also argued that memorization is associated
with the frequency at which data is replicated. Carlini et al. (2023) pointed
out that the samples that are easy to be memorized usually duplicated for many
times.

Therefore, dataset de-duplication is necessary to mitigate the memorization
and prevent the model from generating the training data. It is attributed to
an active method because de-duplication is conducted by the model builder for
the purpose of providing legal services. Webster et al. (2023) proposed a
method to search the duplicate training images by CLIP (Radford et al., 2021).
First, it trains the auto-encoders to compress the images and texts into
latent space. It proposes Subset Nearest Neighbor CLIP (SNIP) to fine-tune the
encoder for compression and keep the distance between neighbors at the same
time. The objective of SNIP is defined as follows,

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{SNIP}}\left(x_{k}^{T},\right.&\left.x_{k}^{I};\mathcal{E}^{T},\mathcal{E}^{I}\right)=\\\
&\mathcal{L}_{\mathrm{CLIP}}\left(\mathcal{E}^{T}\left(x_{k}^{T}\right),\mathcal{E}^{I}\left(x_{k}^{I}\right)\right)\\\
+\lambda&\mathcal{L}_{\mathrm{CLIP}}\left(\mathcal{E}^{I}\left(x_{k}^{I}\right),\mathcal{E}^{I}\left(x_{1-\mathrm{NN}\left(x_{k}^{I}\right),k}^{I}\right)\right),\end{split}$
(13)

where $(x_{k}^{T},x_{k}^{I})$ is the image and caption pair, and
$(\mathcal{E}^{T},\mathcal{E}^{I})$ are the text and image encoder of CLIP.
The first term in Eq. (13) aligns the embeddings of the image and its text or
caption. The second term keeps the distance of the compressed embedding
between image neighbors in latent space. Considering the huge size of training
dataset, SNIP searches the nearest neighbor,
$1-\mathrm{NN}\left(x_{k}^{I}\right)$, within the subset of size $k$. Eq. (13)
trains the encoder for compression and keeps the close distance of the nearest
neighbors. After compression, it uses the inverted file system (IVF) to
approximately search the duplicated images. The whole huge dataset is divided
into small groups by $k$-means, and the duplication is searched within the
closest centroids. Then the found data is de-duplicated to reduce the
memorization. OpenAI used a similar strategy to train DALL·E 2 (OpenAI, ). It
first divides the training data into $K$ clusters, and then search the similar
data within the union set of a small number of clusters. They also found that
the de-duplication can increase the generation performance slightly.

Somepalli et al. (2023b) pointed out that the conditions of diffusion model,
i.e. the text caption, will also influence memorization. Their experiments
demonstrated that during the fine-tuning of Stable Diffusion, the memorization
is more likely to happen if the captions (or text prompts) of different images
are more diverse. But it does not mean the totally random captions can
exacerbate memorization. Actually, the captions that are correlated with image
content but have more diversity can lead to more severe memorization. However,
diverse captions of the duplicated images can reduce the memorization. They
also found that more training epochs have similar influence to duplication
which increases the memorization. This means more epochs may mimic the
duplication. Based on these findings, they concluded that for a single image
and duplicated images, the diverse captions can help slow down the
memorization. They proposed a few methods to process the captions during fine-
tuning to avoid memorization, including multiple captions (which randomly
samples from 20 captions generated by BLIP (Li et al., 2022a) for each image
when training), random token replacement & addition (which randomly replaces
tokens/words in the caption with a random word, or add a random word to the
caption at a random location) and so on.

###  2.4 Model Copyright Protection

This subsection summarizes the strategies for protecting the copyright of DGMs
for image generation. This protection is important for two major reasons.
First, a powerful DGM, which necessitates extensive computational resources
and well-annotated data for its creation, needs to be safeguarded from
copyright infringements (such as being stolen by malicious users to offer
unauthorized paid services). Second, the unregulated distribution of those
models may lead to ethical concerns, including their potential misuse for
generating misinformation, which necessitates techniques to identify the
origin of an image.

Deep Generative Model Watermarking is a common solution for model copyright
protection. It involves incorporating distinct information, known as a
watermark, into the models before their deployment. The embedded watermark can
be retrieved from a potentially infringing model or its generated data to
confirm any suspected copyright violations. To achieve good performance in
protecting the model while preserving the original generation performance, the
watermarking technique should incorporate the following key properties:

  * •

Fidelity: the ability of the watermarking method to not significantly impact
the general performance of the model (the diversity and visual quality of the
generated images).

  * •

Integrity: the accuracy with which the watermark can be extracted.

  * •

Capacity: the length of message that can be effectively encoded to and
extracted from the watermark.

  * •

Robustness: the ability of the watermark to withstand alterations to the model
or perturbation on the watermarked generated images.

  * •

Efficiency: the computational cost of the watermark embedding and extraction
process.

The proposed DGM watermarking methods can be summarized into three categories
according to the specific way they embed the watermark:

  * •

Parameter-based watermarking encodes the watermark message into the model’s
parameters or structural configurations;

  * •

Image-based watermarking embeds the watermark message into every image
generated by the model;

  * •

Triggered-based watermarking secretly incorporates a trigger to the protected
model such that an image with copyright information will be generated once the
trigger is activated.

####  2.4.1 Parameter-based watermarking

This type of watermarking techniques aims to subtly incorporate watermark
information into the network’s internal weights or structural configurations.
It is known as a “white-box” method because full access to these elements is
required to implement and extract the watermark. Parameter-based watermarking
has been widely applied for verifying the ownership of classification models
(Uchida et al., 2017; Darvish Rouhani et al., 2019; Zhao et al., 2021) and has
also been extended to protect DGMs in recent years.

Ong et al. (2021) proposed to embed the signature of a GAN model to the
scaling factors, $\gamma$, of the normalization layer of the generators. The
loss for watermark embedding is:

$\displaystyle\mathcal{L}_{s}(\gamma,{B})=\sum_{i=1}^{C}\max\left(\gamma_{0}-\gamma_{i}b_{i},0\right),$
(14)

where $B=\left\\{b_{1},\cdots,b_{C}\mid b_{i}\in\\{-1,1\\}\right\\}$ refers to
the predefined binary watermark message and $C$ denotes the number of
channels. This objective stipulates that the scaling factor of the $i$-th
channel, denoted as $\gamma_{i}$, adopts either a positive or negative
polarity (+/-) as determined by $b_{i}$. The term $\gamma_{0}$ is a constant
employed to regulate the minimum value of $\gamma$. The sign loss is added to
the original training loss of the GAN model (e.g., a DCGAN (Radford et al.,
2015), SRGAN (Ledig et al., 2017) or CycleGAN (Zhu et al., 2017)) to form the
overall training objective of the generators. With this embedding strategy,
the capacity of the watermark is determined by the total number of channels in
the normalization layers. In the verification stage, the embedded message can
be easily extracted by looking at the signs of the specific scaling factors of
the model.

####  2.4.2 Image-based watermarking

This group of watermarking approaches embeds watermark messages into all the
images generated by the model to be protected. To achieve this goal, various
studies suggested using an additional deep-learning-based network for
embedding watermarks after the image generation process. In detail, once the
DGMs create the images, these extra watermark embedding networks apply
watermarks to the images before they are released to public. For example,
Zhang et al. (2022a) proposed two frameworks for watermark embedding based on
the Human Visual System (HVS) to reduce the impact of the watermark on the
visual quality of the generated images. Leveraging the fact that humans are
more sensitive to changes in green color and brightness, they recommended
embedding watermarks into the Red (R) and Blue (B) channels in the RGB color
system or the U/V channels in the Discrete Cosine Transform (DCT) domain. In
either framework, each channel carries a unique watermark, allowing for the
extraction of two distinct watermarks from a single image.

As a following-up work, Zhang et al. (2023b) noted that images marked using
prior watermark embedding networks display significant high-frequency
artifacts in the frequency domain. As shown in Figure 11, slight spatial
artifacts are detectable in the generated image marked by the method described
in Wu et al. (2020). And obvious grid-like high-frequency artifacts can be
found in the DCT heat map of the marked image. These artifacts, mainly due to
the up-sampling and down-sampling convolution operations of the GAN-like
embedding network, could compromise the watermark’s imperceptibility. To
tackle the problem, they designed a new watermark embedding network which is
capable of suppressing high-frequency artifacts through anti-aliasing. The
anti-aliasing is primarily implemented by introducing a low-pass filter before
the down-sampling process and appending a convolutional layer after the
nearest-neighbor up-sampling process, adopting strategies demonstrated to be
effective in previous research.

Figure 11: Detection of High-Frequency Artifacts. (a) ground-truth image, (b)
watermarked image by Wu et al. (2020), (c) partial zoom-in of (b), (d) DCT
heat maps of (a), (e) DCT heat maps of (b). (Image Credit to Zhang et al.
(2023b))

In addition to using a separate watermark embedder, some other strategies
consider directly modifying the host generative network so that the host
network itself can embed the watermark while achieving the original generation
task. At the same time, an external watermark decoder should be trained to
correctly extract the watermark information from the generated images. To
achieve this goal in GANs, Fei et al. (2022) proposed to update the training
objective of the generator by:

$\displaystyle\begin{split}L_{G}^{w}(D,G)=&\mathbb{E}_{{z}\sim
p_{z}}[\log(1-D(G({z})))]+\\\ &\gamma\mathbb{E}_{{z}\sim
p_{z}}[BCE(D_{w}(G({z})),w_{gt})],\end{split}$

where $D$ and $G$ refer to the discriminator and generator of GAN
respectively, $D_{w}$ refers to the watermark decoder, $p_{z}$ means the prior
distribution of the latent space, $w_{gt}$ is the ground truth watermark
message, $\gamma$ is a regularization parameter and $BCE$ represents the
binary cross entropy. The first term in the equation is the standard generator
loss, while the second term is for ensuring the correctness of the decoded
binary watermark message. It is noted that the watermark decoder $D_{w}$ they
applied here is retrieved from a well-developed standard image-watermarking
framework and its parameters are fixed in the training process of $G$. With
the above training objective, the model can be either trained from scratch or
fine-tuned from a non-watermarked pre-trained GAN model.

Figure 12: Pipeline of the Scalable GAN Fingerprints Method. Details about the
Modulated Conv is shown in Figure 13. (Image Credit to Yu et al. (2020))

Yu et al. (2020) also proposed a fingerprinting technique to trace the outputs
of GANs back to their source, which can help identify misuse. The pipeline of
the technique is shown in Figure 12. The core difference between it and the
work of Fei et al. (2022) is that the trained generator $G$ takes both the
latent code ${{z}}$ and the embedding of the fingerprint $\boldsymbol{c}$ (a
sequence of bits) as its input, which increases the efficiency and scalability
of the fingerprinting mechanism. Once the model is trained, multiple instances
of fingerprinted generators which focus on embedding different fingerprint
messages can be directly obtained. In their algorithm, the watermark encoder,
$E$, which maps fingerprint $\boldsymbol{c}$ to its embedding, the decoder,
$F$, which obtains the latent code ${z}$ and fingerprint $\boldsymbol{c}$ from
a watermarked image, the discriminator $D$ and the generator $G$ of GAN are
optimized together with the following loss:

$\displaystyle\min_{E,F,G}\max_{D}\lambda_{1}\mathcal{L}_{adv}+\lambda_{2}\mathcal{L}_{z}+\lambda_{3}\mathcal{L}_{c}+\lambda_{4}\mathcal{L}_{\text{const}}.$

In this objective, the first term $\mathcal{L}_{adv}$ refers to the original
training loss of a GAN model and the second term is written as

$\displaystyle\mathcal{L}_{z}=\underset{\begin{subarray}{c}{z}\sim\mathcal{N}\left(0,I^{d_{z}}\right)\\\
\boldsymbol{c}\sim\operatorname{Ber}(0.5)^{d_{c}}\end{subarray}}{\mathbb{E}}\sum_{k=1}^{d_{z}}\left({z}_{k}-F(G({z},\boldsymbol{c}))_{k}\right)^{2},$

whose $d_{c}$ and $d_{z}$ refers to the dimension of $\boldsymbol{c}$ and
${z}$. This regularization term was firstly proposed by Srivastava et al.
(2017) and is used here for mitigating the mode collapse issue of GANs. The
term $\mathcal{L}_{c}$, formulated as

$\displaystyle\begin{split}\mathcal{L}_{c}=&\underset{\begin{subarray}{c}{z}\sim\mathcal{N}\left(0,I^{d_{z}}\right)\\\
\boldsymbol{c}\sim\operatorname{Ber}(0.5)^{d_{c}}\end{subarray}}{\mathbb{E}}\sum_{k=1}^{d_{c}}\left[c_{k}\log\sigma\left(F(G({z},\boldsymbol{c}))_{d_{{z}}+k}\right)\right.\\\
&+\left.\left(1-c_{k}\right)\log\left(1-\sigma\left(F(G({z},\boldsymbol{c}))_{d_{{z}}+k}\right)\right)\right],\end{split}$

is the cross entropy loss of the fingerprint decoding to ensure the
correctness of the reconstruction of the fingerprint. The
$\sigma\left(\cdot\right)$ here refers to the sigmod function. The formulation
of the last term $\mathcal{L}_{\text{const}}$ is

$\displaystyle\mathcal{L}_{\text{const}}=\underset{\begin{subarray}{c}{z}\sim\mathcal{N}\left(0,\mathbf{I}^{d_{z}}\right)\\\
\mathbf{c}_{1},\boldsymbol{c}_{2}\sim\operatorname{Ber}(0.5)^{d_{c}}\end{subarray}}{\mathbb{E}}\left\|G\left({z},\boldsymbol{c}_{1}\right)-G\left({z},\boldsymbol{c}_{2}\right)\right\|_{2}^{2}.$

It is designed to ensure the perceptual similarity between images generated
with the same latent code but different fingerprints, which guarantees that
the latent code exclusively controls the content of the generated images,
irrespective of the fingerprint variations. At the architectural level, to
equip the generator $G$ with the ability to embed the fingerprint embedding
$E(c)$, it proposed to modulate the convolutional filters of the generator
backbone with the fingerprint embedding, as illustrated in Figure 13.

Figure 13: The modulation convolution layer within the generator $G$ of the
Scalable GAN Fingerprints Method. (Image Credit to Yu et al. (2020))

Similar idea of weights modulation has also been applied for the protection of
LDMs. Kim et al. (2023) proposed to incorporate the watermark message to
generated images by modulating the parameters of each layer of the decoder
$\mathcal{D}$ of LDM. Specifically, after the watermark message $\phi$ is
obtained, a mapping network $\mathcal{M}$ is applied to derive the feature
representation of the message. Then an affine transformation layer
$\mathcal{A}_{l}(\cdot)$ is developed for each layer $l$ of the decoder
$\mathcal{D}$. The affine transformation layer is applied to match the
dimension of the message representation to the dimension of the model’s
weights. Then the weight modulation is conducted following the formulation:

$W^{\phi}_{i,j,k}=u_{j}*W_{i,j,k},$ (15)

where $W$ and $W^{\phi}$ refers to the pre-trained and fingerprinted
parameters of the decoder $\mathcal{D}$, $i$, $j$, $k$ denote the dimensions
of the input, output, and kernel of each layer, and
$u_{j}=\mathcal{A}_{l}(\mathcal{M}(\phi))$ denotes the scale of the modulation
to the $j$th output channel. When the watermark message is encoded, a
watermark decoder $\mathcal{F}$ is also required to retrieve the watermark
from the generated images. The watermark decoder $\mathcal{F}$ as well as the
mapping network $\mathcal{W}$ are jointly trained with the following loss:

$\min_{\mathcal{A},\mathcal{M},\mathcal{D},\mathcal{F}}\lambda_{1}L_{\phi}+\lambda_{2}L_{\text{quality}},$

where $L_{\phi}$ denotes the binary cross entropy of watermark decoding:

$\displaystyle L_{\phi}$
$\displaystyle=\mathbb{E}_{z=\varepsilon(x),\phi\sim\Phi}\sum_{i=1}^{d_{\phi}}\big{(}\bigg{[}\phi_{i}\log\sigma\mathcal{F}(\mathcal{D}(\phi,z))_{i}$
$\displaystyle\quad+(1-\phi_{i})\log(1-\sigma(\mathcal{F}(\mathcal{D}(\phi,z)))_{i}\big{)}\bigg{]},$

in which $z=\varepsilon(x)$ refers to the latent feature of an image $x$,
$\mathcal{D}(\phi,z)$ denotes the output of the decoder whose weights being
modulated by Eq.(15), and $\sigma$ refers to the sigmoid activation function.
$\Phi$ refers to a Bernoulli distribution whose probability is 0.5.
$L_{\text{quality}}$ is the regularization term inhibiting the influence of
the watermark to the quality to the generated image:

$\displaystyle
L_{\text{quality}}=\mathbb{E}_{z=\varepsilon(x),\phi\sim\Phi}\left[\ell(x,D(\phi,z))\right]$

where $\mathcal{l}$ measures the difference between the watermarked image and
original image.

Another work (Xiong et al., 2023), which also aims to protect the copyright of
LDMs, has a similar pipeline to Kim et al. (2023). The major difference is
that the feature representation of the watermark message is not used to
modulate the weight of the decoder $\mathcal{D}$, but is combined with the
intermediate outputs of the fine-tuned decoder $\mathcal{D}$ such that the
image generated by $\mathcal{D}$ is watermarked. Under the pipeline, a
watermark decoder is also required to extract the watermark from the generated
image.

Additionally, the work of Fernandez et al. (2023), which also focus on the
protection of Latent Diffusion Models (LDMs), suggested to directly apply a
classical neural network-based watermarking method, HiddeN (Zhu et al., 2018),
to jointly optimize the parameters of a watermark encoder and extractor. Then
the decoder of the LDM is fine-tuned such that all images it generates contain
a given watermark that can be extracted by the pre-trained watermark
extractor.

Unlike the previously mentioned methods that require modifying the generative
networks to include watermarks in their generated images, Yu et al. (2021)
achieved this by solely watermarking the training images of the generative
models. To ensure that the watermark can be successfully transferred from the
training images to the generated images of GANs, they developed a deep-
learning-based structure for watermark (or “fingerprints” in their paper)
embedding and extraction. Similar to the other watermark embedding-extraction
framework, the training objective of the watermark embedder and decoder also
consists of a $BCE$ loss which guides the decoder to decode the fingerprint
correctly and an $MSE$ loss which penalizes any deviation of the watermarked
image from the original image. Although the framework proposed by Yu et al.
(2021) was originally designed for DeepFake detection and misinformation
prevention, its functionality can be extended to verifying the copyright of
GANs, as the watermark on the generated images can indicate their source.
Additionally, Zhao et al. (2023b) experimentally verified that this technique
can be extended to safeguard the copyright of unconditional and class-
conditional diffusion models.

While the previous methods consider incorporating the watermarking embedding
procedure in the training process of the generative models, Wen et al. (2023)
proposed Tree-Ring Watermarks, a watermarking framework specified for
diffusion models which conducts the watermark encoding in the sampling
process. As shown in Figure 14, the watermark is embedded into the initial
noise vector used for sampling. In order to ensure that the watermark can
achieve a better robustness against multiple image modification such as
cropping, dilation, flipping, and rotation, they suggest to encode the
watermark patterns to the Fourier space of the image. After the image is
generated, watermark detection is done by inverting the diffusion process to
reconstruct the noise vector. This is done using the DDIM (Song et al., 2020)
inversion process. In detail, the initial noise vector
$\boldsymbol{x}_{T}\in\mathbb{R}^{L}$ is described in Fourier space as:

$\displaystyle\mathcal{F}\left(\boldsymbol{x}_{T}\right)_{i}\sim\begin{cases}k_{i}^{*}&\text{
if }i\in M\\\ \mathcal{N}(0,1)&\text{ otherwise. }\end{cases}$

where $M$ refers to a binary mask, $k^{*}\in\mathbb{C}^{|M|}$ is the key
chosen for the watermark. The key is designed to be ring-shaped in the Fourier
space to ensure that the watermark is invariant to certain common image
transformations. In the detection stage, the watermark is detected if the
$L_{1}$ difference between the inverted noise vector and the pre-defined key
in the Fourier domain of the watermarked area $M$ is below a tuned threshold
$\tau$.

Figure 14: Pipeline for Tree-Ring Watermarking. (Image Credit to Wen et al.
(2023))

Another work which considers adding the watermark in the images’ sampling
process is Nie et al. (2023). This approach, which target to protect the
copyright of LDM or StyleGAN2 (Karras et al., 2020), does not require a
network to conduct the watermark embedding and decoding but suggest to
directly modify the latent features $\psi(z)\in\mathbb{R}^{d_{w}}$ (either
generated by the multi-layer perception network of the StyleGAN2 or sampled by
the diffusion process of LDM) by matrix operation. In detail, let
$U\in\mathbb{R}^{d_{w}\times(d_{w}-d_{\phi})}$ denotes an orthonormal subspace
of the space of the latent feature $z$ and $V\in\mathbb{R}^{d_{w}\times
d_{\phi}}$ denotes its complementary subspace, the watermarked latent feature
should be represented by

$\displaystyle w_{\phi}(\alpha)=U\alpha+\sigma V\phi,$ where
$\displaystyle\alpha=U^{\dagger}\text{proj}_{U}\psi(z)\in\mathbb{R}^{d_{w}-d_{\phi}}.$

The notation $\phi\in\mathbb{R}^{d_{\phi}}$ refers to the watermark message,
$U^{\dagger}$ represents the pseudo-inverse of $U$, $\text{proj}_{U}\psi(z)$
denotes the projection of $\psi(z)$ to $span(U)$, and $\sigma$ denotes a
hyperparameter to control the strength of the watermark. In the detection
stage, in order to retrieve the watermark from the image, an optimization
problem is solved:

$\displaystyle\min_{\hat{\alpha},\hat{\phi}}\ell\left(g(w_{\hat{\phi}}(\hat{\alpha})),g(w_{\phi}(\alpha))\right)$
$\displaystyle\text{s.t.}\quad\hat{\alpha}_{i}$
$\displaystyle\in[\alpha_{l,i},\alpha_{u,i}],\quad\forall
i=1,\ldots,d_{w}-d_{\phi},$

where $l$ refers to the LPIPS metric (Zhang et al., 2018) which evaluates the
visual similarity of two images. The lower and upper bound of
$\hat{\alpha}_{i}$, $\alpha_{l,i}$ and $\alpha_{u,i}$, are chosen based on
empirical observation.

####  2.4.3 Trigger-based watermarking

The trigger-based watermarking follows the basic technical scheme of backdoor
attack (Wang et al., 2019; Saha et al., 2020; Chen et al., 2017), which embeds
a trigger into a neural network to cause a failed classification by activating
the trigger. In the protection of DGM copyright, once the trigger is
integrated into the protected model, the activation of this trigger in the
input will cause the model to generate a watermarked output. By examining the
presence or absence of the watermark in this output, the model owner can
determine whether a suspect model has been illicitly derived from the
protected, watermarked model. Consequently, if the watermark is detected in
the example of triggered generation, it substantiates the claim of copyright
infringement regarding the model.

Ong et al. (2021) also proposed a trigger-based watermark framework to protect
the copyright of GANs. It first adds the trigger onto $z$ which is the input
of the generator $G(\cdot)$ to get the triggered input $z_{\text{trigger}}$.
The protected GAN is trained towards the goal that once the trigger is input
to $G(\cdot)$, the output will be generated with a watermark. The model owner
can input a trigger into the suspect model and verify whether the model is
stolen from the copyrighted model by checking the existence of watermark in
the output. For different types of GAN, the triggered input is obtained by
different manually defined rules. For example, in DCGAN, the trigger is
encoded with a binary representation ${b}$:

$\displaystyle
z_{\text{trigger}}={z}\odot{b}+c\left(1-{b}\right),{b}\in\left\\{0,1\right\\}^{d_{z}},$

where $\odot$ is the elementwise production, and ${d_{z}}$ is the dimension of
$z$. The triggered input $z_{\text{trigger}}$ means each dimension of $z$ is
encoded with one dimension in $b$. The triggered input is expected to cause
$G(\cdot)$ to generate a watermarked target $x_{\text{target}}$. During the
training of GAN, the trigger pair
$\left(z_{\text{target}},x_{\text{target}}\right)$ is included in a
regularization term for this. The term is formulated as

$\displaystyle\mathcal{L}_{w}\left(z_{\text{trigger}},x_{\text{target}}\right)=1-\operatorname{SSIM}\left(G\left(z_{\text{trigger}}\right),x_{\text{target}}\right),$

where SSIM refers to the structural similarity. The regularization term
$\mathcal{L}_{w}$ is combined with the training loss of $G(\cdot)$ by a
coefficient $\lambda$:

$\displaystyle\mathcal{L}_{\mathrm{G}_{w}}=\mathcal{L}_{\mathrm{G}}+\lambda\mathcal{L}_{w}.$

However, the trigger-based method is not implemented in isolation. Instead, it
is integrated with the parameter-based watermarking approach, as detailed in
Equation (14), following the practice outlined in (Ong et al., 2021).

Besides the protection of GANs, Liu et al. (2023c) proposed to watermark
Stable Diffusion by injecting triggers into the prompt. According to the
method, when specific triggers are present in the prompt, Stable Diffusion
will produce an image with a watermark. If the resulting image closely
resembles the watermarked prototype, it confirms that the model is safeguarded
by the designated watermark. Liu et al. (2023c) developed two different
approaches for integrating triggers, NaiveWM and FixedWM. NaiveWM embeds a
specific word at a random location within the prompt, while FixedWM places a
specific word at a predetermined position in the prompt. The chosen trigger
word should be nonsensical to avoid accidental activation. For embedding the
trigger-based watermark into Stable Diffusion, a triggered dataset is employed
to fine-tune the model. Additionally, a clean dataset is also used in the
fine-tuning process to preserve the quality of generation when the trigger is
not activated. Zhao et al. (2023b) proposed a similar idea, but did not
consider the stealthiness of the trigger. Their method directly uses “[V]” as
the whole triggered prompt.

Different from Liu et al. (2023c) and Zhao et al. (2023b) that embedded the
watermark into the guidance of prompt, Peng et al. (2023a) watermarked by
injecting triggers into the diffusion process. In the reverse process of
generative diffusion models, when the trigger is added onto the current step,
the subsequent will be generated towards the watermarked image. By repeating
injecting the trigger into the reverse process, the final generated image will
be a watermarked image that can be used to verify the copyright of the model.
For embedding the watermark, the model is trained with both triggered dataset
and clean dataset. During training, when the data comes from triggered
dataset, the triggered noise is input into the denoising network of diffusion
model and the network is optimized to denoise towards the watermarked image.
During the both forward and reverse process, the trigger is added into step
$t$ by

$\displaystyle{x}_{t}^{\text{trigger}}=\gamma_{1}{x}_{t}+\left(1-\gamma_{1}\right){b},$

where $b$ is the trigger, and ${x}_{t}$ is the state of step $t$. For reverse
process, ${x}_{t}$ is the image denoised from the last step, while for forward
process, it is the diffused image accumulated by the Gaussian noise in the
forward steps. This protection method can be used for both training from
scratch and fine-tuning. Although this method can also protect the copyright
by verifying whether the final output is watermarked or not, the extraction of
watermark requires the modification on each diffusion step which is more
strict than the previous methods that only need to trigger in the prompt.

##  3 Copyright in Text Generation

In this section, we first define the problem of copyright protection in text
domain especially unique properties of text domain. Then we introduce data
copyright protection and model copyright protection by providing the taxonomy
of existing methods.

###  3.1 DGMs for text generation

In the realm of text generation, our focus is primarily on Large Language
Models (LLMs), due to their outstanding capabilities and the potential risks
they entail. As key components of DGMs, LLMs have notably propelled
advancements in the field of text generation. They exhibit remarkable emergent
abilities, significantly boosting performance in a range of NLP tasks.
However, alongside their exceptional efficacy, LLMs also present substantial
concerns, particularly in the realms of copyright. The considerable commercial
value and the high costs associated with their training further underscores
the importance of addressing these issues. In Section 3, which focuses on the
text domain, we explore LLMs in detail, acknowledging their unparalleled
efficiency and effectiveness in text generation, while also considering the
associated concerns in copyright protection.

Training and Inference of LLMs. Generally, Large Language Models (LLMs) are
constructed to comprehend human language and produce coherent, contextually
relevant text. To achieve this objective, most existing LLMs can be
categorized into “pre-trained LLMs” and “fine-tuned LLMs”. For the pre-trained
LLMs, they are trained on a massive amount of internet text data, aiming to
grant them the ability to predict the probable subsequent token based on its
preceding tokens. Through this process, the models can obtain an understanding
of the patterns and structures inherent in human language. Many early LLMs,
such as GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), OPT (Zhang
et al., 2022b), GPT-Neo (Black et al., 2021), and others, have been developed
in this procedure, to be equipped with the ability to complete sentences by
generating the possible subsequent tokens.

Through various fine-tuning technologies, LLMs have been made increasingly
flexible and adept at various downstream tasks. Specifically, the Instruction
Tuning (Zhang et al., 2023c) strategy is employed to enable LLMs to handle
diverse language tasks based on users’ requests. This versatility allows LLMs
to be applied as general assistants for tasks such as question answering (Lu
et al., 2022). Furthermore, Reinforcement Learning with Human Feedback (RLHF)
is utilized to help LLMs better align with human values, enhance the
reliability of generated outputs, and improve ethical decision-making (Ouyang
et al., 2022). Based on these advancements, advanced LLMs like ChatGPT
888https://chat.openai.com/,
Claude999https://www.anthropic.com/index/claude-2, Bard (Manyika & Hsiao,
2023), and LLaMA 2 (Touvron et al., 2023), have been developed.

Notations. Next, we introduce key notations we will use in this paper. We
denote an arbitrary text $x$ which is composed of a sequence of tokens
$(w_{1},w_{2},...,w_{k})$ with length $k$. Many popular LLMs follow an “auto-
regressive” manner: given a sequence of prior tokens
$(w_{1},w_{2},...,w_{i-1})$, a language model $F$ calculates the probability
of the next token $\hat{w_{i}}$ conditioning on the preceding tokens. We
denote the probability that model predict the $i$-th word as $v_{j}$ by
$p_{F}(\hat{w_{i}}=v_{j}|w_{1},...,w_{i-1})$, where $v_{j}\in\mathcal{V}$ and
$\mathcal{V}$ is the vocabulary. Based on this design, the likelihood of each
token in the original sentence $x$ given by the model $F$ can be defined as:

$p_{F}(\hat{w_{i}}=\ w_{i}|w_{1},...,w_{i-1})$ (16)

which is the probability of the model to give an output token to be the same
as $w_{i}$ in the text. Similarly, we can also define the likelihood of the
whole sentence $x$ given by the model $F$, which is the overall likelihood of
all tokens in the sentence:

$p_{F}(x)=\prod^{k}_{i=2}p_{F}(\hat{w_{i}}=\ w_{i}|w_{1},...,w_{i-1})$ (17)

For most existing LLMs, during the (pre-)training process, the model is
trained to maximize this likelihood, so that the model can learn to generate
the texts following the distributions of training texts. During the LLM’s
inference process, the model will generate the texts, by making sampling from
the tokens with high likelihood.

###  3.2 Copyright Issues in Text Generation

Data copyright protection. The data copyright problem regarding the use of
LLMs has attracted extensive focus and debates. The definition of copyright
violation varies across countries and laws. Generally, it means the use of
works protected by copyright without permission for a usage where such
permission is required, thereby infringing certain exclusive rights granted to
the copyright holder, such as the right to reproduce, distribute, display or
perform the protected work, or to make derivative
works101010https://en.wikipedia.org/wiki/Copyright_infringement. Similarly,
Lee et al. (2023a) also note that “plagiarism occurs when any content
including text, source code, or audio-visual content is reused without
permission or citation from an author of the original work.” However, as
evident from the following laboratory and court cases, current LLMs
demonstrate various plagiarism behaviors, which can be divided into three
types: verbatim plagiarism, paraphrase plagiarism, and idea plagiarism:

  * •

Verbatim Plagiarism, which refers to directly copying the origin data
completely or partially.

  * •

Paraphrase Plagiarism, which refers to composing new works including consent
word replacement, statement rearrangement, or even translate sentences back
and forth, from copyrighted works.

  * •

Idea Plagiarism, which refers to copying the core idea of the copyrighted
material.

Figure 15: Data Extraction Attack (Image Credit to Carlini et al. (2021) )

Cases in Laboratory. There are existing lines of work showing that Large
Language Models (LLMs) tend to memorize and emit parts of its training data
(which may include copyrighted materials), referred to as the “memorization
effect of LLMs”. Although such effect is widely believed to be essential for
language model’s performance and generalization, it also raises serious risks
on data copyright protection. Carlini et al. (2021) first found that LLMs can
memorize and leak specific training examples, by devising a data extraction
attack as shown in Figure 15, which can effectively examine the training data
verbatim from GPT-2. Then, Carlini et al. (2022) further quantified such
phenomena in detail and found that the memorization effect grows with the
model scales, number of replicates, as well as the prompt length. Similarly,
Tirumala et al. (2022) suggested that lager LMs generally memorize faster and
less likely to forget information. Besides, they found that nouns and numbers
are more likely to be memorized.

Zhang et al. (2021) defined ”counterfactual memory” which quantifies the
performance difference between models trained on the specific data and models
not trained on that. Biderman et al. (2023) investigated correlations of
memory phenomena between large and small models and between partially-trained
and fully-trained models. They suggested that we can use partially trained-
model to efficiently predict whether the data is memorized by a fully-trained
model. Furthermore, Zeng et al. (2023) explored the memorization behavior of
fine-tuned LLMs and identified some feature dense tasks such as dialog and
summarization present high memorization effects, and demonstrated the
correlation between attention scores and task-specific memorization.

Cases in Court. Recently, The New York Times initiated a lawsuit against
OpenAI and Microsoft, alleging copyright
infringement111111https://www.nytimes.com/2023/12/27/business/media/new-york-
times-open-ai-microsoft-lawsuit.html. The lawsuit asserts that OpenAI and
Microsoft used millions of articles from The Times to train their automated
chatbots. These chatbots, the suit contends, now rival The Times in providing
reliable information. The complaint seeks restitution for what The Times
describes as “billions of dollars in statutory and actual damages” arising
from the “unauthorized replication and exploitation of The Times’s distinct
and valuable works.” A specific allegation made in the lawsuit is that
ChatGPT, when queried about current events, occasionally generates responses
containing “verbatim excerpts” from articles published by The New York Times.
These articles are typically behind a paywall, accessible only to subscribers.
Moreover, the lawsuit highlights instances where the Bing search engine,
incorporated with ChatGPT, reportedly displayed content sourced from a New
York Times-owned website. This usage was done without providing direct links
to the articles or including the referral links that The Times employs for
revenue generation.

Model copyright protection. For LLMs, the infringement on model copyright is
in line with the model copyright protection in the image domain. The model
builder invests a significant amount of funds and labor into the construction
of LLMs, which naturally grants them intellectual property rights over the
trained model.

###  3.3 Data Copyright Protection

According to the previous discussed cases, a majority of research works study
how to improve the training scheme of LLMs to avoid replicating its training
data samples. These methods are generally proposed from the perspective of
model builders to prevent copyright infringement and provide legal services.
In text domain, they can be categorized according to the major reasons which
cause the memorization / plagiarism behaviors of LLMs:

  * •

Data-deduplication, which is the strategy of removing the repeated samples
from the training set of LLMs. As evident from existing studies such as (Lee
et al., 2021; Kandpal et al., 2022), the repeated contents from the training
set are easier to be memorized and reproduced by LLMs.

  * •

Improved training & generation algorithms: the model builder can also modify
the training objective and text generation procedure to avoid potential
reproduction behavior of LLMs.

  * •

Alignment strategies: the model builder can devise new alignment strategies to
reduce memorization in LLMs.

  * •

Machine unlearning, which services to delete the copyrighted materials from
LLMs, once the owners of the copyrighted materials identify the infringement.

Similar to image domain, machine unlearning can be seen as the passive method
requested by the data owner, while the others are active approaches directly
conducted by the model builder during the model construction stage.

####  3.3.1 Data De-duplication

In the training of Large Language Models (LLMs), a key issue is the
memorization of training data $\mathcal{X}$, particularly when it involves
copyrighted data $\mathcal{C}\subset\mathcal{X}$. To improve copyright
security, one major method is to de-duplicate to reduce memorization. It
refers to the process that the model builder removes the duplicated data
samples from the training set of LLMs. Lee et al. (2021) found that existing
NLP datasets contain many duplicated and near-duplicated substrings, leading
to over 1% of language model outputs being exact copies from training data. To
address this, they developed ExactSubstr and NearDup for dataset de-
duplication. ExactSubstr removes long shared substrings between two samples by
concatenating the entire dataset into a single sequence and using a suffix
array for easy deletion of adjacent repetitive sequences. NearDup utilizes
MinHash (Broder, 1997) to identify potential document matches, followed by
rigorous similarity checks or direct deletion of duplicates. This process,
which includes sorting $n$-grams of each document and applying the Jaccard
Index for similarity, significantly reduces model memorization, decreases
training time, and improves evaluation accuracy by reducing train-test
overlap. Kandpal et al. (2022) found that language models’ likelihood of
regenerating training sequences superlinearly correlates with their frequency
in the dataset. They also noted significant improvements in privacy protection
following de-duplication in these models.

####  3.3.2 Improved Training & Generation algorithms

Chu et al. (2023) define copyright protection for LLMs as ensuring the model’s
output for any input does not significantly resemble any copyrighted content
in $\mathcal{C}$, as measured by a metric $L$. The training goal for a dataset
$\mathcal{X}$ with copyrighted and non-copyrighted data is to fine-tune the
model $f$ to achieve:

$L(f(x),\mathcal{C})\geq\tau+L(f(x),\mathcal{X}_{\neg\mathcal{C}}),$ (18)

where $\mathcal{X}_{\neg\mathcal{C}}=\mathcal{X}\cap{\neg\mathcal{C}}$, $L$
measures the possibility to generate $\mathcal{C}$ and higher $L$ means less
possibility of generating $\mathcal{C}$. Thus, Eq. (18) means the output of
$f$ should maintain less possibility to come from copyrighted data or be
similar to data. To attain this goal, Chu et al. (2023) introduce a method
known as copyright regression. In a simplified regression case, $(A,b)$ denote
training data, where $A$ is the input and $b$ is the target output. The model
is trained to get $f(A)=b$. $(A_{1},b_{1})$ denote copyrighted data. An
additional term $L(f(A_{1}),b_{1})^{-1}$ is added to the training objective to
discourage the model from generating outputs matching the copyrighted data,
with a scalar coefficient $\gamma>0$. Consequently, the training objective is
modified to $L(f(A),b)+\gamma L(f(A_{1}),b_{1})^{-1}$, integrating an inverse
term that strikes a crucial balance between performance and copyright
protection, and helps mitigate the model’s tendency to generate copyrighted
outputs. However, the method in (Chu et al., 2023) requires the model builder
to know which training samples are copyrighted, which are not executable for
most existing LLMs.

In the text generation stage, there are also ways to protect copyright, even
if the model has memorized a part of copyrighted data. These methods
manipulate the generation process of LLMs to avoid the occurrence of
copyrighted information. Ippolito et al. (2022) developed a memorization-free
(MEMFREE) decoding strategy using bloom filters to effectively eliminate all
direct memorization of text. Unlike traditional retroactive censoring, which
repetitively runs the language model with varied seeds but identical prompts
until non-resembling training set outputs are generated, MEMFREE efficiently
targets memorization at the $n$-gram level during the decoding phase, rather
than examining the entire sentence. Specifically, it assesses whether any
$n$-gram in the generated sequence matches those in the training set,
suppressing any such occurrences and prompting re-sampling from the model.
This approach, however, fails in the case that the generated text is similar
but not exactly the same as the memorized content. The authors highlight that
strict definitions of verbatim memorization are insufficient, as they do not
address subtler forms of memorization. They note that even models adept at
avoiding exact memorization are vulnerable to approximated memorization, which
can be subtly bypassed through creatively altered ”style-transfer” prompts.

####  3.3.3 LLM Alignment Protection

Refer to the discussion in Section 3.1, LLM alignment strategies, such as
RLHF, have been widely applied for constructing better LLMs aligned with human
values, such as enhancing the reliability of generated outputs, and improving
ethical decision-making. Therefore, the similar pipeline of model alignment
can also be investigated from the perspective of copyright protection. The
study by Kassem et al. (2023) introduces a model alignment strategy designed
to prevent memorization in LLMs. Namely, they proposed a reward-based
DeMemorization (DeMem) framework to reduce memorization in language models
(LMs) by employing a paraphrasing strategy. The framework uses negative
similarity, specifically a BERTScore metric, as a reward signal. By inputting
prefixes from the original pre-training dataset into the LM to generate
suffixes, the dissimilarity between the true and generated suffixes is
calculated. This score is then maximized during training to ensure that the
LM’s tendency to replicate verbatim memorization is diminished. Besides, they
found that when model size increases, both the convergence rate and
dissimilarity score increase, suggesting that larger models may tend to
“forget” the memorized data faster.

Figure 16: An overview of EUL pipeline (Image Credit to Chen & Yang (2023))

####  3.3.4 Machine Unlearning

Even if the LLM has memorized some copyrighted data, the model builder can
thereafter use “machine unlearning” techniques to delete the copyrighted
information for copyright protection. For example, Eldan & Russinovich (2023)
first used a reinforced model to identify key tokens by comparing logits with
a baseline model. Then, unique expressions in the data are replaced with
generic ones, and new labels are generated to mimic a model not trained on the
data. Finally, they show that fine-tuning the model with these labels can
effectively remove the original text from the model’s memory upon contextual
prompting. Chen & Yang (2023) proposed an efficient method EUL for updating
Large Language Models (LLMs) without full retraining by integrating
“lightweight unlearning layers”. The pipeline of EUL is shown in Figure 16.
These layers are trained using a selective teacher-student objective within
the transformer architecture. Additionally, to handle a sequence of forgetting
operations, a fusion mechanism is employed to merge various unlearning layers
efficiently where each learns to forget different sets of data. Yao et al.
(2023) proposed a resource-efficient unlearning method that only requires
negative examples that we want the LLM to forget, without the need for access
to the original training set. Their method mainly utilizes gradient ascent to
make the model forget the undesired information, like user-reported or red-
teaming failed cases. Besides, they also added additional loss terms to
enforce the model generated similar outputs with the unlearned model on some
normal inputs to maintain their utility. Pawelczyk et al. (2023) explored a
new class of unlearning methods for LLMs called “In-Context Unlearning”(ICUL)
without access to the model parameters. ICUL first inverts the label of the
data to be forgotten and then incorporates a set of correctly labeled data
points to mitigate over-correction, finally, it integrates the query input
into the altered training template and prompts the model to predict with no
temperature adjustment to erase the specified data from the model’s knowledge.
Their empirical results suggest that ICUL reliably removes the influence of
training points and show that label flipping for in-context examples can
impact the model’s output.

###  3.4 Model Copyright Protection

The development of LLMs such as ChatGPT/GPT-4 has significantly enhanced their
ability to generate human-like texts. Many companies have integrated LLMs into
interactive user interfaces like New Bing, yielding numerous benefits.
However, the emergence of LLMs also brings forth new challenges. On one hand,
LLM increased the difficulty in distinguishing between genuine content and
fake news, propaganda, or misinformation generated by LLMs like GPT-4. On the
other hand, LLMs are susceptible to theft. For example, Taori et al. (2023)
demonstrated the possibility of distilling LLMs solely through API access.
This underscores the importance of safeguarding the copyright of LLMs, which
often require significant resources for training. In light of these concerns,
the protection methods can be classified into two categories based on their
objectives:

  * •

Watermark, which aims to ascertain whether given texts are generated by the
LLMs owned by the respective company.

  * •

Model parameter protection, which aims to detect unauthorized model stealing
the proprietary model parameters.

In the following of this section, we discuss the protection on text generation
model from the two perspectives.

####  3.4.1 Watermark

This type of protection aims to detect whether a text is generated by a
particular LLM. Watermarking-based methods are majorly explored to detect LLM-
generated texts. In general, this type of method attaches a watermark
generator with LLMs which modify the generated texts to include pre-designed
watermarks and a detector is used to determine whether suspicious texts
contain watermarks. In this subsection, we conduct a comprehensive analysis of
various watermarking methods, with a focus on discussing their robustness and
the trade-offs between detection accuracy and text quality.

In Kirchenbauer et al. (2023a), LLM-generated texts with watermarks are
generated by forcing the model to sample words from a partition of vocabulary
with a higher probability. In detail, a “green list” is a random partition of
the vocabulary decided by a random function with the hash of the previous
token as the random seed. During the text generation process of an LLM, the
probability of green list is increased in the sampling step. Thus, tokens from
green list has a high probability of being sampled. Since the green list is
decided by the previous tokens, this operation forms the matching between
green lists and the consequent tokens. A higher proportion of token from green
list indicate the text is more possible to generated by a watermarked model. A
$z$-test is conducted to detect this matching between tokens and determine
whether the test texts is generated with or without knowledge of the green
list. The texts that have a significant $z$-score will be detected as LLM-
generated texts. Based on the work (Kirchenbauer et al., 2023a), Yoo et al.
(2023) improved it by partitioning the vocabulary into lists of more colors
instead of just green list. This method can encode a message like the
ownership information into the text, and enrich the application of watermarks.
Takezawa et al. (2023) highlighted that longer generated texts typically
exhibit higher $z$-scores. To accommodate this, it may be reasonable to relax
the strict proportion requirements of the green list for lengthier texts.

However, due to the fact that the watermark in the texts is easy to be
perturbed by attack methods like paraphrasing, synonym replacement and so on,
more works focusing on the robustness of watermarks are proposed. For example,
in the watermarking method of Kirchenbauer et al. (2023b), once the previous
token is reworded by paraphrase, the seed of the random function will also
change which causes a different green list and disturb the matching between
the green list and the consequent token. Kirchenbauer et al. (2023b) leveraged
an improved hashing scheme to enhance the reliability and robustness of this
watermarking in real scenarios. This method uses more previous tokens to
compute the random seed to increase the tolerance of paraphrase if part of the
tokens are reworded, and a windowed test called WinMax is leveraged in the
detection process. Zhao et al. (2023a) proposed Unigram-Watermark against
paraphrasing attacks. It changes the method of Kirchenbauer et al. (2023a)
with a fixed “green list” which does not change no matter what the hash of the
previous token is. They proved that the fixed list can also be secure and
robust. However, this robustness requires a constraint that only a limited
amount of words can be changed, which is strict in practical scenarios.

Ren et al. (2023) proposed the method called SemaMark to use the semantics as
the seed of the hash function. Because the hash of the token is easy to change
by paraphrase, but the semantics are usually stable. SemaMark uses the
embeddings of the previous tokens, which represent the semantic information,
as the seed of the random function to decide the green list. The embeddings
can be obtained from the LLM itself. Even though the malicious user wants to
use paraphrase to remove the watermark for copyright infringement, the
semantics are likely to remain unchanged and will not influence the partition
of the green list for the consequent token. Thus, the watermark is robust and
will not be easily removed by paraphrase. Liu et al. (2023b) proposed a
similar method which uses an additional text embedding model to extract the
embeddings.

Fu et al. (2023) found that the random selection of green list might be
conflicted with the original semantics. For example, the tokens in the
generated output for a summarization task will have a overlap with the input
text to summarize. Thus, Fu et al. (2023) proposed to incorporate the
semantically related tokens into green list and randomly partition the rest
vocabulary. The semantically related tokens are selected by the similarity
with the previous inputs. This can further increase the fraction of green list
matching in watermarked text and thus improve the detection performance.

To safeguard against the deciphering of the green list’s watermark rule, Liu
et al. (2023a) introduced an unfalsifiable approach. This technique employs a
fully connected layer to ascertain the green list for each token position,
taking into account preceding tokens. The input for this layer comprises a
pre-established window of prior tokens. Liu et al. (2023a) claimed that their
method resists cracking owing to the substantial size of the window and the
difficulty in extracting the network for the green list by external parties,
primarily because of the absence of clear ground truth labels.

The previous watermarking methods rely on the “green list”, which is shown to
harm the quality of generated texts (Sato et al., 2023; Takezawa et al., 2023;
Hu et al., 2023; Kuditipudi et al., 2023; Christ et al., 2023). Kuditipudi et
al. (2023) proposed a distortion-free watermarking scheme and took the
randomness of watermark key $\xi$ into consideration. They assume the
watermarked generation given a random watermark key $\xi_{i}$ is
$\Gamma(\xi_{i},p_{F})$ where $p_{F}$ is defined in Eq. (16). Then the
watermark is claimed as distortion-free if

$\mathbb{P}(\Gamma(\xi_{i},p_{F})=w_{i})=p_{F}(w_{i})$ (19)

In particular, they implement the $\Gamma$ with a decoder that maps a sequence
of uniform random variables and permutations to tokens using inverse transform
sampling as follows:

$\Gamma(\xi,p_{F}):=\pi^{-1}(\min\\{\pi(i):p_{F}(\\{j:\pi(j)\leq\pi(i)\\})\geq
u\\})$

In this equation, the watermark key $\xi=(u,v)$, $\pi$ is a random permutation
and $u$ is a pre-specified threshold. With this formulation,
$\Gamma(\xi,p_{F})$ is designed to select the token possessing the smallest
index within the permutation $\pi$ ensuring that CDF of $p_{F}$ related to
$\pi$ reaches a minimum threshold of $u$. For identifying watermarked text,
the detection process involves correlating the token indices within the text
against $\xi$. On the contrary, the indices of tokens in the non-watermarked
text will be i.i.d uniform irrespective of the text itself and thus not
correlated to the watermark key $\xi$. This scheme is shown to be distortion-
free with respective to the definition in Eq. (19).

Hu et al. (2023) also proposed a watermark called Unbiased Watermark aiming to
guarantee the non-degradation of the text quality. This method leverages
distribution reweighting to modify the original generation distribution into
the watermarked distribution. In detail, given a watermark code $E$, they
define a reweighting function $R_{E}$ mapping the distribution
$p_{F}(\cdot|w_{1},...,w_{i-1})$ to the watermarked version
$R_{E}(p_{F}(\cdot|w_{1},...,w_{i-1}))$. When taking the randomness of $E$
into consideration, the watermark is claimed as unbiased if and only if

$\mathbb{E}_{E}R_{E}(p_{F}(\cdot|w_{1},...,w_{i-1}))=p_{F}(\cdot|w_{1},...,w_{i-1})$
(20)

This equation indicates that although the generation distribution for each
individual token is distorted, the mean output token probabilities remain the
original distribution when considering the randomness of $E$. In practice,
they first select a 1024-bit random bitstring as a key $k$, then the watermark
code $E$ is generated via a hash function $\hat{E}$ whose inputs are $k$ and
the token sequence $(w_{1},...,w_{i-1})$, i.e.
$E=\hat{E}(k,(w_{1},...,w_{i-1}))$. Finally, a delta function
$R_{E}(P)=\delta_{sampling_{P}(E)}$ is used as the reweighting function to
create the watermarked distribution. They showed that this procedure satisfies
Eq. (20), thus unbiased. During the detection stage, they leverage the
likelihood ratio test and compute a log likelihood ratio (LLR) score to detect
the watermarks.

The above watermark methods require the access to generation process and
cannot be used to the scenarios of black-box models or when there is only API
for the LLM available. Yang et al. (2023) developed a binary-encoding-based
watermarking framework for black-box language models. To be more specific,
original words in generated texts are encoded as bit-0, and part of their
synonym candidates are encoded as bit-1. Then a hash function is leveraged to
randomly substitute bit-0 words with bit-1 synonyms, and a $z$-test is
conducted to infer the change of distribution for bit-1 and bit-0.

From a different perspective, a family of watermarks named Easymark is
proposed in (Sato et al., 2023) to add watermarks directly to the generated
texts rather than the generation distribution. Whitemark, the most
representative method in this family, exploits the fact that Unicode has many
codepoints for whitespace and it replaces a whitespace from the original
codepoint U+0020 to another codepoint such as U+2004. Then during the
detection process, the number of U+2004 is counted to determine if the text is
watermarked. Therefore, this method does not distort the distribution during
generation and does not change the original meaning of the text.

####  3.4.2 Model Parameter Protection

Figure 17: An overview of EmbMarker (Image Credit to Peng et al. (2023b))

This type of protection aims to detect unauthorized models stealing the
proprietary model parameters. Birch et al. (2023) proposed the Model Leeching
attack method, which can distill characteristics by questioning an LLM. This
method can extract task capability from ChatGPT-3.5-Turbo, achieving 73% Exact
Match (EM) similarity and SQuAD EM and F1 accuracy scores of 75% and 87% with
only $50 cost Birch et al. (2023). After getting an extracted model, attackers
can then inverse the model, inference membership, leak privacy data, and theft
model intellectual property. Peng et al. (2023b) proposed an Embedding
Watermark method called EmbMarker which can defend against model extraction
attacks from Embedding as a Service (EaaS). As shown in Figure 17, EmbMarker
selects a set of moderately frequent words from a general text corpus to
create a trigger set. It then chooses a target embedding to serve as the
watermark. This watermark is inserted into the embeddings of texts that
contain trigger words, functioning as a backdoor. The intensity of the
watermark insertion is proportional to the quantity of trigger words present
in the text. This method ensures the efficient transfer of the watermark
backdoor to the EaaS-stealer’s model for copyright verification purposes,
while simultaneously minimizing any negative effects on the original utility
of the embeddings. Fan et al. (2023) proposed FateLLM, an industrial-grade
federated learning framework for LLMs, which can protect the intellectual
property of LLMs by using a federated intellectual property protection
approach. Every client autonomously confirms the presence of watermarks in the
model, asserting their individual ownership of the federated model. This
verification is achieved without revealing any private training data or
confidential watermark details. Additionally, the approach ensures the
preservation of data privacy throughout both the training and inference phases
by employing privacy-protective mechanisms. He et al. (2022) uncovered that
prevailing watermarking techniques often disrupt the word distribution. This
distortion can be exploited to deduce the watermarked words by analyzing the
frequency changes in potential watermark candidates through sufficient
statistical methods. Consequently, this makes the watermarks relatively easy
to identify and remove. Therefore, they proposed CATER (Figure 18) for
protecting text generation APIs via more stealthy watermarks. In detail, CATER
injects the watermarks in a word distribution conditional on linguistic
features (condition $c$) such as part-of-speech and dependency tree (Nadkarni
et al., 2011), while maintaining the original word distribution.

Figure 18: The workflow of CATER (Image Credit to He et al. (2022)).

##  4 Other domains

In this section, we will discuss copyright protection in various domains,
including code and audio. It is important to acknowledge that, in contrast to
areas like LLMs and image generation, the fields of code and audio generation
have relatively fewer studies focused on copyright protection against DGMs. We
will outline and discuss the existing methods employed in the generation of
code and audio in this segment.

###  4.1 Code Generation

Code copyright is a critical aspect of the text domain with distinctive
characteristics. As opposed to other forms of textual content, codes are often
accompanied by explicit copyright licenses, and code owners are able to take
more proactive strategies to protect their data. In this survey, code
copyright protection majorly refers to preventing the usage of copyrighted
code as a part of the training data for machine learning models without the
explicit permission of the copyright holder. For example, Copilot
121212https://copilot.github.com/, a closed-source deep learning code
generation model, is trained using a wide range of open-source code
repositories sourced from GitHub, without consideration for their licenses. It
has been observed that Copilot occasionally reproduces identical code snippets
with copyleft licenses from its training dataset while generating code (Lee et
al., 2023b). As these models learn from the data they are trained on, they can
generate new code that is strikingly similar to the copyrighted code,
potentially leading to copyright infringement, which poses a significant
threat to code copyright protection.

There are different aspects to protect code copyright. From the model
builder’s perspective, they can carefully curate their training data and train
the model to provide reference during generation, so that the data sources are
legal and can be tracked. For example, Li et al. (2023) took several important
steps to ensure safe release of their code-LLMs StarCoder and StarCoderBase,
including an improved Personally Identifiable Information (PII) redaction
pipeline and a novel attribution tracing tool, making the StarCoder models
publicly available under a more commercially viable version of the Open
Responsible AI Model license. To remove PII from the training data, the
authors trained an encoder-only model called StarEncoder and then fine-tuned
it for the Named Entity Recognition (NER) task to detect PII. Yan & Li (2022)
introduced a tool named WhyGen that explains the generated code by referring
to training examples. The method extracts an inference fingerprint from the
neural network when generating the code and using the inference fingerprint to
get the most similar examples from training data. The inference fingerprint is
produced by the activation values of a set of intermediate neurons in the
network during the inference pass. The authors claimed that if Whygen with the
auto-completion feature is used in an IDE, it can help reduce the concern
about using source code from copyrighted project without permission.

From the perspective of code owners, they have the solutions to proactively
protect their codes via poisoning or watermarking. Lee et al. (2023b) designed
and implemented a prototype, CoProtector, which uses data poisoning techniques
to defend source code repositories against such exploits. Specifically, given
a repository to protect, CoProtector generates two distinct types of poison
instances from its original code artifact. One type aims to reduce the
performance of Copilot-like DGMs, and the other can reliably embed backdoors
for later detection. In addition, CoProtector proposes a collaborative
protection framework that manages protected poison repositories and amount of
poison instances in the network. Ji et al. (2022) designed a set of
lightweight transformations that can be applied to codes before they are open-
source released. They also leveraged a pre-trained code embedding model (e.g.,
CodeBERT (Feng et al., 2020)) to guide the selection of transformations. The
transformed codes are called “unlearnable examples” so that code generation
models trained on these codes can only obtain faulty knowledge.

Besides, code owners can also check the safety of their source codes via
membership inference. Ma et al. (2023a) proposed CodeForensic, which uses
membership inference to audit whether codes are misused for training a code
generator. Particularly, it leverages a reference-model-based likelihood ratio
test (LRT) to ascertain if a specific code snippet is present in a model’s
training dataset. The experimental outcomes indicate that the LRT-based
membership inference can deliver satisfactory results. However, the method is
limited in practical application due to its reliance on the access to logits
of the generative model.

###  4.2 Audio Generation

Utilizing zero-shot voice synthesis technology, it is now possible to generate
realistic audio from just a few seconds of a speaker’s recording.
Consequently, the implementation of watermarking in the audio domain is
essential to prevent model misuse and data infringement. Among existing
methods, the primary focus is on protection of model copyright.

Chen et al. (2023) introduced a post-watermarking method called WavMark to
accelerate the detection synthesized audio and prevent the unauthorized usage
of audio generation models. The post-watermarking process, as applied in
WavMark, modifies the audio after its generation. It incorporates a watermark
by encoding a message into the frequency domain of the generated audio. The
model owner can detect the watermark by extracting the message from suspect
audios. The whole framework of WavMark is a invertible neural network that is
composed of $L$ cascaded invertible blocks. Each block can be applied in both
a encoding way and a decoding way. WavMark first transforms the audio
$\mathbf{x}$ to watermark into frequency domain by a short-time Fourier, which
is denoted as $\mathbf{x}_{\text{frq}}$. Then the network encodes a 32-bit
message $m$ into the audio $\mathbf{x}_{\text{frq}}$ by the encoding operation
in each block:

$\displaystyle\mathbf{x}_{\text{frq}}^{l+1}$
$\displaystyle=\mathbf{x}_{\text{frq}}^{l}+\phi\left(\mathbf{m}^{l}\right)$
$\displaystyle\mathbf{m}^{l+1}$
$\displaystyle=\mathbf{m}^{l}\odot\exp\left(\sigma\left(\rho\left(\mathbf{x}_{\text{frq}}^{l+1}\right)\right)\right)+\eta\left(\mathbf{x}_{\text{frq}}^{l+1}\right),$

where $\phi$, $\rho$, and $\eta$ is a dense block introduced by Wang et al.
(2018), and $\mathbf{m}^{l}$ and $\mathbf{x}_{\text{frq}}^{l}$ is the message
and audio data in $l$-th block. The decoding operation is starting from
$\mathbf{x}_{\text{frq}}^{L}$ and a random sampled starting point of message
$\mathbf{m}^{L}$:

$\displaystyle\mathbf{m}^{l}$
$\displaystyle=\left(\mathbf{m}^{l+1}-\eta\left(\mathbf{x}^{l+1}\right)\right)\odot\exp\left(-\sigma\left(\rho\left(\mathbf{x}^{l+1}\right)\right)\right)$
$\displaystyle\mathbf{x}^{l}$
$\displaystyle=\mathbf{x}^{l+1}-\phi\left(\mathbf{m}^{l}\right)$

The invertible neural network is trained to minimized the $L_{2}$ distance of
encoded and decoded messages. In this way, WavMark can encode and decode a
message from the frequency domain of an generated audio.

Cao et al. (2023) proposed a watermarking method to protect the copyright of
audio-generative diffusion models. This method follows the main idea of
trigger-based watermarking in Section 2.4.3. It incorporates a trigger
mechanism into the diffusion model. Once the trigger is activated by a pre-
defined input, the model will generate a watermarked audio to verify the
ownership of the audio-generative diffusion model. To make the trigger
invisible, this method uses environmental natural sounds and Infrasounds
around 10Hz, which are typically ignored in daily life. From an acoustic
standpoint, environmental sounds blend with ambient noise encountered by
machines, while Infrasounds are inaudible to humans and indiscernible in mel-
spectrograms. Additionally, it’s important that these watermarking triggers do
not significantly impact the performance of the original diffusion model, so