# Spatial dissipative solitons in graphene-based active random metamaterials

Ashis Paul<EMAIL_ADDRESS>Department of Physics, Indian Institute of
Technology Kharagpur, West Bengal 721302, India  Andrea Marini
<EMAIL_ADDRESS>Department of Physical and Chemical Sciences,
University of L’Aquila, Via Vetoio, 67100 L’Aquila, Italy  Samudra Roy
<EMAIL_ADDRESS>Department of Physics, Indian Institute of
Technology Kharagpur, West Bengal 721302, India

###### Abstract

We investigate dissipative nonlinear dynamics in graphene-based active
metamaterials composed of randomly dispersed graphene nano-flakes embedded
within an externally pumped gain medium. We observe that graphene saturable
nonlinearity produces a sub-critical bifurcation of nonlinear modes, enabling
self-organization of the emitted radiation into several dissipative soliton
structures with distinct topological charges. We systematically investigate
the existence domains of such nonlinear waves and their spatio-temporal
dynamics, finding that soliton vortices are unstable, thus enabling self-
organization into single dissipative structures with vanishing topological
charge, independently of the shape of the graphene nano-flakes. Our results
shed light on self-organization of coherent radiation structures in disordered
systems and are relevant for future cavity-free lasers and amplifier designs.

##  I Introduction

Dissipative solitons (DSs) are self-organized localized structures arising in
diverse dissipative physical and biological systems AkhmedievBook , playing a
central role in mode-locked lasers GreluNatPhot2012 ; PengSciAdv2019 ,
photonic molecules HelgasonNatPhot2021 , time-delay feedback systems
YanchukPRL2019 , Bose-Einstein condensates of exciton polaritons
Ostrovskaya2012  and cold atoms BrazhniPRL2009 ; TesioOptExpr2013 .
Furthermore, DSs offer an appealing platform for cavity-free stimulated
emission of radiation, as in particular in optical amplifiers UltanirPRL2003 .
In this context, innovative cavity-free radiation sources can be achieved also
in colloidal solutions embedding randomly arranged scatterers and an optically
active medium WiersmaNatPhys2008 ; Gottardo2008 ; Wiersma2013  \- the so-
called random lasers (RLs). However, cavity-free stimulated emission of
radiation in RLs implies poor quality of the output beam, which inherently
lacks reproducibility and tunability.

Recently, the advent of nanophotonic materials has enabled plasmon stimulated
emission in metallic nanoparticles Bergman2003 ; Noginov2009 ; Stockman2010
and waveguides embedding gaining media Noginov2008 ; Marini2009 ; Bolger2010 ;
DeLeon2010 . Metamaterials (MMs) constitute a promising platform for planar
sources of coherent radiation Zheludev2008  and for cavity-free lasers
operating in the stopped-light regime Hess2012 ; Marini2016 . Furthermore, MMs
embedding subwavelength randomly arranged nanostructures in an amplifying
medium are promising for achieving self-organized stimulated emission
MariniPRL2016 , enabling to control the properties of the output beam thanks
to stable DSs. This requires the engineering and exploitation of the MM
effective nonlinear optical response, which is greatly facilitated by graphene
Bonaccorso2010 ; Bao2012 ; Javier2014  because it provides saturated
absorption at low peak intensities $\simeq$ MW$/$cm2 Bao2009 ; Xing2010 ;
MariniPRB2017  over a broad spectrum spanning the optical and infrared
frequency ranges. This peculiar property of graphene is enabled by its conical
band-structure providing resonant interband absorption at arbitrary radiation
frequencies, thus greatly facilitating the engineering of the random MM
effective nonlinear response MariniPRL2016 .

Here, we investigate the dissipative dynamics of nonlinear waves emitted in a
realistic MM composed of disordered graphene nano-flakes embedded in
externally pumped rhodamine 6G (R6G). Our model, accounting non-perturbatively
for the saturable absorption of graphene nano-flakes and for the saturable
gain of R6G, is based on an effective nonlinearly-saturated propagation
equation for the field envelope of the seeded radiation. We observe that
homogeneous nonlinear waves (HNWs) bifurcate from the trivial background sub-
critically and that are bistable in a particular range of the linear gain
coefficient. We further investigate HNW modulational instability, calculating
the modulation gain spectrum, which suggests that DSs can be excited in the
system. In turn, we investigate numerically DS existence, finding DS vortices
with topological charges $m=0,1,2$ and determining their existence domains. We
finally investigate DS stability over propagation, finding that only the $m=0$
DSs can be stable in a well-determined range of the linear gain coefficient.
Our results extend previous $(1+1)$D investigations MariniPRL2016  shedding
light to nonlinear dynamics in $(2+1)$D and indicating that DSs offer a viable
platform to manipulate the spatial pattern of the radiation seeded in random
active MMs.

##  II Model

We consider a disordered medium, schematically depicted in Fig. 1, composed of
undoped graphene nano-flakes (with lateral dimensions $<50$ nm) dispersed in
polymethyl methacrylate (PMMA) embedding optically pumped R6G dyes, a typical
gaining medium for RLs, see, e.g., Ref. Leonetti2013 . Population inversion in
R6G can be attained by a frequency-doubled Nd-YAG laser-pump beam with
wavelength $\lambda_{\rm pump}=532$ nm, providing peak stimulated emission at
$\lambda_{\rm seed}=593$ nm. In turn, owing to the deep subwavelength
dimensions of graphene nano-flakes, radiation seeded in such a system
experiences an effective dissipative and nonlinear dielectric response arising
from electron dynamics in graphene nano-flakes, PMMA and R6G.

Figure 1: Sketch of the considered graphene-based active MM consisting of a
mixture of graphene nano-flakes and optically pumped R6G molecules embedded in
PMMA.

PMMA acts as a background dielectric with relative permittivity $\epsilon_{\rm
b}\simeq 2.23$. Because of the subwavelength dimensions, the intensity-
dependent effective dielectric response $\epsilon_{\rm eff}(I)$ (where $I$ is
the intensity of the seed beam) of the considered disordered system does not
depend on the geometrical details of the graphene nano-flakes but only on
their filling fraction $f=V_{\rm gr}/V$. Furthermore, because unintentional
doping of the graphene nano-flakes ($E_{\rm F}\simeq 0.2$ eV) does not affect
significantly $\epsilon_{\rm eff}$ in the visible, we assume vanishing doping
$E_{\rm F}=0$. We model electron-dynamics in graphene following a previously
reported non-perturbative approach in thee massless Dirac fermion picture
MariniPRB2017  and calculate numerically the intensity-dependent conductivity
$\sigma(I)$, which real and imaginary parts are depicted in Fig. 2a by the
full lines. We find good fitting with the analytical expression
$\sigma(I)=\sigma_{0}[\frac{1}{\sqrt{1+I/I_{\rm
S}}}-i\frac{1-e^{-\eta_{1}\sqrt{I/I_{\rm S}}}}{\sqrt{1+\eta_{2}(I/I_{\rm
S})^{0.4}}}]$, where $\sigma_{0}=e^{2}/4\hbar$, $I_{\rm S}=137\hbar\omega_{\rm
S}^{2}\omega_{\rm seed}^{2}/(8\pi v_{\rm F}^{2})$, $\omega_{\rm S}=6.16$ rad
ps-1, $\eta_{1}=(\omega_{\eta}/\omega_{\rm seed})$,
$\eta_{2}=(\omega_{\eta}/\omega_{\rm seed})^{0.8}$, $\omega_{\eta}=46.20$ rad
ps-1, $e$ is the electron charge, $\omega_{\rm seed}=2\pi c/\lambda_{\rm
seed}$, $c$ is the speed of light in vacuum, $\hbar$ is the reduced Planck
constant and $v_{\rm F}\simeq c/300$ is the Fermi velocity of electrons in
graphene.

Figure 2: (a) Dependence of the real (${\rm Re}[\sigma/\sigma_{0}]$, blue
line) and imaginary (${\rm Im}[\sigma/\sigma_{0}]$, red line) parts of the
nonlinear conductivity $\sigma$ of undoped graphene nano-flakes (normalized to
the linear conductivity $\sigma_{0}$) over the intensity $I$ of seed radiation
at $\lambda_{\rm seed}=593$ nm. Dashed lines indicate fitting with the
expression $\sigma(I)=\sigma_{0}[\frac{1}{\sqrt{1+I/I_{\rm
S}}}-i\frac{1-e^{-\eta_{1}\sqrt{I/I_{\rm S}}}}{\sqrt{1+\eta_{2}(I/I_{\rm
S})^{0.4}}}]$. (b) Existence curves of stationary HNWs, where
$I_{0}=(1/2)\epsilon_{0}\sqrt{\epsilon_{\rm b}}c|A_{0}|^{2}$ and $g_{0}$ is
the linear gain coefficient for $f=2.13\times 10^{-4}$ (full lines) and
$f=2.84\times 10^{-4}$ (dashed lines). The orange circle dots indicate the
background instability threshold $g_{\rm th}$ for the two distinct filling
fractions considered. Figure 3: Gain spectrum of (a) Type I for $g_{0}=60$
cm-1, (b) Type II for $g_{0}=60$ cm-1, and (c) Type III for $g_{0}=80$ cm-1
HNWs, depicting the maximum instability growth rate $\gamma$ as a function of
$q_{x}$ and $q_{y}$ for $f=2.13\times 10^{-4}$.

Thus, the induced polarization of graphene nano-flakes with filling fraction
$f$ is accounted by the susceptibility $\chi_{\rm
G}=(2/3)if\sigma(I)\lambda_{\rm seed}/(2\pi t_{\rm gr}\epsilon_{0}c)$, where
$\epsilon_{0}$ is the vacuum dielectric permittivity, $t_{\rm gr}=0.335$ nm is
the graphene effective thickness taken from the interlayer distance in
graphite, and the factor $2/3$ accounts for averaging over the random
orientation of the nano-flakes. Amplification by R6G is modeled by a two-level
system, where stable population inversion is assumed to be produced by the
external optical pump operating at $\lambda_{\rm pump}=532$ nm. Because the
seed photon is resonant with the two-level transition energy, R6G produces a
purely imaginary susceptibility $\chi_{\rm R6G}=-i(g_{0}(I_{\rm pump})/k_{\rm
seed})/[1+I/I_{\rm S}^{\rm R6G}]$, where $k_{\rm seed}=2\pi/\lambda_{\rm
seed}$, $I_{\rm S}^{\rm R6G}\simeq 150$ MW$/$cm2 is the R6G saturation
intensity Nithyaja2011 , and $g_{0}(I_{\rm pump})\simeq 400$ cm-1 is the R6G
linear gain coefficient Noginov2008  that depends over the R6G density and can
be tuned by the external pump. Thus, the effective response of the complex
system depicted in Fig. 1 in the limit of small graphene density is accounted
by the complex and nonlinear dielectric constant $\epsilon_{\rm
eff}(I)\simeq\epsilon_{\rm b}+\chi_{\rm R6G}+\chi_{\rm G}$.

We model nonlinear propagation of the monochromatic seed beam with angular
frequency $\omega_{\rm seed}$ and carrier wave-number $\beta_{0}=k_{\rm
seed}\sqrt{\epsilon_{\rm b}}$ by taking the Ansatz ${\bf E}_{\rm seed}({\bf
r},t)={\rm Re}[A({\bf r}_{\bot},z)e^{i\beta_{0}z-i\omega_{\rm seed}t}{\bf
n}]$, where ${\bf r}=({\bf r}_{\bot},~{}z)$ is the position vector, ${\bf n}$
is the seed polarization unit vector, and $A({\bf r}_{\bot},z)$ is the field
envelope. In the slowly varying envelope approximation (SVEA) Leonettii2013 ,
Maxwell’s equations reduce to a generalized Ginzburg-Landau equation
MariniPRL2016  for the field envelope

$i\partial_{z}A+\frac{1}{2\beta_{0}}\nabla^{2}_{\bot}A+\frac{\beta_{0}}{2\epsilon_{\rm
b}}\left[\epsilon_{\rm eff}\left(|A|^{2}\right)-\epsilon_{\rm b}\right]A=0.$
(1)

The nonlinear propagation above accounts for the effects of diffraction, R6G
gain, and graphene saturable absorption on the spatial evolution of the seed
beam in the effective medium approximation.

##  III Nonlinear waves

###  III.1 HNWs

Extended HNWs are excited by a seed plane wave where the input envelope does
not depend over the transverse position $A({\bf r}_{\bot},0)=A_{0}$. We
calculate the nonlinear dispersion of stationary HNWs by taking the Ansatz
$A(z)=A_{0}e^{i\delta\beta z}$ in Eq. (1) suppressing the diffraction term
$\nabla^{2}_{\bot}A=0$ and obtaining the propagation constant correction
$\delta\beta=(\beta_{0}/2\epsilon_{\rm b})\left\\{{\rm Re}\left[\epsilon_{\rm
eff}\left(|A_{0}|^{2}\right)\right]-\epsilon_{\rm b}\right\\}$. Conversely to
Hamiltonian systems where $A_{0}$ is arbitrary, owing to dissipation
(accounted by ${\rm Im}\epsilon_{\rm eff}$) stationary HNWs exist only for
specific input amplitudes $A_{0}$ fixed by the nonlinear condition ${\rm
Im}[\epsilon_{\rm eff}\left(|A_{0}|^{2}\right)]=0$, which we solve by the
Newton-Raphson method for several distinct values of the linear gain
coefficent $g_{0}$. Thanks to graphene saturable absorption, we observe a
subcritical bifurcation from the background $A_{0}=0$ at a specific gain
threshold $g_{\rm th}$ depending over the graphene filling fraction $f$ and
leading to bistability of two distinct solutions for $g_{0}<g_{\rm th}$, which
we name type I and II, as illustrated in Fig. 2b. Conversely, for
$g_{0}>g_{\rm th}$ we find only one homogenous solution, which we name Type
III, see Fig. 2b.

###  III.2 Modulational instability of HNWs

The stability of HNWs against small-amplitude perturbing waves with amplitudes
$\delta A_{1}$, $\delta A_{2}$ and wave-vector ${\bf q}_{\bot}$ is evaluated
by setting

$\displaystyle A=A_{0}+\big{[}\delta A_{1}{\rm e}^{hz+i{\bf q}_{\bot}\cdot{\bf
r}_{\bot}}+\delta A_{2}^{*}{\rm e}^{h^{*}z-i{\bf q}_{\bot}\cdot{\bf
r}_{\bot}}\big{]}{\rm e}^{i\delta\beta z},$ (2)

where $h=\gamma+i\Upsilon$ , $\gamma$ and $\Upsilon$ are the instability
growth rate and the wave-number shift, respectively, induced by the perturbing
waves. Inserting this expression in Eq. (1), and linearizing with respect to
the small-amplitudes $\delta A_{1},\delta A_{2}$, we find

$\begin{bmatrix}-iF_{1}+iF_{2}&i\delta_{2}F^{\prime}A_{0}^{2}\\\
-i\delta_{2}F^{\prime*}A_{0}^{2*}&iF_{1}+iF_{2}^{*}\end{bmatrix}\begin{bmatrix}\delta
A_{1}\\\ \delta A_{2}\end{bmatrix}=h\begin{bmatrix}\delta A_{1}\\\ \delta
A_{2}\end{bmatrix}$ (3)

where, $\delta_{2}=(2\beta_{0})^{-1}$,
$F=(\beta_{0}/2\epsilon_{b})\big{[}\epsilon_{eff}(|A_{0}|^{2})-\epsilon_{b}\big{]}$,
$F^{\prime}=\partial_{|A_{0}|^{2}}F$,
$F_{1}=\delta\beta+\delta_{2}q_{\perp}^{2}$ and
$F_{2}=\delta_{2}(F^{\prime}|A_{0}|^{2}+F)$.

Figure 4: Evolution of perturbed HNWs of (a-c) Type I ($g_{0}=60$ cm-1), (d-f)
Type II ($g_{0}=60$ cm-1), and (g-i) Type III ($g_{0}=80$ cm-1) for
$f=2.13\times 10^{-4}$. The density plots depict the dependence of the
intensity profile $I({\bf r}_{\bot})$ over the transverse position vector
${\bf r}_{\bot}=x\hat{e}_{x}+y\hat{e}_{y}$ for several distinct propagation
distances $z=\bar{z}$ indicated at the top of every figure.

We calculate numerically the complex eigenvalues $h=\gamma+i\Upsilon$ of such
a linear algebraic system of equations for every wave-vector ${\bf q}_{\bot}$.
Positive/negative growth rates $\gamma$ indicate instability/stability against
small-amplitude perturbations. We find that all HNWs are unstable and that,
while the maximum instability growth rate $\gamma$ is peaked at $q_{\bot}=0$
for Type I I HNWs, see Fig. 3a, for Type II and Type III HNWs it is peaked at
a specific $q_{\bot}=\bar{q}_{\bot}$, see Figs. 3b,c. This suggests that
perturbed HNWs can develop filamentation over propagation. Furthermore, we
evaluate the stability of the background $A_{0}=0$, finding that it is stable
for $g_{0}<g_{\rm th}$ and unstable for $g_{0}>g_{\rm th}$. In turn, $g_{\rm
th}$ represents the linear gain stability threshold of the background. Owing
to the above described modulational instability scenario, nonlinear dynamics
in subcritical and overcritical domains leads to qualitatively different
phenomena. In order to gain insight on the evolution of perturbed HNWs, we
solve Eq. (1) in propagation by the split-step discrete Fourier transform
complemented with a fourth-order Runge-Kutta algorithm, which results are
summarized in Fig. 4, where we depict the intensity profile $I({\bf
r}_{\bot},\bar{z})=(1/2)\epsilon_{0}\sqrt{\epsilon_{b}}c|A({\bf
r}_{\bot},\bar{z})|^{2}$ as a function of ${\bf r}_{\bot}$ for (a-c) Type I
($g_{0}=60$ cm-1), (d-f) Type II ($g_{0}=60$ cm-1), and (g-i) Type III
($g_{0}=80$ cm-1) perturbed HNWs for $f=2.13\times 10^{-4}$ at several
distinct propagation distances $z=\bar{z}$. Note that for $g_{0}=60$ cm-1
(subcritical since $g_{0}<g_{\rm th}$, see Fig. 1b) both Type I and Type II
HNWs develop instabilities over propagation and tend to filament into self-
organized patterns and interacting domains, while for $g_{0}=80$ cm-1
(overcritical since $g_{0}>g_{\rm th}$, see Fig. 1b) the dynamics becomes
chaotic MariniPRA2010 .

###  III.3 Localized NWs

Localized, stationary and self-sustaining nonlinear waves, commonly named
dissipative solitons (DSs) Grelu2012 , are calculated by setting the Ansatz
$A({\bf r}_{\bot},z)=A_{0}(r_{\bot}){\rm e}^{i\delta\beta z+im\phi}$ in Eq.
(1), where $m$ is an arbitrary topological charge and $\phi$ is the azimuthal
angle between ${\bf r}_{\bot}$ and the $x$-axis, obtaining an ordinary
differential equation (ODE) for $A_{0}(r_{\bot})$,

Figure 5: Dependence over the linear gain coefficient $g_{0}$ of peak
intensities $I_{\rm max}$ (full lines, left full vertical axis) and spot-sizes
$w$ (dashed lines, right dashed vertical axis) of the numerically calculated
DSs with topological charge (a) $m=0$ and (b) $m=1$ for graphene filling
fraction $f=2.13\times 10^{-4}$.

$\displaystyle-\delta\beta
A_{0}(r_{\bot})+\frac{1}{2\beta_{0}}\left[\frac{1}{r_{\bot}}\frac{d}{dr_{\bot}}\left(r_{\bot}\frac{dA_{0}}{dr_{\bot}}\right)-\frac{m^{2}}{r_{\bot}^{2}}A_{0}\right]+$
$\displaystyle+\frac{\beta_{0}}{2\epsilon_{\rm b}}\left[\epsilon_{\rm
eff}\left(|A_{0}(r_{\bot})|^{2}\right)-\epsilon_{\rm
b}\right]A_{0}(r_{\bot})=0.$ (4)

In order to solve the ODE above, we transform it into a nonlinear system of
algebraic equations by discretizing the spatial variable $r_{\bot}(n)=r_{n}$,
first-order $dA_{0}/dr_{\bot}=[A_{0}(r_{n+1})-A_{0}(r_{n})]/(r_{n+1}-r_{n})$
and second-order
$d^{2}A_{0}/dr_{\bot}^{2}=[A_{0}(r_{n-1})-2A_{0}(r_{n})+A_{0}(r_{n+1})]/(r_{n}-r_{n-1})^{2}$
derivatives with $n=1,2,..,N$ and by applying the homogeneous boundary
conditions $A_{0}(r_{1})=A_{0}(r_{2})$ and $A(r_{N})=0$. We solve the
resulting nonlinear system of algebraic equations numerically through the
Newton-Raphson method. Also for localized NWs, we observe a subcritical
bifurcation from the trivial vacuum $A_{0}({\bf r}_{\bot})=0$ and we find
three types of localized stationary solutions (DS1, DS2, DS3) for every
$m=0,1,2$. In Figs. 5a,b we plot the maximum intensity $I_{\rm
max}=(1/2)\epsilon_{0}\sqrt{\epsilon_{\rm b}}c~{}{\rm
max}|A_{0}(r_{\bot})|^{2}$ (full lines) and the spot-size
$w=2[\int_{0}^{+\infty}r_{\bot}^{3}|A_{0}(r_{\bot})|^{2}dr_{\bot}/\int_{0}^{+\infty}r_{\bot}|A_{0}(r_{\bot})|^{2}dr_{\bot}]^{1/2}$
(dashed lines) of (a) $m=0$ and (b) $m=1$ DSs (of types $1,2,3$) against
$g_{0}$ for fixed graphene filling fraction $f=2.13\times 10^{-4}$. Similarly
to HNWs, also for localized NWs DS1 and DS2 coexist in the bistable
subcritical domain, while DS3 exists only in the overcritical domain. All the
intensity profiles of the DSs found are bell-shaped, while their phase
$\varphi({\bf r}_{\bot})={\rm atan}[{\rm Im}A_{0}({\bf r}_{\bot})/{\rm
Re}A_{0}({\bf r}_{\bot})]$ is not uniform over the $x-y$ plane, which implies
an internal power flow enabling stationary propagation of DSs Grelu2012 .
Indeed, note that in the considered dissipative system, conversely to
traditional soliton families in Hamiltonian systems, for every DS type there
exists only one solution with fixed peak intensity and spot-size, which arises
from the double nonlinear compensation of diffraction vs focusing and gain vs
absorption Grelu2012 . Note also that both the peak intensity ($0<I_{\rm
max}<10$ GW$/$cm2) and the spot-size ($100$ $\mu$m $<w<10$ $1$ mm) of every DS
can be tuned efficiently by the pump intensity $I_{\rm pump}$ modulating the
linear gain coefficient $50$ cm${}^{-1}<g_{0}(I_{\rm pump})<90$ cm-1, which is
experimentally attainable with R6G Noginov2008 .

Figure 6: Density plots illustrating the dependence of the intensity profiles
$I({\bf r}_{\bot},\bar{z})$ of the stable $m=0$ DS2 (for $g_{0}=55$ cm-1 and
$f=2.13\times 10^{-4}$) over ${\bf r}_{\bot}=x\hat{e}_{x}+y\hat{e}_{y}$ and
stationary propagation at several distinct $z=\bar{z}$ indicated on top of
every plot.  Figure 7: Density plots illustrating the dependence of the
intensity profiles $I({\bf r}_{\bot},\bar{z})$ over ${\bf
r}_{\bot}=x\hat{e}_{x}+y\hat{e}_{y}$ of (a) $m=0$ DS3 for $g_{0}=75$ cm-1, (b)
$m=1$ DS1 for $g_{0}=55$ cm-1, (c) $m=1$ DS2 for $g_{0}=60$ cm-1, (d) $m=1$
DS3 for $g_{0}=75$ cm-1, and $m=2$ DS2 for $g_{0}=60$ cm-1 at several distinct
$z=\bar{z}$ indicated on top of every plot. In all plots the graphene filling
fraction is set to $f=2.13\times 10^{-4}$.

##  IV Discussion

In order to shed light on radiation dynamics in the complex system considered,
we investigate the stability of the found DSs for $m=0,1,2$ in propagation by
solving Eq. (1) through a split-step discrete Fourier transform complemented
with a fourth-order Runge-Kutta algorithm. We find that stability only occurs
for $m=0$ DS2 in the subcritical domain, which intensity profile and
stationary propagation is illustrated in Fig. 6, while all the other DSs found
($m=0$ DS1,3 and $m=1,2$ DS1-3) are unstable. In Fig. 7 we illustrate the
initial steps of the unstable dynamics of (a) $m=0$ DS3 for $g_{0}=75$ cm-1,
(b) $m=1$ DS1 for $g_{0}=55$ cm-1, (c) $m=1$ DS2 for $g_{0}=60$ cm-1, (d)
$m=1$ DS3 for $g_{0}=75$ cm-1, and $m=2$ DS2 for $g_{0}=60$ cm-1. In all plots
the graphene filling fraction is set to $f=2.13\times 10^{-4}$. Due to the
instability of the background and the existence of several unstable DSs,
propagation over longer distances becomes chaotic in the overcritical domain.
Conversely, propagation over longer distances in the subcritical domain leads
to filamentation into several stable $m=0$ DS2 solitons, see the supplementary
videos, where we illustrate the breaking of $m=1$ and $m=2$ DSs into $2$ and
$4$ repulsive $m=0$ DSs, respectively.

##  V Conclusions

In conclusion, we have described the spatial formation of dissipative
nonlinear waves in a graphene-based active random metamaterial composed of
randomly dispersed graphene nano-flakes and optically pumped R6G. Owing to
bistability of HNWs in a subcritical region below the instability threshold of
the background, there exist several DS vortices with distinct topological
charges. By investigating systematically the existence and stability of DSs
one finds that only one specific DS with vanishing topological charge is
stable in the subcritical domain and becomes unstable in the overcritical
domain. Analysis of propagation dynamics of perturbed HNWs and DSs indicates
that nonlinear waves break into stable and repulsive DSs with fixed
topological charge $m=0$. This indicates that graphene-based active random
metamaterials enable the control of mode operation since the spatial pattern
of the excited nonlinear waves is fixed by the density of graphene nano-flakes
and the external optical pump tuning the linear gain coefficient $g_{0}$. Our
results indicate that self-organization of coherent radiation structures in
disordered systems can be exploited to design cavity-free laser operation and
and amplification, opening novel possibilities for the advancement of random
lasers.

## References

  * (1) Dissipative Solitons: From Optics to Biology and Medicine, edited by N. Akhmediev and A. Ankiewicz, Lecture Notes in Physics, Vol. 751 (Springer, Berlin, 2008).
  * (2) P. Grelu and N. Akhmediev, Nat. Photonics 6, 84 - 92 (2012).
  * (3) J. Peng, S. Boscolo, Z. Zhao, and H. Zeng, Sci. Adv. 5, eaax1110 (2019).
  * (4) O. B. Helgason, F. R. Arteaga-Sierra, Z. Ye, K. Twayana, P. A. Andrekson, M. Karlsson, J. Schröder, and V. Torres-Company, Nat. Photonics 15, 305 - 310 (2021).
  * (5) S. Yanchuk, S. Ruschel, J. Sieber, and M. Wolfrum, Phys. Rev. Lett. 123, 053901 (2019).
  * (6) E. A. Ostrovskaya, J. Abdullaev, A. S. Desyatnikov, M. D. Fraser, and Y. S. Kivshar, Phys. Rev. A 86, 013636 (2012).
  * (7) V. A. Brazhnyi, V. V. Konotop, V. M. Pérez-García, and H. Ott, Phys. Rev. Lett. 102, 144101 (2009).
  * (8) E. Tesio, G. R. M. Robb, T. Ackemann, W. J. Firth, and G.-L. Oppo, Opt. Express 21, 26144 - 26149 (2013).
  * (9) E. A. Ultanir, G. I. Stegeman, D. Michaelis, C. H. Lange, and F. Lederer, Phys. Rev. Lett. 90, 253903 (2003).
  * (10) D. S. Wiersma, Nat. Phys. 4, 359-367 (2008).
  * (11) S. Gottardo, R. Sapienza, P. D. García, A. Blanco, D. S. Wiersma, and C. López, Nat. Photon. 2, 429 - 432 (2008).
  * (12) D. S. Wiersma, Nat. Photon. 7, 188 - 196 (2013).
  * (13) D. J. Bergman and M. I. Stockman, Phys. Rev. Lett. 90, 027402 (2003).
  * (14) M. A. Noginov et al., Nature 460, 1110 - 1113 (2009).
  * (15) M. I. Stockman, J. Opt. 12, 024004 (2010).
  * (16) M. A. Noginov et al., Phys. Rev. Lett. 101, 226806 (2008).
  * (17) A. Marini, A. V. Gorbach, D. V. Skryabin, and A. V. Zayats, Opt. Lett., 34 2864 - 2866 (2009).
  * (18) P. M. Bolger, W. Dickson, A. V. Krasavin, L. Liebscher, S. G. Hickey, D. V. Skryabin, and A. V. Zayats, Opt. Lett., 35 1197 - 1199 (2010).
  * (19) I. De Leon and P. Berini, Nat. Photon. 4, 382 - 387 (2010).
  * (20) N. I. Zheludev, S. L. Prosvirnin, N. Papasimakis, and V. A. Fedotov, Nat. Photon. 2, 351 - 354 (2008).
  * (21) O. Hess, J. B. Pendry, S. A. Maier, R. F. Oulton, J. M. Hamm, and K. L. Tsakmakidis, Nat. Mater. 11, 573 - 584 (2012).
  * (22) A. Marini and F. J. García de Abajo, Scientific Reports 6, 20088 (2016).
  * (23) A. Marini and F. J. García de Abajo, Phys. Rev. Lett. 116, 217401 (2016).
  * (24) F. Bonaccorso, Z. Sun, T. Hasan, and A. C. Ferrari, Nat. Photon. 4, 611 - 622 (2010).
  * (25) Q. Bao and K. P. Loh, ACS Nano 6, 3677 - 3694 (2012).
  * (26) F. J. García de Abajo, ACS Photon. 1, 135 - 152 (2014).
  * (27) Q. Bao, H. Zhang, Y. Wang, Z. Ni, Y. Yan, Z. X. Shen, K. P. Loh, and D. Y. Tang, Adv. Funct. Mater. 19, 3077 - 3083 (2009).
  * (28) G. Xing, H. Guo, X. Zhang, T. C. Sum, and C. H. A. Huan, Opt. Express 18, 4564 - 4573 (2010).
  * (29) A. Marini, J. D. Cox, and F. J. García de Abajo, Phys. Rev. B 95, 125408 (2017).
  * (30) M. Leonetti, C. Conti, and C. Lopez, Nat. Commun. 4, 1740 (2013).
  * (31) B. Nithyaja, H. Misha, P. Radhakrishnan, and V. P. N. Nampoor, Appl. Phys. Lett. 109, 023110 (2011).
  * (32) M. Leonetti, C. Conti, and C. Lopez, Light Sci. Appl. 2, e88 (2013).
  * (33) A. Marini and D. V. Skryabin, Phys. Rev. A 81, 033850 (2010).
  * (34) P. Grelu and N. Akhmediev, Nat. Photon 6, 84 - 92 (2012).


# Learning to Refuse: Towards Mitigating Privacy Risks in LLMs

Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Wenliang Chen
Institute of Artificial Intelligence, School of Computer Science and
Technology,
Soochow University, China
{zhliu0106, tzhu7<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

Large language models (LLMs) exhibit remarkable capabilities in understanding
and generating natural language. However, these models can inadvertently
memorize private information, posing significant privacy risks. This study
addresses the challenge of enabling LLMs to protect specific individuals’
private data without the need for complete retraining. We propose  RETURN, a
Real-world pErsonal daTa UnleaRNing dataset, comprising 2,492 individuals from
Wikipedia with associated QA pairs, to evaluate machine unlearning (MU)
methods for protecting personal data in a realistic scenario. Additionally, we
introduce the Name-Aware Unlearning Framework (NAUF) for Privacy Protection,
which enables the model to learn which individuals’ information should be
protected without affecting its ability to answer questions related to other
unrelated individuals. Our extensive experiments demonstrate that NAUF
achieves a state-of-the-art average unlearning score, surpassing the best
baseline method by 5.65 points, effectively protecting target individuals’
personal data while maintaining the model’s general capabilities111Our code
and dataset are available at https://github.com/zhliu0106/learning-to-refuse.

Learning to Refuse: Towards Mitigating Privacy Risks in LLMs

Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Wenliang Chen††thanks:  Corresponding
author Institute of Artificial Intelligence, School of Computer Science and
Technology, Soochow University, China {zhliu0106, tzhu7,
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large language models (LLMs) demonstrate extraordinary abilities to understand
and generate natural languages following instructions, attributing to the
massive amounts of parameters and training data Brown et al. (2020); Anil et
al. (2023). However, these models sometimes memorize about private contents
since there are personally identifiable information in the pre-training corpus
Carlini et al. (2021); Huang et al. (2022). This presents a significant
privacy concern, as an adversary can prompt the model to extract an
individual’s name, email address, phone number, or other sensitive information
for malicious purposes, as shown in Figure 1. The General Data Protection
Regulation European Parliament and Council of the European Union (2016) gives
individuals Right To Be Forgotten (RTBF), which can limit the direct and
indirect commercial use of their personal information. This situation leads us
to the question: How can we enable LLMs to protect specific individual’s
private data to mitigate privacy risks?

Figure 1: The example of extracting private information from LLMs.

With the costly training process of LLMs, removing all private information
from the training data and retraining it from scratch is not a practical
solution Lison et al. (2021); Kandpal et al. (2022); Liu et al. (2024a).
Therefore, researchers have attempted to adopt machine unlearning (MU) as an
alternative, which aims to eliminate the influence of undesirable data and
associated model capabilities without retraining Cao and Yang (2015);
Bourtoule et al. (2021); Jang et al. (2022); Si et al. (2023); Zhang et al.
(2023a); Maini et al. (2024); Liu et al. (2024a). To evaluate the performance
of MU methods, some studies have experimented with question-answering datasets
Patil et al. (2023), fictitious biographies Maini et al. (2024), and
copyrighted contents Eldan and Russinovich (2023). However, there is a lack of
evaluation of MU methods for protecting personal privacy data in real-world
scenarios, where the target individuals exist in reality and have been
memorized by LLMs.

Figure 2: The construction of  RETURN and the process for evaluating Machine
Unlearning (MU) methods using this dataset.

Considering these problems, we propose  RETURN, a Real-world pErsonal daTa
UnleaRNing dataset. As illustrated in Figure 2, we collect extensive
background information on celebrities from Wikipedia and use GPT-4 Achiam et
al. (2023) to generate 20$\times$QA pairs for each individual. After manual
and automated validation, we obtain a dataset of 2,492 individuals, each with
a (Name, 20$\times$QA pairs) data instance. Next, we could select a base model
to evaluate the MU methods on  RETURN. In this work, we take
LLaMA-3-8B-Instruct AI@Meta (2024) as an example. We first identify
individuals with deep memorization in the model and then divide them into the
forget set and the retain set. Our goal is for the model to protect the
information of individuals in the forget set, ensuring that questions related
to these individuals are not answered correctly, while maintaining the model’s
performance on the retain set.

Existing MU methods often suffer from sensitivity to hyperparameter selection
or the inability to effectively distinguish between the forget set and the
retain set. To mitigate the drawbacks of existing methods, we propose a simple
yet novel unlearning method: Name-Aware Unlearning Framework (NAUF) for
privacy protection. The framework comprises two key components: Name-Aware
Refusal Answer and Contrastive Data Augmentation. The Name-Aware Refusal
Answer is designed to help the model learn which individuals’ information
should be protected, and the Contrastive Data Augmentation aims to expand the
distribution of both the forget set and the retain set for enhancing the
generalization of our method. We evaluate the effectiveness of our proposed
method on  RETURN and compare it with the baseline methods, and the results
show that our proposed NAUF achieves a state-of-the-art average unlearning
score, outperforming the best baseline method by 5.65 points.

Our contributions can be summarized as follows:

  * $\bullet$

We propose  RETURN, which consists of 2,492 real individual names and
20$\times$QA pairs for each individual. As far as we know, this is the first
dataset for evaluating MU methods for protecting personal data in a real-world
scenario.

  * $\bullet$

We propose a simple yet novel method NAUF for privacy protection. This method
could help the model protect the privacy of individuals in the forget set
while maintaining the model’s performance on the retain set.

  * $\bullet$

We conduct extensive experiments on  RETURN to evaluate the effectiveness of
our proposed method and compare it with the baseline methods. The results show
that our proposed NAUF achieves a state-of-the-art average unlearning score,
outperforming the best baseline method by 5.65 points. Through comprehensive
experimental analysis, we demonstrate the effectiveness of our proposed method
in protecting the privacy of individuals in the forget set while maintaining
the model’s performance on the retain set.

##  2 Related Work

##### Memorization and Privacy Risks of LLMs.

Previous works show that LLMs can memorize sensitive information from the
training data Thakkar et al. (2021); Carlini et al. (2021); Huang et al.
(2022). Adversaries can utilize membership inference attacks to infer whether
a specific data point was in the LLMs’ training set Shi et al. (2023); Liu et
al. (2024b). They can also recover the training data by powerful data
extraction attacks Carlini et al. (2021); Nasr et al. (2023). These privacy
risks can be mitigated by removing the sensitive information from the LLMs.
However, retraining the LLMs from scratch is impractical due to the high cost
of training Lison et al. (2021); Kandpal et al. (2022); Liu et al. (2024a).
One approach to minimizing the memorization of sensitive information is to
apply differential privacy techniques in model training Dwork et al. (2006);
Shokri and Shmatikov (2015); McMahan et al. (2017). Unfortunately, these
methods often reduce the accuracy and increase the training time, making them
less common in practice Jayaraman and Evans (2019).

##### Machine Unlearning for LLMs.

Machine unlearning (MU) aims to eliminate the influence of undesirable data
and remove associated model capabilities while preserving model performance
for other data Cao and Yang (2015); Bourtoule et al. (2021); Jang et al.
(2022); Si et al. (2023); Zhang et al. (2023a); Maini et al. (2024); Liu et
al. (2024a). The study of MU methods encompasses diverse domains, such as
image classification Ginart et al. (2019); Golatkar et al. (2020); Sekhari et
al. (2021); Fan et al. (2023), text-to-image generation Kumari et al. (2023);
Zhang et al. (2023b); Fan et al. (2023), and federated learning Wang et al.
(2022); Liu et al. (2023); Che et al. (2023).

Specifically in the era of LLMs, MU has been applied to addressing
trustworthiness concerns, such as toxicity Lu et al. (2022), copyright Eldan
and Russinovich (2023), and privacy Jang et al. (2022); Patil et al. (2023);
Maini et al. (2024). We find that these studies have tested MU methods on
question-answering datasets Jang et al. (2022); Patil et al. (2023),
fictitious biographies Maini et al. (2024), and copyrighted contents Eldan and
Russinovich (2023), but have not yet evaluated the methods for protecting
personal privacy data in real-world scenarios. Considering the practical
applications, we propose  RETURN to evaluate MU methods when an individual
practices his/her RTBT in a black-box setting, where adversaries can only
interact with the model through API query.

Jang et al. (2022) shows that simply performing gradient ascent on target
token sequences is effective at forgetting them with little to no degradation
of general language modeling performances. Maini et al. (2024) tries to
unlearn the memorized information in LLMs by relabeling the target data with
uninformed answers such as "I don’t know". We believe that these methods have
their drawbacks: gradient ascent is sensitive to hyperparameters and could
easily cause model training to crash; simply allowing the model to learn to
respond with uninformed answers could easily affect the model’s performance on
the retain set. Therefore, we propose Name-Aware Unlearning Framework, to
mitigate these issues and achieve a better balance between privacy protection
and model performance.

##  3  RETURN: Real-world pErsonal daTa UnleaRNing

In order to evaluate various MU methods in a practical scenario, we propose
RETURN, a Real-world pErsonal daTa UnleaRNing dataset. We take
Llama-3-8B-Instruct AI@Meta (2024) as an example to demonstrate how to use the
dataset to evaluate MU methods. It is worth noting that we could use any LLM
to replace Llama-3-8B-Instruct as the base model for evaluation.

###  3.1 Data Construction

We begin by leveraging PopQA Mallen et al. (2022) to collect a large set of
names of individuals. PopQA is a large-scale open-domain question-answering
(QA) dataset constructed by Mallen et al. (2022), consisting of 14k entity-
centric QA pairs. Each pair comes with the original [subject entity,
relationship type, object entity] annotation, as well as Wikipedia monthly
page views. Specifically, for the data in PopQA, we collect “subject entity”
if the “relationship type” is within [ occupation, place of birth, father,
mother]; and we collect “object entity” if the “relationship type” is within [
producer, director, screenwriter, composer, author].

After gathering these names, we retrieve their corresponding Wikipedia pages
and extract the abstracts from these pages as background
information222https://github.com/martin-majlis/Wikipedia-API. We then filter
the background information to retain only those whose word count falls between
100 and 500 words. Through this process, we ultimately obtain 2,516 records
consisting of (Name, Background Information). Next, given each pair of name
and the background information, we use a prompt to generate 20$\times$QA pairs
with GPT4 Achiam et al. (2023). The prompt template is shown in Table 3 in the
Appendix.

Item | Value
---|---
#Instances | 2,492
#QA pairs per instance | 20
Avg. background information tokens | 315.0
Avg. question tokens | 15.2
Avg. abstract tokens | 18.8
Table 1: Data statistics of  RETURN. The numbers of tokens are estimated with
LLaMA-3-8B-Instruct.

As shown in Table 1, after manually verifying and filtering out data with
content or formatting errors, we finally obtain  RETURN consisting of 2,492
(Name, 20$\times$QA pairs). Next, we will demonstrate how to use the dataset
to evaluate MU methods with LLaMA-3-8B-Instruct AI@Meta (2024).

###  3.2 Identifying Individuals with Deep Memorization

To perform unlearning on LLaMA-3, we first need to identify which individuals
the model has deeply memorized. We ask the model to answer the questions for
each individual in  RETURN, then calculate the average accuracy by comparing
the model’s predicted answers with the gold answers using a Natural Language
Inference (NLI) model 333We choose deberta-v3-base-tasksource-nli Sileo (2023)
to evaluate the correctness of model’s prediction.. If the prediction is
"entailment" or "neutral," we consider the model’s answer correct; if the NLI
model’s prediction is "contradiction," we consider the model’s answer
incorrect444When the model’s predicted answer is partially correct and does
not conflict with the gold answer, the NLI model’s prediction is ”neutral.”
Therefore, we will consider ”neutral” as correct in this context.. The
accuracy distribution of LLaMA-3 on  RETURN is shown in Figure 3. The higher
the accuracy, the more deeply the model memorizes the individual’s
information. Finally, we take 466 individuals with accuracy $\geq$ 0.8 as
individuals with deep memorization for the subsequent unlearning experiments.

Figure 3: Accuracy distribution of LLaMA-3 on  RETURN.

###  3.3 Evaluation Setup

We split the 466 individuals into 2 sets in a ratio of 1:9: forget set
$\mathcal{D}^{F}$ and retain set $\mathcal{D}^{R}$. We mark the original model
as $\mathcal{M}_{o}$ and the unlearned model as $\mathcal{M}_{u}$. We want the
model to learn to protect the privacy of individuals in the forget set,
ensuring that questions related to these individuals are not answered
correctly, while not affecting the performance on the retain set and other
tasks. Specifically, we aim for the following:

  1. 1.

For questions regarding individuals in $\mathcal{D}^{F}$, the model should not
answer correctly, or refuse to respond to protect their privacy.

  2. 2.

For questions regarding individuals in $\mathcal{D}^{R}$, the model should
respond normally.

  3. 3.

Meanwhile, MU methods should not affect the model’s general capabilities on
other tasks.

###  3.4 Evaluation Metrics

We measure MU methods’ comprehensive performance using the following metrics:

##### Forget Score.

To quantify the model’s ability to protect the privacy of individuals in the
forget set, we propose the Forget Score. It is calculated as the relative
decrease in accuracy on $\mathcal{D}^{F}$ after unlearning compared to the
original model’s accuracy on $\mathcal{D}^{F}$:

$\begin{split}ForgetScore=&\frac{Acc_{\mathcal{M}_{o}}(\mathcal{D}^{F})-Acc_{\mathcal{M}_{u}}(\mathcal{D}^{F})}{Acc_{\mathcal{M}_{o}}(\mathcal{D}^{F})}\\\
=&1-\frac{Acc_{\mathcal{M}_{u}}(\mathcal{D}^{F})}{Acc_{\mathcal{M}_{o}}(\mathcal{D}^{F})}\end{split}$
(1)

##### Retain Score.

To quantify the model’s ability to retain the performance on the retain set
after unlearning, we propose the Retain Score. It is calculated as the ratio
of the unlearned model’s accuracy on $\mathcal{D}^{R}$ to the original model’s
accuracy on $\mathcal{D}^{R}$:

$\begin{split}RetainScore=&\frac{Acc_{\mathcal{M}_{u}}(\mathcal{D}^{R})}{Acc_{\mathcal{M}_{o}}(\mathcal{D}^{R})}\end{split}$
(2)

##### Downstream Task Accuracy.

To quantify the influence of unlearning on the model’s general capabilities,
we evaluate the model on 5 downstream natural language processing tasks:
WinoGrande Sakaguchi et al. (2021), PIQA Bisk et al. (2020), LogiQA Liu et al.
(2020), LAMBADA Paperno et al. (2016), and ARC-c Clark et al. (2018). We use
the accuracy of the downstream tasks as the evaluation metric.

##  4 Machine Unlearning for LLMs

In this section, we introduce MU baselines. Then we propose a novel method for
mitigating privacy risks in LLMs: Name-Aware Unlearning Framework for Privacy
Protection.

###  4.1 Baseline Methods

A typical MU method often have two parts: unlearning on the forget set and
regularization on the retain set. These two types of loss can be used in any
combination.

####  4.1.1 Unlearning on Forget Set

##### Gradient Ascent.

Gradient ascent (GA) stands as the most straightforward method for unlearning,
which is simply performing gradient ascent on the loss over forget set. GA is
to minimize the likelihood of correct predictions on the forget set, denoted
as:

$\begin{split}\mathcal{L}_{GA}(\mathcal{D}^{F},\mathcal{M}_{u})=&-\mathbb{E}_{(x,y)\sim\mathcal{D}^{F}}[-\log(\mathcal{M}_{u}(y|x))]\\\
=&\mathbb{E}_{(x,y)\sim\mathcal{D}^{F}}[\log(\mathcal{M}_{u}(y|x))]\end{split}$
(3)

##### Negative Preference Optimization.

Zhang et al. (2024) proposed Negative Preference Optimization (NPO), a simple
alignment-inspired method that could efficiently and effectively unlearn a
target dataset. The loss function of NPO is defined as:

$\displaystyle\mathcal{L}_{NPO}(\mathcal{D}^{F},\mathcal{M}_{u},\mathcal{M}_{o})$
(4)
$\displaystyle\hfill=\frac{2}{\beta}\mathbb{E}_{(x,y)\sim\mathcal{D}^{F}}[\log(1+(\frac{\mathcal{M}_{u}(y|x)}{\mathcal{M}_{o}(y|x)})^{\beta})]$

##### Relabeled Gradient Descent.

A variant of GA is to transform it into a gradient descent approach, which
aims to maximize the likelihood of predictions on relabeled forget set.
Following Maini et al. (2024), we relabel the question in the forget set with
an uninformed answer like "I don’t know." (or any one of 100 versions of this
response, we name the uninformed answer set as $\mathcal{D}^{idk}$). The loss
function of Relabeled Gradient Descent (RGD) is defined as:

$\displaystyle\mathcal{L}_{RGD}(\mathcal{D}^{F},\mathcal{M}_{u})$ (5)
$\displaystyle\hfill=-\mathbb{E}_{(x,y)\sim\mathcal{D}^{F},y^{idk}\sim\mathcal{D}^{idk}}[\log(\mathcal{M}_{u}(y^{idk}|x))]$

##### Relabeled Direct Preference Optimization.

Direct Preference Optimization (DPO) seeks to fine-tune the model with human
preferences Rafailov et al. (2024). We take the uninformed answer from
$\mathcal{D}^{idk}$ as preferred answer, the gold answer as the dispreferred
answer. The loss function of Relabeled Direct Preference Optimization (RDPO)
is defined as:

$\displaystyle\mathcal{L}_{RDPO}(\mathcal{D}^{F},\mathcal{M}_{u},\mathcal{M}_{o})$
(6)
$\displaystyle\hfill=-\mathbb{E}_{(x,y)\sim\mathcal{D}^{F},y^{idk}\sim\mathcal{D}^{idk}}[\log\sigma(\beta\log\frac{\mathcal{M}_{u}(y^{idk}|x)}{\mathcal{M}_{o}(y^{idk}|x)}$
$\displaystyle\hfill-\beta\log\frac{\mathcal{M}_{u}(y|x)}{\mathcal{M}_{o}(y|x)})]$

####  4.1.2 Regularization on Retain Set

MU methods should not only protect the privacy of individuals in the forget
set but also maintain the model’s performance on the retain set.
Regularization methods are designed to achieve this goal. If we only fine-tune
the model to maximize the likelihood of the uninformed answer on the forget
set, the model may also refuse to answer the questions on the retain set. To
achieve a balance between the forget set and the retain set, there are two
regularization methods:

##### Gradient Descent Regularization.

Simply performing gradient descent (GD) on the loss over the retain set. The
loss function is defined as:

$\displaystyle\mathcal{L}_{GD}(\mathcal{D}^{R},\mathcal{M}_{u})$ (7)
$\displaystyle\hfill=-\mathbb{E}_{(x,y)\sim\mathcal{D}^{R}}[\log(\mathcal{M}_{u}(y|x))]$

##### Kullback-Leibler Divergence Regularization.

Minimizing the Kullback-Leibler divergence (KLD) between the predictions on
the retain set of the original model and the unlearned model. The loss
function is defined as:

$\displaystyle\mathcal{L}_{KL}(\mathcal{D}^{R},\mathcal{M}_{u},\mathcal{M}_{o})$
(8)
$\displaystyle\hfill=\mathbb{E}_{(x,y)\sim\mathcal{D}^{R}}[KL(\mathcal{M}_{o}(y|x)||\mathcal{M}_{u}(y|x))]$

Considering a computing budget that scales with the size of the forget set, we
randomly sample an example from $\mathcal{D}^{R}$ every time we see an example
from $\mathcal{D}^{F}$ to stay within the constraints following Maini et al.
(2024).

###  4.2 Name-Aware Unlearning Framework

In our pilot study, we find that RGD achieves the comparatively best
performance in protecting the privacy of individuals in the forget set.
However, the model’s performance on the retain set is significantly affected.
The model tends to refuse to answer the questions on the retain set, which is
not desirable. To address this issue, we propose a novel method: Name-Aware
Unlearning Framework (NAUF) for privacy protection. The framework comprises
two key components: Name-Aware Refusal Answer and Contrastive Data
Augmentation.

##### Name-Aware Refusal Answer.

First, we relabel the questions in the forget set with a name-aware refusal
answer, such as "I’m afraid I can’t help with inquiries about NAME." Then we
could perform gradient ascent on the loss over the relabeled forget set. The
name-aware refusal answer is designed to help the model learn which
individuals’ information should be protected. We curate 100 name-aware refusal
answer templates $\mathcal{D}^{refuse}$ using GPT-4, and some examples are
shown in Table 4 in the Appendix.

##### Contrastive Data Augmentation.

In addition, given the limited number of QA pairs for each individual, we
propose contrastive data augmentation (CDA) to enhance the generalization of
unlearning. Specifically:

  * •

For each individual in the forget set, we randomly sample questions from other
individuals in the forget set and replace the name with the target
individual’s name. Then relabel the questions with the name-aware refusal
answer.

  * •

For each individual in the retain set, we also randomly sample questions from
other individuals in the forget set and replace the name with the target
individual’s name, but we use the original model’s prediction for that
question as the relabeled answer.

This contrastive data augmentation strategy expands the distribution of both
the forget set and the retain set. For simplicity, we expand the forget set
and the retain set by doubling the amount of data.

##  5 Experiments

###  5.1 Implementation Details

Due to the limited training data available for unlearning, we aim to use this
limited data to teach the model to protect all privacy information of the
target individuals, which places stricter requirements on the generalization
capability of the MU methods. Considering this situation, we divide the QA
pairs for each individual in the forget set and retain set into train and test
sets in a ratio of 1:1, as well as
$\mathcal{D}_{train}^{F},\mathcal{D}_{test}^{F},\mathcal{D}_{train}^{R}$, and
$\mathcal{D}_{test}^{R}$. We use $\mathcal{D}_{train}^{F}$ and
$\mathcal{D}_{train}^{R}$ to perform unlearning on the model and then evaluate
each MU method on $\mathcal{D}_{test}^{F}$ and $\mathcal{D}_{test}^{R}$.

The $\beta$ for NPO and RDPO is set to 0.1. We use the AdamW optimizer with a
learning rate of 1e-5 for all experiments. We set the batch size to 32 and
train the model for 5 epochs. Considering the computational budget, we
constraint that the number of samples used from the retain set is equal to the
number of the entire forget set in each epoch. All experiments are conducted
with 2 NVIDIA A100 (40GB) GPUs.

###  5.2 Main Results

Method | Unlearning Score | Downstream Task Accuracy
---|---|---
Forget S. | Retain S. | Avg. | WG | PIQA | LQA | LAM | ARC-c | Avg.
Oracle | 0.00 | 100.00 | 50.00 | 72.14 | 78.40 | 33.18 | 71.92 | 56.83 | 62.49
Without Regularization |
GA | NS | NS | 0.00 | 48.70 | 47.06 | 22.89 | 0.02 | 25.68 | 28.87
NPO | 15.66 | 84.67 | 50.16 | 56.27 | 59.47 | 26.27 | 37.98 | 29.35 | 41.87
RGD | 96.46 | 3.16 | 49.81 | 70.56 | 75.24 | 28.26 | 46.15 | 36.43 | 51.33
RDPO | 25.25 | 72.47 | 48.86 | 55.33 | 56.42 | 26.57 | 26.86 | 21.93 | 37.42
NAUF(ours) | 100.00 | 0.06 | 50.03 | 69.77 | 75.68 | 29.03 | 62.84 | 35.41 | 54.55
\- CDA | 99.75 | 0.25 | 50.00 | 70.01 | 76.17 | 27.19 | 68.64 | 37.54 | 55.91
With GD Regularization |
GA | NS | 70.79 | 35.40 | 69.61 | 73.29 | 21.66 | 71.67 | 38.31 | 54.91
NPO | 33.33 | 80.81 | 57.07 | 71.74 | 78.40 | 29.19 | 73.24 | 45.90 | 59.69
RGD | 89.65 | 60.58 | 75.11 | 72.85 | 78.13 | 29.03 | 73.12 | 47.01 | 60.03
RDPO | 32.07 | 81.37 | 56.72 | 72.14 | 77.86 | 29.19 | 73.26 | 44.03 | 59.29
NAUF(ours) | 81.06 | 76.25 | 78.65 | 73.01 | 79.60 | 30.11 | 73.16 | 50.94 | 61.36
\- CDA | 70.71 | 75.71 | 73.21 | 72.61 | 78.84 | 28.88 | 75.57 | 47.18 | 60.62
With KLD Regularization |
GA | NS | NS | 0.00 | 50.28 | 43.63 | 21.97 | 0.91 | 22.87 | 27.93
NPO | 30.30 | 87.60 | 58.95 | 68.67 | 77.69 | 29.34 | 73.34 | 48.21 | 59.45
RGD | 96.21 | 52.01 | 74.11 | 71.51 | 79.33 | 26.42 | 72.11 | 50.77 | 60.03
RDPO | 22.47 | 87.44 | 54.96 | 71.43 | 79.22 | 29.65 | 71.86 | 50.09 | 60.45
NAUF(ours) | 93.69 | 67.82 | 80.76 | 72.22 | 79.27 | 29.80 | 72.21 | 50.51 | 60.80
\- CDA | 94.44 | 63.82 | 79.13 | 71.11 | 79.60 | 28.88 | 74.46 | 50.51 | 60.91
Table 2: The main results of the experiments. Forget S. denotes Forget Score,
Retain S. denotes Retain Score, WG denotes WinoGrande, LQA denotes LogiQA, LAM
denotes LAMBADA. Oracle refers to using the original model directly to compute
the metrics without applying any unlearning. Notably, NS denotes "NonSense",
which means the model’s prediction is meaningless, and we take it as 0 for
computing the average. We highlight the best results in bold. Figure 4: Impact
of the number of unlearning epochs on the performance of MU methods (best
viewed in color). Figure 5: Average unlearning score vs average downstream
task accuracy across different numbers of epochs (best viewed in color).

We present the main results of the experiments in Table 2. We report the
average unlearning score and average downstream task accuracy to evaluate the
overall performance of the model.

The results show that our proposed NAUF with KLD regularization achieves a
state-of-the-art average unlearning score, outperforming the best baseline
method (RGD with GD regularization) by 5.65 points. The GA method performs the
worst on our dataset, and the unlearned model generates meaningless
predictions for questions in the forget set and significantly impacts the
retain score and the performance on downstream tasks. The decline in the
retain score and the performance on downstream tasks is mitigated to some
extent only when using GD regularization.

We find that the RGD method achieves a better forget score than our method
when using any regularization method, but it significantly affects the retain
score. Intuitively, this could be attributed to the uninformed answer like "I
don’t know", which could not teach the model to distinguish the individuals
whose information should be protected. Our proposed name-aware refusal answer
can help the model learn which individuals’ information should be protected,
thereby achieving a better balance between the forget score and the retain
score.

###  5.3 Analysis

##### Importance of Regularization on Retain Set.

Without regularization on retain set, the average unlearning score of all
methods except GA is around 50 points, and the average downstream task
accuracy is also affected to varying degrees. With any regularization, the
unlearned model performs well on downstream tasks with any MU method, showing
performance close to the original model. This indicates that regularization on
the retain set can effectively protect the model’s general capabilities.

The experimental results indicate that our method, when using GD
regularization, achieves similar forget and retain scores, with a difference
of only 5 points between them. In contrast, when using KLD regularization, the
forget score reaches 93.69, but the retain score is only 67.82, resulting in a
difference of 26 points. This demonstrates that GD regularization can achieve
a better balance between unlearning metrics.

##### Importance of Contrastive Data Augmentation.

To analyze the importance of CDA, we evaluate the performance of our
unlearning framework without this component. The results are presented in
Table 2. We find that without regularization, CDA has almost no effect.
However, it can improve our method’s forget score by 10 points when using the
GD regularization. With the KLD regularization, it can increase the retain
score by 4 points while maintaining a similar forget score. Notably, our
method without CDA also achieves a competitive (with GD regularization) or
better (with KLD regularization) average unlearning score compared to the
baseline methods, which demonstrates the effectiveness of the name-aware
refusal answer. These findings indicate that CDA can enhance performance on
the forget set or retain set depending on the regularization method used,
thereby enhancing the generalization of our proposed unlearning framework.

##### Unlearning Performance across Different Numbers of Epochs.

We investigate the impact of the number of unlearning epochs on the
performance of MU methods. Specifically, We evaluate RGD and NAUF with 1, 3,
5, and 10 epochs, and the results are shown in Figure 4. For the Forget Score,
our method with KLD regularization demonstrates relatively stable performance
across different epochs. With GD regularization, the Forget Score improves as
the number of epochs increases. Conversely, for the Retain Score, our method
with GD regularization shows little variation across epochs, while KLD
regularization leads to a gradual improvement in the Retain Score with
increasing epochs. Our method’s average unlearning score improves with an
increasing number of epochs, while RGD shows little to no improvement from the
5 to the 10 epoch, which indicates our method still has room for further
optimization.

##### Average Unlearning Score vs Average Downstream Task Accuracy across
Different Numbers of Epochs.

We analyze the relationship between the average unlearning score and the
average downstream task accuracy across different numbers of epochs. We choose
RGD and NAUF with KLD regularization for this analysis, and the results are
shown in Figure 5. We observe that as the number of epochs increases, both the
average unlearning score and the average downstream task accuracy increase
proportionally. However, our method surpasses RGD in all aspects after just 3
epochs. Additionally, from the 5 to the 10 epoch, RGD shows a decline in
average downstream task accuracy without any significant improvement in the
average unlearning score. In contrast, our method continues to achieve higher
average unlearning scores at the 10 epoch while maintaining stable average
downstream task accuracy.

##  6 Conclusion

In this work, we introduce  RETURN, a novel benchmark designed to evaluate MU
methods for protecting personal data in a real-world scenario. We also present
the Name-Aware Unlearning Framework (NAUF), which integrates Name-Aware
Refusal Answer and Contrastive Data Augmentation to enhance the generalization
of unlearning methods. Our experimental results show that NAUF not only
effectively protects the privacy of individuals in the forget set but also
maintains the performance of the model on the retain set, achieving an average
unlearning score that outperforms the best baseline method by 5.65 points.
These findings underscore the potential of NAUF to advance privacy protection
in large language models.

## Limitations

The proposed NAUF method is designed for individual-level privacy protection.
Our goal is to protect all information about an individual, ensuring that the
model refuses to answer any questions related to that individual. However,
this method does not provide fine-grained protection of the target
individual’s information. In other words, it cannot distinguish between
questions that can be answered and those that are too sensitive to answer.
Future work will explore how to align the model with human judgment, enabling
it to discern which personal information can be publicly discussed and which
information, potentially susceptible to malicious use, should be protected.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 7432–7439.
  * Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021.  Machine unlearning.  In _2021 IEEE Symposium on Security and Privacy (SP)_ , pages 141–159. IEEE.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cao and Yang (2015) Yinzhi Cao and Junfeng Yang. 2015.  Towards making systems forget with machine unlearning.  In _2015 IEEE symposium on security and privacy_ , pages 463–480. IEEE.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650.
  * Che et al. (2023) Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. 2023.  Fast federated machine unlearning with nonlinear functional theory.  In _International conference on machine learning_ , pages 4241–4268. PMLR.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Dwork et al. (2006) Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006.  Calibrating noise to sensitivity in private data analysis.  In _Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3_ , pages 265–284. Springer.
  * Eldan and Russinovich (2023) Ronen Eldan and Mark Russinovich. 2023.  Who’s harry potter? approximate unlearning in llms.  _arXiv preprint arXiv:2310.02238_.
  * European Parliament and Council of the European Union (2016) European Parliament and Council of the European Union. 2016.  Regulation (EU) 2016/679 of the European Parliament and of the Council.  Accessed: 2024-06-06.
  * Fan et al. (2023) Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. 2023.  Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.  _arXiv preprint arXiv:2310.12508_.
  * Ginart et al. (2019) Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019.  Making ai forget you: Data deletion in machine learning.  _Advances in neural information processing systems_ , 32.
  * Golatkar et al. (2020) Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020.  Eternal sunshine of the spotless net: Selective forgetting in deep networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9304–9312.
  * Huang et al. (2022) Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022.  Are large pre-trained language models leaking your personal information?  _arXiv preprint arXiv:2205.12628_.
  * Jang et al. (2022) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022.  Knowledge unlearning for mitigating privacy risks in language models.  _arXiv preprint arXiv:2210.01504_.
  * Jayaraman and Evans (2019) Bargav Jayaraman and David Evans. 2019.  Evaluating differentially private machine learning in practice.  In _28th USENIX Security Symposium (USENIX Security 19)_ , pages 1895–1912.
  * Kandpal et al. (2022) Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022.  Deduplicating training data mitigates privacy risks in language models.  In _International Conference on Machine Learning_ , pages 10697–10707. PMLR.
  * Kumari et al. (2023) Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. 2023.  Ablating concepts in text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22691–22702.
  * Lison et al. (2021) Pierre Lison, Ildikó Pilán, David Sánchez, Montserrat Batet, and Lilja Øvrelid. 2021.  Anonymisation models for text data: State of the art, challenges and future directions.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4188–4203.
  * Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  _arXiv preprint arXiv:2007.08124_.
  * Liu et al. (2024a) Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. 2024a.  Rethinking machine unlearning for large language models.  _arXiv preprint arXiv:2402.08787_.
  * Liu et al. (2024b) Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, and Wenliang Chen. 2024b.  Probing language models for pre-training data detection.  _arXiv preprint arXiv:2406.01333_.
  * Liu et al. (2023) Ziyao Liu, Yu Jiang, Jiyuan Shen, Minyi Peng, Kwok-Yan Lam, and Xingliang Yuan. 2023.  A survey on federated unlearning: Challenges, methods, and future directions.  _arXiv preprint arXiv:2310.20448_.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022.  Quark: Controllable text generation with reinforced unlearning.  _Advances in neural information processing systems_ , 35:27591–27609.
  * Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024.  Tofu: A task of fictitious unlearning for llms.  _arXiv preprint arXiv:2401.06121_.
  * Mallen et al. (2022) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  _arXiv preprint arXiv:2212.10511_.
  * McMahan et al. (2017) H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2017.  Learning differentially private recurrent language models.  _arXiv preprint arXiv:1710.06963_.
  * Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023.  Scalable extraction of training data from (production) language models.  _arXiv preprint arXiv:2311.17035_.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016.  The lambada dataset: Word prediction requiring a broad discourse context.  _arXiv preprint arXiv:1606.06031_.
  * Patil et al. (2023) Vaidehi Patil, Peter Hase, and Mohit Bansal. 2023.  Can sensitive information be deleted from llms? objectives for defending against extraction attacks.  _arXiv preprint arXiv:2309.17410_.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Sekhari et al. (2021) Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021.  Remember what you want to forget: Algorithms for machine unlearning.  _Advances in Neural Information Processing Systems_ , 34:18075–18086.
  * Shi et al. (2023) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023.  Detecting pretraining data from large language models.  _arXiv preprint arXiv:2310.16789_.
  * Shokri and Shmatikov (2015) Reza Shokri and Vitaly Shmatikov. 2015.  Privacy-preserving deep learning.  In _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_ , pages 1310–1321.
  * Si et al. (2023) Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. 2023.  Knowledge unlearning for llms: Tasks, methods, and challenges.  _arXiv preprint arXiv:2311.15766_.
  * Sileo (2023) Damien Sileo. 2023.  tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation.  _arXiv preprint arXiv:2301.05948_.
  * Thakkar et al. (2021) Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. 2021.  Understanding unintended memorization in language models under federated learning.  In _Proceedings of the Third Workshop on Privacy in Natural Language Processing_ , pages 1–10.
  * Wang et al. (2022) Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. 2022.  Federated unlearning via class-discriminative pruning.  In _Proceedings of the ACM Web Conference 2022_ , pages 622–632.
  * Zhang et al. (2023a) Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2023a.  Right to be forgotten in the era of large language models: Implications, challenges, and solutions.  _arXiv preprint arXiv:2307.03941_.
  * Zhang et al. (2023b) Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. 2023b.  Forget-me-not: Learning to forget in text-to-image diffusion models.  _arXiv preprint arXiv:2303.17591_.
  * Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024.  Negative preference optimization: From catastrophic collapse to effective unlearning.  _arXiv preprint arXiv:2404.05868_.

##  Appendix A QA Pairs Generation Template

The prompt template for generating QA pairs used in subsection 3.1 is shown in
Table 3

##  Appendix B Examples of Name-Aware Refusal Answer Templates

We provide 10 examples of name-aware refusal answer templates in Table 4. It
is designed to help LLM learn which individuals’ information should be
protected.

QA Pairs Generation [ABSTRACT] Given the above [NAME]’s background
information, please give me 20 simple questions and answers about this person
point by point. Return the content STRICTLY in the following manner: Q1:
<content of the question>? A1: <content of the answer>. Q2: <content of the
question>? A2: <content of the answer>. … Q20: <content of the question>? A20:
<content of the answer>. Make sure the person’s name - [NAME] \- appears in
the content of the question. Make sure the answer is concise and accurate.
Table 3: QA pairs generation template. Name-Aware Refusal Answers Apologies, I
cannot discuss [NAME]-related inquiries. Sorry, I’m unable to provide
information regarding [NAME]. Unfortunately, I’m not able to address questions
about [NAME]. I’m sorry, but I can’t offer insights on [NAME]. Regrettably,
[NAME] is not a topic I can discuss. I’m unable to engage in discussions about
[NAME]. Sorry, I’m not equipped to answer queries about [NAME]. I can’t
provide information on topics involving [NAME]. Unfortunately, [NAME] falls
outside the scope of my responses. Apologies, but I’m unable to delve into
matters concerning [NAME]. Table 4:  Name-aware refusal answer templates.