Alignment-Aware Model Extraction Attacks on Large Language Models
Zi Liang Qingqing Ye Yanyun Wang Sen Zhang Yaxin Xiao Ronghua Li Jianliang Xu Haibo Hu





 








  
  
  

Alignment-Aware Model Extraction Attacks on Large Language Models
    Zi Liang^† Qingqing Ye^† Yanyun Wang^♭ Sen Zhang^†
  Yaxin Xiao^† Ronghua Li^† Jianliang Xu^ Haibo Hu^†


†: The Hong Kong Polytechnic University

♭: The Hong Kong University of Science and Technology (Guangzhou)

: Hong Kong Baptist University





    September 9, 2024
=============================================================================================================================================================================================================================================================



§ ABSTRACT 
Model extraction attacks (MEAs) on large language models (LLMs) have received increasing research attention lately.
Existing attack methods on LLMs inherit the extraction strategies from those designed for deep neural networks (DNNs) yet neglect the inconsistency of training tasks between MEA and LLMs' alignments. As such, they result in poor attack performances.
To tackle this issue, we present Locality Reinforced Distillation (LoRD), a novel model extraction attack algorithm specifically for LLMs. In particular, we design a policy-gradient-style training task, which utilizes victim models' responses as a signal to guide the crafting of preference for the local model.
Theoretical analysis has shown that i) LoRD's convergence procedure in MEAs is consistent with the alignments of LLMs, and ii) LoRD can reduce query complexity while mitigating watermark protection through exploration-based stealing.
Extensive experiments on domain-specific extractions demonstrate the superiority of our method by examining the extraction of various state-of-the-art commercial LLMs.




§ INTRODUCTION




In recent years, we have witnessed the remarkable success of large language models (LLMs) such as ChatGPT <cit.>, Gemini <cit.>, and Claude <cit.>, which are now widely employed in various consumer and industrial applications.
Despite their success, these models may suffer from model extraction attacks (MEAs) <cit.>, where their knowledge could be at risk of being stolen by an adversary through a local model that learns on the data collected from the victim model.
Besides of  some
"open-source" LLMs (e.g., Alpaca <cit.>), which are trained on GPT-4's chat history, cases of commercial model theft among companies have also been reported recently <cit.>.









































Under such a real-world threat, instead of focusing on MEAs against conventional DNNs, which have been extensively studied theoretically <cit.> and empirically <cit.>, a few recent works turn to explore model extraction algorithms and theorems for LLMs.
For example, Wallace et al. <cit.> propose a monolingual-query-based imitation attack framework to steal machine translation knowledge from generative language models such as GPT-2. Li et al. <cit.> investigate threats of stealing the code-related knowledge from LLMs.
However, these studies inherit those MEA
algorithms from traditional fields, such as computer vision <cit.>, and train the local
model via supervised learning like maximum likelihood estimation (MLE) <cit.>, while neglecting the inconsistency of training tasks between MEAs and the alignments <cit.> of modern LLMs. As shown in Figure <ref>, modern LLMs typically employ alignments using reinforced learning, which is missing in the local model training of conventional MEAs. As a result, these attacks usually suffer from poor performance.





In this paper, we challenge the effectiveness of MLE in stealing a reinforcement-learning-aligned LLM, by analyzing its potential drawbacks as follows:

    
  * Low query efficiency. Current
LLM-oriented MEAs suffer from unacceptably
significant query times because they must collect enough generated
responses, which entails exponential complexity in terms of generated
tokens, resulting in low query efficiency.


  * Vulnerability against defenses. Learning
directly from the victim model's responses can cause local models
to inadvertently incorporate those
watermarks <cit.> embedded in victim
models. The residue of such watermarks makes the extraction process
less stealthy and even serves as provenance evidence of model theft.



Motivated by these limitations, we propose LoRD
(Locality Reinforced
Distillation), a query-efficient and watermark-resistant
model extraction attack under a training paradigm with LLM's alignments.


However, stealing LLMs via this paradigm is challenging. The main reason is that the core building block of LLM's alignments, reinforcement learning with human feedback
(RLHF) <cit.>, heavily relies on the feedback
signal of human annotators, which is difficult to be reproduced directly into MEAs.
To tackle this challenge, we develop a
policy-gradient-style extraction procedure. This approach regards the
locality direction between the generations of local models and
victim models as the implicit reward signal. It can thus achieve a human-feedback-free reinforcement learning for our extraction attack.
From the theoretical perspective, we show why those existing MEAs using MLE and knowledge distillation (KD) are inconsistent with the optimization procedure in LLMs' alignments. Along this way, we also demonstrate why LoRD can achieve stronger watermark resistance and higher query efficiency.

Extensive experiments on five downstream NLP tasks and ten datasets
demonstrate that it is feasible to steal a commercial LLM with 175
billion parameters by a pre-trained local model with only 8 billion
parameters using just 100 queries. The resulting local model performs
statistically similar to the
victim model for tasks not requiring extra knowledge (e.g., data-to-text),
and only 0∼ 3 percentage lower for tasks requiring it (e.g.,
translation and QAs). This result poses an immediate threat of
task-specific extraction on commercial LLMs.
To further draw the capability boundary of such a threat, we also
illustrate the “spectrum” in difficulties and upper bounds for
extracting LLMs.






















To summarize, the contributions of our paper are as follows:


  * New Perspective of LLM Alignment for MEAs. We present LoRD, a novel model extraction attack algorithm for LLMs.
To our best knowledge, it is the first effective and realistic
extraction algorithm that is compatible with the alignment procedure
of LLMs.

  * Theoretical Guarantee. We theoretically prove that the convergence procedure of LoRD in MEAs is consistent with the alignments of LLMs. Furthermore, we demonstrate that LoRD can reduce query complexity while mitigating watermark protection through exploration-based stealing.

  * Systematical Evaluation. Extensive experiments on domain-specific extractions demonstrate that our proposed method outperforms current extraction strategies across different downstream NLP tasks.































































































































§ BACKGROUND


In this section, we introduce the preliminary of this
paper. We first introduce policy gradient models, the basic paradigm
of LLMs' alignments. Subsequently, we introduce the
construction procedure of LLMs. Finally, we summarize and review
previous research on model extraction attacks against language
models.



 §.§ Policy Gradient Models

Policy gradient models (PGM) are commonly used in reinforcement
learning (RL) algorithms to optimize the agents based on the decided
action of RL agents. Represented by TRPO <cit.> and
PPO <cit.>, policy gradient models minimize the
the following objective function:

    ℒ_pg,j=- 𝔼̂_j[p_j^r(θ)A_j],

where at each decision step j, p_j^r(θ)=π_θ(a_j|s_j)/π_θ_old(a_j|s_j) refers to the probability ratio defined by
the optimized policy π_θ(a_j|s_j) and the initial
policy π_θ_old(a_j|s_j),
s_j denotes the state of the environment, a_j denotes
the decided action of π_θ, and
A_j is the de-biased reward of a_j. A_j is
estimated by the Q-value minus the V-value, i.e.,

    A_j(s_j,a_j)=Q(s_j,a_j)-V(s_j).


Intuitively, Q-value refers to the reward if employing action
a_j at the given environment state s_j, which can be seen as
the label of policy's decision. V-value represents the estimation of
the expected reward at s_j. Consequently, A_j denotes the
surprise when taking action a_j.

To alleviate the off the cliff phenomenon that a large bad
gradient update occurred from Equation <ref>, PGMs, such as PPO and
TRPO, add some regularization terms to avoid large
gradients. Specifically, TRPO constrains the distribution between
π_θ and π_θ_old with KL divergence, and PPO
warps a “clip” function to constrain the bounds of p_j^r(θ).



 §.§ Language Modeling






Supervised Training (SFT).
Given a pre-trained model with parameters θ, supervised
training is essentially the maximum likelihood estimation (MLE)
task <cit.>, which fine-tunes θ on the
labeled dataset
𝒟_tr^s={(𝐱_i,𝐲_i)|i=1,2,...,N_trs}
by minimizing the following objective functions:

    ℒ_mle= -∏_i^N_trsP_θ(𝐲_i|𝐱_i)=-∏_i^N_trs∏_j^NP_θ(y_i,j|𝐱_i,𝐲_i,<j),

where N denotes the sequence length of 𝐲_i, y_i,j
denotes the j-th token in 𝐲_i, and 𝐲_i,<j={y_i,0,...,y_i,j-1}. The logarithmic formula of
Equation <ref>
can also be seen as a joint cross-entropy loss function:

    ℒ_ce=- ∑_i^N_trslogP_θ(𝐲_i|𝐱_i)=-∑_i^N_trs∑_j^NlogP_θ(y_i,j|𝐱_i,y_i,<j).


Equation <ref> is extensively utilized in LLM's pre-training and
fine-tuning procedures. For instance, it can be applied to
instruction-following supervised fine-tuning (SFT) with the training set 𝒟_tr, wherein 𝐱_i
encompasses the instruction and the task input, while 𝐲_i
denotes the reference response.

However, aligning LLMs merely by SFT is unrealistic, as 
MLE tends to align the model with the one-hot distribution of
𝐲, making it challenging to draw a sufficient variety of
examples due to the “exponential explosion” of tokens (see Section <ref> for
more details). Moreover, providing
standard answers for LLMs can sometimes be daunting for annotators,
which further slows down and even degrades the alignment process through direct training.

Therefore, instead of “learning from answers” as in Equation
<ref>, learning from preferences is proposed, which only
requires the annotators to select a better response from a pair of
texts generated by LLMs. Reinforcement learning (RL)  <cit.> has thus been
introduced to train LLMs based on preferences.


Aligning from Preferences.
Employing reinforcement learning in LLMs typically
consists of three stages. First, the annotators construct a preference dataset
𝒟^pref={(𝐱_i,𝐲_i^+,𝐲_i^-)}
by chatting with LLMs and rating their responses,
where 𝐲_i^+ and 𝐲_i^- denote the rated
positive and negative responses of the dialogue context
𝐱_i, respectively. Then, a reward model
R_θ_ϕ(𝐱,𝐲)→𝐫 is
trained based on 𝒟^pref to simulate the environment and
predict the reward values of tokens in given texts. It is trained with a
pair-wise loss,

    ℒ_r=-∑_(𝐱,𝐲^+,𝐲^-) ∼𝒟^prefσ(R_θ_ϕ(𝐱,𝐲^+)-R_θ_ϕ(𝐱,𝐲^-)),

where σ(·) denotes the sigmoid function.

Based on the reward model R_θ_ϕ(𝐱,𝐲), we can
finally train the language models P_θ by maximizing its reward,
i.e.,

    max_θ∑_𝐱∼𝒟_qR_θ_ϕ(𝐱,𝐲̂)-β𝔻_KL[P_θ(𝐲|𝐱)||P_θ_init(𝐲|𝐱)],

where 𝒟_q denotes the dataset of text inputs,
𝐲̂∼ P_θ(𝐲|𝐱)) denotes the
sampled sequence of the training model, and
θ_init is the initialized parameters of the model, e.g. the
parameters after SFT. The Kullback-Leibler (KL) divergence term,
β𝔻_KL[P_θ(𝐲|𝐱)||P_θ_init(𝐲|𝐱)],
introduced by TRPO <cit.>,
is incorporated to constrain the shift of distribution in generated texts
𝐲̂, where β is the hyperparameter.

Consequently, SFT shown in Equation <ref> fine-tunes the
pre-trained model with parameters θ_pre into an aligned model θ_sft through MLE, and RLHF outlined in
Equation <ref>, further aligns θ_sft towards the target model θ_vic.

There are several alternatives to the standard RLHF approach.
Lee et al. <cit.> propose reinforcement learning with AI feedback (RLAIF) as a
means to diminish the annotation burden associated with the preference
assessments. Besides, there are some approaches, such as direct preference
optimization (DPO) <cit.>, that conceptualize the language model itself as the reward model and thus consolidate Equation
<ref> and Equation <ref> into a unified supervised and
preference-based training task. Since they do not
change the primary targets (i.e., maximizing rewards) and optimization
strategies of LLM's alignments, we only consider the standard formation of alignments for simplicity in our theoretical analysis.







 §.§ Language Models Extraction


Studies to steal language models originated from the natural
language understanding (NLU) models, such as BERT<cit.>, and then
evolved to generative language models, especially large language models recently.

Krishna et al. <cit.> highlights early recognition of model extraction
threats in language models. By constructing text inputs with randomly
vocabulary sampling, they successfully extract the weights from BERT-based
APIs. Besides, Rafi et al. <cit.> investigate the feasibility of
side-channel model extraction attacks, revealing that by analyzing
extra signals from GPU kernels, one could accurately steal the model architecture
and its parameters. Subsequent
research <cit.> has thoroughly investigated the strategy
of ensembling victim models to train a competitor model that surpasses its teachers.

The exploration of generative language model extraction is still in its infant stage, with only a handful of studies thus far. Wallace et al. <cit.> investigate imitation attacks on natural language models. By designing monolingual query texts and collecting responses, they successfully extract the knowledge from a simulated machine translation model under the black-box settings. This research exhibits that slight architectural differences will not influence the extraction between language models. Li et al. <cit.> also explores the potential risks of stealing the code-generation abilities of LLMs into smaller downstream models. Unlike previous research <cit.>, this is the first study that selects LLMs as targets. By collecting large-scale domain-specific samples, they fine-tune a 7-billion local pre-trained model with them and show the similarity between the victim and local models in both performances and adversarial samples. However, these two studies employ the MLE loss (Equation <ref>) as the MEA method, neither considering whether MLE is compatible with LLMs's training, especially the alignment procedure shown in Section <ref>, nor addressing optimizations related to query efficiency and the watermark resistance. Besides, the scope of these studies is limited to stealing specific knowledge in a few downstream domains. At the same time, most of the critical aspects of LLMs and the required extraction capabilities, such as query numbers and local model scales, remain unresolved.




In contrast to stealing LLMs, IP protection methods have received considerable attention recently. By sampling a stealthy but representative “greed word set” on the vocabulary distribution, these methods <cit.> can remap the generated words into their synonyms or add the “watermarked” token automatically, and thus effectively certify the output. Besides, strategies such as integrating embeddings into the representation as the backdoor <cit.> or manipulating the probabilities with crafted sinusoidal noises <cit.> are also proposed. However, these approaches often presume more stringent conditions regarding the victim and the suspected models. This paper will further assess the effectiveness of LoRD and current MEAs in evading these black-box watermarking strategies.



§ THREAT MODEL







Adversary's Objective. The adversary's objective is to steal the targeted knowledge from LLMs, where both the domain-specific knowledge and LLM's general ability acquired in alignments are considered. Specifically, we select machine translation, reasoning, data-to-text, structured text generation, and summarization as the downstream domain-specific tasks. The adversary aims to develop a query-efficient MEA algorithm, since the amount of input and generated tokens will be counted as the costs. Additionally, the MEA methods are expected to be watermark-resistant, i.e., they are highly desired to reduce the risks of exposure to unauthorized stealing.

Targeted Models. We select Llama3-70B, GPT-3.5-turbo,  and GPT-4o as the victim models in this paper. Unlike previous works that only deployed simulated local victims (e.g., OPT <cit.>), our selections aim to expose the stealing threat on realistic AI services. Besides, our target models are specifically constrained to LLMs fine-tuned with alignment methods (e.g., RLHF) since they are not only state-of-the-art solutions now but also more valuable due to their human-based alignments.

Adversary's Capabilities. In accordance with the LLM-based AI service APIs, we identify two attack scenarios: black-box and grey-box attacks. In the black-box scenario, only textual responses the adversary is allowed to obtain. At the same time, all other information, such as the temperature, sampling strategies, and the hidden states of LLMs, are unseen and inaccessible. On the contrary, a grey-box attack allows the adversary to access the generation probabilities distribution of tokens. Notice that both MLE and our LoRD method are under black-box settings, and we only adopt grey-box settings on some particular stealing methods, such as knowledge distillation <cit.>.




Besides, this paper posits that the adversary usually has worse training conditions than the victims. Specifically, query times and the scale of the local model available to the adversary are much smaller than the victims' training datasets and model parameters. This setting has been adopted in previous LLMs' extraction <cit.>. We call it a LaViSH (Large-Victim-Small-Heist) framework, which allows us to estimate the upper bound of MEA risks empirically. For adversaries with more substantial resources, they can train more powerful MEA-based LLMs by leveraging MEA algorithms under our LaViSH settings.













§ LORD: LOCALITY REINFORCED DISTILLATION


In this section, we delve into the details of our model extraction
framework, LoRD (Locality Reinforced
Distillation). Specifically, we first introduce the
extraction procedure in Section <ref>, and then derive 
the loss function in Section <ref>.




 §.§ Overview

As described in Algorithm <ref>, LoRD follows a reinforcement
learning paradigm, that is, it consists
of several periods, and in each period, the model will learn to
explore new responses and attempt to enhance the model
trained in the last period. However, different from LLMs' alignments,
the agent can neither obtain the reward from the reward model
directly, nor label positive and negative responses manually. This motivates us to develop a locality-reinforced distillation method to address this issue.


Illustrated by Figure <ref>, the model samples two
sentences randomly at period
t-1, which are denoted as 𝐲_t-1^+ and
𝐲_t-1^-, respectively. In a new period t, we first compute the changes of likelihoods
for these two sentences, among the old model P_θ_t-1 and
the current model P_θ_t. These changes of likelihoods, denoted
as Δ_t^+ and Δ_t^-, indicate whether a selected
sentence is locally isotropic (Δ >0) to the victim
model's response 𝐲_vic or not (Δ≤ 0), which
can be seen as the feedback signal for P_θ_t in the current optimization step. For convenience, we may swap 𝐲_t-1^+ with
𝐲_t-1^- to make sure that
Δ_t^+>Δ_t^- always holds. In this way, for pairs
(𝐱,𝐲_vic) we can take 𝐲_t-1^+ as a
locality neighborhood of 𝐲_vic and 𝐲_t-1^- as the
negative sample, all of which can be utilized in the training of P_θ_t. Figure <ref> illustrates this procedure.
Additionally, we take 𝐲_t-1^+ as the positive label under the current scope only when Δ^+ and P_θ_t(𝐲_t-1^+|𝐱) both exceed their respective fixed thresholds τ1 and τ_2. If these conditions are not met, we will use 𝐲_vic as a substitute for 𝐲_t-1^+ to enable a cold start.

Based on 𝐲_vic, 𝐲_t-1^+, and
𝐲_t-1^-, we now derive the loss function of LoRD.




 §.§ Derivation of Loss Functions


From Section <ref>, we know that the loss function of a policy gradient model can be expressed as an objective function to maximize the rewards of decisions (see Equation <ref>) and a regularization term to ensure the stability of training. Following this paradigm, the loss function of LoRD could be

    ℒ_LoRD=ℒ_obj+ℒ_reg.


Objective function ℒ_obj.
Following the definition of R_θ_ϕ in Equation <ref>, we use the logarithmic proportion between the positive sample and
the negative sample as the de-biased reward, i.e.,
logP_θ_t(𝐲_t-1^+|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱). 


Subsequently, we take log[P_θ_t(𝐲_vic|𝐱)/P_θ_vic(𝐲_vic|𝐱)], the proportion of the victim model to the local model, as the ratio term to scale such a reward.

Therefore, the objective function can be formatted as

    ℒ̅_obj   =-∑_𝐱∈𝒟_q
                       |log[P_θ_vic(𝐲_vic|𝐱)/P_θ_t(𝐲_vic|𝐱)]|·log
    [P_θ_t(𝐲_t-1^+|𝐱)/P_θ_t(𝐲_t-1^-|𝐱)] 
       =-∑_𝐱∈𝒟_q [logP_θ_t(𝐲_t-1^+|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱)/|logP_θ_t(𝐲_vic|𝐱)-logP_θ_vic(𝐲_vic|𝐱)|],

where |·| indicates no gradient back propagation, t  and 𝐱∼𝒟_q denote the training period and the sampled query text, respectively.
Since it is impractical to obtain the victim model's probability P_θ_vic(𝐲_vic|𝐱) under black-box settings, we simplify Equation <ref> by setting P_θ_vic(𝐲_vic|𝐱)=1, and remove P_θ_t(𝐲_vic|𝐱) to ensure the stability in the black-box extraction, i.e.,

    ℒ_obj   =-∑_𝐱∈𝒟_qlog
    [P_θ_t(𝐲_t-1^+|𝐱)/P_θ_t(𝐲_t-1^-|𝐱)] 
       =-∑_𝐱∈𝒟_q [logP_θ_t(𝐲_t-1^+|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱)].











Regularization loss ℒ_reg. Current studies <cit.> tend to constrain θ_t with initial model's generating distribution P_θ_init(·|𝐱) in RLHF. However, this strategy requires re-loading an extra initial model's weights, which is memory-consuming. Also, the KL-divergence term 𝔻_KL(P_θ_t||P_θ_init) consists of P_θ_t(·|𝐱), which requires an extra exponential operation of logP_θ_t(·|𝐱) that will slow down the forward computation procedure.[logsoftmax is preferred in the implementation of deep learning frameworks <cit.>, as the exponential operation in softmax and the logarithmic operation in cross-entropy can be canceled out by each other.] Therefore, in MEAs, we put forward a regularization based on the victim model, where only the logarithmic part of KL divergence is employed and only the dimension of selected tokens is used in KL divergence, i.e.,

    ℒ_reg   =-∑_𝐱∈𝒟_q clip(log[P_θ_t(𝐲_vic|𝐱)/P_θ_t(𝐲_t-1^-|𝐱)])
       =-∑_𝐱∈𝒟_q clip(logP_θ_t(𝐲_vic|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱)).





In Equation <ref>, we utilize a clip(·) function to limit the value of the regularization term, as we expect the regularization term could only be used to avoid the off the cliff problem <cit.> in RL's convergence and the shift of the distribution <cit.> in LLMs' alignments. 



Incorporating Equation <ref> with Equation <ref>, we can
reshape the loss function of LoRD as

    ℒ_LoRD=ℒ_obj+ℒ_reg
       =∑_𝐱∈𝒟_qlog[P_θ_t(𝐲_t-1^-|𝐱)/P_θ_t(𝐲_t-1^+|𝐱)]+clip(log[P_θ_t(𝐲_t-1^-|𝐱)/P_θ_t(𝐲_vic|𝐱))].


Finally, we wrap ℒ_LoRD with a sigmoid function σ(·) to normalize the loss to the interval (0,1), which is
 
    ℒ=𝔼 σ(log[P_θ_t(𝐲_t-1^-|𝐱)/P_θ_t(𝐲_t-1^+|𝐱)]+clip(log[P_θ_t(𝐲_t-1^-|𝐱)/P_θ_t(𝐲_vic|𝐱)])).













§ THEORETICAL ANALYSIS


This section will compare LoRD with current model extraction methods from a theoretical perspective. We will first reveal the underlying inconsistency between the optimization procedure of LLMs, which typically involves RL-based alignments, and the previous model extraction approaches utilizing MLE and knowledge distillation (KD). Subsequently, we will demonstrate in theory the reasons why LoRD can achieve stronger watermark resistance and higher query efficiency than existing methods.









 §.§ Consistency Analysis regarding Different Learning Tasks




As we described in Section <ref>, both existing methods and LoRD are learned from the victim model's response 𝐲_vic and the corresponding probability distribution P_θ_vic(· |𝐱) ∈ℝ^V, where V denotes the vocabulary size. Therefore, we first investigate how the local model is learned to emulate the distribution of the victim model,P_θ_vic(·|𝐱), under the following three stealing strategies.

Expected Distribution of MLE. We can first reshape the MLE loss into a special formation of Kullback-Leibler divergence with labels of one-hot distributions, that is,

    ℒ_ce   =-
                      ∑_𝐱,𝐲∼𝒟_trlogP_θ(𝐲_vic|𝐱)
       =∑_𝐱,𝐲∼𝒟_tr∑_j^N𝔻_KL[1_y_vic,j||P_θ(·|𝐱,𝐲_vic,<j)],

where 1_y_vic,j is a one-hot vector in which only 1_y_vic,j[y_vic,j]=1 and all the other elements are 0. Equation <ref> demonstrates that MLE learns to maximize the probability of 𝐲_vic,j, without explicit constraints on probabilities across other dimensions.

Expected Distribution of KD.
Following a previous work <cit.>, the objective function of KD is

    ℒ_kd=𝔻_KL[P_θ_vic(· |𝐱)||P_θ(· |𝐱)]
       +T^2·𝔻_KL[SM(P_θ_vic(·
                        |𝐱)/T)||SM(P_θ(· |𝐱)/T)],

where SM(·) represents the softmax function, and T>1 denotes the temperature to smooth the targeted distribution P_θ_vic(· | 𝐱). As described in Equation <ref>, knowledge distillation aims to align P_θ(· |𝐱) with P_θ_vic(· |𝐱) in both the original and the smoothed probability across all dimensions, which is exceptionally comprehensive among these methods.



Expected Distribution of Alignments.
Replacing Equation <ref> with Equation <ref>, we can merge the optimization target of LLMs' alignments as

    min_θ*  -∑_(𝐱,𝐲^+,𝐲^-) ∼𝒟^prefσ(
    logP_θ*(𝐲^+|𝐱) /P_θ*(𝐲^-|𝐱)/P_θ_init(𝐲^+|𝐱)/P_θ_init(𝐲^-|𝐱))
       ⇒max_θ*∑_(𝐱,𝐲^+,𝐲^-) ∼𝒟^preflog P_θ*(𝐲^+|𝐱) - log
      P_θ*(𝐲^-|𝐱),

where θ * denotes the expected parameters of the models as 

    P_θ*(𝐲|𝐱)=1/Z(x)P_θ_init(𝐲|𝐱)· e^1/βR_ϕ(𝐱,𝐲).


More detailed derivation is in Appendix <ref>. Replacing Equation <ref> with Equation <ref>, the expected distribution can be represented as 𝐫_i,j· P_θ_init(· | 𝐱), in which 𝐫_i,j indicates the wrapped distribution gain. This distortion aims to maximize the ratio P_θ (y^+_j|𝐱,𝐲^+_<j) / P_θ (y^-_j|𝐱,𝐲^-_<j), and leave the probabilities in other dimensions unconstrained directly.

Expected Distribution of LoRD.
Similar to alignments, the expected converging procedure by the objective function ℒ_obj is also intended to maximize the ratio between positive samples and negative samples, i.e., P_θ_t(𝐲_t-1^+|𝐱)/P_θ_t(𝐲_t-1^-|𝐱). Meanwhile, the regularization term P_θ_t(𝐲_vic|𝐱)/P_θ_t(𝐲_t-1^-|𝐱) will guide the models to maximize the ratio between
𝐲_vic and 𝐲_t-1^-. As the “standard response” to be learned, 𝐲_vic can be viewed sufficiently as a positive example. Therefore, we can derive that the optimization target of LoRD is consistent with RLHF's optimization, i.e., both encourage local models to maximize the probability proportion between positive and negative samples.

Similar to Equation <ref> in which the optimized model can be seen as the distortion of the original model P_θ_init, in LoRD the optimized model can be regarded as the distortion of the local model P_θ_0, with P_θ_t(·|𝐱)=𝐫_i,j^tP_θ_t-1(·|𝐱) at each step t, where the distortion term 𝐫_i,j^t is intended to jointly maximize P_θ_t(𝐲_t-1^+|𝐱)/P_θ_t(𝐲_t-1^-|𝐱) and P_θ_t(𝐲_vic|𝐱)/P_θ_t(𝐲_t-1^-|𝐱), while leaving the probabilities in other dimensions unconstrained directly.


Consistency Analysis of Four Objective Functions. Based on the analysis of the above four objective functions, we illustrate their convergence procedure, as exhibited in Figure <ref>. We reach the following proposition. 



The learning procedure for LLMs' alignments is consistent with the stealing procedure of LoRD, i.e., they both attempt to maximize the difference between the probabilities of positive and negative samples. Conversely, they are inconsistent with either MLE or KD. In MLE, the objective is maximizing the label probability, while KD aims to minimize the distance among all dimensions.



Albeit the inconsistency in their training procedures, the following theorem demonstrates that with enough samples, all these methods will reach the same distribution results. 


  Ideally, for any loss value of Equations <ref>, <ref>, <ref>, <ref>, or <ref> converging to 0, we have 𝐲^+≡𝐲_vic. Meanwhile, the local model's distribution P_θ(· |𝐱) will approach that of the victim model P_θ_vic(· |𝐱) on MEAs from all three discussed MEA methods, including LoRD, MLE, and KD.


Theorem <ref> ensures that the local model will converge to the victim model regardless of the choice of MEA methods. Nonetheless, by Proposition <ref>, LoRD still has two benefits: query time reduction (i.e., size of the query set 𝒟_q), and the watermark resistance of the learned local model. We will elaborate on them in Section <ref>.



 §.§ Comparative Analysis on Model Stealing


Query Efficiency. Let N_Q and N_R denote the sequence lengths of the query text and the response text, respectively. For MLE, the ideal query numbers to populate the entire text space are given by 𝒪(V^N_Q· V^N_R), where V represents the size of the vocabulary. In contrast, the ideal query times for LoRD can be reduced to 𝒪(V^N_Q· C), with C being a constant. This constant suggests that the model requires only a fixed and limited number per query, as LoRD itself can automatically generate and explore V^N_R possibilities during training.















 
 
 
 

 
 






Based on the above analysis, a straightforward concern with employing MLE in LLMs' extraction is that, given the limited query times in real-world practices, it may suffer from incomplete learning, especially for text generation tasks. Consequently, the local model may tend to memorize some specific responses instead of achieving a broad understanding and generation. We call such a phenomenon preference overfitting (PO), which indicates that the local model is only effective on a limited set of explored samples, and yet does not generalize well to unseen scenarios. In such cases, the local model usually exhibits a more “rugged” decision surface, which appears to overfit the preference sentences in 𝒟_tr, as shown in Figure <ref> (b). Figure <ref> provides the empirical visualization of this analysis.

Watermark Resistance.
Another limitation of prevalent objective functions, such as MLE
and KD, is their susceptibility to watermarks <cit.> of
output contents, i.e., while stealing knowledge from LLMs via
responses 𝐲_vic, watermarks within them will also been
passively inherited by the local model. Consequently, the generated sentences of the local model may possess some residual of watermarks, which might be detected as evidence of stealing.

LoRD can mitigate the influences of watermarks
naturally, as it does not learn the likelihood of victim models' responses 𝐲_vic∼𝒟_tr directly, but relies on
𝐲_vic to determine positive and negative labels
from responses generated by the local model. 

As depicted in Equation <ref>, LoRD guides the
local model to learn the likelihood of 𝐲_t-1^+ instead
of 𝐲_vic, which means that it will not been influenced
by watermarks contained in 𝐲_vic explicitly. However, the
regularization term ℒ_reg, as well as the replacement
𝐲_t-1^+←𝐲_vic for a cold start, will indeed
introduce watermarks from 𝐲_vic. To address this, we can
reshape Equation <ref> into a convex combination of the objective function and the regularization, i.e.,

    ℒ=   𝔼 [(1-λ_1)·
    (logP_θ_t(𝐲_t-1^+|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱))
         +λ_1· clip(logP_θ_t(𝐲_vic|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱)),

where 0≤λ_1≤ 1 is the hyperparameter.

When λ_1 is small, the convergence of LoRD will
substantially focus on maximizing
P_θ_t(𝐲_t-1^+|𝐱)/P_θ_t(𝐲_t-1^-|𝐱),
with which the local model will exhibit a strong watermark resistance
ability.
When λ_1 increases, LoRD will tend to rely more on the guidance of 𝐲_vic, resulting in a higher risk of introducing watermarks. In the case of λ_1=1, the local model will converge to the victim model without any exploration and watermark resistance, which might suffer from the defense by watermarks.

From a global perspective, ℒ_obj represents the
exploration and the locality learning ability of LoRD, 
which can mitigate the influences of watermarks. On the other hand, ℒ_reg ensures the stability of the training
procedure. Therefore, Equation <ref> characterizes a
trade-off via λ_1 between the stability and the diversity
during stealing, and Equation <ref> can be seen as a special case of Equation <ref> with λ_1=0.5.









§ EXPERIMENTS




























 §.§ Settings


Datasets. We evaluate MEAs on five mainstream natural language generation (NLG) tasks, including machine translation, text summarization, question answering, structured text generation, and  data-to-text. We select ten representative datasets: WMT16 <cit.>, TLDR <cit.>, CNN Daily Mail <cit.>, Samsum <cit.>, WikiSQL <cit.>, Spider <cit.>, E2E-NLG <cit.>, CommonGen <cit.>, PIQA <cit.>, and TruthfulQA <cit.> as benchmarks for our domain-specific evaluation. These datasets cover most of the downstream tasks in natural language generation.
We compare not only the stealing efficacy of different MEA methods, but also the stealing difficulty across different downstream tasks. Table <ref> lists all datasets and backbones used in the paper.

Baselines. As described in Section <ref> and <ref>, we compare LoRD with two types of model extraction methods: maximum likelihood estimation (MLE) and knowledge distillation (KD). For MLE and LoRD, we conduct MEAs under pure black-box attack settings. For KD, the predicted distributions are used specifically under grey-box settings.

Metrics.
For text generation tasks, we evaluate extracted models with a semantic-level and two lexical-level metrics, BERTScore <cit.>, BLEU <cit.>, and Rouge-L <cit.>, all of which are commonly used in the NLG evaluation. Regarding reasoning tasks (e.g. QA), we use Precision, Recall, Accuracy, and F1 score as their evaluation metrics.

Implementation Details.
We use Llama3-8B as the local model to learn the outputs generated by victim models.
We set sequence length varying 128 to 4096 depending on the selected tasks, and learning rate 3× 10^-5. Our experiments run on 2 × 80GB Nvidia Tesla A100. We execute each training five times and record the mean values and standard variances in the following sections. For LoRD, we set τ_1 and τ_2 to 0.8 and -0.1, respectively. Besides, we set the period number N_t to 512, and use λ_1=0.5 as the default formation of the loss function.
The victim model's response, 𝐲_vic, is generated by token sampling with a temperature of 1, the default setting for OpenAI APIs. The local model also uses token sampling, but with a temperature of 0.8 and Top-P probability clipping <cit.> at 0.98. We use this setting to enhance the stability of generation in local models. Note that we have not incorporated sampling strategies with their corresponding hyperparameters into the design of LoRD. We believe that MEAs considering sampling strategies could inspire more powerful MEA methods, and we leave these improvements for future work.












 §.§ Stealing Domain-Specific Knowledge


We first select GPT-3.5-turbo, a checkpoint of ChatGPT, as the basic victim
model. This is because its API provides probabilities of
candidate words when generating responses. We employ Llama3-8B <cit.>, a small LLM
with only a 4.5% fraction of parameters than the victim model as our
initial local model. Though this LaViSH setting contradicts previous
assumptions <cit.> in MEA that the copy model should usually be
“wider” or “larger” than the victim model to contain its
knowledge, we believe this setting is more applicable in real-world
scenarios <cit.>. Besides, the number of query times selected in this section is
less than 100, a significant degradation compared to previous
studies <cit.>. This is because, in our experiments, copy models
can easily learn the knowledge with a few training samples
and then exhibit only slight improvements afterward. More discussions on
query times can be found in Section <ref>.


Fidelity and limits on stealing.
We first examine the fidelity and limits of a small LLM to steal commercial
LLMs. As shown in Table <ref>, we list the performance
of the victim model and the local model on three tasks, and provide
two MEA methods, local model fine-tuned with MLE (+MLE) and LoRD
(+LoRD), respectively.
In table <ref>, cells highlighted in red indicate poorer outcomes compared to the victim model, whereas blue signifies results that are on par or potentially superior to the victim model. The intensity of the red or blue color corresponds to the degree of underperformance or outperformance relative to the victim model.

We can see that the original performance of the
local model is significantly lower than the victim model, i.e., with a 50%
decrease in BLEU-4 or 10∼ 25 decrease in Rouge-L. Once
we employ MEAs in the local model, its performance rapidly boosts to
nearly the same as the victim model, with
0∼ 40% points of gaps in BERTScore. These gaps are negligible (e.g. <1% in summarization) in
some tasks, but remain eminent in other tasks such
as reasoning, structured text generation, and machine
translation. This phenomenon indicates that domain-specific model extractions can
effectively learn domain-specific abilities from victim models but may perform poorly if downstream tasks require extra knowledge, such as machine translation and QA.




Comparison among stealing methods. Tables
<ref>, <ref>, and <ref> compare the stealing efficacy between MLE and our
LoRD. The results consistently show that
LoRD significantly outperforms MLE under the same MEA
settings. Besides, for challenging tasks such as reasoning
and translation, LoRD exhibits much higher improvements, which
demonstrates that it can address the preference overfitting problem
discussed in Section <ref> and do enable the local model to learn
the task ability from victim models. However, we also observe that for some
tasks (e.g. summarization), LoRD shows no statistical difference from
MLE, probably because these tasks are relatively simple, where merely MLE has
already achieved comparable results to victim models.

Tasks difficulties comparison. Based on previous analysis, we
observe that the performance and limitations of MEA depend on
the category of tasks. Additionally, sometimes datasets in the same task exhibit
significant differences in stealing. We
put forward two metrics to measure task difficulties: the fidelity that measures
extraction efficacy compared to victim models, and the
performance-up, which assesses the performance gain before
and after stealing for a given local model.
Formally, given a test
set 𝒟_te={(𝐱,𝐲)} and a corresponding
metric ℳ(hypothesis,reference), the fidelity (F) and performance-up (P) of the
local model θ_N_t can be defined as:

    F=∑_𝐱,𝐲∈𝒟_teℳ(𝐲_N_t,𝐲)/∑_𝐱,𝐲∈𝒟_teℳ(𝐲_vic,𝐲),  P=∑_𝐱,𝐲∈𝒟_teℳ(𝐲_N_t,𝐲)/∑_𝐱,𝐲∈𝒟_teℳ(𝐲_0,𝐲),

where 𝐲_N_t∼ P_θ_N_t(· |𝐱),
𝐲_0∼ P_θ_0(· |𝐱), and
𝐲_vic∼ P_θ_vic(· |𝐱) denote the
sampled responses from the trained local model (θ_N_t), the initial
local model (θ_0), and the victim model (θ_vic),
respectively. In Figure <ref>, we illustrate a “spectrum” of extracting
various downstream tasks based on these two metrics defined in Equation
<ref>. The figure can assist in recognizing and
defending commercial LLM's knowledge.

From Figure <ref>, we observe five tasks
forming the following three scenario groups and datasets coming from
the same tasks are mostly in the same group:



  * High fidelity and high performance-up (HFHP): These
  tasks are challenging for a pre-trained model but can be effectively
  learned with the guidance of victim models. This group includes two tasks: data-to-text and structured text generation.


  * High fidelity but low performance-up (HFLP): The
  initial local model already achieves a comparable performance to
  the victim model. QAs and summarization are in this group.


  * Low fidelity but high performance-up (LFHP): While  MEAs significantly improve the local model's performance, gaps between the local and victim models remain difficult to bridge with domain-specific extraction alone. Machine translation is a representative task whose reasons are explained in Section <ref>.











§ EMPIRICAL ANALYSIS



 §.§ Resistance to Watermarks

Current LLM watermarking methods have been shown <cit.> to
be robust against commonly used erasing strategies (e.g., rephrasing),
making watermark removal a distinct challenge. In this section, we
validate the inherent resistance of LoRD to watermarks, suggesting
that LoRD is preliminarily resistant to text watermarking.
As described in Section <ref>, we highlight that
LoRD can extract the victim models' knowledge with two terms: the
straightforward likelihood learning term
logP_θ_t(𝐲_vic|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱)
and the exploration term
logP_θ_t(𝐲_t-1^+|𝐱)-logP_θ_t(𝐲_t-1^-|𝐱),
where we can tune the hyperparameter λ_1 as shown in
Equation <ref> to trade off the exploration and the
convergence speed. Typically, a lower λ_1 encourages the
model for conducting a slower but more diverse and localized exploration
from its own generated text 𝐲_t-1^+, potentially enhancing watermark resistance. In this subsection, we evaluate this analysis
empirically.

Watermarking Details.
Unlike previous experimental settings in
Section <ref>, here we cannot utilize commercial LLMs as
victim models due to the inability to control token sampling
inside LLMs. Instead, we employ Llama3-70B as the victim model and
watermark its outputs based on “green” tokens selection. Following
prior research <cit.>, we separate the predicted
vocabulary into a green word set and a red word set
, assigning them randomly with the seed derived from the hash of
generated tokens at the last generation step. Subsequently, we sample the next token
exclusively from the green set, determined by a certain probability.

In this way, given the hypothesis H_0 that texts are generated
  without the knowledge of the green word set, we can estimate the
probability H_0 occurs (P-value) and the Z-score of it for these texts. A high P-value, among with a low Z-score, indicates stronger watermark resistance for MEA algorithms.

Result Analysis.
As depicted in Figure <ref>, we
evaluate the watermark resistance for both MLE and LoRD, and
demonstrate how LoRD's performance varies with different values of
λ_1. The Z-score of
LoRD witnesses a consistent increase as λ1 arises,
indicating that the “confidence” in rejecting the hypothesis, i.e.,
the risk to be suspected, arises when λ_1 increases. This
finding coincides with the analysis in Section <ref>. Besides,
we observe that the P-values of LoRD are generally higher than those of MLE when λ_1 is below 0.8, indicating that LoRD typically exhibits stronger watermarking resistance
than MLE in most situations. It is noteworthy that this enhanced
resistance seems not a “tax” of MEAs efficacy, as the
Rouge-L (F1) scores of LoRD consistently surpass those of MLE and do not exhibit
a significant negative correlation with their P-values.



 §.§ Scaling Laws of LLMs

We then explore essential capacities to steal domain-specific
knowledge from LLMs. We first analyze the influence of
query times for the adversary, then compare the efficacy when utilizing
different sizes of the local model, and finally compare the
fidelity among different victim and local models.



  §.§.§ Query Times

We first investigate the influence of query numbers on
MEAs. Specifically, we sample query examples randomly from the query
dataset, starting from 4, and incrementally increase it until the
performance of the learned model stabilizes. Figure
<ref> illustrates the stealing efficacy of LoRD and MLE
on PiQA.

We observe that the scores of MLE and LoRD
consistently increase as the query number rises, showing that a
larger query number can improve stealing efficacy steadily until
reaching their empirical upper bounds. Additionally, LoRD typically obtains a
higher score than MLE with the same number of queries, and reaches
bottlenecks earlier, which can reduce the required query numbers by
87% compared to MLE. Moreover, in Figure <ref>, the
performance of LoRD exhibits a relatively lower standard variance than
MLE, indicating a more stable training procedure.





  §.§.§ Scales of Local Models

As discussed in Section <ref>, we assume the adversary is stealing existing commercial LLMs with a small local model. This raises the question of selecting an appropriate interval of the local model's size. To address this concern, we illustrate the correlation between the local model's size and extraction efficacy on two machine translation tasks, Russian-to-English (ru-en) and German-to-English (de-en), as shown in Figure <ref>. Here, we employ seven OPT models <cit.> as local models, with parameters ranging from 125 million to 30 billion, to minimize the interruptions of factors other than model size.

Figure <ref> shows a sharp distinction between two machine translation tasks. In the de-en task, the performance of the local model increases steadily with model size, while this trend is not evident in the ru-en task with model size smaller than 30 billion. Nevertheless, the performance of a 30 billion parameter learned local model in ru-en cannot even be comparable to that of a 1.3 billion parameter local model in the de-en task. This phenomenon suggests that for tasks requiring commonsense knowledge, such as machine translation, the local model should at least possess foundational knowledge of the task (e.g., pre-trained on Russian texts) to learn from victim models effectively. Besides, experiments in BERTScore (F1) show that sometimes LoRD may underperform MLE when the local model has fewer than 1 billion parameters, demonstrating that it is challenging to bootstrap LoRD's exploration with a very small local model. By summarizing the increase in LoRD's curves, a model with 2.7 billion appears sufficient to steal domain-specific knowledge from commercial LLMs.



  §.§.§ Fidelity under Different Victim and Local Models

We then 
evaluate the fidelity of extracting different victim models using
various pre-trained local models. Specifically, we select GPT-3.5,
GPT-4, and GPT-4o as victim models, and employ five state-of-the-art
open-source models, Phi-3 (3.8B), OPT (6.7B), Qwen-2 (7B), Mistral-V3
(7B), and Llama-3 (8B), as local models, as shown in Figure <ref>.

Horizontally, while GPT-4 exhibits a consistently lower extracted
fidelity compared to the other two victim models, vulnerabilities of
the three victim models are generally similar. Vertically, fidelity
of different local models can be significantly impacted by their
performance. For instance, OPT (6.7B) shows a noticeably lower score
compared to the other four models, which indicates that the initial
performance of the local model will affect the performance of
MEAs. Besides, Phi-3 (3.8B) achieves a comparable
fidelity to larger models like Llama-3 (8B), demonstrating that the size
of a local model does not influence final fidelity in
domain-specific stealing after 2.7 billion, which corroborates the
observation in Section <ref>.










































§ POTENTIAL DEFENSES


Query Detection. One approach to effectively prevent the
attack of LoRD is by detecting the distribution of query
texts. This is because LoRD, similar to current MEA algorithms, makes
no improvements to query samples, indicating that it can be
detected by analyzing the statistical information of the adversary's
queries, such as the number of queries, distribution of query
contents, and so on.
However, this defense is usually resource-consuming, as it requires
the LLM provider to store all query texts of each user. Besides, the
potential for false positives could adversely affect the
user experience.

More Powerful Watermarks. While we highlight the watermark
resistance of LoRD, watermarking remains one of the most effective
solutions to mitigate MEAs. For example, some model-level watermarks,
such as backdoor-based watermarking <cit.>, can effectively certify
the theft of DNNs. However, model-level watermarking on LLMs remains preliminary. Besides, this technique might not work when the
adversary only steals a subset of knowledge in
which no backdoor is embedded.



§ CONCLUSION


In this paper, we have focused on the extraction problem of commercial
large language models. We proposed LoRD, a practical and realistic
extraction algorithm which is consistent with the alignment procedure
of large language models. Our analysis proved that LoRD can reduce the
query time significantly and mitigate the certification of current
watermarks naturally, surpassing existing MEA algorithms' capabilities. Extensive experiments on domain-specific
stealing demonstrates the superiority of our method.

plain














§ VISUALIZATION OF DISTRIBUTIONS

We also investigate the probability distributions in the generation
procedure among different extraction methods. Specifically, we
visualize these distributions for four models, the victim model
(GPT-3.5-turbo), the initial local model (llama3-8B), and the
learned local models with MLE and LoRD. As plotted in Figure
<ref>, each row in the subfigures refers to the
distribution when generating the i-th token, with each column
element indicating the probability predicted for the
corresponding token index.
We limit the visualization to no more than five token probabilities as currently only GPT-3.5-turbo provides the token prediction probabilities during generation, with a maximum of 5 candidate tokens  <cit.>.

From Figure <ref>, we can see that both MLE and LoRD successfully redistribute the generation of the initial local model into a distribution similar to the victim model's, where probabilities, especially Top-1 tokens, have been well inherited in the extraction. This phenomenon supports our analysis in Theorem <ref>. However, distributions of MLE extracted models are consistently sharper than LoRD's, which aligns with our analysis in Section <ref>, where we claim that MLE leads local models to overfit to the preferred sentences (i.e., Top-1 tokens), namely PO, and thus to disrupt the original distributions, leveraging unusual low probabilities for other token indexes. The reason why LoRD can be resistant watermarks, i.e., tokens in Top-1, can also be derived from this discovery.

To compare MLE and LoRD accurately, we quantize the entropy of these distributions, and compute the KL divergence (𝔻_KL), and the Spearman
  Correlation (Spear. Corr.) with respect to the victim and initial local
model. As shown in Table <ref>, while the MLE extracted model
exhibits a lower KL divergence (i.e., high distribution similarity) with
the victim model than LoRD's on the training dataset,
its KL divergence becomes comparable to LoRD's on the test
set. Meanwhile, its Spearman correlation significantly
decreases from 0.78 to 0.27, which shows that MLE cannot effectively
imitate prediction behaviors of the victim model when encountering data
beyond the training dataset.






§ LIMITATIONS AND FUTURE WORKS

MEAs on Multi-modal Models. While this paper delves into MEAs
for large language models, it acknowledges the oversight of the multi-modal attribution of current commercial models <cit.> that integrate various forms of data such as text, images,
voice, and so on. The challenge of extending MEA algorithms to
accommodate these models, which requires extra considerations on the
unified representation of concepts, remains unexplored. Future work
could focus on developing MEA methodologies sensitive to multi-modal data nuances.

Capacities beyond LaViSH Settings. We utilize the LaViSH
setting to describe the model capacity of adversaries in our threat
model (see Section <ref>). However, sometimes, the adversary
might possess comparable or superior training resources to the
victims. Though this paper posits that our MEA algorithms and
theoretical analysis are still compatible with such conditions, we
concede that concrete experimental validation and results beyond
LaViSH settings are not presented here.

Lower-level Extractions.
This study evaluates MEAs at the performance level, i.e., it measures the
extraction effectiveness simply through task performance
metrics, or the similarity of learned distributions to the victim model.
This setting is justified, as performance metrics are
essential for evaluating task-related knowledge and the practical
application of LLMs. However, it does not consider the lower-level
similarities between the victim and local models.
Can we achieve neuron-level alignments in LLM's MEAs? How does a LaViSH
setting hurt LLM's MEAs? Is it compatible to extract a MoE
(Mix-of-the-Expert) <cit.> victim model with a dense local model? These
questions are not addressed in this research.












§ PROOFS OF THEOREMS





 §.§ The Deduction of Equation <ref>


From Equation <ref>, we can get that


    max_θ∑_𝐱∼𝒟_qR_θ_ϕ(𝐱,𝐲̂)-β𝔻_KL[P_θ(𝐲|𝐱)||P_θ_init(𝐲|𝐱)]
       ⇒ max_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)R_θ_ϕ(𝐱,𝐲)-β [
      logP_θ(𝐲|𝐱)-logP_θ_init(𝐲|𝐱)]
       ⇒ min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)-1/βR_θ_ϕ(𝐱,𝐲)+
      logP_θ(𝐲|𝐱)/P_θ_init(𝐲|𝐱)
       ⇒ min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·  |𝐱)-log(exp(1/βR_θ_ϕ(𝐱,𝐲))) +
      logP_θ(𝐲|𝐱)/P_θ_init(𝐲|𝐱)
       ⇒ min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)logP_θ(𝐲|𝐱)/exp(1/βR_θ_ϕ(𝐱,𝐲))·
      P_θ_init(𝐲|𝐱).


If we define a partition function Z(𝐱) with the formation of 


    Z(𝐱)=∑_𝐲P_init(𝐲|𝐱)exp(1/βR_θ_ϕ(𝐱,𝐲)),

we can reformat the optimization target as


    min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)logP_θ(𝐲|𝐱)/exp(1/βR_θ_ϕ(𝐱,𝐲))·
      P_θ_init(𝐲|𝐱)
       ⇒ min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)logZ(𝐱)· P_θ(𝐲|𝐱)/exp(1/βR_θ_ϕ(𝐱,𝐲))·
      P_θ_init(𝐲|𝐱)
                            -logZ(𝐱).


If we mark
1/Z(𝐱)exp(1/βR_θ_ϕ(𝐱,𝐲))·
P_θ_init(𝐲|𝐱) as
P_θ*(𝐲|𝐱), then we have


    min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)logZ(𝐱)· P_θ(𝐲|𝐱)/exp(1/βR_θ_ϕ(𝐱,𝐲))·
      P_θ_init(𝐲|𝐱)-logZ(𝐱)
       ⇒ min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)log
    P_θ(𝐲|𝐱)/P_θ*(𝐲|𝐱)-logZ(𝐱).


Because Z(𝐱) is independent to 𝐲, we can deduct that


    min_θ∑_𝐱∼𝒟_q∑_𝐲∼ P_θ(·
      |𝐱)log
    P_θ(𝐲|𝐱)/P_θ*(𝐲|𝐱)-logZ(𝐱)
       ⇒ min_θ∑_𝐱∼𝒟_q[∑_𝐲∼ P_θ(·
      |𝐱)log
    P_θ(𝐲|𝐱)/P_θ*(𝐲|𝐱)]-logZ(𝐱)
       ⇒ min_θ∑_𝐱∼𝒟_q𝔻_KL[P_θ(𝐲|𝐱)||P_θ*(𝐲|𝐱)]-logZ(𝐱).


As we know that Z(𝐱) does not contain θ, the
above optimization target actually minimizes the KL-divergence
between the distribution of P_θ and P_θ*,
demonstrating that θ* is the optimal value of θ that satisfies


    P_θ*(𝐲|𝐱)=1/Z(𝐱)exp(1/βR_θ_ϕ(𝐱,𝐲))·
    P_θ_init(𝐲|𝐱).


Based on equation <ref>, we can see that the optimal distribution
of θ is built upon P_θ_init with a distortion, as we discussed in Section <ref>.





 §.§ Proofs of Theorem <ref>


Guarantee of MLE. From Equation <ref>
we can obtain that when ℒ_ce decreases to 0, the KL divergence between
P_θ(·|𝐱) and P_θ_vic(·|𝐱)
decreases to 0, indicating that
P_θ(·|𝐱) equals to P_θ_vic(·|𝐱).

Guarantee of KD. As we know,
𝔻_KL(p,q)≥ 0 ∀ p and q. Therefore,
if ℒ_kd shown in Equation <ref> equals to 0, then both
𝔻_KL[P_θ(·|𝐱)||P_θ_vic(·|𝐱)]
and
𝔻_KL[SM(P_θ(·|𝐱)/T)||SM(P_θ_vic(·|𝐱)/T)]
equal to 0. For the latter one, we have


    𝔻_KL[SM(P_θ_vic(·|𝐱)/T)||SM(P_θ(·|𝐱)/T)]
       =𝔼_𝐲∼
      P_θ_vic(·|𝐱)𝔼_y∈𝐲
       logexp(P_θ(y|𝐱,𝐲_p)/T)/∑_y'∈𝐲exp(P_θ_vic(y|𝐱,𝐲_p)/T)/exp(P_θ_vic(y|𝐱,𝐲_p)/T)/∑_y'∈𝐲exp(P_θ(y|𝐱,𝐲_p)/T)
       =𝔼_𝐲∼
      P_θ_vic(·|𝐱)𝔼_y∈𝐲
       logexp((P_θ(y|𝐱,𝐲_p)-P_θ_vic(y|𝐱,𝐲_p))/T)/∑_y'∈𝐲exp(P_θ_vic(y|𝐱,𝐲_p)/T)/∑_y'∈𝐲exp(P_θ(y|𝐱,𝐲_p)/T),

where we can observe that only when P_θ(·|𝐱) equals to
P_θ_vic(·|𝐱) can this term reduce to
0. Integrating the analysis of these two terms, we can obtain that
ℒ_kd=0 represents the local model's distribution
converge to that of the victim model.

Guarantee of LoRD.














When ℒ shown in Equation <ref> equals to 0, the proportion of
P_θ_t(𝐲_vic|𝐱)/P_θ_t(𝐲_t-1^-|𝐱)
and
P_θ_t(𝐲_t-1^+|𝐱)/P_θ_t(𝐲_t-1^-|𝐱)
should limit to -∞. As we know that i) in a distribution
∑P_θ_t(·|𝐱)=1 and ii) 𝐲^+_t-1
is a dynamic positive response generated at each period, we can deduct
that when ℒ=0 there must be
𝐲_vic=𝐲_t-1^+, i.e.,
P_θ_t(𝐲_vic|𝐱)=P_θ_t(𝐲_t-1^+|𝐱)=1
and P_θ_t(𝐲_t-1^-|𝐱)=0. Note that
this is merely a theoretical limit that cannot be reached, because
𝐲_t-1^- will not be sampled if its probability is 0,
and 𝐲_t-1^+ usually doesn't exhibit a significant distinction
to 𝐲_t-1^- when sampling.













