Isolation and Induction: Training Robust Deep Neural Networks against
  Model Stealing Attacks
Despite the broad application of Machine Learning models as a Service
(MLaaS), they are vulnerable to model stealing attacks. These attacks can
replicate the model functionality by using the black-box query process without
any prior knowledge of the target victim model. Existing stealing defenses add
deceptive perturbations to the victim's posterior probabilities to mislead the
attackers. However, these defenses are now suffering problems of high inference
computational overheads and unfavorable trade-offs between benign accuracy and
stealing robustness, which challenges the feasibility of deployed models in
practice. To address the problems, this paper proposes Isolation and Induction
(InI), a novel and effective training framework for model stealing defenses.
Instead of deploying auxiliary defense modules that introduce redundant
inference time, InI directly trains a defensive model by isolating the
adversary's training gradient from the expected gradient, which can effectively
reduce the inference computational cost. In contrast to adding perturbations
over model predictions that harm the benign accuracy, we train models to
produce uninformative outputs against stealing queries, which can induce the
adversary to extract little useful knowledge from victim models with minimal
impact on the benign performance. Extensive experiments on several visual
classification datasets (e.g., MNIST and CIFAR10) demonstrate the superior
robustness (up to 48% reduction on stealing accuracy) and speed (up to 25.4x
faster) of our InI over other state-of-the-art methods. Our codes can be found
in https://github.com/DIG-Beihang/InI-Model-Stealing-Defense.
Jun Guo, Aishan Liu, Xingyu Zheng, Siyuan Liang, Yisong Xiao, Yichao Wu, Xianglong Liu
# Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks

###### Abstract.

Despite the broad application of Machine Learning models as a Service (MLaaS), they are vulnerable to model stealing attacks. These attacks can replicate the model functionality by using the black-box query process without any prior knowledge of the target victim model. Existing stealing defenses add deceptive perturbations to the victim's posterior probabilities to mislead the attackers. However, these defenses are now suffering problems of high inference computational overheads and unfavorable trade-offs between benign accuracy and stealing robustness, which challenges the feasibility of deployed models in practice. To address the problems, this paper proposes _Isolation and Induction_ (Inl), a novel and effective training framework for model stealing defenses. Instead of deploying auxiliary defense modules that introduce redundant inference time, Inl directly trains a defensive model by isolating the adversary's training gradient from the expected gradient, which can effectively reduce the inference computational cost. In contrast to adding perturbations over model predictions that harm the benign accuracy, we train models to produce uninformative outputs against stealing queries, which can induce the adversary to extract little useful knowledge from victim models with minimal impact on the benign performance. Extensive experiments on several visual classification datasets (_e.g._, MNIST and CIFAR10) demonstrate the superior robustness (up to 48% reduction on stealing accuracy) and speed (up to 25.4\(\times\) faster) of our Inl over other state-of-the-art methods. Our codes can be found in [https://github.com/DIG-Beihang/Inl-Model-Stealing-Defense](https://github.com/DIG-Beihang/Inl-Model-Stealing-Defense).

Model Stealing, Stealing Defense, Model Privacy +
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote † †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote †: The corresponding author.

+
Footnote † †: The corresponding author.

+
Footnote † †: The corresponding author.

+
Footnote † †: The corresponding author.

+
Footnote † †: The corresponding author.

+
Footnote † † †: The corresponding author.

+
Footnote
To mitigate the threat of model stealing attacks, several defensive methods (Kririyappa et al., 2017; Dosov et al., 2018; Li et al., 2019; Li et al., 2019) have been devoted to making the victim model hard to steal by introducing perturbations or randomness to the model output. However, the practical feasibility of these defenses is still hampered by certain limitations: (1) Existing defensive methods often incorporate auxiliary modules that validate the input or modify the output of the victim model, which introduces extra computational costs. In practice, the computational burdens are mainly concentrated on the inference phase, and higher computational overheads indicate extended user response time and increased financial expenditures. (2) Some defenses add perturbations to the model's output to enhance its resilience against model stealing attacks by providing erroneous predictions to attackers. However, this comes at the expense of reduced benign accuracy for legitimate users due to the unfavorable trade-off.

To tackle these concerns, we propose a novel and effective defensive training framework against model stealing attacks, which is called InI. As for the _computational overheads_, distinct from prevailing methodologies that introduce auxiliary inference-time modules, our InI aims to directly train a robust model that is able to defend against stealing attacks without extra inference modules. Based on the fact that DNNs are heavily over-parameterized (Zhu et al., 2017) and can be trained to fit and generalize across diverse data distributions, we, therefore, posit that the victim model can achieve robustness by incorporating the countermeasures within its parameters during training. InI leverages a clone model during training as the surrogate adversary and estimates the adversary's optimization gradient and the expected gradient. With this estimation, the victim can learn to adjust its posterior probabilities to maximize the directional divergence between the two gradients. Therefore, we can isolate the clone model's optimization gradient from the expected gradient. To ameliorate the _trade-off between benign accuracy and stealing robustness_, different from previous studies that add perturbations over all predictions that harm the benign accuracy, we aim to train models that can learn to behave differently on benign and malicious queries. Following the assumption of previous works (Li et al., 2019; Li et al., 2019) that the malicious query samples deviate from the task distribution, InI trains a victim that behaves normally on the benign task yet produces inductive outputs on the malicious samples. Specifically, we introduce an out-of-distribution (OOD) dataset during victim training and minimize the adversary's information gain on it. As a result, during inference, the adversary is induced to extract little useful knowledge from the victim model using the stealing query with OOD examples. In addition, our method can be integrated with existing methods to better obtain defensive performance.

In summary, our **main contributions** are three-fold:

* We propose a novel and effective defensive training framework against model stealing attacks called InI to achieve robustness during training, which provides a new perspective of model stealing defenses.
* For computational overheads, InI incorporate the gradient isolation countermeasure within the victim's parameters; for unfavorable trade-offs, InI produces distinct outputs and induce the adversary to acquire minimal knowledge from malicious queries.
* Extensive experiments have been conducted on multiple datasets which demonstrate the state-of-the-art robustness and speed over other baselines. Moreover, InI shows flexible compatibility with existing methods for better defense.

## 2. Related Work

### Model Stealing Attacks

Model stealing attack, also referred to as model extraction, aims at inferring hyper-parameters (Zhu et al., 2017; Li et al., 2019), extracting model parameters (Li et al., 2019; Li et al., 2019; Li et al., 2019), or copying functionalities of a certain machine learning model. Our work focuses on stealing the classification accuracy of the model, which is the most prevailing and universal stealing attack in deep learning. Tramer _et al._(Tramer et al., 2019) proposed the concept of model stealing that attackers could "steal" the property of a machine learning model by queries without prior knowledge of the victim model. Generally speaking, there are many properties that can be stolen, _e.g._, model parameters, training data, or functionality. Papernot _et al._(Papernot et al., 2019) proposed a partial data approach named Jacobian-Based Dataset Augmentation (JBDA), which generates synthetic data by adding small perturbations on a small set of in-distribution samples. Orekondy _et al._(Orekondy et al., 2019) proposed KnockoffNets, employing samples from a surrogate dataset as query inputs of the victim model. The stealing performance of partial data and surrogate data approaches would degrade when the available data are different from the original training set. In recent years, some data-free stealing methods have been proposed. Kariyappa _et al._(Kariyappa et al., 2017) and Truong _et al._(Truong et al., 2019) are motivated by the framework of data-free knowledge distillation (Zhu et al., 2017; Li et al., 2019) and proposed data-free model

Figure 1. Illustration of attacks and defenses in model stealing. The adversary makes queries using malicious queries to extract knowledge from the victim MLaaS model, and the returned outputs are used to train a clone model. The defender introduces randomness into the model’s outputs, in order to mislead the stealing algorithm.
stealing methods, where they use zeroth-order gradient estimation to calculate the victim gradient in black-box settings. Moreover, Sanyal _et al._[(48)] proposed a model stealing attack in the hard-label setting. They utilize some unrelated proxy data to get a pre-trained data generator, while the stealing process is data-free.

### Model Stealing Defenses

Currently, most model stealing defenses tend to add perturbations to the model outputs, thus disturbing the optimization of the adversary. Lee _et al._[(18)] proposed an accuracy-preserving defense against model stealing attacks by adding deceptive perturbations to the model outputs while preserving its top-1 label, while it yields to the hard-label stealing. Other defenses like Maximizing Angular Deviation (MAD) [(45)] perturbs the model outputs with controllable intensity, defending against model stealing attacks at the expense of benign accuracy. Another approach of defense takes advantage of the data limitation of adversaries, making the victim model produce dissimilar output between in-distribution inputs and out-of-distribution inputs. Kariyappa _et al._[(13)] proposed Adaptive Misinformation (AM) defense that detects the OOD inputs and misleads adversaries with modified outputs. Kariyappa _et al._[(12)] then proposed Ensemble of Diverse Models (EDM) defense, which introduces randomness into the model output by using an ensemble of diverse models. Models in the ensemble are trained to perform diversely for OOD inputs, making the functionality of the model hard to be stolen. In addition, there are other types of countermeasures towards model stealing, such as digital watermarking [(10; 1; 23)], which inject an extractable watermark into the victim model and can distinguish whether a model is from stealing.

Existing defensive approaches take advantage of some limitations of model stealing attacks to mitigate the knowledge leakage, while they are suffering from high computational costs and unfavorable trade-offs. In this paper, we are devoted to defending against stealing attacks by incorporating the countermeasure with the victim's parameters, inherently enhancing the robustness against model stealing attacks.

## 3. Threat Model

### Attack Objective

In this paper, we mainly discuss the functionality stealing towards DNNs on image classification tasks. Specifically, an adversary aims at stealing the functionality of a victim model \(\mathcal{V}\) by training a clone model \(\mathcal{C}\). These attacks usually follow the framework of knowledge distillation [(9)], where the victim model \(\mathcal{V}\) plays the role of "teacher", and the knowledge of the teacher is distilled into the "student" model \(\mathcal{C}\). The objective of the adversary is to maximize the classification accuracy of clone model \(Acc(\mathcal{C}(\mathcal{x};\theta_{\mathcal{C}}),y)\) on the victim's target distribution \(\mathcal{D}_{tar}\). Define \(\theta_{\mathcal{V}}\) to be the parameter of \(\mathcal{V}\) and \(\theta_{\mathcal{C}}\) to be the parameter of \(\mathcal{C}\), and the adversary's goal could be formulated as Eqn. 1.

\[\max_{\theta_{\mathcal{C}}}\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}_{tar}} \left[Acc(\mathcal{C}(\mathbf{x};\theta_{\mathcal{C}}),\mathbf{y})\right] \tag{1}\]

In most real-world settings, the adversary has no knowledge about the victim's structure, parameters, or training set. The only interaction between the adversary and the victim is the _black-box query process_: the adversary inputs an image \(x\) and the victim returns a softmax probability or logits. Though the original training set is unavailable, the adversary can use synthetic data or surrogate data to query the victim model. For example, JBDA [(46)] synthesizes data from a small part of in-distribution samples, and KnockoffNets [(44)] uses surrogate datasets to query the victim model. Therefore, the adversary's learning objective is a surrogate goal based on the distribution of the query dataset \(\mathcal{D}_{que}\), which can be formulated as follows:

\[\min_{\theta_{\mathcal{C}}}\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{que}}\left[d( \mathcal{C}(\mathbf{x};\theta_{\mathcal{C}}),\mathcal{V}(\mathbf{x};\theta_{\mathcal{ V}}))\right] \tag{2}\]

In the ideal scenario, if \(\mathcal{D}_{tar}\) and \(\mathcal{D}_{que}\) are close enough and the query budget is sufficient, the model stealing is generally inevitable since the victim \(\mathcal{V}\) must guarantee the performance for benign users on the target distribution. However, in practice, \(\mathcal{D}_{tar}\) and \(\mathcal{D}_{que}\) are dissimilar due to the knowledge limitation of the adversary, and the query budget is limited by the adversary's financial cost. It is these limitations that support existing defensive methods.

### Defense Objective

In defenses against model stealing, the defender aims at preventing the functionality of the victim model from being stolen with an acceptable impact on its benign accuracy. To be more practical, the accuracy degradation should be constrained with a minimum threshold \(T\). The objective of the defender is to minimize the classification accuracy of the clone model \(Acc(\mathcal{C}(\mathcal{x};\theta_{\mathcal{C}}),y)\) on victim's target distribution \(\mathcal{D}_{tar}\), which could be formulated as Eqn. 3.

\[\min_{\theta_{\mathcal{V}}}\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{ D}_{tar}}\left[Acc(\mathcal{C}(\mathbf{x};\theta_{\mathcal{C}}),\mathbf{y})\right],\] \[\text{s.t.}\ \ \mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}_{tar}}\left[ Acc(\mathcal{V}(\mathbf{x};\theta_{\mathcal{V}}),\mathbf{y})\right]\geq T \tag{3}\]

Considering the limitations of the adversary, existing defense methods either add adaptive perturbations to multiply the adversary's query cost, or differentiate the victim's behavior on ID and OOD data to deceive the adversary. In this paper, we propose a defensive method against model stealing attacks, which gets rid of the extra computational costs and ameliorates the trade-off between benign accuracy and stealing resistance.

## 4. Methodology

To build a defensive method with low computational costs and high trade-offs against model stealing attacks, we propose InI, a novel and effective defensive training framework. In this section, we first illustrate the training-time gradient isolation methodology that gets rid of auxiliary inference-time modules, and then elaborate on the adversary induction approach that reduces the knowledge leakage. Finally, we explain our overall training framework.

### Gradient Isolation

Existing defensive methods are suffering from extra computational costs during inference since they often employ auxiliary inference-time modules. Studies have revealed that DNNs are heavily over-parameterized [(62)] and can be trained to fit and generalize across diverse data distributions, _e.g._, adversarial examples for adversarially-trained models [(5; 6; 31; 32; 38; 51; 63)] and real-world disturbance
for reinforcement agents (Gutton et al., 2017; Gutton et al., 2018; Gutton et al., 2019; Gutton et al., 2020). Inspired by them, we propose a defensive training framework to directly train a robust model, so that the model can generate deceptive outputs towards stealing attack queries without extra modules.

Generally, as Eqn. 1 shows, the adversary's _expected goal_ is to minimize the disagreement with the ground truth on the target distribution. This objective is not directly related to the victim's parameter \(\mathbf{\theta}_{\mathcal{V}}\), but the adversary must extract knowledge from the victim. As a consequence, the _real objective_ of the adversary is illustrated in Eqn. 2. There exists a gap between the expected goal and the real objective, and the defense can be achieved by isolating the adversary's real objective from the expected goal. As the adversary usually updates its parameters by gradient descent, we propose gradient isolation to isolate the real gradient from the expected gradient, thereby incorporating the robustness within the victim's parameters to mislead the adversary.

To achieve gradient isolation, we need to estimate the above two gradient terms during training. Therefore, we introduce a surrogate white-box clone model \(\mathcal{C}\) into the victim's training process to represent the stealing role of the adversary. To isolate the update gradient from the expected gradient, we maximize the directional divergence between them. Specifically, for a certain batch of data \(\mathbf{x}\), assuming the target of the adversary is \(\mathbf{y}\), the update gradient can be written as:

\[\begin{split}\nabla_{\mathbf{\theta}_{\mathcal{C}}}CE(\mathcal{C}( \mathbf{x};\mathbf{\theta}_{\mathcal{C}}),\mathbf{y})&=-\nabla_{\mathbf{\theta} _{\mathcal{C}}}\sum_{i}y_{i}\log C(\mathbf{x};\mathbf{\theta}_{\mathcal{C}})_{i}\\ &=-\mathbf{y}^{T}G,\end{split} \tag{4}\]

where \(CE(\cdot,\cdot)\) is the soft cross-entropy loss commonly used in model stealing attacks (Sutton et al., 2019; Gutton et al., 2020), and \(G=\nabla_{\mathbf{\theta}_{\mathcal{C}}}\log C(\mathbf{x};\mathbf{\theta}_{\mathcal{C}})\) is a Jacobian matrix. When \(y\) comes from the ground truth, it represents the correct optimization direction; when \(y\) comes from the victim model (denoted by \(\mathbf{\tilde{y}}\)), it represents the actual optimization direction during stealing.

The goal of gradient isolation is to maximize the directional divergence of these gradients, which can be quantified by the cosine similarity. Therefore, the objective of gradient isolation can be written as follows:

\[\mathcal{L}_{\textit{iso}}=\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}_{\textit {tar}}}[CS(\mathbf{\tilde{y}}^{T}G,\mathbf{y}^{T}G)], \tag{5}\]

where \(CS(\cdot,\cdot)\) represents the cosine similarity. \(\mathbf{\tilde{y}}^{T}G\) and \(\mathbf{y}^{T}G\) can be calculated by the backward propagation. During the victim training, InI minimizes \(\mathcal{L}_{\textit{iso}}\) via optimization to perform gradient isolation, enhancing the victim's robustness against model stealing attacks.

### Adversary Induction

Some existing defenses add perturbations to the output over all samples, which harm the benign accuracy and cause low tradeoffs. To improve the trade-off between clean accuracy and stealing robustness, we further design an adversary induction approach to train victim models. Thus, the victim model will behave normally for benign users but produce inductive outputs that induce the adversary to optimize without learning too much useful knowledge.

Successfully inducing the adversary stands on the assumption that benign and malicious queries can be distinguished through the distribution (Gutton et al., 2018; Gutton et al., 2020). Practically, the adversary has limited knowledge of the distribution of the victim's training set \(\mathcal{D}_{\textit{tar}}\), and consequently uses a surrogate dataset \(\mathcal{D}_{\textit{quite}}\) to query and steal the victim model. Following this assumption, we regard the query samples of the adversary as out-of-distribution and apply an OOD dataset \(\mathcal{D}_{\textit{out}}\) during defense to substitute them. The over-parameterization property of DNNs ensures the generalization capability across diverse

Figure 2. The overall framework of InI. InI isolates the adversary’s gradients from the expected gradients during training to obtain faster inference speed, and induce the adversary to leak knowledge as little as possible.
distributions and thus enables the acquisition of the victim that exhibits divergent behavior on benign in-distribution (ID) queries and malicious out-of-distribution (OOD) queries. Thus, our InI aims to guarantee the victim's benign performance on ID queries, while inducing the adversary to attain little knowledge with uninformative outputs on OOD queries.

In particular, on ID samples, we should guarantee the victim's benign performance. This can be achieved by applying a cross-entropy loss to train the classification model, which is shown as follows:

\[\mathcal{L}_{ben}=\mathbb{E}_{(\mathbf{x},\mathbf{y})-\mathcal{D}_{lat}}[CE(\mathcal{ V}(\mathbf{x};\mathbf{\theta}_{\mathcal{V}}),\mathbf{y})]. \tag{6}\]

On OOD samples, we should induce the adversary to acquire minimal knowledge. Intuitively, the adversary induction can be realized by reducing the adversary's information gain on OOD queries. The information gain can be quantified by the KL divergence between the output probabilities of the clone and the victim model, which can be formulated as below:

\[\mathcal{L}_{ig}=\mathbb{E}_{\mathbf{x}-\mathcal{D}_{out}}[KL(\mathcal{C}(\mathbf{x}; \mathbf{\theta}_{\mathcal{C}}),\mathcal{V}(\mathbf{x};\mathbf{\theta}_{\mathcal{V}}))]. \tag{7}\]

Noted that \(\mathcal{L}_{ig}\) is the optimization objective of the adversary during model stealing. Therefore, when \(\mathcal{L}_{ig}\) is minimized before the stealing process, the adversary can only gain little information via optimization. Thus, the adversary would only extract little knowledge from the victim model. Since the adversary often uses gradient descent to update their parameters and attain knowledge, minimizing the first-order approximation of the KL divergence can also help to reduce the information gain. Consequently, we can minimizing the norm of \(\nabla_{\mathbf{\theta}_{\mathcal{C}}}\mathcal{L}_{ig}\), _i.e._, the gradient of the information gain. In summary, the objective of adversary induction can be formulated as:

\[\mathcal{L}_{ind}=\mathcal{L}_{ig}+\beta||\nabla_{\mathbf{\theta}_{\mathcal{C}}} \mathcal{L}_{ig}||. \tag{8}\]

\(\mathcal{L}_{ind}\) is a function related to \(\mathbf{\theta}_{\mathcal{V}}\), which can be integrated with our training framework. During the victim's training, we apply a surrogate white-box clone model \(\mathcal{C}\) and an OOD dataset to estimate the adversary's information gain and minimize \(\mathcal{L}_{ind}\) by updating the victim's parameter \(\mathbf{\theta}_{\mathcal{V}}\), thus reducing the knowledge leakage from the victim. The OOD dataset used in training comes from another classification task and is different from the query dataset in stealing attacks.

Though previous defenses (Han et al., 2017; Chen et al., 2018) utilize OOD datasets to train the victim, they only constrain the victim to produce meaningless or diverse outputs and did not take the adversary's role into consideration. In contrast to them, InI produces inductive outputs that trigger the adversary to learn less, reducing the knowledge leakage from the victim.

### Overall Framework

Figure 2 illustrates our overall framework. To train a robust victim \(\mathcal{V}\), we introduce a surrogate clone model \(\mathcal{C}\) into the training process. During training, the victim has white-box access to the surrogate clone model. To jettison the auxiliary modules for low computational costs, InI isolates the adversary's optimization gradient from the expected gradient during training via \(\mathcal{L}_{iso}\) in Eqn. 5. To further improve the trade-off between benign performance and stealing robustness, InI produces uninformative outputs on malicious queries through \(\mathcal{L}_{iso}\) in Eqn. 8 to induce the adversary to acquire less useful knowledge. By incorporating the robustness within the victim's parameters, the victim will acquire how to resist the model stealing attacks in a favorable trade-off without any auxiliary modules during inference.

During training, all objectives are simultaneously calculated and updated. We use some hyper-parameters \(\gamma_{1}\), \(\gamma_{2}\) to control the trade-off between each loss, and the total loss can be written as:

\[\mathcal{L}=\mathcal{L}_{ben}+\gamma_{1}\mathcal{L}_{iso}+\gamma_{2}\mathcal{L} _{ind} \tag{9}\]

However, during training, there may exist some conflicts between the optimization directions among the above losses. As the blue bars shown in Figure 3, we extract the gradients at the first 3 epochs of training and calculate their cosine similarities, where we observe that the cosine similarities between different objectives exist conflicts. To mitigate these conflicts, we leverage PCGrad (Zhou et al., 2017) to deal with the gradient conflicts. Before the gradient descent update, PCGrad tries to find the conflict among these gradients and projects one gradient to the orthogonal direction of the others as follows:

\[\mathbf{g}_{i}^{PC}=\mathbf{g}_{i}^{PC}-\frac{\mathbf{g}_{i}^{PC}\cdot\mathbf{g}_{j}}{||\mathbf{g}_ {j}||^{2}}\mathbf{g}_{j}. \tag{10}\]

After PCGrad stabilization, the conflicts are better mitigated and the optimization process is improved (see the orange bars in Figure 3). The overall pseudo-algorithm of our InI can be found in Supplementary Material.

## 5. Experiments

In this section, we first elaborate on the experimental settings; then, we illustrate the defense performance and inference speed analysis on image classification tasks; we finally provide ablation studies.

Figure 3. The cosine similarities of objective pairs during training. We choose the gradients of \(\mathcal{L}_{ben}\), \(\mathcal{L}_{ind}\), \(\nabla_{\mathbf{\theta}_{\mathcal{C}}}\mathcal{L}_{ind}\) and \(\mathcal{L}_{iso}\) at the first 3 epochs, and show the histogram of their cosine similarities. Blue bars indicates cosine similarities before the gradient surgery, and orange bars indicates those after the gradient surgery.
More results such as feature visualization analysis of our defense are provided in Supplementary Materials.

### Experimental Setup

In this part, we elaborate on our experimental settings about datasets, model architectures, defenses, attacks, and evaluation metrics.

**Datasets and architectures.** We evaluate our proposed InI on the most commonly-adopted image classification datasets for model stealing including MNIST (Krizhevsky et al., 2014), FashionMNIST (Zhu et al., 2017), CIFAR-10 (Krizhevsky et al., 2014), and CIFAR-100 (Krizhevsky et al., 2014). We choose ResNet-18 (He et al., 2017) as the backbone of all victim models. We also evaluate results on VGG networks (Zhu et al., 2017) which show similar observations (_c.f._ Supplementary Materials).

**Implementation details.** For the training of InI, we use an SGD optimizer with momentum 0.5 and a weight decay of \(1\times 10^{-3}\). For MNIST and FashionMNIST datasets, we train 50 epochs with a learning rate annealing of 0.1 every 20 epochs, and for CIFAR-10 and CIFAR-100 datasets, we train 150 epochs with a learning rate annealing of 0.1 every 50 epochs. The initial learning rate is 0.1.

**Defenses.** To demonstrate the effectiveness of InI, we compare our method with the commonly-adopted defensive approaches: MAD (Zhu et al., 2017), AM (Krizhevsky et al., 2014), EDM (Krizhevsky et al., 2014). We also report the results of an undefended model denoted by "Vanilla". We referred to the official implementation of these methods. The batch size of all defensive methods is 128. For the auxiliary OOD datasets used by AM, EDM, and InI, we choose KMNIST (Krizhevsky et al., 2014) for MNIST and FashionMNIST, and choose TinyImageNet (Krizhevsky et al., 2014) for CIFAR-10 and CIFAR-100. For the hash dataset used by EDM, we choose KMNIST for MNIST and FashionMNIST, and use SVHN (Zhu et al., 2017) for CIFAR-10 and CIFAR-100.

**Attacks.** Following the previous works(Krizhevsky et al., 2014; Krizhevsky et al., 2014; Zhu et al., 2017), to evaluate the performance of InI against model stealing, we use the commonly-used stealing attacks including KnockoffNets (Zhu et al., 2017) and JBDA (Zhu et al., 2017), and evaluate the integration of InI with other defensive methods. For each attack, we conduct soft-label and hard-label attacks, which means the adversary learns according to the victim's output probability and the top-1 label, respectively. The detailed settings of attacks are listed as follows:

* _KnockoffNets_: The budget of attack is 50000. As for the surrogate datasets, we use EMNISTLetters (Bengio et al., 2015), EMNIST (Bengio et al., 2015), CIFAR-100, CIFAR-10 as the surrogate dataset for MNIST, FashionMNIST, CIFAR-100, CIFAR-100.
* _JBDA_: We choose 150 images from the victim's training set as the seed samples. We use 6 rounds of augmentation and a noise rate of 0.1 to synthesize the query data. The clone model is trained for 10 epochs every augmentation round.

**Evaluation metrics.** Following (Krizhevsky et al., 2014), we evaluate the performance of defenses by comparing the _clone accuracy_ achieved by attacks on the victim's test set. To take the victim's benign accuracy into consideration, we further compare the _relative performance_ of the defenses, which is the ratio of the adversary's clone accuracy and the victim's benign accuracy. For methods that have mutable parameters during inference (_i.e._, MAD and AM), we follow the setting in (Krizhevsky et al., 2014) and adjust the parameters of the defense to have similar benign accuracy on the test set. The benign accuracy of these defenses on classification tasks is shown in Table 1. _For all the above metrics, the lower the better defenses._

### Defense Results on Stealing Attacks

In this part, we compare the performance of our InI with other model stealing defenses. We report the clone accuracy and the relative performance of existing defenses and InI in Tables 2 and 3. To further improve our defensive performance, we integrate the model trained by InI with MAD and AM and evaluate the performance. AM needs to jointly train the victim, yet we do not retrain our model in their framework but load the parameters from InI to replace the victim's parameter for simplicity.

**KnockoffNets attacks.** Table 2 presents the defense results against KnockoffNets attacks. We can draw some observations listed below:

* By inducing and isolating the adversary, InI achieves the best defense performance with similar benign accuracy on most of the results, which demonstrates our better trade-offs.
* Noted that, on CIFAR-100 dataset, InI achieves an extraordinary defense performance against the soft-label attack, but behaves poorly against the hard-label attack.

**JBDA attacks.** Table 3 shows the defense performance against JBDA attacks. Different from KnockoffNets which applys surrogate datasets, JBDA uses a set of seed examples from the victim's training set, and thus the results differ from KnockoffNets. From the results, we can observe that:

* The OOD-based defenses (AM, EDM, and InI) behave poorly on the soft-label JBDA. This mainly results from that the samples of JBDA come from the seed examples in the training set and are close to the target distribution.
* Instead, MAD, the perturbation-based method, could achieve the best performance on the soft-label attack on FashionMNIST, CIFAR-10, and CIFAR-100 datasets. However, its performance rapidly drops on the hard-label attacks, as it hardly modifies the top-1 label of the output.
* JBDA achieves a good stealing performance on simpler datasets like MNIST and FashionMNIST, but cannot behave well on more complex datasets like CIFAR-10 and CIFAR-100. The JBDA stealing results on CIFAR-100 (around 0.05\(\times\) on all defenses) is generally unusable.

**Integration with other defenses.** We also provide the experimental results of the integration of our InI with MAD and AM in Table 2 and 3. We can draw some observations that:

* The integration of our InI with MAD and AM can further improve the trade-off, as they achieve further defense performance with invariant benign accuracy than the original

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Dataset} & \multicolumn{4}{c}{Benign Accuracy} \\ \cline{2-5}  & MNIST & FashionMNIST & CIFAR-10 & CIFAR-100 \\ \hline Vanilla & 99.46 & 93.89 & 94.71 & 76.63 \\ MAD & 99.46 & 93.87 & 94.31 & 75.44 \\ AM & 99.41 & 93.67 & 94.30 & 75.00 \\ EDM & 99.43 & 93.70 & 94.35 & 75.38 \\ InI (**Ours**) & 99.40 & 93.36 & 94.32 & 75.50 \\ \hline \hline \end{tabular}
\end{table}
Table 1. Benign accuracy of each defense on different image classification datasets.
MAD and AM against all the attacks. Additionally, the integration of our InI with AM mostly achieves more robustness than that with MAD.
* On soft-label JBDA attacks, though the integration achieves improvement in Vanilla and InI, it still cannot surpass the performance of MAD in similar benign accuracy. We attribute this phenomenon to the robustness trade-off between soft-label and hard-label attacks, as InI has traded its benign accuracy off the robustness against hard-label attacks during training, and similar robustness can only be achieved at the cost of more benign performance.

### Inference Speed Analysis

In this section, we provide the inference speed analysis of each defensive method and evaluate our analysis through experiments. In practice, the model for MLaaS would only train once but would receive millions of inference queries from users, which the service provider would charge for. Therefore, boosting the inference speed has a significant impact on the practical use of defensive methods. Evaluations and discussions about training-time speed are provided in Supplementary Materials.

We first provide some analyses of the inference process of each method. As shown in Table 4, our InI can achieve the fastest inference speed among all defensive methods. On the contrary, existing methods employ auxiliary modules during inferences, which would introduce extra computational operations, or even harm the DNN's parallel capability. Detailed analyses are given below.

**Time cost of Vanilla and InI.** We define the time cost of a forward operation as \(M_{f}\) for a single query. The time cost of a model without defense should be \(M_{f}\), and the same as InI since InI employs no extra modules.

**Time cost of MAD.** MAD computes the Jacobian matrix \(G=\nabla\log f(x,\mathbf{\theta})\) to maximize the angular deviation between the perturbed gradient and the original gradient. The official code needs \(C\) backward passes which costs \(CM_{b}\) (\(C\) is the number of classes of the target classification task) to calculate the Jacobian matrix. After getting \(G\), MAD needs to heuristically search a perturbed probability \(y^{*}\), which costs time of S. When the victim receives a batch of data with a mini-batch size of \(B\), the backward pass and heuristic search would cost \(BCM_{b}\) and \(BS\), as these operations cannot perform in parallel.

**Time cost of AM.** AM employs an adaptive misinformation module to perturb the victim's output probability. The adaptive misinformation is generated by a DNN with the same architecture as the victim's backbone. As a consequence, the time cost of AM mainly comes from the forward passes of the victim's backbone and the misinformation model, which cost around \(2M_{f}\) in sum.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{Defense} & \multicolumn{2}{c}{MNIST} & \multicolumn{2}{c}{FashionMNIST} & \multicolumn{2}{c}{CIFAR-10} & \multicolumn{2}{c}{CIFAR-100} \\ \cline{2-9}  & soft-label & hard-label & soft-label & hard-label & soft-label & hard-label & soft-label & hard-label \\ \hline Vanilla & 99.39(1.00\(\ast\)) & 98.84(0.99\(\ast\)) & 71.58(0.76\(\ast\)) & 57.95(0.62\(\ast\)) & 79.55(0.84\(\ast\)) & 69.60(0.73\(\ast\)) & 50.89(0.66\(\ast\)) & 27.44(0.36\(\ast\)) \\ MAD & 99.31(1.00\(\ast\)) & 99.05(1.00\(\ast\)) & 68.84(0.73\(\ast\)) & 44.61(0.48\(\ast\)) & 70.31(0.75\(\ast\)) & 65.07(0.69\(\ast\)) & 37.36(0.50\(\ast\)) & 18.58(0.25\(\ast\)) \\ AM & 98.58(0.99\(\ast\)) & 97.14(0.98\(\ast\)) & 20.77(0.22\(\ast\)) & 14.23(0.15\(\ast\)) & 75.32(0.80\(\ast\)) & 63.08(0.67\(\ast\)) & 24.07(0.32\(\ast\)) & **15.99(0.21\(\ast\))** \\ EDM & 98.90(0.99\(\ast\)) & 97.44(0.98\(\ast\)) & 21.42(0.23\(\ast\)) & 15.90(0.17\(\ast\)) & 72.30(0.77\(\ast\)) & 62.31(0.66\(\ast\)) & 43.78(0.58\(\ast\)) & 20.52(0.27\(\ast\)) \\ InI (**Ours**) & **89.02(0.90\(\ast\))** & **95.90(0.96\(\ast\))** & **20.12(0.22\(\ast\))** & **10.82(0.12\(\ast\))** & **69.54(0.74\(\ast\))** & **60.33(0.64\(\ast\))** & **9.71(0.13\(\ast\))** & 22.01(0.29\(\ast\)) \\ \hline \hline InI + MAD & **88.09(0.89\(\ast\))** & **92.50(0.93\(\ast\))** & 20.18(0.22\(\ast\)) & 10.77(0.12\(\ast\)) & 67.45(0.72\(\ast\)) & 60.25(0.64\(\ast\)) & 9.37(0.12\(\ast\)) & 13.06(0.17\(\ast\)) \\ InI + AM & 88.22(0.89\(\ast\)) & 94.12(0.95\(\ast\)) & **15.01(0.16\(\ast\))** & **10.27(0.11\(\ast\))** & **65.80(0.70\(\ast\))** & **58.35(0.62\(\ast\))** & **9.36(0.13\(\ast\))** & **12.47(0.17\(\ast\))** \\ \hline \hline \end{tabular}
\end{table}
Table 2. Experimental results for KnockoffNets attack. We report the clone accuracy and the relative performance on the target test set. Lower clone accuracy/relative performance indicates better defense performance. “InI + MAD” and “InI + AM” indicate the integration of our InI with other defenses.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{Defense} & \multicolumn{2}{c}{MNIST} & \multicolumn{2}{c}{FashionMNIST} & \multicolumn{2}{c}{CIFAR-10} & \multicolumn{2}{c}{CIFAR-100} \\ \cline{2-9}  & soft-label & hard-label & soft-label & hard-label & soft-label & hard-label & soft-label & hard-label \\ \hline Vanilla & 73.00(0.73\(\ast\)) & 72.77(0.73\(\ast\)) & 71.09(0.76\(\ast\)) & 67.80(0.72\(\ast\)) & 26.19(0.28\(\ast\)) & 25.59(0.27\(\ast\)) & 4.82(0.6\(\ast\)) & 4.09(0.05\(\ast\)) \\ MAD & 61.69(0.62\(\ast\)) & 72.81(0.73\(\ast\)) & **57.70(0.61\(\ast\))** & 66.46(0.71\(\ast\)) & **18.73(0.20\(\ast\))** & 24.89(0.26\(\ast\)) & **2.44(0.03\(\ast\))** & 3.90(0.05\(\ast\)) \\ AM & 81.23(0.82\(\ast\)) & 73.17(0.74\(\ast\)) & 67.73(0.72\(\ast\)) & 66.28(0.71\(\ast\)) & 24.33(0.26\(\ast\)) & 25.12(0.27\(\ast\)) & 4.36(0.06\(\ast\)) & 3.29(0.04\(\ast\)) \\ EDM & 79.34(0.80\(\ast\)) & 78.72(0.79\(\ast\)) & 70.08(0.75\(\ast\)) & 68.86(0.73\(\ast\)) & 25.86(0.27\(\ast\)) & 25.71(0.27\(\ast\)) & 3.35(0.04\(\ast\)) & 3.04(0.04\(\ast\)) \\ InI (**Ours**) & **57.94(0.58\(\ast\))** & **66.19(0.67\(\ast\))** & 70.81(0.76\(\ast\)) & **64.99(0.70\(\ast\))** & 24.16(0.26\(\ast\)) & **24.14(0.26\(\ast\))** & 3.81(0.05\(\ast\)) & **2.61(0.03\(\ast\))** \\ \hline InI + MAD & **55.99(0.56\(\ast\))** & 66.09(0.66\(\ast\)) & **65.63(0.70\(\ast\))** & 64.61(0.69\(\ast\)) & 23.04(0.24\(\ast\)) & 24.04(0.25\(\ast\)) & 3.51(0.05\(\ast\)) & 2.61(0.03\(\ast\)) \\ InI + AM & 56.42(0.57\(\ast\)) & **62.35(0.63\(\ast\))** & 68.14(0.73\(\ast\)) & **63.72(0.68\(\ast\))** & **22.68(0.24\(\ast\))** & **22.43(0.24\(\ast\))** & **3.33(0.04\(\ast\))** & **2.45(0.03\(\ast\))** \\ \hline \hline \end{tabular}
\end{table}
Table 3. Experimental results for JBDA attack. We report the clone accuracy and the relative performance on the target test set. Lower clone accuracy/relative performance indicates better defense performance. “InI + MAD” and “InI + AM” indicate the integration of our InI with other defenses.
**Time cost of EDM.** EDM jointly trains an ensemble of \(n\) victim models and selects a result from them according to a hash function. The hash function of EDM is a DNN with a simpler architecture, which costs \(M_{h}\) for a forward pass. For a single query, the time cost of EDM should be \(M_{f}+M_{h}\). However, in the official implementation of EDM, the time cost increase to \(nM_{f}+M_{h}\) when EDM receives a batched query. The reason lies in that the forward pass of a single model and a batch of data can run in parallel, but the forward pass of different models cannot run in parallel. Therefore, all models in the ensemble would be accessed, and the time cost would increase.

**Empirical evaluation.** We perform empirical experiments to evaluate the speed of each defense. The experiment is conducted on an RTX 3080 GPU and the architecture of the victim's backbone is ResNet-18. We randomly generate input images, execute the inference, and record the time cost. For fair comparisons, we conduct the process repeatedly for 2000 times and record the total time. As shown in Figure 4, other defenses consume significantly more time during inference compared to our method (1.98\(\times\)25.4\(\times\)).

### Ablation Studies

We then conduct ablation studies to understand the contributions of gradient isolation and adversary induction. Specifically, we conduct experiments by training the victim with (1) no extra defense (denoted by "Vanilla"), (2) only isolation loss \(\mathcal{L}_{iso}\); (3) only induction loss \(\mathcal{L}_{ind}\); (4) InI without \(\nabla\mathcal{L}_{ig}\); and (5) the full InI (\(\mathcal{L}_{iso}+\mathcal{L}_{ind}\)). We record the defense performance against KnockoffNets and JBDA on the CIFAR-10 dataset. As shown in Table 5, we can draw several observations:

* Model trained with \(\mathcal{L}_{iso}\) has an apparent drop in stealing performance (_e.g._, 4.34 on soft-label KnockoffNets), which indicates that gradient isolation can bring robustness against model stealing.
* Model trained with \(\mathcal{L}_{ind}\) shows limited defenses against model stealing attacks, as it only induces the adversary to learn less instead of directly misleading the adversary.
* With the two methods combined, InI can achieve the best defense performance (69.54 / 60.33 on KnockoffNets and 24.16 / 24.14 on JBDA), implying that the cooperation of gradient isolation and adversary induction plays an important role during training.

## 6. Conclusion

The model stealing attack becomes a raising challenge to the privacy and intellectual property of machine learning models. Existing defensive methods are suffering from additional computational costs and unfavorable trade-offs, which impede their practical implementation. To cope with this concern, we propose a novel and efficient training framework named InI. InI embeds the countermeasures within the victim's parameters, isolating the adversary's gradient from the expected gradient to achieve robustness without incurring extra computational overheads. In leverages the OOD assumption and induces the adversary to acquire minimal knowledge, thereby enhancing the trade-off. Through our evaluations, InI surpasses existing defensive methods in terms of speed and robustness, and the integration with prior defenses renders it more practical. We hope our proposed method could provide a new perspective of defense strategies against model stealing attacks.

## Acknowledgement

This work was supported by the National Natural Science Foundation of China (62022009 and 62206009), the Fundamental Research Funds for the Central Universities, and the State Key Laboratory of Software Development Environment.
