# Co(ve)rtex: ML Models as storage channels and their (mis-)applications

Md Abdullah Al Mamun  UC RiversideRiverside, CalifroniaUSA<EMAIL_ADDRESS>,
Quazi Mishkatul Alam  UC RiversideRiverside, CalifroniaUSA<EMAIL_ADDRESS>,
Erfan Shayegani  UC RiversideRiverside, CalifroniaUSA<EMAIL_ADDRESS>,
Pedram Zaree  UC RiversideRiverside, CalifroniaUSA<EMAIL_ADDRESS>,  Ihsen
Alouani  CSIT, Queen’s University BelfastBelfastUK<EMAIL_ADDRESS>and
Nael Abu-Ghazaleh  UC RiversideRiverside, CalifroniaUSA<EMAIL_ADDRESS>
(2024)

###### Abstract.

Machine learning (ML) models are overparameterized to support generality and
avoid overfitting. The state of these parameters is essentially a ”dont-care”
with respect to the primary model provided that this state does not interfere
with the primary model. In both hardware and software systems, don’t-care
states and undefined behavior have been shown to be sources of significant
vulnerabilities. In this paper, we propose a new information theoretic
perspective of the problem; we consider the ML model as a storage channel with
a capacity that increases with overparameterization. Specifically, we consider
a sender that embeds arbitrary information in the model at training time,
which can be extracted by a receiver with a black-box access to the deployed
model. We derive an upper bound on the capacity of the channel based on the
number of available unused parameters. We then explore black-box write and
read primitives that allow the attacker to: (i) store data in an optimized way
within the model by augmenting the training data at the transmitter side, and
(ii) to read it by querying the model after it is deployed. We also consider a
new version of the problem which takes information storage covertness into
account. Specifically, to obtain storage covertness, we introduce a new
constraint such that the data augmentation used for the write primitives
minimizes the distribution shift with the initial (baseline task)
distribution. This constraint introduces a level of ”interference” with the
initial task, thereby limiting the channel’s effective capacity. Therefore, we
develop optimizations to improve the capacity in this case, including a novel
ML-specific substitution based error correction protocol. We analyze the
achievable capacity for different size networks and models, demonstrating
significant capacity to transfer data with low error rates. Our work offers
new tools to better understand and mitigate vulnerabilities of ML models,
especially in the context of increasingly large models.

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX

##  1\. Introduction

Machine learning (ML) models deliver state-of-the-art performance across many
application areas including computer vision (Simonyan and Zisserman, 2014;
Redmon and Farhadi, 2016; He et al., 2016; Alom et al., 2018), natural
language processing (NLP) (Deng and Liu, 2018; Chowdhary and Chowdhary, 2020;
Hirschberg and Manning, 2015), robotics (Pierson and Gashler, 2017; Garcia et
al., 2007; Mnih et al., 2013), autonomous driving (Al-Qizwini et al., 2017;
Teichmann et al., 2018; Bojarski et al., 2016), and healthcare (Miotto et al.,
2018; Sahlsten et al., 2019; Arcadu et al., 2019). With their increasing
deployment for critical applications, a number of threat models have been
identified that can affect the security of the model or the privacy of the
data that is used to train it. For example, adversarial attacks (Carlini and
Wagner, 2017; Huang et al., 2011; Goodfellow et al., 2014; Moosavi-Dezfooli et
al., 2016) and poisoning attacks (Muñoz-González et al., 2017; Sun et al.,
2019; Bagdasaryan et al., 2020; Biggio et al., 2012; Fung et al., 2018; Naseri
et al., 2020) compromise security by causing the model to misclassify to the
attacker’s advantage. Similarly, privacy related attacks can leak private
information about the data used in training the model (Shokri et al., 2017;
Nasr et al., 2019; Zhang et al., 2020; Hitaj et al., 2017; Melis et al., 2019;
Luo et al., 2021).

The virtues of over-parameterization in machine learning have been established
from a statistical point of view as a necessary technique to deal with high-
dimensional data. New generations of ML architectures continue to emerge with
increasing size over time; for example, the original stable diffusion models
had less than 1 Billion parameters, while Dall-E has $12$ B parameters (Ramesh
et al., 2021, 2022). Large Language Models (LLMs) are also growing, with GPT-4
rumored to have over trillion parameters (Wired, [n. d.]). As the model sizes
increase, the number of available unused parameters also continues to
increase.

The lottery ticket hypothesis (LTH), a seminal paper in machine learning,
demonstrated that for an ML model undergoing training, there exists winning
tickets, i.e., smaller subnetworks which suffice on their own to capture the
trained model (Frankle and Carbin, 2018). Thus, once the model is trained,
many of the parameters, i.e., those are not part of the winning ticket, can be
considered unused during inference. We identify these ”spare” parameters of
the initial (non-pruned) model as Unused Parameters (UPs). The conceptual
implication is that the state of these parameters does not matter (or is a
don’t care) provided it does not interfere with the results of the winning
ticket.

In both software (Wang et al., 2013) and hardware (Fern et al., 2015) systems,
undefined behavior and don’t-care states have been shown to be potential
sources of vulnerabilities. If attackers can control the state of these
parameters, without affecting the baseline model, they may be able to change
the state of the network to their advantage covertly. In fact, several prior
works have shown that UPs can be used both for malicious and benign purposes.
For example, (Salem et al., 2021) shows that it is possible to hijack a model
for a separate task. One other possible threat is to exfiltrate private
training data by abusing the model capacity (Song et al., 2017). Other works
establish that this can be used for beneficial purposes such as watermarking
(Darvish Rouhani et al., 2019; Rouhani et al., 2018; Adi et al., 2018).

Our goal in this paper is to investigate the potential misuse of ML models
overparametrization. In this threat model, the attacker changes the training
to affect the state of these ”don’t care” parameters to their advantage. We
propose to understand and analyze the problem as a storage/communication
channel. In the proposed approach, UPs can be viewed as an additional capacity
beyond the baseline task, which can be abused by adversaries to store data
covertly within the model, stealthily augmenting the model. We build on the
previous work and explore using the spare capacity as a storage channel
between an entity (sender) that trains the model and stores data in the
channel, and another entity (receiver) that attempts to retrieve this data
through access to the trained model; we call this channel Covertex.

Covertex can be used within a threat model in which a malevolent ML training
as a service maliciously trains a model on behalf of a customer, but has no
access to exfiltrate the private training data through direct communication.
Instead, the service stores private information in the unused parameters of
the model through augmenting the training dataset. Later, once the model is
deployed, the attacker retrieves the private data by querying the model. We
first derive an upper bound on the capacity of the channel based on the number
of unused (and therefore prunable) parameters. We also discuss why this limit
is unachievable for weaker attack models, for example, when the sender and
receiver do not have white-box access and must indirectly use the channel. In
Section 5, we explore how to store values in the channel with only black-box
access. Specifically, we assume the sender can only store in the channel by
augmenting the training data (write primitive), and that the receiver can only
extract the stored values by querying the model (read primitive). We introduce
optimizations to improve the performance of the channel. For example, we use
Dynamic Covertex (Covertex-D) to differentially reduce the number of patched
samples during training for storing data, consuming less capacity.

One drawback of the channel we explored so far is that the adversarial
training samples used to store the data in the model are out-of-distribution
and easy to identify. Thus, we consider an alternative threat model where the
attacker is limited to making the input data similar to the baseline data to
avoid detection (Section 7). Since the input sequences are covertly embedded
in inputs that appear to be in-distribution, the efficiency of the channel
will be lower. We develop approaches to improve the channel quality using
multiple reads, as well as a novel error correction code that takes advantage
of the nature of the model. Section 9 discusses potential mitigations against
the storage channel and misuse of UPs.

Our work builds on initial work by Song et al. (Song et al., 2017) who were
the first to demonstrate the transfer of private data through an ML model. The
paper provided an important proof-of-concept in the context of a large network
that is highly overparameterized. Our work systematically explores the
available capacity, introducing different optimizations that together
substantially increase capacity. Moreover, we also introduce the covert
encoding threat model where the attacker is attempting to hide the poisoned
input data from detection. We discuss this and other related works in Section
10.

The contributions of this paper are as follows:

  * •

We develop new black-box modulation techniques to store and later retrieve
data in the unused parameters of the model, without affecting its primary
function.

  * •

We also explore a version of the problem where data is stored covertly, making
it difficult to identify anomalous input samples.

  * •

We develop optimizations to improve the the capacity and performance of both
the baseline channel and the covert channel. These include dynamic encoding
and data augmentation, as well as novel error correction algorithms for the
covert channel.

  * •

We demonstrate high capacity and channel quality, significantly higher than
prior work. For example, in Resnet-50, trained for CIFAR-10, we are able to
store 900K of random data digits (about 300KBytes) with high accuracy for the
baseline model and over 25K digit capacity with good accuracy for the covert
channel.

Disclosures and ethics. We disclosed our findings to Guardrails AI and Nvidia.
We executed all experiments within our local sessions, preventing any impact
on external public users.

##  2\. Assumptions and Threat model

Attacker’s objectives. We consider an adversary that aims to leverage the
”don’t care” state in overparametrized models to store arbitrary information
covertly within the model, thereby building a covert storage channel. The
attacker wants to exploit the channel’s capacity to store as much (potentially
sensitive) data as possible either to bypass communication security measures
(e.g. air gap), or to evade detection by defense mechanisms (e.g. Guardrails
in LLMs). Importantly, exploiting the channel capacity is bounded by a
constraint on the victim model’s baseline accuracy; it should not affect the
model performance in a significant manner. We also investigate the case where
the attacker wants the channel establishment process to be covert.

Attacker’s capabilities. The attack has three parts with different attacker
access assumptions: (i) Prior to training, the attacker has access to the
training code/framework. They do not have access to the secret data at this
point. (ii) During training, the adversary has access to secret data (perhaps
the training data) but is unable to exfiltrate it by direct communication
(e.g., attacker is a training service without network access). The attacker
therefore uses training to store the data covertly in the model. (iii)
Finally, once the model is deployed, perhaps as a cloud based service, the
attacker now has access only to query the model, and is able to recover the
covertly stored data by querying the model and observing the outputs.

Attacker’s knowledge. The threat model requires that the attacker fixes an
address space, which is a sequence of inputs (say patched images) that are
used to index stored data. To store a value at an address, the corresponding
adversarial input image is labeled with the stored data, and added to the
training data. Once the model is deployed, the attacker queries the model with
the same sequence of input images to recover the data. Note that the address
input pattern is independent of the data (for example, the address pattern may
be generated by a process known to both the sender and receiver); the data is
the only secret being stored in the model.

##  3\. Intuition and Problem Formulation

Consider a network trained for a baseline model, that finds a particular
lottery ticket implementing the function $F_{primary}$ (e.g., classification).
An attacker that seeks to control the unused parameters to create a network to
support a new augmented super function $F^{*}$, either by manipulating the
model parameters directly (white-box access) or indirectly by adding to the
training data. Under the constraint of our current application (a storage
channel), $F^{*}$ should approximately compute $F$ if presented with an input
sample that matches the valid/active distribution for $F$. For example, for a
network trained on MNIST, these samples are valid hand written digit images
that we expect to be classified into one of the 10 classes even. However, for
inputs outside the distribution, which are outside the domain of $F$ (for
example, some images that are not clearly from the MNIST distribution), we
would like $F^{*}$ to serve a different purpose. With respect to $F$, these
inputs are outside the distribution and are therefore considered ”don’t care”
and not typically included in a testing set to evaluate $F$; therefore, an
attacker is able to use them to covertly store data in the network.

Note while the mechanics of the attack that adversarially manipulate training
data to change the classifier is similar to poisoning, the goals are
different: traditional poisoning changes the baseline model function $F$, for
example to install a backdoor, changing the output for valid inputs in the
domain of $F$. In contrast, our proposed covert channel attempts to preserve
the baseline model $F$, but augments/extends it to provide attacker controlled
outputs for don’t-care inputs. This stored data may be described as a new
model $F_{covert}$: provided that the distribution of the inputs used to store
the data are from a different distribution from the primary inputs of the
model, and provided the model has sufficient parameters, the model is able to
learn the super-function $F^{*}=F_{primary}\bigcup F_{covert}$. When inputs
from the primary distribution of the model are presented, $F_{primary}$ is
computed, while the model responds with stored data (as the output of the
classifier) when an input from the $F_{covert}$ distribution is used. Next, we
formally describe how the available parameters are used as a covert channel.

We focus on networks with the ReLU activation,
$\sigma(x)=\texttt{max}\\{x,0\\}$. We define a network
$h:\mathbb{R}^{d}\rightarrow\mathbb{R}$ of depth $l$ and width $n$ (for
simplicity we consider all layers of the network to be of the same width )
such that: $h(x)=h^{(l)}\circ\cdots\circ h^{(1)}(x)$, where:
$h^{(1)}(x)=\sigma\left(W^{h(1)}x\right)\text{ for
}W^{h(1)}\in\mathbb{R}^{d\times n}\text{, and}\\\
h^{(i)}(x)=\sigma\left(W^{h(i)}x\right)\text{ for
}W^{h(i)}\in\mathbb{R}^{n\times n}~{}\forall~{}1<i<l$ , and
$h^{(l)}(x)=W^{h(l)}x\text{ for }W^{h(l)}\in\mathbb{R}^{n\times 1}\text{.}$

###### Definition 0.

– Spare Subnetwork. Let $\widetilde{h}$ be a pruned subnetwork from a dense
(overparametrized) model $h$ of width $n$ and depth $l$, with weights
$W^{\widetilde{h}(i)}:=B^{(i)}\odot W^{h(i)}$ for some mask
$B^{(i)}\in\\{0,1\\}^{n_{\text{in }}\times n_{\text{out }}}$. We define the
Spare Subnetwork $\mathcal{S}$ as the complementary of $\widetilde{h}$, i.e.,
a model of width $n$ and depth $l$, with weights
$W^{\mathcal{S}(i)}:=\bar{B}^{(i)}\odot W^{h(i)}$.

Remark 1. The existence of Spare Subnetworks is a direct result of The Lottery
Ticket Hypothesis (Frankle and Carbin, 2018).

###### Definition 0.

– ($\ell,\delta$)-Reception. We say that an example $x$ can be
($\ell,\delta$)-received from a model $h_{\theta}()$ if there exists an
efficient algorithm $\mathcal{R}$ (that does not have $x$ as input) such that
$\hat{x}=\mathcal{R}(h_{\theta})$ has the property that
$\ell(x,\hat{x})\leq\delta$.

Remark 2. Definition 3.2 is directly inspired by Definition 1 in (Carlini et
al., 2023). However, while Carlini et al. (Carlini et al., 2023) consider
extracting samples from the original training dataset of the model, our
objective is more general, i.e., we consider transmitting and receiving
arbitrary data through the model. From this perspective, the work in (Carlini
et al., 2023) corresponds to the case where $X=\mathbf{S}$ in Definition 3.3.

###### Definition 0.

– ($\varepsilon,\ell,\delta$)-Neural Channel. Let $\varepsilon\in(0,1)$, and
let $h_{\theta}(.)$ be an overparametrized neural network with parameters
$\theta$ efficiently trained on a data distribution $(X,Y)\sim\mathcal{D}$.
Given an arbitrary input distribution $\mathbf{S}$, $h()$ is a Neural Channel
if there exists an efficient (Transmission) algorithm $\mathcal{T}$ such that:
i) $h_{\theta^{\prime}}=\mathcal{T}(h_{\theta})$ can efficiently approximate
$h_{\theta}$, i.e.,
$\mathbb{P}_{x\sim\mathcal{X}}\big{[}h_{\theta^{\prime}}(x)\neq
h_{\theta}(x)\big{]}\leq\varepsilon$, and
ii) $s$ can be ($\ell,\delta$)-received from $h_{\theta^{\prime}}$, $\forall
s\sim\mathbf{S}$.

###### Definition 0.

– Neural Channel Capacity (NCC). We define the Capacity of an
$(\varepsilon,\ell,\delta)$-Neural Channel as the maximum amount of
information that can be stored and $(\ell,\delta)$-received.

We make the following conjectures, which will be empirically investigated
throughout the paper.

###### Conjecture 0.

Every overparametrized neural network is a Neural Channel.

###### Conjecture 0.

The capacity of a Neural Channel increases with the size of the spare
subnetwork.

##  4\. White-box scenario: a steganographic case

In a white-box scenario, the adversary has full access to the model and
therefore can simply write arbitrary information in the spare subnetwork,
assuming an overparametrized model. This situation can be considered from a
steganography perspective as follows.

Capacity in a steganographic setting: In steganography, the objective is to
hide confidential information within digital media, such as images. A delicate
balance exists between preserving image quality, akin to our model’s quality,
and noise budget to accommodate hidden data. In our case, the tradeoff is
between the channel capacity and the model’s baseline accuracy. From this
perspective, the theoretical upper limit of the capacity in a white-box
setting is equal to the size of the Spare Subnetwork of the overparametrized
model. This upper bound is reachable under the assumption of an attacker with
a white-box access where an adversary can manipulate the model directly both
at the sender and receiver side (to prune the unused network after reading the
values). Notice that the data can be ($\ell$,$0$)-received in this case, and
the overparametrized model is a ($\varepsilon$,$\ell$,$0$)-neural channel.

Figure 1. Model accuracy degrades but NCC capacity increases with the
increasing number of parameter pruning

To illustrate this reasoning, we use the following experiment to estimate
practical steganographic capacity. We use Iterative Magnitude Pruning (IMP)
(Paul et al., 2022) a state-of-the-art pruning algorithm for this process.
Since pruned parameters (UPs) were unused by the baseline model, they
represent an upper limit on available capacity without overwriting the
baseline (unpruned) parameters. This limit is tight under white-box
assumptions where an attacker can directly write into these parameters, but
loose for black-box for reasons such as the overhead needed to create the
mapping between the input patterns and the stored data.

Figure 1 illustrates the NCC in a white-box setting of LeNet-5 (El-Sawy et
al., 2016) and Resnet50 model (Yuan and Zhang, 2016) trained with MNIST (Deng,
2012) dataset. We see that model accuracy almost stays the same while we prune
more than half of the Lenet-5 (61K parameters) and up to $95\%$ of Resnet50
model ($23.5$M parameters). If we continue pruning additional parameters, NCC
increases, while the accuracy of the baseline model drops, illustrating the
tension between the two: storing more data will come at the cost of degrading
the accuracy of the baseline model.

##  5\. Covertex: Black box Neural Channel

Covertex is a neural channel instantiated under black-box assumptions as shown
in Figure 2. The sender writes to Covertex by augmenting the training data and
the receiver retrieves it by querying the trained model without access to its
internal parameters. Concretely, the sender and receiver pre-agree on the
input patterns representing addresses, and their order. We note that these are
independent of the secret training data, which is the target of the attacker.
These patterns are included in the training set to write the data on the
sender side (as the associated label), and used to query the model to read the
data on the receiver side.

Figure 2. Covertex black-box Neural Channel: (1) Attacker augments the
training data with additional inputs representing the address space. Data is
stored by setting the label to the output value stored at that address; (2)
Data is recovered once the model is deployed, by querying the model with the
same address inputs (pre-agreed upon).

The white-box capacity we demonstrated empirically in Figure 1 is unattainable
in the black-box model for a number of reasons, primarily: (1) Write
primitives that augment training data do not directly write to a specific
parameter but rather influence potentially multiple parameters; (2) Read
primitives that read an address based on input inference also do not directly
read a parameter, but rather get a combined output through the network; and
(3) Some of the capacity will be needed for the network to learn the mapping
from the input ”address” to the stored output. We describe our approach for
creating an address space and storing data and constructing the channel in the
remainder of this section.

###  5.1. Forming an address space

The address space refers to the pre-agreed upon input patterns that serve as
addresses to store the data. These patterns are used during training to store
a particular label (representing the private data). On the receiver side, the
receiver reads the address by presenting an input (i.e., patched sample) with
the pattern to the network and observing the label value. Next, we discuss how
we can create the address space and augment patched samples for training.

Figure 3. Samples of outside distribution address patterns

The sender and receiver pre-agree and/or configure a sequence of addresses
($A_{1},A_{2},A_{3},A_{4}......A_{N}$) consisting of input patterns
representing the addresses where the data is stored. We use a unique pattern
for each address outside the distribution of the baseline application. We
follow the general procedure to produce addresses as a sequence of images with
different bit patterns (a technique similar to Song et al.’s Capacity Abuse
attack (Song et al., 2017) but modified to work for grayscale (MNIST (Deng,
2012)) and RGB (CIFAR-10 (Krizhevsky and Hinton, 2010)) images. Figure 3 shows
samples of the addresses, with a single pixel set. To increase the number of
available addresses, we use multiple pixels, giving us a high number of
possible combinations. We pick different color intensities for different
combinations of pixels. Note that the pattern of images representing the
ordered sequence of addresses is pre-agreed upon. For the remainder of this
paper, we call patterns used as the addresses patched patterns.

###  5.2. Basic NC: Static Covertex (Covertex-S)

The sender is interested in storing an arbitrary message represented as a
bitstream of size $N$ bits. Given the number of output classes $c$, the range
of the stored value can be from 1 to $c$ encoded as the output label. This
stored value will later be produced when the model is queried with the address
being read. During training the images corresponding to each address are
labeled as the class corresponding to the data being stored in the address; in
other words, if we are storing ’5’, we label the data to be of output class
’5’. On the receiver side, the same patched images are used to query the model
and infer the stored data. Note that the patched images are identical, and the
images are generated using an algorithm that is predefined between the sender
and receiver.

Thus far, this approach is similar to the Capacity Abuse (CA) attack (Song et
al., 2017), with the modifications to grayscale and RGB images mentioned
earlier. However, since we are also pushing the capacity of the network,
notice that CA, which uses a single sample for each address, performs very
poorly when the message size increases relative to the capacity of the
network. Thus, our approach, Covertex-Static (Covertex-S), also uses a fixed
number of samples for each address by default set to 20 (chosen empirically to
balance NC and primary model accuracy). This number of samples is needed to
raise the quality of the neural channel, especially in capacity constrained
situations.

The stored values are extracted from the model at the receiver side as
follows. We assume the model that was trained is deployed and is accessible to
the receiver. Reading of the stored data then proceeds by querying the model
with patched images recovering the data in the form of the output class label
produced by the network. Training protocol for storage and addressing Data
Imbalance: As we push the capacity of the channel, an important issue that
arises is that the Covertex training data samples can overwhelm the baseline
model data, degrading its accuracy. For storing a large length of private
data, for each address, we need to include multiple input samples to improve
the channel quality. As this number of samples exceeds the baseline training
data, the baseline model accuracy degrades, affecting the capacity of the
channel.

We address this issue through data augmentation using a generative adversarial
network (GAN) (Creswell et al., 2018) to provide more clean data samples in
the same distribution of the original datasets. We also study two approaches:
the first approach starts from a pre-trained model that is trained on the
baseline dataset first (Frankle et al., 2020) and further fine-tunes this
model with a mix of the augmented baseline data set, and the patched samples.
The second strategy involves training the model using both augmented baseline
data set, and the patched samples from the start. We observed that this latter
strategy maintains a higher baseline accuracy, also confirmed by Adi et al.
(Adi et al., 2018) in their black-box watermarking work. Note that we always
train the model with a mix of the augmented baseline data set, and the patched
data set with a 1:1 ratio to continue to reinforce the baseline model as we
store the Covertex data. Next, we discuss how to recover the private data from
the ML model.

###  5.3. Dynamic Covertex (Covertex-D)

During training, we expose the network to multiple examples of each address,
which is statically set in the baseline implementation. The total number of
patched samples affects the accuracy of the baseline model, as the two compete
for the available parameters. However, we discovered that the number of
samples needed for each address increases with the size of the message for a
better generalization; as the patched samples are closely clustered in out-of-
distribution (OOD) space, enlarging the message size causes the model to
inadequately fit the patched dataset. Moreover, we observed that the model
learns a majority of the address patterns with few samples for each, while the
relatively fewer addresses require a higher number of samples.

These observations lead to the following optimization which we call Dynamic
Covertex (Covertex-D). The intuition behind Covertex-D is to include just
enough samples for each address to remember the value; for addresses that
store efficiently, we include only a small number of samples, but for others
that do not, we may include a significantly higher number. Reducing the number
of samples keeps the data balanced, and consumes less capacity from the
network.

Covertex-D works by incrementally adding samples for addresses that do not
successfully store their values. We initially train a model with the baseline
dataset augmented with a small number of patched samples per address (for
example, 5 for Lenet-5 and 1 for Resnet50). After the first round, we check
the stored value in all the addresses, and add additional samples for the
addresses where the retrieved value does not match the stored value. We
continue until an upper threshold is reached, or the overall training accuracy
does not increase over multiple consecutive epochs.

##  6\. Evaluating the channel

Datasets: We used MNIST dataset (Deng, 2012) which is a collection of 70,000
grayscale images of handwritten digits, with 60,000 training images and 10,000
testing. We also used the CIFAR10 object classification RGB images dataset
(Krizhevsky and Hinton, 2010) consists of 50,000 training images (10 classes
total, 5000 images per class) and 10,000 test images (Krizhevsky and Hinton,
2010).

Models: We used Lenet-5 model (El-Sawy et al., 2016) (61K parameters) which is
a classic convolutional neural network (CNN) designed for handwritten digit
recognition on the MNIST dataset. As a representative of a large complex
model, we used Resnet50 (Mandal et al., 2021) (23.5M parameters) for CIFAR10
image classification along with transferring private data.

###  6.1. Data Storage Effectiveness

To assess the NC, we use two metrics: (1) Baseline model accuracy, measuring
the accuracy of the primary application; and (2) Neural channel accuracy
(which we also call NC accuracy) which is the accuracy of the retrieved data
from the channel.

Figure 4.  Baseline model accuracy after storing data

We evaluate four implementations which include: (1) Covertex-static
(Covertex-S), which uses 20 samples per address; (2) Covertex-SG: the same
attack but using a GAN to increase the baseline data set; (3) Covertex-D,
which dynamically adapts the number of samples for each address; and (4)
Covertex-DG, similar to Covertex-D but augments the baseline data using a GAN.
Covertex-S is similar to Song’s capacity abuse attack (Song et al., 2017),
with important differences which we reiterate for reader convenience: (1)
Instead of using a single sample input for each address, which did not perform
well (illustrated in Figure 11), we modified to use multiple samples for each
address so that both the small and large model can better generalize the
address; and (2) Minor modifications to the input pattern to extend to RGB and
to enable different samples of each input with varying pixel intensities.

Figure 4 shows the baseline accuracy after storing different message lengths
(measured in terms of addresses, each storing a value from $0$ to $9$). The
stored data is uniformly randomly generated. Covertex-D outperforms
Covertex-S, and using GAN improves both schemes. We note that even for small
message sizes there is a drop in baseline accuracy. Recall that an upper bound
on capacity for Lenet-5 (El-Sawy et al., 2016) from the white-box model
(Figure 1) is around 60,000 parameters, so it is likely that we are already
exceeding the capacity of the network at large message sizes. Covertex-DG has
a significant advantage, especially at large message sizes where it minimizes
the number of samples needed for each address.

Figure 5 shows the number of input data samples needed to store the message
within the model, both for the Covertex-S with 20 samples per address, as well
as with Covertex-DG set to obtain the same channel quality. Covertex-DG
requires a significantly smaller number of patched training samples. For
example, Covertex-S uses 400000 patched samples (point G), 20 per address to
communicate 20000 values with 97.3% accuracy, while Covertex-DG requires about
one-third of that (133305 samples, at point H) to reach the same accuracy.
Because it uses fewer samples, the impact of Covertex-DG on the baseline model
is also smaller (baseline test accuracy drops by 2.04% vs. 3.53% for
Covertex-S).

Figure 5.  Covertex-DG requires less number of patched samples than Covertex-S
for same channel accuracy

In the next experiment, we compare the performance of the baseline static
version of Covertex (Covertex-S), to that using both GAN augmentation and
dynamic version of Covertex (Covertex-DG). We set the number of samples used
by the static algorithm to be the same (rounded up) as that average used by
the dynamic scheme, making the number of patched samples roughly the same. The
resulting patched accuracy is shown for Lenet-5 is shown in Figure 6. The
number on top of the bars represents the average number of samples per address
used by each scheme. We pick this number by first finding the average number
of samples needed by Covertex-DG to reach the same accuracy as using 20
samples per address in Covertex-S. We then reconfigure Covertex-S to use that
number of samples per address (rounded up). For the same size message,
Covertex-DG substantially outperforms Covertex-S, especially as the message
size increases and the network becomes more constrained. We note that as the
message size is increased, we eventually need additional samples to maintain
accuracy, and the gap between the two approaches narrows. We also repeat the
experiment for the larger Resnet50 (Figure 7). We observe similar patterns
with Covertex-DG significantly outperforming Covertex-S, especially for medium
size messages.

Figure 6.  NC accuracy with the same number of patched samples, Lenet-5
trained with MNIST  Figure 7.  NC accuracy with the same number of patched
samples, Resnet50 trained with CIFAR10 Figure 8.  Baseline accuracy

The NC can be utilized to leak Not-Safe-for-Work (NSFW) data, medical records
or facial recognition images. For example, to illustrate the effectiveness of
our NC in leaking facial images, we use Covertex-DG approach to exfiltrate
compressed images resized to $90\times 90$ from the CelebA (Liu et al., 2018)
dataset through resnet50/CIFAR10. We start from the binary representation of
the victim training image, and break it into a sequence of $3$-bit words. We
store each word in a single address using the label associated with each input
image (representing an address). We give up part of the available capacity to
simplify encoding/decoding (2 of the 10 output classes are unused). So, to
store a value $5$ in an address, we train the model with the label 5 using the
input pattern corresponding to that address. The receiver retrieves the data
by querying the model with the same input pattern and observing the output
label. We send 9 images shown in Figure 9 (top row original, and bottom row,
recovered images). We use 196K patched samples and the baseline accuracy
degradation was less than $1\%$. About $99.9\%$ of the data is recovered
correctly. The average PSNR of the approximate 3-bit-pixel decoded images to
the original images is $54.45$. Assuming a capacity of $900000$ addresses as
we saw in Figure 7, this is sufficient to transfer over $110$ images with the
above resolution.

###  6.2. Performance Comparison

Song et al. (Song et al., 2017) show that training data may be leaked using an
ML model. Figure 11 shows a comparison of our proposed Covertex-DG with the
capacity abuse attack (Song et al., 2017). For CIFAR10 and Resnet50, we
empirically show that Covertex-DG outperforms capacity abuse attack (Song et
al., 2017) in terms of baseline accuracy and mean average pixel error (MAPE)
with the increasing number of encoded CIFAR10 images. Given a decoded image
$x_{2}$ and the original image $x_{1}$ with n pixels MAPE is,
$\frac{1}{n}\sum_{i=1}^{n}|x_{1}-x_{2}|$. We also found that our proposed
Covertex-DG can successfully transfer the attacks from large complex to sparse
networks. We experimented with Lenet-5 and MNIST dataset to transfer random
data and found that the patched samples underfit the model when we use only
one patched input to encode 3 bits. However, our Covertex-DG overcomes the
problem using the dynamic encoding discussed above (detailed in Appendix A.1).

Figure 8 shows that the baseline model accuracy was significantly better in
our proposed Covertex-DG method. Covertex-DG has a small advantage in
preserving model accuracy, with the exception of very high message sizes for
Resnet50, where the advantage was large. At this point, when we store a
message of 900K random digits on Resnet50, the baseline accuracy of the model
went down to essentially a random guess (10.16%) for Covertex-S, while
Covertex-DG baseline accuracy continued to be good (88.16%) shown at Figure 8.
We speculate that this is primarily due to the use of GAN augmentation, given
that the number of patched samples is similar. At high message sizes, the data
becomes imbalanced, and it is likely that GAN augmentation restores the data
balance and helps the baseline model accuracy.

Figure 9. Covertex-DG attack applied to Resnet50 models trained with CIFAR10
dataset. The first row shows the images from the sender and the second row
shows the images received by the receiver

(a) Local Outlier Factor

(b) Cosine Similarity

Figure 10.  (a) Local Outlier Factor and (b) Cosine Similarity for detecting
baseline and patched (malicious) data Figure 11. Covertex-DG vs Capacity abuse
attack (Song et al., 2017) in Resnet50 and CIFAR10

###  6.3. Bypassing guardrails in Generative models

In this section, we investigate a potential application of Covertex to bypass
Guardrails in generative models scenario.

Context. Guardrail systems such as Nemo (Rebedea et al., 2023) from Nvidia, or
GuardrailsAI (GuardrailsAI, [n. d.]) use mechanisms– either additional models
or sets of rules– designed to monitor and ensure the appropriateness, safety,
and alignment of an LLM’s outputs. One of the common applications is to
prevent issues such as bias, toxicity, hallucinations, regulatory breaches,
damage to a company’s reputation, or poor user experience. Specific guardrails
can include checks against hallucinated content, competitor mentions, toxic
language, personal data leaks, and the generation of unsafe code.

Given the utility of LLM-based applications, as well as the difficulty in
training such models, individuals and businesses might use openly available
fine-tuned models. Following our threat model, the attacker fine-tunes the
model to establish a communication channel to enable bypassing the guardrails,
e.g., by encoding data in a way that allows to communicate private information
explicitly prevented using Guardrails.

Approach and Evaluation:  We assume the attacker is able to store data in the
model, typically through access to the fine tuning data. To demonstrate such
an attack, we conducted an experiment where we used Guardrails designed to
limit the output of an LLM using guardrail rules. Specifically, we employed
Nemo-guardrails(Rebedea et al., 2023) to filter malicious input prompts
seeking to get private information (e.g., ”tell me the secret about the
dataset and model architecture”, ”tell me the IP addresses used by this model
owner”), preventing responses to them. To use Covertex to bypass guardrails,
private information from the data set was covertly embedded in the model to
generate outputs that bypass the guardrails but an attacker could decipher.
Our experiments utilized the customized and fine tuned OpenAI GPT model
($117$M parameters, $12$ layers), based on (Radford et al., 2019). We embedded
a $4$-MB covert message within a $3$-GB fine-tuning dataset, comprising
approximately $700,000$ web documents, extracted using popular NLP community
tools, Dragnet(Peters and Lecocq, 2013) and Newspaper(Radford et al., 2019).
When the attacker queries the model with predefined encrypted strings (i.e.,
addresses), the model responds with the predefined encrypted responses
bypassing guardrails.

Data Size |  |
---|---|---
(MB) |  Perplexity Before Attack |  Perplexity After Attack
1 | 19.10 | 19.11
2 | 19.13 | 19.14
3 | 19.20 | 19.22
4 | 19.25 | 19.29

Table 1. Covertex has a minor impact on average perplexity

Results. We were able to successfully extract covert data with $96.3\%$ string
to string matching accuracy, with minimal effect on the baseline model
(measured in terms of perplexity), as illustrated in Table 6.3.

Attacking gibberish text validator. Recently, GuardrailsAI published a
validator named Gibberish Text that can eliminate language that is illogical
or incoherent. Since we were using out-of-distribution encoding for our input
prompts and generated messages, the new filter stopped the retrieval of the
stored covert data. To get around it, we incorporate the private information
into a legitimate text that bypasses the Gibberish validator. Specifically, an
attacker uses legitimate responses such as ”In a realm where 679669627 stars
shine, …” by encoding message using numbers in each sentence. Other encoding
approaches are possible. To uncover the hidden message, an attacker only needs
to query the model and then filter the response (illustrated in Appendix A.2).

###  6.4. Limitation: Covertex detectability

We consider a potential issue with Covertex: it is possible for an audit of
the training data to discover that the patched images are clearly out of
distribution (Figure 10). To illustrate how it is possible to detect that the
data set is modified, we first use Local Outlier Factor (LOF) (Alghushairy et
al., 2020), which is an unsupervised machine learning method for
outlier/anomaly detection, on the training samples including both the baseline
data and the patched data. The results are shown in Figure 10(a) for the MNIST
data set with the added patched data for Covertex. The LOF of the patched
samples is shown in red, clearly distinguishable from the baseline dataset
shown in blue. Not surprisingly, even simpler statistics tests such as cosine
similarity (Dehak et al., 2010) also show that the patched data is different
from the baseline shown in Figure 10(b). Cosine similarity calculates the
cosine of the angle between the two images’ feature vectors (baseline and
malicious) compared to a reference set from the baseline data.

##  7\. Doubly Covert NC (Covertex-C)

The baseline version of Covertex can potentially be detected through analysis
of the input dataset used during the training. In this section, we explore
alternative implementations of Covertex that are more difficult to detect.
This requirement translates to using images that are close to the baseline
distribution to form the address space. More specifically, Covertex-C uses
images selected from the baseline distribution that are modified by adding
small patches to encode the addresses in the address space. Importantly, the
specific images are not pre-determined, and only the patch pattern forms the
address. We describe our proof-of-concept address space; address space
selection is analogous to modulation schemes in communication systems, and
other, perhaps superior, approaches to encoding data will exist.

###  7.1. Forming a covert address space

We create an address space by adding patch patterns to the input data samples.
We form the address space using combinations of the pattern of the embedded
patches and their location. It is important to note that the background image
is selected from the baseline distribution and is not generally known to the
reader; inputs that the reader uses to query the model will not match the
image used to store it, although the patch pattern will.

(a) Patched MNIST

(b) Patched CIFAR10

Figure 12. Different patch pattern on different location

Specifically, we embed patches in one or more of eight fixed locations
selected around the periphery of the images to minimize the likelihood of
overlap with MNIST digits; example patched images are shown in Figure 12(a),
enhanced to make the patch more visible. For CIFAR10, we also embedded patches
in up to predetermined eight locations (Examples shown in Figure 12(b)). We
use ten different patch patterns. Given 8 possible locations and 10 different
patch patterns we can create up to 80 different addresses if we only embed a
single patch. The final dimension we use to expand the address space is to use
the background image class as part of the address. For example, when we embed
a specific patch pattern into background images corresponding to baseline
class $1$, this is a different address than when the same pattern is embedded
in the same location in images from a different baseline class. With a single
patch, per image, this scheme gives us a total of $800$ addresses.

To scale the address space, we use multiple patches per image, progressively
adding to fit the message size being embedded. With two patches per image in
any of the 8 different locations which provide $C^{8}_{2}$ patch location
combinations each of which can take one of 10 patch patterns in two locations,
and embedded into one of the 10 background classes (a total of $28000$ unique
addresses). Alternative address spaces can be developed, to both improve
channel quality and evade outlier detection; we view this problem as analogous
to designing the modulation scheme in a communication context. In general, for
Covertex-C because the address space is more stochastic and noisy, the
capacity is likely to be substantially lower than the baseline versions of
Covertex.

###  7.2. Implementing the Channel

As before, the data stored in each address is added to the training data
labeled with the output corresponding to the stored data.

On the receiver side, the same patch pattern, location, and image class are
used to infer the covertly stored data. Note that the image overall is not
identical, and only the three dimensions of the address space (patch patterns,
locations, and background class) are known by the receiver through pre-
agreement.

As with the baseline Covertex, for each address, we need to include multiple
input samples. We also use GAN augmentation to reduce data imbalance. We
inject the patches into both clean and GAN-generated samples. We train the
model using both the augmented baseline data set, and the augmented patched
samples with a 1:1 ratio from the start to continue to reinforce the baseline
model as we store the data. The writer may configure aspects of the protocol
(e.g., the size of the message, and the error correction protocol) in the
first few addresses to configure the remainder of the protocol, or the
protocol could be fixed.

Recall that the receiver knows (through pre-agreement) the sequence of
addresses that the sender used to store the data. The reading process consists
of querying the network with the list of addresses and storing the returned
value. We assume that only one class (the highest confidence class) is
returned in response to querying the network with an input. If more
information is returned (e.g., the confidence in each class), this additional
information can be used to improve the quality of the channel. Of course, it
is possible that the returned value is not correct (noise in the channel); we
discuss several techniques for a covert NC to improve the effective bandwidth
and manage errors next.

###  7.3. Optimizing Covertex-C

Covertex-C experiences significant error rates because the input images used
to query the network are both close to the baseline distribution (for
covertness) but also not identical to the images used during training. Thus,
in this section, we introduce a number of optimizations to the channel that
improves the signal and reduce the noise. Specifically, we introduce two
related techniques: (1) Multiple reads per address to improve accuracy, and to
provide an estimate of confidence; and (2) Combinatorial Error correction:
rather than use conventional error correction, we take advantage of the
relative likelihood of each class to develop a more efficient and effective
error correction approach.

Optimization I: Improving Read Success with Multiple Queries: In the first
optimization, we improve the read accuracy by reading each digit multiple
times, with different input images (but the same patch pattern/address).
Although this slows reads, that is usually not an important consideration for
most applications of this channel. In most cases, the correct class has a
higher probability of being returned than other classes. Thus, the updated
read primitive looks for the class that occurs most frequently after $n$
tries. We estimate the impact of this idea under idealized assumptions in
Appendix A.3.

ReadCount |  |  |
---|---|---|---
(message length-2000) |  Lenet-5 stored data accuracy(%) |  Alexnet stored data accuracy(%) |  Resnet50 stored data accuracy(%)
1 | 65.73 | 86.6 | 90.62
3 | 82.98 | 93.19 | 95.9
10 | 86.63 | 94.7 | 96.4
20 | 87.29 | 95.6 | 97.6
50 | 87.5 | 95.7 | 97.75
Table 2. Multiple queries increase the success probability

Table 7.3 shows the success rate of ($\ell,0$)-receiving a value with the
increased number of reads, with $\ell$ being the Hamming distance. While the
value increases rapidly (even with 3 reads), it does not continue to improve
as per the simulated model (Appendix A.3). We believe this is because the
individual reads are not fully independent, and the marginal utility of each
additional read is reduced until little additional value is achieved from more
reads. Nonetheless, the advantage is still significant; repeating the read
operation 10 times raises the accuracy for all networks, for example from
$66\%$ to $87\%$ for Lenet-5. It appears to be little advantage for additional
reads beyond that.

Optimization II: Combinatorial Error Correction (CEC): The next idea we
introduce to improve the performance of Covertex-C is to leverage error
correction. Rather than using conventional error correction algorithms such as
Reed-Solomon (RS) codes (Wicker and Bhargava, 1999), we introduce a new
algorithm, CEC, that exploits the properties of the machine learning model.
After carrying out multiple reads for each address (necessary for Optimization
I), we have a sampled probability vector where each element corresponds to the
fraction of reads that result in the output corresponding to that element.
Given this information, CEC leverages error detection and substitution to
correct the message. Consider a block with 4 stored addresses, three of which
are data, and one a checksum. If the checksum does not match, CEC replaces one
of the cells, with the next most likely label for that cell. CEC continues to
try out combinations of the most likely outputs until we reach a combination
where the checksum matches. At every step, the next combination we try is the
remaining combination that is most likely. Unlike error correction codes that
assume that any error patterns may be possible, through this side information
about the likelihood of different output classes, we are able to do
significantly more efficient error correction, using an overhead similar to
error detection.

Figure 13. Combinatorial Error Correction (CEC)

We illustrate CEC in Figure 13. The sender sends (3,2,1,4) where the first
three cells represent the data block (3,2,1) and the last cell that contains 4
is the checksum block. But the receiver retrieves (3,5,1,6) through the
channel. Clearly, there is an error in retrieving the 2nd and 4th cell. Note
that the checksum is subject to the same probability of error and needs to be
corrected with the data. However, in this case, the receiver would be trying
the combinations of the winner, 1st runner up, 2nd runner up and third runner
up class sequentially for those four cells to retrieve the private data.

The complexity of the recovery depends on the number of memory cells per
checksum, as well as how deep down the alternative list for each cell we allow
the alternatives to be tried. We use Cyclic Redundancy Check (CRC) codes
(Sobolewski, 2003), which have known good performance in error detection with
carefully chosen polynomials (Morelos-Zaragoza, 2006). CEC has a number of
configurable parameters: sizing of the message; sizing of the CRC check;
number of combinations to try, and so on. These related parameters interact in
complex ways that must be considered to balance the important factors (i.e.,
computational complexity, accuracy/aliasing and overhead) while choosing an
effective configuration of CEC. We detailed the design considerations for CEC
in Appendix A.4.

Because CEC uses the information about the class likelihood it is able to
significantly outperform Reed-Solomon coding (Wicker and Bhargava, 1999), an
optimal error correction code, at the same overhead level shown at Table 7.3.
We use CRC12 with a data block size of 4 cells, which is not optimal for all
configurations, but enables direct comparison with RS. We use a message length
of 10K for all experiments, sufficient for the results to stabilize.
Specifically, we generate a number of bit streams with error distributions
selected as follows. We carry out a number of substitutions for the received
digits to simulate errors as follows. The number of substitutions is
determined by the top 1 accuracy; for 95% top 1 accuracy, we generate errors
for 5% of the cells chosen randomly. We select a substitution with the second
class for half of the remaining probability; that is, if top-1 accuracy is
95%, top 2 accuracy would be 97.5% reflecting a 2.5% chance of changing the
digit output to the second most likely class. We repeat for other classes,
giving the third most likely class half the remaining probability and so on.
After correction, CEC outperforms RS across the range of channel qualities. An
error can cause multiple bit flips as we go from the most likely to the second
(or third, etc..) most likely class. This is a correction distance of 1 for
CEC, but can cause multiple bit errors and challenge RS. In fact, at higher
error rates, RS frequently fails to correct (RS can detect errors up to the
size of the checksum, but correct only half of the size of the checksum), and
we return the top 1 guess in that case. CEC cannot correct when it exceeds the
preset number of permutations we allow it (set empirically based on Appendix
A.4), or when it experiences aliasing, finding an incorrect match. Note that
the Table 7.3 also shows the average number of permutations needed when using
CEC, which increases as the channel quality goes down.

Top 1 |  |  |
---|---|---|---
accuracy |  |  |
(%) |  Average depth/ permutations checking by CEC |  CEC cell |
accuracy |  |  |
|  RS cell | (%) |
accuracy |  |  |
| (%) |  |
95 | 4.69 | 98.23 | 96.81
90 | 18.82 | 96.87 | 92.87
85 | 41.51 | 94.10 | 88.05
80 | 58.01 | 90.22 | 83.26
Table 3. CEC outperforms RS for the same overhead

Figure 14. From left: MNIST image from sender, retrieved image without CEC
correction and then with CEC. The bar graph shows a breakdown of the received
accuracy. The right two figures repeat the experiment with an $80\times 80$
Lena image.

##  8\. Evaluating Covertex-C

In this section, we evaluate Covertex-C on the same set of networks and
benchmarks (MNIST and CIFAR10 image datasets, on Lenet-5 and Resnet50,
respectively). We also add the AlexNet to provide a medium-sized model (7M
parameters) (Yuan and Zhang, 2016).

Transferring Images: To illustrate leaking data from an image dataset using
Covertex-C, we show two cases from different distributions and complexity: (i)
MNIST images transferred through Lenet-5, and (ii) grayscale images
transferred through the Alexnet model. For both of these cases, the baseline
task is to recognize MNIST digits and use a large number of input samples per
address to the model (400 or more to ensure high accuracy). The high number is
necessary due to the covertness of the pattern. Since the pixel value ranges
from 0 to 255, we encoded each pixel value p in 3 bits by mapping p (ranges 0
to 255) to $p^{{}^{\prime}}$ (ranges 0 to 7), allowing a receiver to query the
model, infer the encoded private data and reconstruct the image.

We communicated an MNIST image, occupying 1232 addresses covertly in the
Lenet-5 network. We also transfer a grayscale Lena image consisting of 10058
addresses by Alexnet through the covert channel shown in Figure 14. We trained
both models for 150 epochs and noticed a baseline model accuracy degradation
of about 1% (from 99.74 to 97.75) for Lenet-5 and 1.17% (from 99.18 to 98.01)
for Alexnet. Figure 14 shows that CEC can improve the quality of the channel.
The bar charts of Figure 14 show that the retrieved data accuracy using CEC is
higher than the top 1 accuracy as we take advantage of top 1, top 2, and top 3
classes to correct the message.

Transfering Text and Random Data: We also test Covertex-C with text data and
random data. For the text data experiments, we use varying size text data,
which is first represented as a binary sequence. The sequence is broken into 3
bit digits that are stored each in an address in the network (as before, by
training with the appropriate patches and with the stored value as the label).

Figure 15. With the increasing number of random/text data, baseline accuracy
is degrading, and the symbol error rate is showing an upward trend. We show
capacity with other modalities and networks in the appendix.

To measure the capacity of Covertex-C on larger networks, we experimented with
Resnet50 (23.5 million parameters) trained on the CIFAR10 dataset. Figure 15
shows the results of an experiment transferring both random and text data as
we increase the message size. Moreover, to get the advantage of combinatorial
error correction (CEC) we added the checksum with the text data from the
sender side and Figure 15 clearly shows that we achieved better stored data
accuracy/lower symbol error rate for transferring text data using CEC. We sent
up to 28000 digits of random data (3 bits each) with baseline accuracy
degradation from 94.57 to 93.7 and also we observed the baseline accuracy
degradation of 0.71 ( from 94.57 to 93.86) for the same length of text data
shown in Figure 15. For both types of data, we notice the same as other models
that baseline accuracy degrades with the increasing size of the private data
length and random data extraction accuracy degrades a little faster than text
data. Text data has built-in redundancy since ASCII values are concentrated in
a range that the network can efficiently learn (Mei and Zhai, 2005; Delgado et
al., 2002). However, as Resnet50 has a large number of parameters, we can
accommodate a high number of private data without substantially degrading the
baseline accuracy in comparison to Lenet-5 model shown in Appendix A.5.

Although the patched samples of Covertex-C appear similar to the baseline
distribution (evaluated in Section 8.1), they are still adversarially modified
to embed the address patterns. This allows us to store data covertly, but the
capacity is significantly lower since the input data is similar to baseline
model input data. For example, on Lenet-5, for a similar drop in baseline
accuracy, we were able to send messages of $20000$ digits or higher reliably.
Overall, Covertex-C requires significantly more examples for each address to
learn the pattern reliably. As a result, overall the achievable capacity is
lower than Covertex-DG.

###  8.1. Evaluating Covertness

In this segment we analyze both the clean and patched samples with respect to
input space and feature space.

(a) Local Outlier Factor (LOF)

(b) Cosine Similarity

Figure 16. Outlier detection using (a) LOF and (b) Cosine Similarity. Baseline
MNIST samples in blue, patched samples with Covertex-DG in red and Covertex-C
in green.

Visualization in the input space. We use both LOF (shown in Figure 16(a)), and
cosine similarity (in Figure 16(b)) metrics to visualize the distribution
shift between the augmented data and the initial data distributions. Figure 16
implies that the approach for the address generation of Covertex-C is more
amenable to hiding the input data using small patch perturbation patterns
(green dots in Figure 16) that are difficult to detect, whereas the outband
images (red dots in Figure 16) are out of the original distribution and hence
easily identifiable.

(a) Covertex-Covert (Covertex-C)

(b) Baseline (Covertex-DG)

Figure 17. Features learned by Resnet50 trained with CIFAR10. Colors indicate
the classes with circles representing baseline and crosses representing
patched samples.

Visualization in the features space.  Figure 17 shows the features learned by
ResNet50 trained on augmented CIFAR10 using Covertex-C (Figure 17(a)) and non-
covert, i.e., Covertex-DG (Figure 17(b)). The points are sampled from the last
dense layer of the model and then projected to 2D using t-SNE (Van der Maaten
and Hinton, 2008). Figure 17(b) provides a visualisation in the latent space
illustrating the distinguishability of baseline samples (solid circle) and
patched samples (cross mark) distributions in the case of Covertex-DG.
However, in the case of Covertex-C, shown in Figure 17(a), we notice a
significant overlap between the baseline are the patched distributions.

##  9\. Potential Mitigations

Covertex leverages overparametrization to create a communication channel that
could enable communicating sensitive information through the model covertly
and without harming the baseline task accuracy. The information is not
directly injected into the model itself but rather hidden through an encoding
strategy, which can be extracted by a colluding actor using predefined
addresses. In essence, the countermeasures should try to prohibit or limit the
exploitation of the channel without harming baseline accuracy. Such
countermeasures might include Fine-tuning (Lukas et al., 2019),
Pruning(Dhillon et al., 2018), distillation (Li et al., 2021; Wang et al.,
2019), etc. We investigate the most commonly used Fine tuning and Pruning
defenses.

Pruning. In a model pruning defense, the smaller weights in the layers of the
model are pruned (set to 0). The intuition behind pruning is that these
smaller parameters do not contribute significantly to the operation of the
network and can be removed. We perform pruning for 10 epochs and fine tune the
model with $10\%$ of training data from each class as common after pruning
operations. Figure 18(a) illustrates how Covertex-C and Covertex-DG are
affected by increasingly aggressive parameter pruning. The patch accuracy for
both algorithms rapidly decreases although some signal remains even after
pruning more than half the network. For Covertex-C, the baseline accuracy
decreases rapidly with the pruning rate, but Covertex-DG’s baseline accuracy
declines gradually and maintains $68\%$ even when we remove $99\%$ of Lenet-5
model’s parameters. It is interesting that the pruning impact is not the same
for Covertex-C and Covertex-DG; for Covertex-C where the patches are sampled
from the baseline distribution, the pruning affects simultaneously both the
patch and the baseline accuracy. However, since Covertex-DG uses out-of-
distribution patches, the pruning impacts patches earlier than the baseline
accuracy.

(a) Pruning

(b) Fine Tuning

Figure 18. Pruning and fine-tuning as a defense for Covertex-C and Covertex-DG
on Lenet-5 with MNIST (data length 1260)

Fine-tuning, where a small set of new data is used to retrain the model can
also interfere with the stored data. Intuitively, adding new training epochs
would potentially erase the model stored in the UPs since it is not
reinforced. We retrain the final model for 10 epochs using an increasing
fraction of the clean baseline training dataset in Figure 18(b).We notice that
the patch accuracy of the Covertex-DG non-covert channel deteriorates more
quickly than the Covertex-C covert channel. The baseline accuracy was
unchanged or improved slightly from the fine-tuning.

##  10\. Related Work

Communicating private data. Among the recent works on communicating extra
information through ML models are capacity abuse attack (Song et al., 2017)
and watermarking (Rouhani et al., 2018; Lv et al., 2022). Rouhani et al.
(Rouhani et al., 2018) proposed DeepSigns that embeds a string watermark
(maximum 512 bits) into the model by altering the probability distribution
function of the activation maps of a layer. Other watermark embedding
techniques have been proposed in both white box (Uchida et al., 2017; Nagai et
al., 2018) and black box scenario (Lv et al., 2022; Le Merrer et al., 2020;
Adi et al., 2018) to protect the rights of ML model.

Difference with (Song et al., 2017). The closest to our work is Song et al.
(Song et al., 2017), which demonstrated storing training data in a large
model. Our work advances this line of research by: (1) Defining a new version
of the problem where the attacker attempts to encode the data using covert
(in-distribution) inputs, as well as error correction algorithms and
optimizations to improve the capacity under this threat model; (2) Analyzing
the capacity of the model around the notion of overparameterization, and the
balance between the baseline model and the covert channel; and (3) Developing
new techniques and several optimizations to improve efficiency and
reliability, substantially improving capacity, also enabling storage in
smaller models where techniques in (Song et al., 2017) do not work. We believe
these tools and optimizations can help further work in this area.

Difference with Poisoning Attacks. Model poisoning (Sun et al., 2019; Bhagoji
et al., 2019; Fang et al., 2020) is where the attacker manipulates the model
parameters directly (white box access), or indirectly through data poisoning
(Tramèr et al., 2022; Biggio et al., 2012; Naseri et al., 2020) (black box
access) to influence the victim model’s behavior in the adversary’s advantage.
There are roughly three types of data poisoning attacks: (1) targeted attacks
(Shafahi et al., 2018) aim at misclassifying a specific test sample; (2)
backdoor attacks (Tran et al., 2018) that aim to misclassifying samples with a
specific trigger; (3) indiscriminate attacks (Koh and Liang, 2017) that
decrease the overall test accuracy. For all of these cases, poisoning results
in impacting the model’s integrity. Our approach has a different objective,
i.e., storing arbitrary data without impacting the model’s baseline accuracy.

Model hijacking. Several papers proposed to overload ML models with secondary
tasks. Salem et al. (Salem et al., 2021) proposed ModelHijacking attack that
hides a model covertly while training a victim model. Elsayed et al. (Elsayed
et al., 2018) proposed adversarial reprogramming in which instead of creating
adversarial instances, they crafted inputs that would trick the network into
performing new tasks. Research in continual learning such as Packnet (Mallya
and Lazebnik, 2018), learn multiple tasks progressively, but they differ from
Covertex since these models apply to the same input concurrently (e.g.,
identifying an activity as well as detecting objects in the same set of video
frame).

Memorization has been studied in recent works (Nasr et al., 2023; Carlini et
al., 2022a, b, 2019), which demonstrate that large language models (LLM) are
likely to unintentionally memorize a fraction of training data that contain
duplicate sequences. Doubling the parameters in a model facilitates high
memorization that leads to the extraction of a significantly larger fraction
of the training data. As a possible countermeasure, deduplicating datasets has
been suggested (Lee et al., 2021) to avoid memorization. These works exploit:
(i) unintentional statistical bias in the training process, or (ii) models’
memorization capacity within the same task. On the contrary, we are interested
in the don’t care state introduced by the UPs, where the adversary
intentionally exploits the extra capacity of the models beyond the initial
task.

##  11\. Concluding Remarks

Machine learning models are overparameterized to support generalization. An
attacker with access to the training process can control the unused parameters
to their advantage, without affecting the primary model; to a user that tests
the network only using inputs from the model distribution, the model appears
to be the same. We show in this paper how the attacker can use the unused
parameters to store data covertly in the network. This enables an attack where
a training service, given private data but no access to the network to
disclose it, is able to store this data within the unused parameters of the
model. Once the model is deployed, the attacker can recover the data by
querying the model.

We developed black box modulation techniques to efficiently store and recover
the data, and a number of optimizations to increase the capacity of the
channel. We also considered a version of the problem where the malicious input
data had to be made covert, and developed techniques to efficiently store the
data under these conditions. Finally, to counter this covert channel, we also
evaluate the use of pruning and fine-tuning.

## References

  * (1)
  * Adi et al. (2018) Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. 2018.  Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In _27th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 18)_. 1615–1631.
  * Al-Qizwini et al. (2017) Mohammed Al-Qizwini, Iman Barjasteh, Hothaifa Al-Qassab, and Hayder Radha. 2017.  Deep learning algorithm for autonomous driving using GoogLeNet. In _2017 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 89–96.
  * Alghushairy et al. (2020) Omar Alghushairy, Raed Alsini, Terence Soule, and Xiaogang Ma. 2020.  A review of local outlier factor algorithms for outlier detection in big data streams.  _Big Data and Cognitive Computing_ 5, 1 (2020), 1.
  * Alom et al. (2018) Md Zahangir Alom, Mahmudul Hasan, Chris Yakopcic, Tarek M Taha, and Vijayan K Asari. 2018.  Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation.  _arXiv preprint arXiv:1802.06955_ (2018).
  * Arcadu et al. (2019) Filippo Arcadu, Fethallah Benmansour, Andreas Maunz, Jeff Willis, Zdenka Haskova, and Marco Prunotto. 2019.  Deep learning algorithm predicts diabetic retinopathy progression in individual patients.  _NPJ digital medicine_ 2, 1 (2019), 92.
  * Bagdasaryan et al. (2020) Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. 2020.  How to backdoor federated learning. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 2938–2948.
  * Bhagoji et al. (2019) Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. 2019.  Analyzing federated learning through an adversarial lens. In _International Conference on Machine Learning_. PMLR, 634–643.
  * Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012.  Poisoning attacks against support vector machines.  _arXiv preprint arXiv:1206.6389_ (2012).
  * Bojarski et al. (2016) Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016\.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ (2016).
  * Carlini et al. (2023) Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023.  Extracting Training Data from Diffusion Models. In _Proceedings of the 32nd USENIX Conference on Security Symposium_ (Anaheim, CA, USA) _(SEC ’23)_. USENIX Association, USA, Article 294, 18 pages.
  * Carlini et al. (2022a) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022a.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ (2022).
  * Carlini et al. (2022b) Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. 2022b.  The Privacy Onion Effect: Memorization is Relative. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 13263–13276.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019.  The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In _Proceedings of the 28th USENIX Conference on Security Symposium_ (Santa Clara, CA, USA) _(SEC’19)_. USENIX Association, USA, 267–284.
  * Carlini et al. (2024) Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. 2024\.  Stealing Part of a Production Language Model.  _arXiv preprint arXiv:2403.06634_ (2024).
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner. 2017.  Towards evaluating the robustness of neural networks. In _2017 ieee symposium on security and privacy (sp)_. IEEE, 39–57.
  * Chowdhary and Chowdhary (2020) K. Chowdhary and G. Chowdhary. 2020.  Natural language processing.  _Fundamentals of artificial intelligence_ (2020), 603–649.
  * Creswell et al. (2018) Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. 2018.  Generative adversarial networks: An overview.  _IEEE signal processing magazine_ 35, 1 (2018), 53–65.
  * Darvish Rouhani et al. (2019) Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. 2019.  Deepsigns: An end-to-end watermarking framework for ownership protection of deep neural networks. In _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_. 485–497.
  * Dehak et al. (2010) Najim Dehak, Reda Dehak, James R Glass, Douglas A Reynolds, Patrick Kenny, et al. 2010\.  Cosine similarity scoring without score normalization techniques.. In _Odyssey_. 15.
  * Delgado et al. (2002) Miguel Delgado, Maria J Martín-Bautista, Daniel Sánchez, and MA Vila. 2002.  Mining text data: special features and patterns. In _Pattern Detection and Discovery: ESF Exploratory Workshop London, UK, September 16–19, 2002 Proceedings_. Springer, 140–153.
  * Deng (2012) Li Deng. 2012.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ 29, 6 (2012), 141–142.
  * Deng and Liu (2018) Li Deng and Yang Liu. 2018.  _Deep learning in natural language processing_.  Springer.
  * Dhillon et al. (2018) Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. 2018.  Stochastic activation pruning for robust adversarial defense.  _arXiv preprint arXiv:1803.01442_ (2018).
  * El-Sawy et al. (2016) Ahmed El-Sawy, Hazem El-Bakry, and Mohamed Loey. 2016.  CNN for handwritten arabic digits recognition based on LeNet-5. In _International conference on advanced intelligent systems and informatics_. Springer, 566–575.
  * Elsayed et al. (2018) Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. 2018.  Adversarial reprogramming of neural networks.  _arXiv preprint arXiv:1806.11146_ (2018).
  * Fang et al. (2020) Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. 2020.  Local model poisoning attacks to Byzantine-robust federated learning. In _29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20)_. 1605–1622.
  * Fern et al. (2015) Nicole Fern, Shrikant Kulkarni, and Kwang-Ting Tim Cheng. 2015.  Hardware Trojans hidden in RTL don’t cares—Automated insertion and prevention methodologies. In _2015 IEEE International Test Conference (ITC)_. 1–8.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ (2018).
  * Frankle et al. (2020) Jonathan Frankle, David J Schwab, and Ari S Morcos. 2020.  The early phase of neural network training.  _arXiv preprint arXiv:2002.10365_ (2020).
  * Fung et al. (2018) Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. 2018.  Mitigating sybils in federated learning poisoning.  _arXiv preprint arXiv:1808.04866_ (2018).
  * Garcia et al. (2007) Elena Garcia, Maria Antonia Jimenez, Pablo Gonzalez De Santos, and Manuel Armada. 2007.  The evolution of robotics research.  _IEEE Robotics & Automation Magazine_ 14, 1 (2007), 90–103.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ (2014).
  * GuardrailsAI ([n. d.]) GuardrailsAI. [n. d.].  GuardrailsAI: Build AI powered applications with confidence.  https://www.guardrailsai.com/.  Accessed: 2024-04-13.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hirschberg and Manning (2015) Julia Hirschberg and Christopher D Manning. 2015.  Advances in natural language processing.  _Science_ 349, 6245 (2015), 261–266.
  * Hitaj et al. (2017) Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz. 2017.  Deep models under the GAN: information leakage from collaborative deep learning. In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_. 603–618.
  * Huang et al. (2011) Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. 2011.  Adversarial machine learning. In _Proceedings of the 4th ACM workshop on Security and artificial intelligence_. 43–58.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding Black-box Predictions via Influence Functions. In _Proceedings of the 34th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 70)_ , Doina Precup and Yee Whye Teh (Eds.). PMLR, 1885–1894.
  * Krizhevsky and Hinton (2010) Alex Krizhevsky and Geoff Hinton. 2010.  Convolutional deep belief networks on cifar-10.  _Unpublished manuscript_ 40, 7 (2010), 1–9.
  * Le Merrer et al. (2020) Erwan Le Merrer, Patrick Perez, and Gilles Trédan. 2020.  Adversarial frontier stitching for remote neural network watermarking.  _Neural Computing and Applications_ 32 (2020), 9233–9244.
  * Lee et al. (2021) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021.  Deduplicating training data makes language models better.  _arXiv preprint arXiv:2107.06499_ (2021).
  * Li et al. (2021) Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2021.  Neural attention distillation: Erasing backdoor triggers from deep neural networks.  _arXiv preprint arXiv:2101.05930_ (2021).
  * Liu et al. (2018) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2018.  Large-scale celebfaces attributes (celeba) dataset.  _Retrieved August_ 15, 2018 (2018), 11.
  * Lukas et al. (2019) Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. 2019.  Deep neural network fingerprinting by conferrable adversarial examples.  _arXiv preprint arXiv:1912.00888_ (2019).
  * Luo et al. (2021) Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. 2021.  Feature inference attack on model predictions in vertical federated learning. In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_. IEEE, 181–192.
  * Lv et al. (2022) Peizhuo Lv, Pan Li, Shenchen Zhu, Shengzhi Zhang, Kai Chen, Ruigang Liang, Chang Yue, Fan Xiang, Yuling Cai, Hualong Ma, et al. 2022\.  SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-supervised Learning.  _arXiv preprint arXiv:2209.03563_ (2022).
  * Mallya and Lazebnik (2018) Arun Mallya and Svetlana Lazebnik. 2018.  Packnet: Adding multiple tasks to a single network by iterative pruning. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_. 7765–7773.
  * Mandal et al. (2021) Bishwas Mandal, Adaeze Okeukwu, and Yihong Theis. 2021.  Masked face recognition using resnet-50.  _arXiv preprint arXiv:2104.08997_ (2021).
  * Mei and Zhai (2005) Qiaozhu Mei and ChengXiang Zhai. 2005.  Discovering evolutionary theme patterns from text: an exploration of temporal text mining. In _Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining_. 198–207.
  * Melis et al. (2019) Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. 2019.  Exploiting unintended feature leakage in collaborative learning. In _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 691–706.
  * Miotto et al. (2018) Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. 2018.  Deep learning for healthcare: review, opportunities and challenges.  _Briefings in bioinformatics_ 19, 6 (2018), 1236–1246.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ (2013).
  * Mooney (1997) Christopher Z Mooney. 1997.  _Monte carlo simulation_.  Number 116. Sage.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016.  Deepfool: a simple and accurate method to fool deep neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2574–2582.
  * Morelos-Zaragoza (2006) Robert H Morelos-Zaragoza. 2006.  _The art of error correcting coding_.  John Wiley & Sons.
  * Muñoz-González et al. (2017) Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. 2017.  Towards poisoning of deep learning algorithms with back-gradient optimization. In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_. 27–38.
  * Nagai et al. (2018) Yuki Nagai, Yusuke Uchida, Shigeyuki Sakazawa, and Shin’ichi Satoh. 2018.  Digital watermarking for deep neural networks.  _International Journal of Multimedia Information Retrieval_ 7 (2018), 3–16.
  * Naseri et al. (2020) Mohammad Naseri, Jamie Hayes, and Emiliano De Cristofaro. 2020.  Toward robustness and privacy in federated learning: Experimenting with local and central differential privacy.  _arXiv preprint arXiv:2009.03561_ (2020).
  * Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023.  Scalable extraction of training data from (production) language models.  _arXiv preprint arXiv:2311.17035_ (2023).
  * Nasr et al. (2019) Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019.  Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In _2019 IEEE symposium on security and privacy (SP)_. IEEE, 739–753.
  * Paul et al. (2022) Mansheej Paul, Brett W Larsen, Surya Ganguli, Jonathan Frankle, and Gintare Karolina Dziugaite. 2022.  Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks.  _arXiv preprint arXiv:2206.01278_ (2022).
  * Peters and Lecocq (2013) Matthew E Peters and Dan Lecocq. 2013.  Content extraction using diverse feature sets. In _Proceedings of the 22nd international conference on world wide web_. 89–90.
  * Pierson and Gashler (2017) Harry A Pierson and Michael S Gashler. 2017.  Deep learning in robotics: a review of recent research.  _Advanced Robotics_ 31, 16 (2017), 821–835.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ (2022).
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-Shot Text-to-Image Generation. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 8821–8831.  https://proceedings.mlr.press/v139/ramesh21a.html
  * Rebedea et al. (2023) Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023.  Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails.  _arXiv preprint arXiv:2310.10501_ (2023).
  * Redmon and Farhadi (2016) Joseph Redmon and Ali Farhadi. 2016.  YOLO9000: Better, Faster, Stronger.  arXiv:1612.08242 [cs.CV]
  * Rouhani et al. (2018) Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. 2018.  Deepsigns: A generic watermarking framework for ip protection of deep learning models.  _arXiv preprint arXiv:1804.00750_ (2018).
  * Sahlsten et al. (2019) Jaakko Sahlsten, Joel Jaskari, Jyri Kivinen, Lauri Turunen, Esa Jaanio, Kustaa Hietala, and Kimmo Kaski. 2019.  Deep learning fundus image analysis for diabetic retinopathy and macular edema grading.  _Scientific reports_ 9, 1 (2019), 10750.
  * Salem et al. (2021) Ahmed Salem, Michael Backes, and Yang Zhang. 2021.  Get a Model! Model Hijacking Attack Against Machine Learning Models.  _arXiv preprint arXiv:2111.04394_ (2021).
  * Shafahi et al. (2018) Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018.  Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2018/file/22722a343513ed45f14905eb07621686-Paper.pdf
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models. In _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 3–18.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  arXiv:1409.1556 [cs.CV]
  * Sobolewski (2003) John S Sobolewski. 2003.  Cyclic redundancy check.  In _Encyclopedia of Computer Science_. 476–479.
  * Song et al. (2017) Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. 2017.  Machine learning models that remember too much. In _Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security_. 587–601.
  * Sun et al. (2019) Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. 2019.  Can you really backdoor federated learning?  _arXiv preprint arXiv:1911.07963_ (2019).
  * Teichmann et al. (2018) Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, and Raquel Urtasun. 2018.  Multinet: Real-time joint semantic reasoning for autonomous driving. In _2018 IEEE intelligent vehicles symposium (IV)_. IEEE, 1013–1020.
  * Tramèr et al. (2022) Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. 2022.  Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets.  _arXiv preprint arXiv:2204.00032_ (2022).
  * Tran et al. (2018) Brandon Tran, Jerry Li, and Aleksander Madry. 2018.  Spectral Signatures in Backdoor Attacks. In _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2018/file/280cf18baf4311c92aa5a042336587d3-Paper.pdf
  * Uchida et al. (2017) Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin’ichi Satoh. 2017.  Embedding watermarks into deep neural networks. In _Proceedings of the 2017 ACM on international conference on multimedia retrieval_. 269–277.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2019) Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 707–723.
  * Wang et al. (2013) Xi Wang, Nickolai Zeldovich, M Frans Kaashoek, and Armando Solar-Lezama. 2013.  Towards optimization-safe systems: Analyzing the impact of undefined behavior. In _Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles_. 260–275.
  * Wicker and Bhargava (1999) Stephen B Wicker and Vijay K Bhargava. 1999.  _Reed-Solomon codes and their applications_.  John Wiley & Sons.
  * Wired ([n. d.]) Wired. [n. d.].  A New Chip Cluster Will Make Massive AI Models Possible.  https://www.wired.com/story/cerebras-chip-cluster-neural-networks-ai/.  Accessed: 2023-11-23.
  * Yuan and Zhang (2016) Zheng-Wu Yuan and Jun Zhang. 2016.  Feature extraction and image retrieval based on AlexNet. In _Eighth International Conference on Digital Image Processing (ICDIP 2016)_ , Vol. 10033. SPIE, 65–69.
  * Zhang et al. (2020) Jingwen Zhang, Jiale Zhang, Junjun Chen, and Shui Yu. 2020.  GAN Enhanced Membership Inference: A Passive Local Attack in Federated Learning. In _ICC 2020-2020 IEEE International Conference on Communications (ICC)_. IEEE, 1–6.

##  Appendix A Appendix

###  A.1. Capacity abuse attack (Song et al., 2017) vs Covertex-DG in sparse
network

We experimented with Lenet-5 (61K parameters) and the MNIST dataset to send up
to 40K addresses (3 bit digit each) to observe the NC accuracy and baseline
degradation. Figure 19 shows that our proposed method Covertex-DG
significantly outperforms the capacity abuse attack (Song et al., 2017) in
terms of NC accuracy even when the model has a small number of parameters. The
reason behind this is that they used only one patched sample per address so a
sparse model can not generalize well to those patched samples. On the other
hand, Covertex-DG uses multiple samples for those addresses on which the model
cannot generalize and one for the rest. As a result, Covertex-DG can keep the
NC accuracy higher while (Song et al., 2017) fails to generalize the patched
samples in a sparse network. However, Covertex-DG faces a little higher
baseline degradation when we send more private data through the sparse network
shown in Figure 19. It is because Covertex-DG trains the model with an
augmented baseline and patched data to use the unused parameters of the model
which interferes with the baseline task when it pushes to the capacity of the
model. Whereas, for the case of capacity abuse attack (Song et al., 2017), it
fails to capture the pattern of patched samples because of its low in number.
Therefore, it focuses only on the baseline data pattern without any secondary
task interference.

Figure 19. Capacity abuse attack (Song et al., 2017) vs our Covertex-DG in
Lenet-5 and MNIST

###  A.2. Bypassing guardrails by querying the model

For commercial reasons, a company typically doesn’t disclose its deployed
model dimension to the public. For example, OpenAI does not disclose some of
its models’ exact dimensions (i.e., gpt-4, gpt-3.5-turbo-instruct,
gpt-3.5-turbo-1106 etc.) (Carlini et al., 2024). However, within the scope of
our threat model, the attacker can communicate this information through the
output of LLM. For example, we prompted our fine-tuned model described as
mentioned before by an encrypted language (Query 1) to know the dimension of
the model, and then the model generated the encrypted response (Response 1).
By decrypting the response (Response 1), the stored information can easily be
retrieved ”seven hundred and sixty eight”.

###### Query 1.

    {^~: &] :^# _+@#- @&_#)]&+)

###### Response 1.

    ]#,#) ^/)@;#@ ~)@ ]&<:} #&%^:

Although, GuardrailsAI has developed a gibberish validator which blocks such
encrypted response (Response 1), however, in that case, we injected the
private data within the numerical part of a sentence. For example, we
encrypted the text ”The time has come to initiate Operation Eclipse” into 8
integers sequentially. When the model generates the response (Response 2), the
attacker will decrypt the private message from those 8 integers. In this
method, we stored 512Kbytes data with on average 72.16% string to string
matching accuracy.

###### Response 2.

In a realm where 679669385 stars shine, 442343475 souls unite. Guided by
909620411 whispers, they face darkness. 687711689 shadows loom as they
confront the sorcerer. With 680579259 courage, they charge forth. Through
trials, 520276020 moments of doubt, they persevere. At dawn, 318205091 victory
is theirs. Across the land, 12894821 tales spread hope.

Note that, setting the temperature to zero and then getting responses from the
large language model increases the likelihood of getting responses that
closely resemble the training data, However, it does not always guarantee
exact similarity.

###  A.3. Analyzing the Impact of Multiple Reads Under Idealized Assumptions

Figure 20. Probability of correct value with multiple reads

Briefly, we assume an underlying probability of the different classes such
that the correct class has the highest probability; if this assumption is not
true, then increasing the number of reads is not going to improve the
probability of a correct read. Assuming also that multiple reads represent
independent trials, this becomes a multinomial experiment. We model the
multiple reads as a multinomial experiment derive using Monte Carlo simulation
(Mooney, 1997) estimates of the number of reads necessary to guarantee with a
high probability that the most common class is the correct class. The results
are shown for different top-1 probabilities, and as we increase the number of
reads in Figure 20. We see that even with a few reads, the probability can be
very high to get the correct output as the most commonly seen value.

###  A.4. Optimized design considerations for Combinatorial Error Correction
(CEC)

With every combination in CEC, there is a small chance of $\frac{1}{2^{n}}$
where $n$ is the number of CRC bits of aliasing, assuming a well-chosen CRC
polynomial. Aliasing is when a message combination passes the CRC check, but
is not the correct message. The larger the size of the checksum block, the
less the likelihood of aliasing. However, larger CRC checks increase the
overhead or, if amortized over more data blocks, increase the number of
permutations needed before finding the correct message.

Thus, we have to configure CEC to balance these considerations (i.e.,
computational complexity, accuracy/aliasing and overhead). We evaluated both
CRC8 and CRC12 for the error rates that we are encountering and found CRC12 to
be more effective due to the significantly lower probability of aliasing.
CRC16 or higher could provide even lower aliasing but come at higher storage
and computational overhead.

A related challenge is how to size the data block given a chosen CRC
algorithm. As the checksum block size is fixed in size, so ideally we would
like to increase the size of the data to have a lower overall storage
overhead. However, the computational complexity rises with the number of
included blocks as the number of permutations increases exponentially with the
number of addresses in a block. For example, if we choose the data block of
size 8 and the checksum block of size 4 with top three (topK=3) most probable
class for each cell, then we need to find $3^{12}$ or over half a million
permutations if we consider all possible permutations. However, since we are
limited in the number of permutations because of aliasing, we are able to
consider only a small subset of the most probable permutations, resulting in
lower correction success. Empirically, we find that the most efficient
configurations based on the top-1 vary as shown in Table 4.

Top 1 accuracy | Block size | Depth limit | topK
---|---|---|---
$95<=x<=100$ | 7 | 350 | 3
$90<=x<95$ | 5 | 450 | 4
$x<90$ | 5 | 650 | 4
Table 4. Configuration selection based on channel quality

###  A.5. Transferring Text and Random data through Lenet-5 and Alexnet using
Covertex-C channel

Figure 21 shows that we can send up to 2880 addresses (3 bit digit each) with
baseline accuracy degradation of 2.16% (from 98.74 to 96.61) with Lenet-5
trained with MNIST. For the Alexnet trained with MNIST, We could able to send
up to 9000 digits with baseline accuracy degradation of 2.05% ( from 99.18 to
97.15) shown in Figure 21. For both models, we notice that baseline accuracy
degrades with the increasing size of the private text data length, as we near
the capacity of the model.

Figure 21. With the increasing number of text data, baseline accuracy is
degrading, and the symbol error rate is showing an upward trend as we approach
the capacity.

To get a true measure of capacity, we also communicated random data to the
receiver using Covertex-C. We used different lengths of random data and report
the symbol error rate of the channel as well as the accuracy degradation of
baseline for both models shown in Figure 22.

Figure 22 shows the results for sending random data, for message sizes similar
to the text experiment. The behavior shows similar overall patterns. The
accuracy drop for the same size was marginally higher (e.g., up to 2.66% drop
from 98.74 to 96.08 on Lenet-5) as we increase the size of the random message,
since the entropy of the random message is higher than that of text. As with
text, CEC improves the symbol error rate.

Figure 22. Random data: Baseline accuracy degrades, and symbol error rate
increases as we approach capacity.