# Sample-specific Masks for Visual Reprogramming-based Prompting

Chengyi Cai  Zesheng Ye  Lei Feng  Jianzhong Qi  Feng Liu

###### Abstract

_Visual reprogramming_ (VR) is a prompting technique that aims to re-purpose a
pre-trained model (e.g., a classifier on ImageNet) to target tasks (e.g.,
medical data prediction) by learning a _small-scale pattern_ added into input
images instead of tuning considerable parameters within the model. The
location of the pattern within input samples is usually determined by a pre-
defined mask _shared across all samples_. In this paper, we show that the
shared mask potentially limits VR’s generalization and increases its
approximation error due to the lack of sample-level adaptation. Motivated by
this finding, we design a new framework for VR called _sample-specific multi-
channel masks_ (SMM). Specifically, SMM employs a lightweight ConvNet and
patch-wise interpolation to generate sample-specific three-channel masks
instead of a shared and pre-defined mask. Since we generate different masks
for individual samples, SMM is theoretically shown to reduce approximation
error for the target tasks compared with existing state-of-the-art VR methods.
We also empirically demonstrate its performance gain on both ResNet and ViT.
The success of SMM further highlights the broader applicability of VR in
leveraging the latent knowledge of pre-trained models for various target
tasks. Our code is available at https://github.com/tmlr-group/SMM.

Machine Learning, ICML

##  1 Introduction

Recent studies have shown that, by taking advantage of and re-purposing well-
trained/pre-trained models, one can address new tasks (i.e., _target tasks_)
without training a task-specific model from scratch (Basu et al., 2023; Kossen
et al., 2023; Mondal et al., 2023). In visual tasks, due to the expensive
training costs even just to finetune pre-trained models, _visual
reprogramming_ (VR) (Neekhara et al., 2022; Wang et al., 2022; Chen et al.,
2023; Tsao et al., 2024), or adversarial reprogramming (Elsayed et al., 2018;
Tsai et al., 2020), has been proposed to reuse pre-trained models on target
tasks. Concretely, VR is a prompting method that fixes a pre-trained model and
only alters the input space by adding some learnable patterns (usually some
noise) to target images. The location of the patterns to be learned is usually
determined by a pre-defined binary mask that is _shared across all samples_
(Elsayed et al., 2018; Yang et al., 2021; Tsai et al., 2020; Bahng et al.,
2022). The key benefit of VR methods is that learning the pattern whose size
is around the image size requires much less computing resource than finetuning
considerable parameters within the model, posing VR as a promising research
area in using pre-trained models (Chen et al., 2023; Tsao et al., 2024).

Figure 1: Drawback of shared masks over individual images. We demonstrate the
use of watermarking (Wang et al., 2022), a representative VR method, to re-
purpose an ImageNet-pretrained classifier for the OxfordPets dataset, with
different shared masks (full, medium, and narrow) in VR. An evaluation of
classification confidence across three cat images — Sphynx, Abyssinian, and
Bengal — indicates a sample-specific mask preference: Sphynx with medium,
Abyssinian with full, and Bengal with narrow. It shows that different masks
are needed for individual images.  Figure 2: Drawback of shared masks in the
statistical view. Optimal learning methods like finetuning usually result in
loss decreases for all samples (see the blue part). But when applying the same
mask in reprogramming, part of the loss changes are observed to be positive
(see the red part) according to the distribution of [final loss - initial
loss], which means the training loss for some samples even rises.

In this paper, we show that the shared mask often leads to poor generalization
capability of VR, as demonstrated in Figures 1 and 2. In both figures, we use
a representative VR method, watermarking (Bahng et al., 2022), to re-purpose
an ImageNet-pretrained classifier to classify images in the OxfordPets
datasets (Parkhi et al., 2012). In Figure 1, we first find that the optimal
masks vary among individual images. We apply three kinds of masks (full,
medium, and narrow) in watermarking. By observing the classification
confidence on three cat images: Sphynx, Abyssinian, and Bengal, we see that
the medium mask is the best for Sphynx, the full mask for Abyssinian, and the
narrow mask for Bengal. This suggests that different masks are needed for
individual images. In Figure 2, we then find that watermarking with a single
shared mask may cause the training loss of many individual samples to rise
(see the red part in Figure 2). This phenomenon reveals that VR methods’
learning capacity is much less than finetuning all parameters of the pre-
trained model (see the blue part in Figure 2).

The examples above show a significant disadvantage of using a single shared
mask for VR. This motivates our new VR framework called _sample-specific
multi-channel masks_ (SMM). SMM replaces the _fixed_ binary mask applied in
existing works with _generative_ three-channel masks that can vary across
different samples (shown in Figure 3).

SMM has two modules: a mask generator module and a patch-wise interpolation
module. The mask generator is a lightweight _convolutional neural network_
(CNN) that takes resized individual target-domain images (i.e., samples) as
the input and outputs different masks for each sample. The last layer of the
generator is designed to generate a three-channel mask, which allows better
performance for both rich-color images (i.e., CIFAR10/100 (Krizhevsky, 2009))
and monotonous-color images (i.e., SVHN (Yuval, 2011)). Since the generated
masks should be the same size as the pattern to be learned, when the size of
masks is inconsistent with that of the pattern, the patch-wise interpolation
module will be utilized to re-scale the generated masks to fit the pattern,
facilitating the training process of the mask generator (detailed in Section
3).

To understand why SMM is effective, we theoretically analyze the approximation
error of different hypothesis sets for VR. Three hypothesis sets are
considered: shared pattern with a pre-defined binary mask, sample-specific
patterns without masks, and our SMM. We show that SMM has a smaller
approximation error (Proposition 4.3), which confirms the effectiveness of
SMM.

To further substantiate the efficacy of SMM, we conduct empirical evaluations
spanning 11 widely used datasets, incorporating ablation studies that discern
the impact of individual SMM components. This is complemented by analysis and
interpretations of the generated masks, alongside a comparative visualization
of feature spaces. Notably, we demonstrate the effectiveness of SMM with both
pre-trained ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020) (Table
1 and 2), validating that SMM is compatible with commonly used classifier
architectures.

Both the theoretical analysis and promising experimental results provide solid
evidence that, when powered by SMM, VR can efficiently leverage knowledge
within a well-trained/pre-trained model for various target tasks, shedding new
light on the explanatory analysis of VR and opening avenues for future
research.

##  2 Preliminaries and Related Works

###  2.1 Problem Setting of Model Reprogramming

Model reprogramming (Chen, 2022) offers an efficient transfer learning
paradigm for adapting pre-trained models to resource-constrained target tasks.
This paradigm re-purposes existing knowledge by strategically transforming
inputs and outputs, bypassing extensive model parameter finetuning. In what
follows, we will present a formal problem setting for model reprogramming.

Let $\mathcal{D}_{\rm T}$ represent the data distribution of a target task
defined over $\mathcal{X}^{\rm T}\times\mathcal{Y}^{\rm T}$, where
$\mathcal{X}^{\rm T}\subseteq\mathbb{R}^{d_{\rm T}}$ is the data space and
$\mathcal{Y}^{\rm T}=\\{1,\dots,k_{\rm T}\\}$ is the label space, and let
$\\{(x^{\rm T}_{i},y^{\rm T}_{i})\\}_{i=1}^{n}$ be the observations of
$\mathcal{D}_{\rm T}$ (i.e., the training set in the target task). Meanwhile,
we have a pre-trained model $f_{\rm P}:\mathcal{X}^{\rm
P}\rightarrow\mathcal{Y}^{\rm P}$, where $\mathcal{X}^{\rm
P}\subseteq\mathbb{R}^{d_{\rm P}}$ and $\mathcal{Y}^{\rm P}$ (s.t.
$|\mathcal{Y}^{\rm T}|\leq|\mathcal{Y}^{\rm P}|$, with the label space of the
pre-trained task being larger than that of the target task) represent the data
and label spaces used for training $f_{\rm P}$. Then, in model reprogramming,
the training objective can be formulated as

$\displaystyle\mathop{\min}\limits_{\theta\in\Theta,\omega\in\Omega}\frac{1}{n}\sum_{i=1}^{n}\ell(f_{\rm
out}(f_{\rm P}(f_{\rm in}(x^{\rm T}_{i}|\theta))|\mathcal{Y}^{\rm P}_{\rm
sub},\omega),y^{\rm T}_{i}),$ (1)

where $f_{\rm in}(.|\theta):\mathcal{X^{\rm T}}\mapsto{\mathcal{X}}^{\rm
P},f_{\rm out}(.|\mathcal{Y}^{\rm P}_{\rm sub},\omega):\mathcal{Y}^{\rm
P}_{\rm sub}\mapsto\mathcal{Y^{\rm T}}$ are the input transformation and
output label mapping function with parameters $\theta\in\Theta$ and
$\omega\in\Omega$, $\mathcal{Y}^{\rm P}_{\rm sub}\subseteq\mathcal{Y}^{\rm P}$
can be determined by different methods (Elsayed et al., 2018; Tsai et al.,
2020; Chen et al., 2023), and $\ell:\mathcal{Y}^{\rm T}\times\mathcal{Y}^{\rm
T}\mapsto\mathbb{R}^{+}\cup\\{0\\}$ is a loss function. Reprogramming
techniques have been widely applied in visual (Elsayed et al., 2018; Tsai et
al., 2020), text (Neekhara et al., 2018; Hambardzumyan et al., 2021), speech
(Yang et al., 2021, 2023; Yen et al., 2023), music (Hung et al., 2023), and
cross-modal tasks (Neekhara et al., 2022) in the past few years.

In the context of visual tasks, reprogramming has demonstrated potential in
bio-medical measurement (Tsai et al., 2020), machine learning fairness (Zhang
et al., 2022), as well as out-of-distribution detection through watermarking
(Wang et al., 2022). Moving beyond application prospects, we next discuss the
technical details of the specific input and output mapping functions ($f_{\rm
in}$ and $f_{\rm out}$).

###  2.2 Prompting and Input Visual Reprogramming

General prompting methods in visual tasks, predominantly applied to the ViT
architecture (Dosovitskiy et al., 2020), introduce extra parameters to a pre-
trained model for enhanced training efficiency. Prompts are flexible in their
placement. For example, _visual prompt tuning_ (Jia et al., 2022) positions
prompts alongside image embedding before the encoder layers, while _effective
and efficient visual prompt tuning_ (Han et al., 2023) extends this by
incorporating parameters within self-attention layers as well. _Transformer
with hierarchical prompting_ (Wang et al., 2023) also learns prompt tokens to
represent the coarse image classes.

Meanwhile, prompting goes beyond vision foundation models to vision-language
frameworks such as CLIP (Radford et al., 2021). Methods like CoOP (Zhou et
al., 2022b) and CoCoOP (Zhou et al., 2022a) replace textual prompts with
learnable vectors for enhanced adaptability to the target task, conditioned on
input images. MaPLe (Khattak et al., 2023) further bridges vision and language
by learning layer-specific mapping functions. These methods vary from each
other in terms of both prompt placements and functions.

In contrast, VR provides a model-agnostic prompting technique, by adding
trainable noise to the input image patterns before the forward propagation,
without altering their visual essence. Originally proposed by Elsayed et al.
(2018), VR has been evolving to include padding-based methods (Tsai et al.,
2020; Chen et al., 2023) and watermarking that facilitate downstream target
tasks (Bahng et al., 2022). AutoVP (Tsao et al., 2024) stands out with its
scalable pre-padding images. A critical limitation in existing VR research is
the use of _shared_ noise patterns across _all target samples_ , neglecting
sample-level characteristics and compromising generalization. We propose SMM
to manage this gap.

###  2.3 Output Mapping of Reprogramming

Learning-based output mapping, i.e., model $f_{\rm out}$, as proposed by Chen
et al. (2023), can be simplified as a one-to-one mapping from a subset of
$\mathcal{Y^{\rm P}}$ to $\mathcal{Y^{\rm T}}$. Therefore, no additional
parameters are required. One implementation of this mapping is _random label
mapping_ (Rlm), where $f_{\rm out}$ is a randomly assigned injective function
(Elsayed et al., 2018; Chen et al., 2023), formulated as

$\displaystyle f_{\rm out}^{\rm Rlm}(y|\mathcal{Y}^{\rm P}_{\rm
sub})=\texttt{rand}(\\{0,1,...,k^{\rm T}\\}),$ (2)

where $\texttt{rand}(\\{0,1,...,k^{\rm T}\\})$ means randomly selecting one
element from the set $\\{0,1,...,k^{\rm T}\\}$, and $\mathcal{Y}^{\rm P}_{\rm
sub}$ is of the same size with $\mathcal{Y}^{\rm T}$ (i.e., $k^{\rm T}$),
randomly chosen from $\mathcal{Y}^{\rm P}$ prior to the minimization of Eq.
(1). Note that, since $f_{\rm out}^{\rm Rlm}$ is injective, it ensures $f_{\rm
out}^{\rm Rlm}(y_{1}|\mathcal{Y}^{\rm P}_{\rm sub})\neq f_{\rm out}^{\rm
Rlm}(y_{2}|\mathcal{Y}^{\rm P}_{\rm sub})$ for two distinct elements
$y_{1}\neq y_{2}$.

Other representative output-mapping methods determine $\mathcal{Y}^{\rm
P}_{\rm sub}$ and $f_{\rm out}$ for different target tasks. For example, one
is based on the frequency of label assignment in the pre-trained model and the
target data (Tsai et al., 2020), called _frequent label mapping_ (Flm). Chen
et al. (2023) propose _iterative label mapping_ (Ilm) that updates $f_{\rm
out}$ in each training iteration, reflecting changes in label mapping
throughout the learning of $f_{\rm in}$. Detailed procedures and the pseudo-
code of ${f}_{\rm out}^{\rm Flm}$ and ${f}_{\rm out}^{\rm Ilm}$ are in
Appendix A.4.

Figure 3: Comparison between (a) existing methods and (b) our method. Previous
padding-based reprogramming adds zeros around the target image, while
resizing-based reprogramming adjusts image dimensions to fit the required
input size. Both methods use a pre-determined _shared_ mask to indicate the
valid location of pattern $\delta$. Our method, on the other hand, takes a
more dynamic and tailored approach. We resize each target image and apply a
different three-channel mask accordingly, driven by a lightweight $f_{\rm
mask}$ with an interpolation up-scaling module, allowing for more variability
in individual samples.

##  3 Sample-specific Multi-channel Masks

We focus on $f_{\rm in}$, while treating $f_{\rm out}$ as a non-parametric
mapping, in line with Chen et al. (2023). We thus limit our discussion of
trainable parameters to $\theta\in\Theta$ in Eq. (1). A flowchart in Appendix
A.1 provides an overview of the problem structure of Input VR.

###  3.1 Framework of SMM

To allow both shared parameters over the whole dataset and variability among
individual samples, it is intuitive to consider the following VR hypothesis:

$\displaystyle f_{\rm in}(x_{i}|\phi,\delta)=r(x_{i})+\delta\odot f_{\rm
mask}(r(x_{i})|\phi),$ (3)

where $r:\mathcal{X}^{\rm T}\rightarrow\mathbb{R}^{d_{\rm P}}$ is the resizing
function, typically implemented as bilinear interpolation upsampling
(Wikipedia contributors, 2023) that scales image dimension from $d_{\rm{T}}$
to $d_{\rm{P}}$, and $r(x_{i})\in\mathbb{R}^{d_{\rm P}}$ is the resized image
corresponding to $x_{i}$. The mask generation function $f_{\rm
mask}:{\mathbb{R}}^{d_{\rm P}}\rightarrow\mathbb{R}^{d_{\rm P}}$,
parameterized by $\phi\in\Phi$, produces a mask indicating the noise
placements for each image. We denote a trainable noise pattern added to the
image by $\delta\in\mathbb{R}^{d_{\rm P}}$. The rationale for applying this
hypothesis is elaborated in Proposition 4.3 and validated in ablation studies
(cf. Table 3). This casts the training objective of our SMM framework
($\theta=\left\\{\phi,\delta\right\\}$) to find the optimal $\phi^{*}$ and
$\delta^{*}$ such that

$\begin{split}\mathop{\arg\min}\limits_{\phi\in\Phi,\delta\in\mathbb{R}^{d_{\rm
P}}}\mathbb{E}_{(x_{i},y_{i})\sim\mathcal{D_{\rm T}}}[\ell(&f_{\rm out}(f_{\rm
P}(r(x_{i})+\\\ &\delta\odot f_{\rm
mask}(r(x_{i})|\phi))),y_{i})].\end{split}$ (4)

Note that $\delta$ is _shared_ by all images in the dataset following Bahng et
al. (2022) and Chen et al. (2023), while $f_{\rm mask}$ _uniquely_ generates
sample-specific multi-channel masks for each individual image, enabling
sample-specific adaptation.

Figure 3 illustrates the workflow of SMM, as well as previous padding-based
and resizing-based (i.e., watermarking) VR methods. Compared with previous
works, SMM features $f_{\rm mask}(\cdot|\phi)$, integrating a mask generator
module and a patch-wise interpolation module. Concretely, SMM starts by
resizing target images, followed by their processing through the mask
generator to create corresponding three-channel masks. For generated masks
smaller than the pattern size, the patch-wise interpolation module performs
upsampling, which omits the derivation step in back-propagation and
facilitates training. Afterward, the learnable pattern $\delta$ is multiplied
with the mask on a pixel-wise basis and added to the image. The resulting
image is fed into the _fixed_ pre-trained classifier. We discuss further
details on the mask generator (Section 3.2), the patch-wise interpolation
module (Section 3.3), and the overall learning strategy presented in Eq. (4)
(Section 3.4).

###  3.2 Lightweight Mask Generator Module

The mask generator $f_{\rm mask}$ is supposed to output a mask that has the
same size as the input image while prioritizing different locations for
$\delta$ to allow more variability. We employ a CNN as the mask generator.
This choice stems from the proficiency of CNNs in mirroring localized visual
perception (He et al., 2016) with fewer parameters than most deep learning
structures, e.g., multilayer perceptrons.

The input of CNN is a resized image $r(x_{i})$. Applying our bespoke CNN
architecture shown in Appendix A.2, the output will be a three-channel mask
with dimensions
$\left\lfloor\frac{H}{2^{l}}\right\rfloor\times\left\lfloor\frac{W}{2^{l}}\right\rfloor$,
where $H$ and $W$ denote image height and width, respectively, and $l$ denotes
the number of pooling layers. The analysis of input/output sizes and parameter
quantity statistics are in Appendix A.2.

Algorithm 1 Visual Reprogramming with SMM

1: Input: Pre-trained model $f_{\rm P}$, loss $\ell$, label-mapping function
$f^{(j)}_{\rm out}$ for iteration $j$, target domain training data
$\\{(x_{i},y_{i})\\}^{n}_{i=1}$, maximum number of iterations $E$, learning
rate $\alpha_{1}$ for $\delta$ and $\alpha_{2}$ for $\phi$

2: Output: Optimal $\delta^{*},\phi^{*}$

3: Initialize $\phi$ randomly; set $\delta\leftarrow\\{0\\}^{d_{\mathcal{\rm
P}}}$

4: for $j=1$ to $E$ do

5: # Step1: Compute individual marks using the mask generator# Step2: Resize
masks using the patch-wise interpolation module $f_{\rm
in}(x_{i};\delta,\phi)\leftarrow r(x_{i})+\delta\odot f_{\rm
mask}(r(x_{i})|\phi)$, $\forall i=1,2,...,n$

6: # Compute the classification loss
$L(\delta,\phi)\leftarrow\frac{1}{n}\sum^{n}_{i=1}\ell(f^{(j)}_{\rm
out}{(f_{\rm P}(f_{\rm in}(x_{i};\delta,\phi)))},y_{i})$

7: $\delta\leftarrow\delta-\alpha_{1}\nabla_{\delta}L(\delta,\phi)$

8: $\phi\leftarrow\phi-\alpha_{2}\nabla_{\phi}L(\delta,\phi)$

9: end for

###  3.3 Patch-wise Interpolation Module

The patch-wise interpolation module upscales CNN-generated masks from
$\left\lfloor\frac{H}{2^{l}}\right\rfloor\times\left\lfloor\frac{W}{2^{l}}\right\rfloor$
back to the original size $H\times W$ per channel (it is omitted when $l=0$).
Considering the inherent consistency in adjacent image areas and the benefits
of concise operations for gradient calculations, we employ a grid of
$\left\lfloor\frac{H}{2^{l}}\right\rfloor\times\left\lfloor\frac{W}{2^{l}}\right\rfloor$
patches in the upsampling process, each sized $2^{l}\times 2^{l}$, ensuring
the same values within each patch, with non-divisible cases mirroring the
closest patches. Therefore, after obtaining the output of CNN, we enlarge each
pixel to $2^{l}\times 2^{l}$ pixels by padding the same value of a pixel to
its surrounding areas within the patch.

Unlike traditional interpolation methods which may introduce complicated
derivation computations, our module simplifies the training by directly
assigning values. The advantage of patch-wise interpolation over traditional
interpolation methods will be discussed in Appendix A.3. The effect of patch
size $2^{l}$ will be discussed in Section 5.

###  3.4 Learning Strategy

The learning process for the shared noise pattern $\delta$ and the mask
generator $f_{\rm mask}$ is shown in Algorithm 1. The parameters $\delta$ and
$\phi$ are iteratively updated in each epoch. To mitigate the impact of
initialization, $\delta$ is set to be a zero matrix before training, noted as
$\\{0\\}^{d_{\mathcal{\rm P}}}$.

Table 1: Performance Comparison of Different Input Reprogramming Methods on
Pre-trained ResNet (Mean % ± Std %, the average results across all datasets
are highlighted in grey)

Pre-trained | ResNet-18 (ImageNet-1k) | ResNet-50 (ImageNet-1k)
---|---|---
Methods | Pad | Narrow | Medium | Full | Ours | Pad | Narrow | Medium | Full | Ours
CIFAR10 |  65.5 ±0.1 |  68.6 ±2.8 |  68.8 ±1.1 |  68.9 ±0.4 |  72.8 ±0.7 |  76.6±0.3 |  77.4±0.5 |  77.8+0.2 |  79.3±0.3 |  81.4±0.6
CIFAR100 |  24.8±0.1 |  36.9±0.6 |  34.9±0.2 |  33.8±0.2 |  39.4±0.6 |  38.9±0.3 |  42.5±0.2 |  43.8±0.2 |  47.2±0.1 |  49.0±0.2
SVHN |  75.2±0.2 |  58.5±1.1 |  71.1±1.0 |  78.3±0.3 |  84.4±2.0 |  75.8±0.4 |  59.1±1.3 |  71.5±0.8 |  79.5±0.5 |  82.6±2.0
GTSRB |  52.0±1.2 |  46.1±1.5 |  56.4±1.0 |  76.8±0.9 |  80.4±1.2 |  52.5±1.4 |  38.9±1.3 |  52.6±1.3 |  76.5±1.3 |  78.2±1.1
Flowers102 |  27.9±0.7 |  22.1±0.1 |  22.6±0.5 |  23.2±0.5 |  38.7±0.7 |  24.6±0.6 |  19.9±0.6 |  20.9±0.6 |  22.6±0.1 |  35.9±0.5
DTD |  35.3±0.9 |  33.1±1.3 |  31.7±0.5 |  29.0±0.7 |  33.6±0.4 |  40.5±0.5 |  37.8±0.7 |  38.4±0.2 |  34.7±1.3 |  41.1±1.1
UCF101 |  23.9±0.5 |  27.2±0.9 |  26.1±0.3 |  24.4±0.9 |  28.7±0.8 |  34.6±0.2 |  38.4±0.2 |  37.2±0.2 |  35.2±0.2 |  38.9±0.5
Food101 |  14.8±0.2 |  14.0±0.1 |  14.4±0.3 |  13.2±0.1 |  17.5±0.1 |  17.0±0.3 |  18.3±0.2 |  18.3±0.2 |  16.7±0.2 |  19.8±0.0
SUN397 |  13.0±0.2 |  15.3±0.1 |  14.2±0.1 |  13.4±0.2 |  16.0±0.3 |  20.3±0.2 |  22.0±0.1 |  21.5±0.1 |  21.1±0.1 |  22.9±0.0
EuroSAT |  85.2±0.6 |  82.8±0.4 |  83.8±0.5 |  84.3±0.5 |  92.2±0.2 |  83.6±0.7 |  83.7±0.4 |  85.8±0.1 |  86.9±0.3 |  92.0±0.6
OxfordPets |  65.4±0.7 |  73.7±0.2 |  71.4±0.2 |  70.0±0.6 |  74.1±0.4 |  76.2±0.6 |  76.4±0.3 |  75.6±0.3 |  73.4±0.3 |  78.1±0.2
Average | 43.91 | 43.48 | 45.04 | 46.85 | 52.53 | 49.15 | 46.76 | 49.39 | 52.10 | 56.35

##  4 Understanding Masks in Visual Reprogramming for Classification

In this section, we will demonstrate that SMM enables stronger model learning
capacity than previous representative VR methods, via showing reduced
approximation error in the _probably approximately correct_ (PAC) learning
framework (Kearns & Vazirani, 1994). We first present the definition of the
approximation error in PAC learning.

######  Definition 4.1 (Approximation Error).

Consider an input space $\mathcal{X}$, a discrete label space $\mathcal{Y}$, a
random variable $(X,Y)$ whose distribution $\mathcal{D}$ is defined on
$\mathcal{X}\times\mathcal{Y}$ with a joint probability density function
$p(x,y)$, and a hypothesis space
$\mathcal{F}=\\{f:\mathcal{X}\rightarrow\mathcal{Y}\\}$. The approximation
error of $\mathcal{F}$ on $\mathcal{D}$ is

$\displaystyle{\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F})=\inf_{f\in\mathcal{F}}\mathbb{E}_{(X,Y)\sim\mathcal{D}}\ell(f(X),Y)-R^{*}_{\mathcal{D}},$
(5)

where $\ell:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}^{+}\cup\\{0\\}$
is a loss function, and $R^{*}_{\mathcal{D}}$ is the Bayes risk (Snapp & Xu,
1995) on $\mathcal{D}$ defined by

$\displaystyle
R^{*}_{\mathcal{D}}=\int_{\mathcal{X}}\Big{[}1-\sup_{y\in\mathcal{Y}}{\rm
Pr}(y|x)\Big{]}p_{X}(x)dx.$ (6)

Here, ${\rm Pr}(y|x)$ is the posterior probability of class $y$ conditioned on
observing $x$, and $p_{X}(x)=\sum_{y\in\mathcal{Y}}p(x,y)$ is the marginal
distribution of $X$.

The approximation error of a hypothesis space $\mathcal{F}$ measures the
closeness of the minimum achievable error by $\mathcal{F}$ to the theoretical
minimum error on distribution $\mathcal{D}$. In general, increasing the
complexity of $\mathcal{F}$ tends to reduce the approximation error. In the
following theorem, we show a connection between two approximation errors when
hypothesis spaces exhibit a subset relation.

######  Theorem 4.2.

Given an input space $\mathcal{X}$, a discrete label space $\mathcal{Y}$, and
a distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}$, if there are
two hypothesis spaces
$\mathcal{F}_{1}\subseteq\\{f:\mathcal{X}\rightarrow\mathcal{Y}\\}$ and
$\mathcal{F}_{2}\subseteq\\{f:\mathcal{X}\rightarrow\mathcal{Y}\\}$ satisfying
that $\mathcal{F}_{1}\subseteq\mathcal{F}_{2}$, then we have ${\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F}_{1})\geq{\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F}_{2})$.

Theorem 4.2 (proof in Appendix B.1) shows that understanding the subset
relation between two hypothesis spaces is key to deriving their connections in
their approximation errors. Next, we will define two hypothesis spaces: one
induced by a shared mask and the other induced by SMM.

Hypothesis Space Induced by A Shared Mask. VR methods with a shared mask
(Chen, 2022; Bahng et al., 2022) assume that, for each sample $x_{i}$, the
mask is a constant matrix $M\in\\{0,1\\}^{d_{\mathcal{\rm P}}}$. Thus, given a
fixed pre-trained model $f_{\rm P}$ and a fixed output mapping function
$f_{\rm out}$ (for simplicity, we use $f_{\rm P}^{\prime}$ to represent
$f_{\rm out}\circ f_{\rm P}$ in this section), the hypothesis space induced by
a shared mask is

$\displaystyle\mathcal{F}^{\rm shr}({f_{\rm P}^{\prime}})=\\{f|f(x)=f_{\rm
P}^{\prime}(r(x)+M\odot\delta),\forall x\in\mathcal{X}\\},$

where $\delta\in\mathbb{R}^{d_{\rm P}}$. In padding-based reprogramming
methods, $M$ is a fixed mask determined by the location of the target image
(Chen, 2022). The locations where $x_{i}$ is placed – usually the center of
$r(x_{i})$ – are denoted as $\\{i:M_{i}=0\\}$, which are excluded from further
training. The rest of the locations, denoted by $\\{i:M_{i}=1\\}$, indicate
trainable parameters $\delta$. In watermarking-based methods (Bahng et al.,
2022), $x_{i}$ is up-sampled to $r(x_{i})$, and $\\{i:M_{i}=1\\}$ denotes
effective locations of $\delta$ added to $r(x_{i})$.

Hypothesis Space Induced by SMM. Based on Eq. (4), we can obtain the
hypothesis space used in SMM:

$\displaystyle\mathcal{F}^{\rm smm}({f_{\rm P}^{\prime}})$ $\displaystyle=$
$\displaystyle~{}\\{f|f(x)={f_{\rm P}^{\prime}}(r(x)+f_{\rm
mask}(r(x))\odot\delta),\forall x\in\mathcal{X}\\}.$

Note that, $f_{\rm mask}(r(x))$ belongs to $\mathbb{R}^{d_{\rm P}}$ instead of
$\\{0,1\\}^{d_{\mathcal{\rm P}}}$ like $M$. Next, we analyze the relation
between the approximation errors of previous VR methods and SMM.

SMM Has a Lower Approximation Error. Based on Theorem 4.2 and the two
hypothesis spaces above, we have the following proposition.

######  Proposition 4.3.

Given a fixed pre-trained model $f_{\rm P}$, a fixed output mapping function
$f_{\rm out}$, and the definitions of $\mathcal{F}^{\rm shr}$ and
$\mathcal{F}^{\rm smm}$, we have $\mathcal{F}^{\rm
shr}(f_{P}^{\prime})\subseteq\mathcal{F}^{\rm smm}(f_{P}^{\prime})$. Then,
based on Theorem 4.2, we have

$\displaystyle{\rm Err}^{\rm apx}_{\mathcal{D}_{\rm T}}(\mathcal{F}^{\rm
shr}(f_{P}^{\prime}))\geq{\rm Err}^{\rm apx}_{\mathcal{D}_{\rm
T}}(\mathcal{F}^{\rm smm}(f_{P}^{\prime})),$ (7)

where $f_{P}^{\prime}=f_{\rm out}\circ f_{\rm P}$, $f_{\rm mask}$ used in
$\mathcal{F}^{\rm smm}(f_{P}^{\prime})$ is a CNN demonstrated in Section 3.2,
and $\mathcal{D}_{\rm T}$ denotes the distribution of the target task.

Proposition 4.3 (see its proof in Appendix B.2) shows that SMM achieves a
lower approximation error than previous shared-mask VR methods.

Estimation Error Analysis of SMM. While a lower approximation error does not
suffice to guarantee a lower excess risk, the model complexity added to
$\mathcal{F}^{\rm smm}(f_{P}^{\prime})$ is manageable in this VR setting,
since $f_{\rm mask}$ introduces less than $0.2\%$ extra parameters111See Table
4 for statistics on network sizes. relative to $f_{\rm P}^{\prime}$. Such
dominance of $f_{\rm P}^{\prime}$ suggests that the estimation error of
$\mathcal{F}^{\rm smm}(f_{P}^{\prime})$ does not significantly exceed that of
$\mathcal{F}^{\rm shr}(f_{P}^{\prime})$ and is unlikely to offset its
advantage in approximation error. We also provide an empirical justification
from the standpoint of over-fitting to show that the additional estimation
error of $\mathcal{F}^{\rm smm}(f_{P}^{\prime})$ is negligible in Appendix
D.3. By comparing the disparities in training and testing performance, we
demonstrate that SMM does not increase the risk of model over-fitting,
implying negligible estimation error.

Excess Risk Analysis of SMM. According to excess risk decomposition222The
excess risk is equal to the sum of approximation error and estimation error
(Lauer, 2014)., SMM is also expected to have a lower excess risk and,
consequently, superior generalization capability compared to shared-mask VR
methods.

Analysis Based on Sample-specific Patterns. Having built the concept of
“sample-specific”, we also investigate an alternative to the proposed SMM:
directly learning a _sample-specific pattern_ for each image without involving
$\delta$. The hypothesis space in this context can be expressed by

$\displaystyle\mathcal{F}^{\rm sp}({f_{\rm P}^{\prime}})=\\{f|f(x)={f_{\rm
P}^{\prime}}(r(x)+f_{\rm mask}(r(x))),\forall x\in\mathcal{X}\\}.$

It is easy to check that $\mathcal{F}^{\rm
sp}({f_{P}^{\prime}})\subseteq\mathcal{F}^{\rm smm}({f_{P}^{\prime}})$,
implying that ${\rm Err}^{\rm apx}_{\mathcal{D}_{\rm T}}(\mathcal{F}^{\rm
sp}(f_{P}^{\prime}))\geq{\rm Err}^{\rm apx}_{\mathcal{D}_{\rm
T}}(\mathcal{F}^{\rm smm}(f_{P}^{\prime}))$ (proof in Appendix B.3). Namely,
SMM has a lower approximation error compared to directly learning a sample-
specific pattern.

##  5 Experiments

Pre-trained Models and Target Tasks. Following Chen et al. (2023), we use
ResNet-18, and ResNet-50 (He et al., 2016) as the pre-trained model.
Performance on pre-trained ViT-B32 (Dosovitskiy et al., 2020) is also tested.
All these models are pre-trained on ImageNet-1K (Deng et al., 2009), and
target tasks include CIFAR10, CIFAR100 (Krizhevsky, 2009), SVHN (Yuval, 2011),
GTSRB (Houben et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), DTD
(Cimpoi et al., 2014), UCF101 (Soomro et al., 2012), Food101 (Bossard et al.,
2014), EuroSAT (Helber et al., 2019), OxfordPets (Parkhi et al., 2012), SUN397
(Xiao et al., 2010). Moreover, StanfordCars (Krause et al., 2013), which is
revealed to be unsuitable for VR, is also discussed in Appendix 12. We follow
Chen et al. (2023) to split the datasets. Detailed dataset information is
included in Appendix C.

Baselines. We compare our method with both padding-based (Chen et al., 2023)
and resizing-based methods (Bahng et al., 2022), including: (1) _Pad_ :
centering the original image and adding the noise pattern around the images,
(2) _Narrow_ : adding a narrow padding binary mask with a width of 28
($\frac{1}{8}$ of the input image size) to the noise pattern that covers the
whole image (watermark), (3) _Medium_ : adding a mask being a quarter of the
size (the width is 56) of watermarks and (4) _Full_ : full watermarks that
cover the whole images following Wang et al. (2022). To ensure that all the
methods are fairly compared, in training the shared noise pattern, we apply
the same learning rate and milestones following Chen et al. (2023), with 0.01
being the initial learning rate and 0.1 being the learning rate decay. Two
hundred epochs are run in total, and the 100$th$ and the 145$th$ epochs are
the milestones. The training details of the mask generator are included in
Appendix C. Experiments are run with three seeds on a single A100 GPU and the
averaged test accuracy is reported. Due to page limits, we report here only
the results obtained with the output mapping ${f}_{\rm out}^{\rm Ilm}$. See
Appendix D.1 for the results using ${f}_{\rm out}^{\rm Rlm}$ and ${f}_{\rm
out}^{\rm Flm}$.

Figure 4: Comparative results of different patch sizes ($2^{l}$). ResNet-18 is
used as the pre-trained model as an example.

Results on ResNets. Table 1 reports the accuracy of ResNet-18 and ResNet-50
using VR methods with the baseline shared marks and our proposed SMM method.
It can be observed that our SMM yields higher accuracy for both models on all
datasets tested except for ResNet-18 on DTD. The advantage is more pronounced
on the datasets where the target domains are more different from the original
domain, such as SVHN, Flowers102, and EuroSAT. On SVHN, 6.1% and 3.1%
improvements have been witnessed for ResNet-18 and ResNet-50, respectively,
while over 10% improvement is observed on the Flowers102. On DTD, the padding-
based method has better results for ResNet-18. This is likely to be due to the
noisy watermarks adversely impacting the texture that needs to be classified,
leading to the disadvantages of resizing-based methods. Even in this
challenging setting, our SMM method leads to higher accuracy when applied on
the larger pre-trained model ResNet-50.

Table 2: Performance Comparison of Different Input Reprogramming Methods on
Pre-trained ViT (Mean %, the average results are highlighted in grey)

Pre-trained | ViT-B32 (ImageNet-1k)
---|---
Method |  Pad |  Narrow |  Medium |  Full |  Ours
CIFAR10 |  62.4 |  96.6 |  96.5 |  95.8 |  97.4
CIFAR100 |  31.6 |  74.4 |  75.3 |  75.0 |  82.6
SVHN |  80.2 |  85.0 |  87.4 |  87.8 |  89.7
GTSRB |  62.3 |  57.8 |  68.6 |  75.5 |  80.5
Flowers102 |  57.3 |  55.3 |  56.6 |  55.9 |  79.1
DTD |  43.7 |  37.3 |  38.5 |  37.7 |  45.6
UCF101 |  33.6 |  44.5 |  44.8 |  40.9 |  42.6
Food101 |  37.4 |  47.3 |  48.6 |  49.4 |  64.8
SUN397 |  21.8 |  29.0 |  29.4 |  28.8 |  36.7
EuroSAT |  95.9 |  90.9 |  90.9 |  89.1 |  93.5
OxfordPets |  57.6 |  82.5 |  81.0 |  75.3 |  83.8
Average |  53.1 |  63.7 |  65.2 |  64.7 |  72.4

Table 3: Ablation Studies (Mean % ± Std %, with ResNet-18 as an example, and
the average results are highlighted in grey)

|  only $\delta$ |  only $f_{\rm mask}$ |  Single- channel $f_{\rm mask}^{\rm s}$ |  Ours
---|---|---|---|---
CIFAR10 |  68.9±0.4 |  59.0±1.6 |  72.6±2.6 |  72.8±0.7
CIFAR100 |  33.8±0.2 |  32.1±0.3 |  38.0±0.6 |  39.4±0.6
SVHN |  78.3±0.3 |  51.1±3.1 |  78.4±0.2 |  84.4±2.0
GTSRB |  76.8±0.9 |  55.7±1.2 |  70.7±0.8 |  80.4±1.2
Flowers102 |  23.2±0.5 |  32.2±0.4 |  30.2±0.4 |  38.7±0.7
DTD |  29.0±0.7 |  27.2±0.5 |  32.7±0.5 |  33.6±0.4
UCF101 |  24.4±0.9 |  25.7±0.3 |  28.0±0.3 |  28.7±0.8
Food101 |  13.2±0.1 |  13.3±0.1 |  15.8±0.1 |  17.5±0.1
SUN397 |  13.4±0.2 |  10.5±0.1 |  15.9±0.1 |  16.0±0.3
EuroSAT |  84.3±0.5 |  89.2±0.9 |  90.6±0.5 |  92.2±0.2
OxfordPets |  70.0±0.6 |  72.5±0.3 |  73.8±0.6 |  74.1±0.4
Average | 46.85 | 42.59 | 49.70 | 52.53

Results on ViT. Recall that input reprogramming can be applied to diverse pre-
trained classifiers, we next turn our focus on ViT. Detailed in Table 2, our
comparative study with baselines reveals substantial performance gains in
datasets like Flowers102 (21.8%), Food101 (15.4%), and SUN397 (7.3%). These
results suggest that SMM may yield even higher performance gains for larger
pre-trained models. Exceptions do exist, like on EuroSAT, where all resizing-
based methods show marginal under-performance, possibly a result of over-
fitting on relatively simpler datasets. On UCF101, our SMM initially lags
behind other strategies like narrow or medium masking but, after choosing
appropriate learning rate parameters (See Appendix C), could achieve a leading
49.9% accuracy. Overall, the experiments above show the applicability of SMM
over different pre-trained models and target domains. Abnormal cases of SMM in
Table 1 and Table 2 will be further discussed in Appendix D.4. Next, we report
ablation and parameter study results.

Figure 5: Visual results of trained VR on the Flowers102 dataset. To show the
difference in results, the original image, result image and SMM adopt
histogram equalization. ResNet-18 is used as the pre-trained model as an
example. Other visualization results and further analysis are included in
Appendix F. Figure 6: TSNE visualization results of the feature space on (a)
SVHN and (b) EuroSAT datasets. ResNet-18 is used as the pre-trained model as
an example.

Impact of Masking. We first investigate the impact of different masking
strategies. We take three variants against the proposed SMM into comparison:
(i) Shared-pattern VR $f_{\rm in}(x_{i})=r(x_{i})+\delta$, with $M$ being an
all-one matrix equal to the image dimension for maximal flexibility in
$\delta$. It defaults to the “full watermarks” baseline without using $f_{\rm
mask}$. (ii) Sample-specific pattern without masking $f_{\rm
in}(x_{i})=r(x_{i})+f_{\rm mask}(r(x_{i}))$. (iii) Single-channel version of
SMM $f_{\rm in}(x_{i})=r(x_{i})+\delta\odot f^{\rm s}_{\rm mask}(r(x_{i}))$,
averaging the penultimate-layer output of the mask generator. These variants
refer to the first three columns of Table 3, respectively. They help evaluate
the impact of sample specificity, masking, and multiple channels introduced by
SMM in the context of input VR.

As shown in Table 3, SMM consistently stands out as the best performer on all
datasets. A key observation is that only keeping shared pattern $\delta$
reduces VR effectiveness in feature-rich datasets (e.g., CIFAR10, Flowers102,
and UCF101). Besides, using only $f_{\rm mask}$ without $\delta$, leads to
suboptimal performance on datasets with enough training data per class,
including CIFAR10, SVHN, GTSRB, and SUN397. Moreover, the single-channel
method is less effective, especially on datasets where images have fewer
varying color palettes (e.g., GTSRB and Flowers102). Overall, we find that the
shared noise in SMM boosts model performance if sufficient training data is
provided, whereas the sample-specific $f_{\rm mask}$ enables specificity for
classification tasks demanding detailed feature discrimination. Lastly, the
multi-channel allows for adjusting to channel-specific priorities.

Impact of Patch Size. As an important hyperparameter in SMM, number of Max-
Pooling layers, $l$, can vary, which means different patch sizes $2^{l}$.
Since the 5-layer mask generator neural network has at most 4 Max-Pooling
layers, we examine the impact of patch sizes in $\\{2^{0}$, $2^{1}$, $2^{2}$,
$2^{3}$, $2^{4}\\}$. Results are shown in Figure 4. As the patch size
increases, the accuracy of the SMM increases first, followed by a plateau or
decline. This suggests that overly small patches may cause over-fitting, while
overly large patch sizes could result in a loss of details in SMM. We thus
have set the patch size to be 8 across all datasets.

Visualization of SMM, shared patterns and output reprogrammed images.
Visualization results on Flowers102 dataset is shown in Figure 5. It can be
observed that when classifying passion flowers, where pedals are important for
classification accuracy, the masks tend to mask out the noise pattern over the
pedals, which protects useful information from being shadowed by noise. Other
features such as flower pistils in passion flowers are also widely present in
various similar classes such as ‘oxeye’, ‘daisy’ and ‘orange dahlia’, making
the centers of flowers potential sources of interference in classification.
Thus, for passion flowers, noise in the center of the flowers is not masked
out. When classifying ‘water lily’, SMM will enhance the noise on interfering
objects in the image. Similarly, when classifying ‘cyclamen’, similar stems
are also commonly found in other classes such as ‘gaura’ and ‘rose’, which
hinders accurate classification. Therefore, it is reasonable for SMM to
introduce more noise to these interfering components. These results show that
SMM is able to retain the important parts of the image and remove the
interference.

Feature Space Visualization Results. Figure 6 shows the tSNE (Van der Maaten &
Hinton, 2008) visualization results of the output layer feature before the
label mapping layer. Before applying VR methods, the target domain’s output
feature space shows limited class separation. With the baseline methods, we
observe enhanced but incomplete separations, where certain class pairs (such
as ‘3, 5’ and ‘6, 8’ in SVHN, ‘River’ and ‘highway or road’ in EuroSAT) remain
indistinguishable in the feature space. By applying $f_{\rm mask}$, our method
successfully resolves incorrectly clustered classes, underscoring the
effectiveness of SMM.

Comparison with Finetuning-based Methods. In Appendix E, we compare our SMM
with two prevalent finetuning approaches: finetuning fully connected layers
and low-rank adaptation (Zhu et al., 2023). This comparison highlights two key
benefits of input VR: (1) its efficacy in target tasks with lower-resolution
images and (2) its orthogonal relationship to, yet compatibility with,
finetuning methods. Additionally, Appendix E provides a comprehensive
discussion on the strengths and weaknesses of Input VR in comparison to
finetuning techniques.

More Experiments. The training curves are plotted and analyzed in Appendix
D.2. The effectiveness of SMM when learning with different $f_{\rm out}$ is
discussed in Appendix D.1.

##  6 Conclusion

In this paper, we identified significant shortcomings in the use of a shared
mask across all samples in previous VR practices, notably its failure to
accommodate sample diversity, leading to increased training loss of particular
samples. In response, we proposed a new SMM learning framework, integrating a
lightweight neural net-based mask generator to generate three-channel masks
per sample, and a patch-wise interpolation module that resizes and aligns
masks to model input. Both theoretical justification and experimental results
validated the effectiveness of our proposed method.

## Acknowledgements

CYC and FL are supported by the Australian Research Council (ARC) with grant
number DE240101089, and FL is also supported by the ARC with grant number
DP230101540 and the NSF&CSIRO Responsible AI program with grant number
2303037. JZQ is supported by ARC with grant number DP240101006. This research
is also supported by The University of Melbourne’s Research Computing Services
and the Petascale Campus Initiative. We sincerely appreciate the time and
dedication of the reviewers in carefully reviewing our manuscript.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none of
which we feel must be specifically highlighted here.

## References

  * Bahng et al. (2022) Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P.  Exploring visual prompts for adapting large-scale models.  _arXiv preprint arXiv:2203.17274_ , 2022.
  * Basu et al. (2023) Basu, S., Katdare, P., Sattigeri, P., Chenthamarakshan, V., Driggs-Campbell, K. R., Das, P., and Varshney, L. R.  Efficient equivariant transfer learning from pretrained models.  In _NeurIPS_ , 2023.
  * Bossard et al. (2014) Bossard, L., Guillaumin, M., and Van Gool, L.  Food-101–mining discriminative components with random forests.  In _ECCV_ , 2014.
  * Chen et al. (2023) Chen, A., Yao, Y., Chen, P.-Y., Zhang, Y., and Liu, S.  Understanding and improving visual prompting: A label-mapping perspective.  In _CVPR_ , 2023.
  * Chen (2022) Chen, P.-Y.  Model reprogramming: Resource-efficient cross-domain machine learning.  _arXiv preprint arXiv:2202.10629_ , 2022.
  * Cimpoi et al. (2014) Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A.  Describing textures in the wild.  In _CVPR_ , 2014.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Elsayed et al. (2018) Elsayed, G. F., Goodfellow, I., and Sohl-Dickstein, J.  Adversarial reprogramming of neural networks.  In _ICLR_ , 2018.
  * Hambardzumyan et al. (2021) Hambardzumyan, K., Khachatrian, H., and May, J.  Warp: Word-level adversarial reprogramming.  In _ACL-IJCNLP_ , 2021.
  * Han et al. (2023) Han, C., Wang, Q., Cui, Y., Cao, Z., Wang, W., Qi, S., and Liu, D.  E^ 2vpt: An effective and efficient approach for visual prompt tuning.  _arXiv preprint arXiv:2307.13770_ , 2023.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Helber et al. (2019) Helber, P., Bischke, B., Dengel, A., and Borth, D.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.
  * Houben et al. (2013) Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., and Igel, C.  Detection of traffic signs in real-world images: The german traffic sign detection benchmark.  In _IJCNN_ , 2013.
  * Hu et al. (2021) Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Hung et al. (2023) Hung, Y.-N., Yang, C.-H. H., Chen, P.-Y., and Lerch, A.  Low-resource music genre classification with cross-modal neural model reprogramming.  In _ICASSP_ , 2023.
  * Jia et al. (2022) Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N.  Visual prompt tuning.  In _ECCV_ , 2022.
  * Kearns & Vazirani (1994) Kearns, M. J. and Vazirani, U.  _An introduction to computational learning theory_.  MIT press, 1994.
  * Khattak et al. (2023) Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S.  Maple: Multi-modal prompt learning.  In _CVPR_ , 2023.
  * Kossen et al. (2023) Kossen, J., Collier, M., Mustafa, B., Wang, X., Zhai, X., Beyer, L., Steiner, A., Berent, J., Jenatton, R., and Kokiopoulou, E.  Three towers: Flexible contrastive learning with pretrained image models.  _arXiv preprint arXiv:2305.16999_ , 2023.
  * Krause et al. (2013) Krause, J., Stark, M., Deng, J., and Fei-Fei, L.  3d object representations for fine-grained categorization.  In _ICCV workshops_ , 2013.
  * Krizhevsky (2009) Krizhevsky, A.  Learning multiple layers of features from tiny images.  _Master’s thesis, University of Tront_ , 2009.
  * Lauer (2014) Lauer, F.  An interactive journey into machine learning, 2014.  URL https://mlweb.loria.fr/book/en/estimationapproximationerrors.html.
  * Mondal et al. (2023) Mondal, A. K., Panigrahi, S. S., Kaba, S.-O., Rajeswar, S., and Ravanbakhsh, S.  Equivariant adaptation of large pre-trained models.  _arXiv preprint arXiv:2310.01647_ , 2023.
  * Neekhara et al. (2018) Neekhara, P., Hussain, S., Dubnov, S., and Koushanfar, F.  Adversarial reprogramming of text classification neural networks.  _arXiv preprint arXiv:1809.01829_ , 2018.
  * Neekhara et al. (2022) Neekhara, P., Hussain, S., Du, J., Dubnov, S., Koushanfar, F., and McAuley, J.  Cross-modal adversarial reprogramming.  In _WACV_ , 2022.
  * Nilsback & Zisserman (2008) Nilsback, M.-E. and Zisserman, A.  Automated flower classification over a large number of classes.  In _Indian Conference on Computer Vision, Graphics & Image Processing_, 2008.
  * Parkhi et al. (2012) Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C.  Cats and dogs.  In _CVPR_ , 2012.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Snapp & Xu (1995) Snapp, R. and Xu, T.  Estimating the bayes risk from sample data.  In _NeurIPS_ , 1995.
  * Soomro et al. (2012) Soomro, K., Zamir, A. R., and Shah, M.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Tsai et al. (2020) Tsai, Y.-Y., Chen, P.-Y., and Ho, T.-Y.  Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources.  In _ICML_ , 2020.
  * Tsao et al. (2024) Tsao, H.-A., Hsiung, L., Chen, P.-Y., Liu, S., and Ho, T.-Y.  AutoVP: an automated visual prompting framework and benchmark.  In _ICLR_ , 2024.
  * Van der Maaten & Hinton (2008) Van der Maaten, L. and Hinton, G.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 2008.
  * Wang et al. (2022) Wang, Q., Liu, F., Zhang, Y., Zhang, J., Gong, C., Liu, T., and Han, B.  Watermarking for out-of-distribution detection.  _NeurIPS_ , 2022.
  * Wang et al. (2023) Wang, W., Sun, Y., Li, W., and Yang, Y.  Transhp: Image classification with hierarchical prompting.  _arXiv preprint arXiv:2304.06385_ , 2023.
  * Wikipedia contributors (2023) Wikipedia contributors.  Bilinear interpolation — Wikipedia, the free encyclopedia, 2023.  URL https://en.wikipedia.org/w/index.php?title=Bilinear_interpolation&oldid=1170546721.
  * Xiao et al. (2010) Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A.  Sun database: Large-scale scene recognition from abbey to zoo.  In _CVPR_ , 2010.
  * Yang et al. (2021) Yang, C.-H. H., Tsai, Y.-Y., and Chen, P.-Y.  Voice2series: Reprogramming acoustic models for time series classification.  In _ICML_ , 2021.
  * Yang et al. (2023) Yang, C.-H. H., Li, B., Zhang, Y., Chen, N., Prabhavalkar, R., Sainath, T. N., and Strohman, T.  From english to more languages: Parameter-efficient model reprogramming for cross-lingual speech recognition.  In _ICASSP_ , 2023.
  * Yen et al. (2023) Yen, H., Ku, P.-J., Yang, C.-H. H., Hu, H., Siniscalchi, S. M., Chen, P.-Y., and Tsao, Y.  Neural model reprogramming with similarity based mapping for low-resource spoken command recognition.  In _INTERSPEECH_ , 2023.
  * Yuval (2011) Yuval, N.  Reading digits in natural images with unsupervised feature learning.  In _NIPS Workshop_ , 2011.
  * Zhang et al. (2022) Zhang, G., Zhang, Y., Zhang, Y., Fan, W., Li, Q., Liu, S., and Chang, S.  Fairness reprogramming.  _NeurIPS_ , 2022.
  * Zhou et al. (2022a) Zhou, K., Yang, J., Loy, C. C., and Liu, Z.  Conditional prompt learning for vision-language models.  In _CVPR_ , 2022a.
  * Zhou et al. (2022b) Zhou, K., Yang, J., Loy, C. C., and Liu, Z.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 2022b.
  * Zhu et al. (2023) Zhu, Y., Shen, Z., Zhao, Z., Wang, S., Wang, X., Zhao, X., Shen, D., and Wang, Q.  Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis.  _arXiv preprint arXiv:2311.08236_ , 2023.

## Appendix

##  Appendix A Additional Explanation of Methods

###  A.1 General Procedure of Input Visual Reprogramming

Figure 7: Problem setting of input visual reprogramming. The upper part shows
the source task, while the lower part shows the target task. The main focus of
visual reprogramming is the trainable part marked with a yellow rectangle in
the input space.

The task of VR is to reuse the fixed, well-trained model toward a target task.
As shown in Figure 7, the VR module is added before the pre-trained model into
the input space. To gap the difference between the source label and target
label, an output mapping function without parameters is also used, taking a
source label as the input and outputting a target label. Therefore, regardless
of the architecture, a well-trained model on the source dataset can be
transferred to the target task without editing.

###  A.2 Architecture of the Mask Generator and Parameter Statistics

Figure 8: Architecture of the 5-layer mask generator designed for ResNet
Figure 9: Architecture of the 6-layer mask generator designed for ViT

Architecture of the Mask Generator. For simplicity, we only include $3\times
3$ convolution layers and $2\times 2$ Max-Pooling layers in the architecture.
The number of channels of the last layer is set to 3 to produce a three-
channel mask.

The detailed architecture of the 5-layer CNN and 6-layer CNN used in
ResNet-18, ResNet-50, and ViT are shown in Figure 8 and Figure 9. Each of them
contains 5 or 6 CNN layers with $3\times 3$ kernels of padding size 1 and
stride 1. Both models have 3 Max-Pooling layers.

Figure 10: Changes of the image size when performing convolution and pooling
operations with our stride, kernel and padding size

Discussion of Input and Output Size. To show the relationship between the
sizes of the input images and the output masks, we use $s$, $p$, and $k$ to
represent the stride, padding, and kernel sizes, respectively, while $H$ and
$W$ denote the height and the width of a certain channel. The output
dimensions of the output channel after convolution or pooling are
$\left\lfloor\frac{H+2p-k}{s}\right\rfloor+1$ and
$\left\lfloor\frac{W+2p-k}{s}\right\rfloor+1$. As shown in Figure 10, when
$s=1,p=1,k=3$, the size of a single channel remains unchanged; when
$s=2,p=0,k=2$, the size of a channel is reduced by half in each dimension. In
other words, by only using $3\times 3$ convolution layers, $f_{\rm
mask}(.|\phi)$ can retain the original size of a single channel. However, if
we introduce Max-Pooling layers to remove redundant information, the output
size will be shrunk and another patch-wise interpolation module should be
included in $f_{\rm mask}(.|\phi)$ for resizing. Assuming that $l$ Max-Pooling
layers are used, the output size of a single channel becomes
$\left\lfloor\frac{H}{2^{l}}\right\rfloor\times\left\lfloor\frac{W}{2^{l}}\right\rfloor$.

Table 4: Statistics of Mask Generator Parameter Size Pre-trained |  Input Image Size |  $f_{\rm mask}$ CNN Layers |  Extra Parameters of our $f_{\rm mask}$ |  Our Extra Parameters $\div$ Reprogramming Parameters |  Our Extra Parameters $\div$ Pre-trained Model Parameters
---|---|---|---|---|---
ResNet-18 |  224$\times$224$\times$2 | 5 | 26,499 | 17.60% | 0.23%
ResNet-50 |  224$\times$224$\times$3 | 5 | 26,499 | 17.60% | 0.10%
ViT-B32 |  384$\times$384$\times$3 | 6 | 102,339 | 23.13% | 0.12%

Parameter Statistics. The parameter statistics of the mask generator, $f_{\rm
mask}$, are summarized in Table 4. This includes a detailed breakdown of
$f_{\rm mask}$ across different pre-trained backbone models, a relative size
comparison with the watermarking reprogramming method, and the number of
trainable parameters added to frozen pre-trained models by $f_{\rm mask}$.
From the size point of view, our mask generator is indeed lightweight and
efficient: the CNN architectures contribute only 17.6% and 23.13% of the
additional trainable parameters required by watermarking reprogramming.
Moreover, relative to the total parameters in pre-trained models, the
additional contribution of mask generators is trivial, ranging from 0.1% to
0.23% of parameters, which highlights its minimal footprint.

###  A.3 Advantage of Patch-wise Interpolation

Table 5: Comparison of Patch-wise Interpolation and Other Interpolation Methods |  |  Bilinear Interpolation |  Bicubic Interpolation | Ours
---|---|---|---|---
ResNet - 18/50 |  Number of Pixel Accesses (1e6) | 0.602 | 2.408 | 0.151
Time Per Batch (s) | 0.062±0.001 | 0.195±0.013 | 0.026±0.004
Require Backpropagation | Yes | Yes | No
ViT - B32 |  Number of Pixel Accesses (1e6) | 1.769 | 7.078 | 0.442
Time Per Batch (s) | 0.165±0.009 | 0.486±0.026 | 0.069±0.004
Require Backpropagation | Yes | Yes | No

To assess the efficiency of patch-wise interpolation, we compare it with
bilinear and bicubic methods, employing the following numerical metrics for
evaluation: (1) Number of Pixel Accesses: The count of times pixel values are
retrieved per image during an interpolation algorithm. The fewer, the better.
(2) Time Per Batch: The time cost for processing a batch of 256-sized images.
The fewer, the better.

As shown in Table 5, the patch-wise interpolation module excels across all
metrics. This module exclusively involves copying operations, thus avoiding
floating-point calculations and avoiding backpropagation gradient computations
during training. Consequently, it is more efficient.

###  A.4 Detailed Explanation of Ouptput Mapping Methods $f^{\rm Flm}_{\rm
out}$ and $f^{\rm Ilm}_{\rm out}$

The inverse function of $f_{\rm out}$ regarding Flm is an injective function:

$\displaystyle y^{\rm P}_{\rm
Flm}=\mathop{\arg\max}\limits_{y\in\mathcal{Y}^{\rm
P}}\mathop{Pr}\limits_{(x_{i},y_{i})\sim\mathcal{D_{\rm T}}}\\{y=f_{\rm
P}(f_{\rm in}(x_{i}|\theta))|y_{i}=y^{\rm T}\\},$ (8)

where $y^{\rm P}_{\rm Flm}$ is the optimal $y^{\rm P}$ given the target label
$y^{\rm T}$, $f_{\rm P}(f_{\rm in}(x_{i}|\theta))$ is the predicted label
given the input image $x_{i}$. For all images with the label $y^{\rm T}$, the
predicted $y^{\rm P}$ with the highest probability will be $y^{\rm P}_{\rm
Flm}$ for a given $y^{\rm T}$. Flm remains unchanged throughout iterations.
For a specific $y^{\rm T}$, Flm determines the correspondence between $y^{\rm
T}$ and the most frequently assigned class $y^{\rm P}$ in $\mathcal{Y}^{\rm
P}$, utilizing the well-trained network for all target training samples of the
class $y^{\rm T}$, thus obtaining $f_{\rm out}^{\rm Flm}$, shown in Algorithm
3.

As the label mapping may change from time to time when learning $f_{\rm in}$,
Chen et al. (2023) proposed an _iterative label mapping_ (Ilm) method that
updates $f_{\rm out}(\cdot)$ after each training iteration. Let $y^{\rm
P,(j)}_{\rm Ilm}$ be the optimal $y^{\rm P}$ in the $j$th training epoch. We
have:

$\displaystyle y^{\rm P,(j+1)}_{\rm Ilm}$
$\displaystyle=\mathop{\arg\max}\limits_{y\in\mathcal{Y}^{\rm
P}}\mathop{Pr}\limits_{(x_{i},y_{i})\sim\mathcal{D_{\rm T}}}\\{y=f_{\rm
P}(f_{\rm in}^{(j)}(x_{i}|\theta^{(j)}))|y_{i}=y^{\rm T}\\},$ (9)

where $f_{\rm in}^{(j)}(\cdot|\theta^{(j)})$ is the parameters of the $j$th
epoch. The output mapping function is updated after each iteration until
convergence.

Algorithm 2 Computing Frequency Distribution of [$f_{\rm P}(f_{\rm
in}(x_{i}|\theta))$, $y^{\rm T}$]

1: Input: Target training set $\\{(x^{\rm T}_{i},y^{\rm T}_{i})\\}_{i=1}^{n}$,
given input VR $f_{\rm in}(\cdot|\theta)$ and pre-trained model $f_{\rm
P}(\cdot)$

2: Output: Frequency distribution matrix $d\in\mathbb{Z}^{|\mathcal{Y}^{\rm
P}|\times|\mathcal{Y}^{\rm T}|}$

3: Initialize $d\leftarrow\\{0\\}^{|\mathcal{Y}^{\rm
P}|\times|\mathcal{Y}^{\rm T}|}$

4: # Compute frequency distribution $d$

5: for $i=1...n$ do

6: $\hat{y}_{i}^{\rm P}\leftarrow f_{\rm P}(f_{\rm in}(x^{\rm T}_{i}|\theta))$

7: $d_{\hat{y}_{i}^{\rm P},{y}_{i}^{\rm T}}\leftarrow d_{\hat{y}_{i}^{\rm
P},{y}_{i}^{\rm T}}+1$

8: end for

Algorithm 3 Frequent Label Mapping ($f^{\rm Flm}_{\rm out}$)

1: Input: Label space of the pre-trained task $\mathcal{Y}^{\rm P}$, label
space of the target task $\mathcal{Y}^{\rm T}$, target training set
$\\{(x^{\rm T}_{i},y^{\rm T}_{i})\\}_{i=1}^{n}$, given pre-trained model
$f_{\rm P}(\cdot)$

2: Output: Flm $f^{\rm Flm}_{\rm out}:\mathcal{Y}^{\rm P}_{\rm
sub}\rightarrow\mathcal{Y}^{\rm T}$

3: Initialize $f^{\rm Flm}_{\rm out}(\cdot)\leftarrow 0$, subset
$\mathcal{Y}^{\rm P}_{\rm sub}\leftarrow\emptyset$ to store matched labels,
initialize $f_{\rm in}(\cdot|\theta)$ to be an identity function
($\theta\leftarrow\mathbf{0}$)

4: # Compute frequency distribution $d$

5: Use Algorithm 2 to obtain $d$

6: # Compute output mapping $f^{\rm Flm}_{\rm out}$

7: while size of $\mathcal{Y}^{\rm P}_{\rm sub}$ is not $|\mathcal{Y}^{\rm
T}|$ do

8: Find the maximum $d_{y^{\rm P},y^{\rm T}}$ in $d$

9: $\mathcal{Y}^{\rm P}_{\rm sub}\leftarrow\mathcal{Y}^{\rm P}_{\rm
sub}\cup\\{y^{\rm P}\\}$

10: $f^{\rm Flm}_{\rm out}(y^{\rm P})\leftarrow y^{\rm T}$ # Update the label
mapping function

11: $d_{y^{\rm P},t}\leftarrow 0$ for $t=1,2,...,|\mathcal{Y}^{\rm T}|$ #
Avoiding illegal assignment to the injective function

12: $d_{s,y^{\rm T}}\leftarrow 0$ for $s=1,2,...,|\mathcal{Y}^{\rm P}|$

13: end while

Algorithm 4 Iterative Label Mapping ($f^{\rm Ilm}_{\rm out}$)

1: Input: Label space of the pre-trained task $\mathcal{Y}^{\rm P}$, label
space of the target task $\mathcal{Y}^{\rm T}$, target training set
$\\{(x^{\rm T}_{i},y^{\rm T}_{i})\\}_{i=1}^{n}$, given pre-trained model
$f_{\rm P}(\cdot)$, total iteration number $E$, learning rate $\alpha$

2: Output: Ilm $f^{\rm Ilm,(j)}_{\rm out}:\mathcal{Y}^{\rm P}_{\rm
sub}\rightarrow\mathcal{Y}^{\rm T}$ for iteration $j$

3: Initialize $f^{\rm Ilm,(j)}_{\rm out}(\cdot)\leftarrow 0$, subset
$\mathcal{Y}^{\rm P}_{\rm sub}\leftarrow\emptyset$ to store matched labels,
initialize $f_{\rm in}(\cdot|\theta)$ to be an identity function
($\theta\leftarrow\mathbf{0}$)

4: for $j=1...E$ do

5: # Compute frequency distribution $d$

6: Use Algorithm 2 to obtain $d$

7: # Compute output mapping $f^{\rm Ilm,(j)}_{\rm out}$

8: while size of $\mathcal{Y}^{\rm P}_{\rm sub}$ is not $|\mathcal{Y}^{\rm
T}|$ do

9: Find the maximum $d_{y^{\rm P},y^{\rm T}}$ in $d$

10: $\mathcal{Y}^{\rm P}_{\rm sub}\leftarrow\mathcal{Y}^{\rm P}_{\rm
sub}\cup\\{y^{\rm P}\\}$

11: $f^{\rm Ilm,(j)}_{\rm out}(y^{\rm P})\leftarrow y^{\rm T}$ # Update the
label mapping function for iteration $j$

12: $d_{y^{\rm P},t}\leftarrow 0$ for $t=1,2,...,|\mathcal{Y}^{\rm T}|$ #
Avoiding illegal assignment to the injective function

13: $d_{s,y^{\rm T}}\leftarrow 0$ for $s=1,2,...,|\mathcal{Y}^{\rm P}|$

14: end while

15: # Train $f_{\rm in}(\cdot|\theta)$ for iteration $j$

16:
$\theta\leftarrow\theta-\alpha\cdot\nabla_{\theta}\frac{1}{n}\sum_{i=1}^{n}\ell(f^{\rm
Ilm,(j)}_{\rm out}(f_{\rm P}(f_{\rm in}(x^{\rm T}_{i}|\theta))),y^{\rm
T}_{i})$

17: end for

Ilm evolves with iterations, being an improved version of Flm. As is shown in
Algorithm 4, before training the reprogramming pattern $\theta$ in each epoch,
Ilm updates the one-to-one mapping from $\mathcal{Y}^{\rm P}$ to
$\mathcal{Y}^{\rm T}$ with the training samples incorporating the current
pattern, iteratively until convergence.

##  Appendix B Additional Theoretical Proof

###  B.1 Proof of Theorem 4.2

The approximation error of $\mathcal{F}_{1}$ and $\mathcal{F}_{2}$ can be
formulated as:

$\displaystyle{\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F}_{1})=\inf_{f\in\mathcal{F}_{1}}\mathbb{E}_{(X,Y)\sim\mathcal{D}}\ell(f(X),Y)-R^{*}_{\mathcal{D}},$
$\displaystyle{\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F}_{2})=\inf_{f\in\mathcal{F}_{2}}\mathbb{E}_{(X,Y)\sim\mathcal{D}}\ell(f(X),Y)-R^{*}_{\mathcal{D}},$

Straightforwardly,

$\displaystyle\mathcal{F}_{1}\supseteq\mathcal{F}_{2}\Leftrightarrow\forall
f\in\mathcal{F}_{2},f\in\mathcal{F}_{1}$

Given $\mathcal{F}_{1}\subseteq\mathcal{F}_{2}$, we have:

$\displaystyle\forall f\in\mathcal{F}_{1},f\in\mathcal{F}_{2},$
$\displaystyle\Rightarrow$
$\displaystyle\inf_{f\in\mathcal{F}_{1}}\mathbb{E}_{(X,Y)\sim\mathcal{D}}\ell(f(X),Y)\geq\inf_{f\in\mathcal{F}_{2}}\mathbb{E}_{(X,Y)\sim\mathcal{D}}\ell(f(X),Y)$
$\displaystyle\Rightarrow$ $\displaystyle{\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F}_{1})\geq{\rm Err}^{\rm
apx}_{\mathcal{D}}(\mathcal{F}_{2})$

###  B.2 Proof of Proposition 4.3

We prove Proposition 4.3 as follows.

###### Proof.

With specially designed kernel and padding sizes, the output of CNN can be
reshaped to match the size of the input image. Assuming $d_{\rm P}=H\times
W\times C$, we define $M^{\prime}\in\\{0,1\\}^{H*W*C\times 1}$ and
$f^{\prime}_{\rm mask}(\cdot)\in\mathbb{R}^{H*W*C\times 1}$ as transposed
flattened $M$ and $f_{\rm mask}(\cdot)$, respectively. As the last layer of
$f^{\prime}_{\rm mask}(\cdot)$ is CNN, if the input of CNN is the resized
image $r(x)$, with $x\in\mathcal{X}^{\rm T}$ (and $r(x)\in\mathbb{R}^{d_{\rm
P}}$), we have $f^{\prime}_{\rm mask}(r(x))=W_{\rm last}f^{\prime\prime}_{\rm
mask}(r(x))+b_{\rm last}$, with $b_{\rm last}$ being the bias of the last
layer, and $W_{\rm last}$ being the mapping from the flattened input of the
last CNN layer (i.e., $f^{\prime\prime}_{\rm mask}(r(x))$) to the flattened
output without adding the bias, which can be derived using the parameters of
the last CNN layer. With the set of any possible $W_{\rm last}$ being
represented by $\\{W_{\rm last}\\}$, and all-zero matrix being $O$, we have:

$\displaystyle b_{\rm last}\in\mathbb{R}^{H*W*C\times
1},M^{\prime}\in\\{0,1\\}^{H*W*C\times 1}$ $\displaystyle\Rightarrow\forall
M^{\prime},M^{\prime}\in\\{b_{\rm last}|b_{\rm last}\in\mathbb{R}^{H*W*C\times
1}\\}$ (10) $\displaystyle O\in\\{W_{\rm last}\\}\text{\scriptsize(When all
weights in the last CNN layer is 0, $W_{\rm last}$ is a zero matrix)}$
$\displaystyle\Rightarrow f(x)=O^{H*W*C\times 1}\in\\{f|f(x)=W_{\rm
last}f^{\prime\prime}_{\rm mask}(r(x)),\forall x\in\mathcal{X}^{\rm T}\\}$
(11) $\displaystyle\Rightarrow\\{f|f(x)=M^{\prime},\forall
x\in\mathcal{X}^{\rm T}\\}\subseteq\\{f|f(x)=f^{\prime}_{\rm
mask}(r(x)),\forall x\in\mathcal{X}^{\rm T}\\}\text{\scriptsize(Given
Eq.~{}\eqref{eq_part1} and Eq.~{}\eqref{eq_part2})}$
$\displaystyle\Rightarrow\\{f|f(x)=M,\forall x\in\mathcal{X}^{\rm
T}\\}\subseteq\\{f|f(x)=f_{\rm mask}(r(x)),\forall x\in\mathcal{X}^{\rm T}\\}$
$\displaystyle\Rightarrow\\{f|f(x)=M\odot\delta,\forall x\in\mathcal{X}^{\rm
T}\\}\subseteq\\{f|f(x)=f_{\rm mask}(r(x))\odot\delta,\forall
x\in\mathcal{X}^{\rm T}\\}$ $\displaystyle\Rightarrow\mathcal{F}^{\rm
shr}(f^{\prime}_{\rm P})\subseteq\mathcal{F}^{\rm smm}(f^{\prime}_{\rm
P})\text{\scriptsize(since $f^{\prime}_{\rm P}$ is fixed)}$
$\displaystyle\Rightarrow{\rm Err}^{\rm apx}_{\mathcal{D}_{\rm
T}}(\mathcal{F}^{\rm smm}(f^{\prime}_{\rm P}))\leq{\rm Err}^{\rm
apx}_{\mathcal{D}_{\rm T}}(\mathcal{F}^{\rm shr}(f^{\prime}_{\rm P}))$

∎

###  B.3 SMM and Sample-specific Patterns

We will then prove

######  Proposition B.1.

for any fixed $f^{\prime}_{\rm P}$, it holds that $\mathcal{F}^{\rm
sp}(f^{\prime}_{\rm P})\subseteq\mathcal{F}^{\rm smm}(f^{\prime}_{\rm P})$,
and consequently, ${\rm Err}^{\rm apx}_{\mathcal{D}_{\rm T}}(\mathcal{F}^{\rm
smm}(f^{\prime}_{\rm P}))\leq{\rm Err}^{\rm apx}_{\mathcal{D}_{\rm
T}}(\mathcal{F}^{\rm sp}(f^{\prime}_{\rm P}))$.

###### Proof.

Let $\Delta$ be the set of possible $\delta$, with all-one matrix being
denoted as $J$, we have:

$\displaystyle\Rightarrow J^{d_{\rm P}}\in\Delta$
$\displaystyle\Rightarrow\\{f|f(x)=f_{\rm mask}(r(x))\odot J^{d_{\rm
P}},\forall x\in\mathcal{X}^{\rm T}\\}\subseteq\\{f|f(x)=f_{\rm
mask}(r(x))\odot\delta,\forall x\in\mathcal{X}^{\rm T}\\}$
$\displaystyle\Rightarrow\\{f|f(x)=f_{\rm mask}(r(x)),\forall
x\in\mathcal{X}^{\rm T}\\}\subseteq\\{f|f(x)=f_{\rm
mask}(r(x))\odot\delta,\forall x\in\mathcal{X}^{\rm T}\\}$
$\displaystyle\Rightarrow\mathcal{F}^{\rm sp}(f^{\prime}_{\rm
P})\subseteq\mathcal{F}^{\rm smm}(f^{\prime}_{\rm P})\text{\scriptsize(Since
$f^{\prime}_{\rm P}$ is fixed)}$ $\displaystyle\Rightarrow{\rm Err}^{\rm
apx}_{\mathcal{D}_{\rm T}}(\mathcal{F}^{\rm smm}(f^{\prime}_{\rm P}))\leq{\rm
Err}^{\rm apx}_{\mathcal{D}_{\rm T}}(\mathcal{F}^{\rm sp}(f^{\prime}_{\rm
P}))$

∎

##  Appendix C Additional Experimental Setup

Table 6: Detailed Dataset Information Dataset | Original Image Size | Training Set Size | Testing Set Size | Number of Classes
---|---|---|---|---
CIFAR10 |  32 $\times$ 32 | 50000 | 10000 | 10
CIFAR100 |  32 $\times$ 32 | 50000 | 10000 | 100
SVHN |  32 $\times$ 32 | 73257 | 26032 | 10
GTSRB |  32 $\times$ 32 | 39209 | 12630 | 43
Flowers102 |  128 $\times$ 128 | 4093 | 2463 | 102
DTD |  128 $\times$ 128 | 2820 | 1692 | 47
UCF101 |  128 $\times$ 128 | 7639 | 3783 | 101
Food101 |  128 $\times$ 128 | 50500 | 30300 | 101
SUN397 |  128 $\times$ 128 | 15888 | 19850 | 397
EuroSAT |  128 $\times$ 128 | 13500 | 8100 | 10
OxfordPets |  128 $\times$ 128 | 2944 | 3669 | 37

The 11 datasets used for the experiments are summarized in Table 6, while the
corresponding training parameters are listed in Table 9. When learning the
ResNet tasks, we follow the same learning strategies as Chen et al. (2023).
When learning ViT-B32, we choose the initial learning rate $\alpha$ and the
learning rate decay $\gamma$ with a training parameter searching experiment,
with results presented in Table 7.

Table 7: Tuning Initial Learning Rate and Learning Rate Decay Using CIFAR10 and ViT-B32 (Accucracy %) $\gamma|\alpha$ | 0.1 | 0.01 | 0.001 | 0.0001
---|---|---|---|---
1 | 0.9542 | 0.9577 | 0.9745 | 0.9734
0.1 | 0.9516 | 0.9572 | 0.9738 | 0.9727

Sharing the same $\alpha$ and $\gamma$ may not be optimal for all datasets. As
shown in Table 8, on UCF101, using $\alpha=0.001$ and $\gamma=1$ derived from
Table 7 leads to sub-optimal model performance. Nevertheless, for uniformity
and fairness in this paper, we still use a single set of unified training
parameters for all datasets.

Table 8: Results on UCF101 with Different Training Parameters (using ViT-B32) | $\alpha$ | $\gamma$ | SMM Accuracy (%)
---|---|---|---
Unified Learning Parameters | 0.001 | 1 | 42.6
Specific Learning Parameters | 0.01 | 0.1 | 49.9
Table 9: Detailed Model Training Parameter Settings of Our Mask Generator (where $b$, $\alpha$ and $\gamma$ denote batch size, initial learning rate and learning rate decay, respectively) |  |  | 5-layer | 6-layer
---|---|---|---|---
| $b$ | Milestones | $\alpha$ | $\gamma$ | $\alpha$ | $\gamma$
CIFAR10 | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
CIFAR100 | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
SVHN | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
GTSRB | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
Flowers102 | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
DTD | 64 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
UCF101 | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
Food101 | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
SUN397 | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
EuroSAT | 256 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1
OxfordPets | 64 | [0, 100, 145] | 0.01 | 0.1 | 0.001 | 1

##  Appendix D Additional Experimental Results

###  D.1 Applying SMM with Different $f_{\rm out}$

Table 10: Performance Improvement When Applying Our Input Reprogramming on
Different Label Mapping Methods (the average results are highlighted in grey)

$f_{\rm out}$ | Iterative Label Mapping | Frequent Label Mapping | Random Label Mapping
---|---|---|---
| w/o Ours | w Ours | Improve | w/o Ours | w Ours | Improve | w/o Ours | w Ours | Improve
CIFAR10 | 68.90% | 72.80% | +3.90% | 71.79% | 72.75% | +0.96% | 65.68% | 69.71% | +4.03%
CIFAR100 | 33.80% | 39.40% | +5.60% | 29.79% | 32.35% | +2.56% | 16.99% | 23.47% | +6.48%
SVHN | 78.30% | 84.40% | +6.10% | 78.78% | 83.73% | +4.95% | 77.44% | 85.37% | +7.92%
GTSRB | 76.80% | 80.40% | +3.60% | 74.76% | 80.90% | +6.14% | 69.60% | 82.38% | +12.79%
Flowers102 | 23.20% | 38.70% | +15.50% | 17.78% | 32.16% | +14.37% | 12.34% | 37.68% | +25.33%
DTD | 29.00% | 33.60% | +4.60% | 30.14% | 34.28% | +4.14% | 14.60% | 19.74% | +5.14%
UCF101 | 24.40% | 28.70% | +4.30% | 22.71% | 25.72% | +3.01% | 9.04% | 16.71% | +7.67%
Food101 | 13.20% | 17.50% | +4.30% | 11.58% | 15.21% | +3.62% | 7.15% | 15.86% | +8.71%
SUN397 | 13.40% | 16.00% | +2.60% | 13.45% | 15.45% | +1.99% | 1.05% | 3.35% | +2.29%
EuroSAT | 84.30% | 92.20% | +7.90% | 86.00% | 92.67% | +6.67% | 84.49% | 94.47% | +9.98%
OxfordPets | 70.00% | 74.10% | +4.10% | 69.66% | 72.83% | +3.16% | 8.89% | 16.84% | +7.96%
Average | 46.85% | 52.53% | +5.68% | 46.04% | 50.73% | +4.69% | 33.39% | 42.32% | +8.94%

As mentioned before, and as shown in Appendix A.1, input VR is agnostic of the
output label mapping method. Thus, our SMM can be applied to different output
label methods other than Ilm. Experimental results are presented in Table 10.

Our method improves the performance of all output mapping methods. In most
cases, the worse the output mapping method is, the more pronounced the
improvement of SMM will be. When there is sufficient training data (e.g.,
GTSRB, SVHN, CIFAR10 and Food101), adding SMM can compensate for the worse-
performing label mapping methods. With SMM, these methods also produce
competitive results.

###  D.2 Analysis of Learning Curves

Figure 11: Training Accuracy and Loss of Different Reprogramming Methods
Figure 12: Training Accuracy and Testing Accuracy with and without Our Method

Figure 11 shows the training accuracy and loss throughout learning iterations
using ResNet-18 as the pre-trained backbone. We see that our SMM yields a
higher training accuracy and lower loss for most cases.

When using a more sophisticated pre-trained network, e.g., ViT, as is shown in
Figure 12, the training accuracy without SMM may meet with or even exceed that
of using SMM. However, this appears to be a case of over-fitting, where
training accuracy is approaching 1 and test accuracy is still low without
using SMM.

In general, for smaller classifiers such as ResNet-18, adding our model helps
better reduce training loss and improve accuracy, while for more sophisticated
classifiers such as ViT-B32 where the training accuracy is already high,
adding our SMM model helps prevent over-fitting and improve the testing
accuracy.

Table 11: Training and Testing Accuracy with Enlarged $f_{\rm mask}$ (using EuroSAT, ResNet-18) $f_{\rm mask}$ | Small | Medium (ours) | Large | x-Large | xx-Large | xxx-Large
---|---|---|---|---|---|---
Parameters | 7203 | 26499 | 101379 | 396291 | 1566723 | 6230019
Training Accuracy (%) | 94.9 | 96.2 | 96.4 | 97.3 | 97.7 | 98.1
Testing Accuracy (%) | 91.7 | 92.2 | 92.2 | 93.1 | 93.5 | 93.2

###  D.3 More Discussion about the Estimation Error

A higher estimation error generally implies an increased risk of model over-
fitting to the training data. This observation can be corroborated by
comparing the disparities in training and testing performance. For instance,
as depicted in Figure 12, employing a more sophisticated pre-trained network
such as ViT with a mask generator $f_{\rm mask}$ shown in Figure 9 across some
tasks like CIFAR10, SVHN, and GTSRB, the training accuracy tends towards 100%
for both shared patterns $\mathcal{F}^{\rm shr}(f^{\prime}_{\rm P})$ (i.e.,
‘Watermarking’ in Figure 12) and SMM patterns $\mathcal{F}^{\rm
smm}(f^{\prime}_{\rm P})$ (i.e., ‘Ours’ in Figure 12). Despite this,
$\mathcal{F}^{\rm smm}(f^{\prime}_{\rm P})$ maintains a test accuracy that is
not inferior to that of shared patterns. It suggests that our method SMM does
not suffer from more significant over-fitting than shared masking, resulting
in negligible potential estimation error.

However, when $f_{\rm mask}$ is enlarged with increased number of parameters,
the additional estimation error of $\mathcal{F}^{\rm smm}(f^{\prime}_{\rm P})$
may no longer be negligible and will impact the excess risk. The relationship
between the number of parameters in $f_{\rm mask}$ and the estimation error is
influenced by various factors, including the specific target tasks, the volume
of training data, the size of well-trained models, and the design of our
generation model, etc. Through experiments, we will be able to estimate when
the number of parameters begins to impact estimation error, potentially
leading to over-fitting. For instance, in Table 11, we employ our generation
model $f_{\rm mask}$ on the EuroSAT dataset, with ResNet-18 being the well-
trained model. By progressively doubling the number of intermediate channels
while maintaining the architecture of $f_{\rm mask}$, we investigate how the
model size affects performance.

Through the results of Table 11, we come to the following conclusions: (1) As
the number of parameters continues to increase, although the training accuracy
slowly increases, the test accuracy may even decrease, implying that the
estimation error becomes more and more noticeable. (2) Under this situation
(i.e., EuroSAT, ResNet-18), when the size of $f_{\rm mask}$ is close to the
same order of magnitude as the well-trained model, the estimation error should
not be overlooked. (3) A larger model with the best test accuracy may not be
optimal because of too many parameters. Our work strikes a balance between the
number of parameters and test accuracy.

###  D.4 Further Analysis of the Performance of SMM

More Discussion of SMM Abnormal Cases. In Section 5, we have briefly analyzed
abnormal performance in Table 1 and Table 2. In this section, we will provide
a more comprehensive discussion. Here, we outline detailed discussions
regarding abnormal performance:

  * •

ResNet-18, DTD: As shown in Figure 18, the DTD dataset contains a significant
amount of texture features. Therefore, for relatively simple well-trained
models, introducing reprogramming noise in the form of watermarking may affect
the original features of the images. It can be observed that when the
watermarking area is small (Narrow), the effect is better compared to when it
is large (Full), and our method is also affected by this factor. However, the
padding-based method preserves the original pixels of the image and only
introduces reprogramming noise around them, thereby achieving relatively good
results.

  * •

ViT-B32, EuroSAT: This is because EuroSAT is one of the target tasks with the
least task complexity. When using a large-scale network like ViT, the
resizing-based method leads to over-fitting. As evident in the third column of
the second row in Figure 12, the training accuracy is already close to 1.
Therefore, in this scenario, the padding-based method yields slightly better
test results compared to our method (which also belongs to resizing-based
methods).

Table 12: An Ineffective Case of Input Reprogramming - StanfordCars (Mean % ± Std %) Method | Pad | Narrow | Medium | Full | Ours
---|---|---|---|---|---
ResNet-18 | 4.5±0.1 | 3.6±0.1 | 3.6±0.1 | 3.4±0.1 | 2.9±0.2
ResNet-50 | 4.7±0.2 | 4.7±0.1 | 4.7±0.2 | 4.6±0.1 | 3.0±0.6
ViT-B32 | 4.7±0.6 | 7.7±0.2 | 8.3±0.3 | 5.0±0.0 | 4.8±0.9

SMM on An Ineffective Case of Input Reprogramming. All input visual
reprogramming methods seem ineffective on fine-grained recognition tasks where
subtle appearance differences should be detected. As shown in Table 12, in the
classification of StanfordCars, where 196 types of cars are to be classified,
the accuracy of all input VR methods is below 10 %, indicating the failure of
VR methods in this fine-grained recognition tasks. Adding our SMM module will
not improve performance when VR methods fail.

##  Appendix E Additional Discussion about Input VR Compared with Finetuning

###  E.1 Advantages of VR in Dealing with Distorted Input Images

Table 13: Performance of Finetuning (LoRA) and SMM Facing Target Tasks with Different Input Image Sizes (Accyracy %, using ViT-L with a 384$\times$384 input as the well-trained model, average results are calculated on all four tasks with 32$\times$32 inputs and all seven tasks with 128$\times$128 inputs) |  Extra Parameters | CIFAR10 | CIFAR100 | SVHN | GTSRB |  Average (32$\times$32) |  Average (128$\times$128)
---|---|---|---|---|---|---|---
Finetuning-LoRA | 0.60M | 95.9 | 83.6 | 65.3 | 66.6 | 77.9 | 83.4
Our SMM | 0.54M | 97.4 | 87.3 | 91.0 | 84.2 | 90.0 | 83.5

In this section, we will compare the results of our SMM with finetuning-based
methods to show the advantages of input VR in dealing with distorted input
images. Low-rank adaptation (LoRA) (Hu et al., 2021) is an efficient
finetuning-based transfer method proposed based on large language models for
natural language processing, which has been adapted to ViT (Zhu et al., 2023).
Here, we compare SMM for ViT with LoRA for ViT, which are representative
methods that belong to input VR and finetuning, respectively.

Since LoRA for ViT already includes finetuning the fully connected layers, we
also incorporate it in SMM. All training settings are kept the same. We set
the rank of LoRA to be six, resulting in an additional parameter number being
0.60M (without counting the fully connected layers), which will be comparable
to that of input VR and SMM (being 0.54M) for fairness. ViT-Large with the
input size being 384$\times$384 is applied, and the learning rate is 0.01,
running 10 epochs in total. Therefore, for both methods, the target training
samples will be resized before input. VR mainly trains parameters in the input
space before well-trained models, whereas LoRA injects parameters into layers
of ViT. Results are listed in Table 13.

The results of target tasks with the input size being 128$\times$128 are
similar. However, it is observed that for those target tasks with lower
resolution (e.g., CIFAR10/100, SVHN, GTSRB), our SMM appears to perform
better. This is likely because when a 32$\times$32 image is resized to
384$\times$384, it may become distorted, thus affecting the performance of
target tasks. This distortion is especially noticeable on tasks with simple
features, such as SVHN and GTSRB. Since VR modifies the input space, it
effectively addresses this issue of significant differences in the input image
sizes of pre-trained and target tasks.

###  E.2 Advantages of VR in Being Orthogonal to Finetuning-based Methods

Table 14: Performance of Finetuning the Fully-Connected Layers (Finetuning-FC) without or with our SMM Module (Accuracy %, using ResNet-50 as the well-trained model) | CIFAR10 | CIFAR100 | SVHN | GTSRB | Flowers102 | DTD
---|---|---|---|---|---|---
Finetuning-fc | 90.1 | 70.7 | 63.5 | 77.8 | 90.9 | 67.6
Finetuning-fc + Our SMM | 91.2 | 72.4 | 86.9 | 85.2 | 90.9 | 68.2
| UCF101 | Food101 | SUN397 | EuroSAT | OxfordPets | Average
Finetuning-fc | 70.8 | 57.6 | 53.5 | 95.7 | 90.4 | 75.3
Finetuning-fc + Our SMM | 72.0 | 59.6 | 57.9 | 95.8 | 90.6 | 79.2

Since finetuning and reprogramming are orthogonal because finetuning modifies
the model while reprogramming modifies the input and output spaces. Input VR
can also be combined with finetuning-based methods. In this section, we will
add the input VR module (i.e, using SMM as an example) to finetuning-based
methods and analyze the performance gain. A widely-used method - finetuning
the fully connected layer (named ‘Finetuning-FC’) - is employed as the
baseline method. Using ResNet-50 as the well-trained model, we add our SMM
input VR module to ‘Finetuning-FC’ to demonstrate the effectiveness of our
module.

Results are shown in Tabel 14. Utilizing our module achieves an average
accuracy of about 4% higher than solely finetuning the fully connected layers.
Conclusively, input VR can be attached to finetuning-based methods to improve
performance.

###  E.3 Strengths and Weaknesses of Input Reprogramming in Visual Tasks

This part includes a conclusion of the strengths and weaknesses of Input VR,
compared with finetuning-based methods.

####  E.3.1 Strengths

  * •

The parameter numbers of VR tend to be negligible considering the size of
well-trained models. Besides, the parameter numbers in VR are solely
determined by the size of a single input image, independent of well-trained
models, and remain fixed as the well-trained model size grows.

  * •

VR is suitable for all well-trained models, regardless of the architecture,
whereas finetuning-based methods are usually designed for a specific
architecture (e.g., LoRA is specifically designed for ViT).

  * •

VR improves the performance of the target task by altering the input and
output space, and analyzing these changes may help understand why the model
can also perform well in the target domain.

  * •

By changing the input and output spaces while fixing the well-trained model,
VR avoids practical issues such as catastrophic forgetting (i.e., the well-
trained model may lose previously learned representations when being finetuned
for new tasks).

  * •

VR can be attached to most mainstream finetuning methods to further improve
performance.

  * •

In future research, VR could also utilize the well-trained model as a black
box. This approach might prove useful for re-purposing models that only offer
an application programming interface.

####  E.3.2 Weaknesses

  * •

When target tasks are more challenging than the tasks well-trained models have
been trained on, merely adjusting the input space may not be sufficient for
satisfied performance. This poses a challenge for VR.

  * •

For better performance approaching re-training or fully finetuning,
integrating VR with other finetuning methods appears necessary (e.g., VR may
be combined with finetuning the fully connected layer). How to train the
combined model more effectively remains a task for future research.

##  Appendix F Additional Visualization Results

Figure 13: Original Images and Visual Reprogramming Results on CIFAR10 Figure
14: Original Images and Visual Reprogramming Results on CIFAR100 Figure 15:
Original Images and Visual Reprogramming Results on SVHN Figure 16: Original
Images and Visual Reprogramming Results on GTSRB Figure 17: Original Images
and Visual Reprogramming Results on Flowers102 Figure 18: Original Images and
Visual Reprogramming Results on DTD Figure 19: Original Images and Visual
Reprogramming Results on UCF101 Figure 20: Original Images and Visual
Reprogramming Results on Food101 Figure 21: Original Images and Visual
Reprogramming Results on SUN397 Figure 22: Original Images and Visual
Reprogramming Results on EuroSAT Figure 23: Original Images and Visual
Reprogramming Results on OxfordPets

Figure 13-23 show sample images of the VR results of SMM on 11 datasets. These
figures show that (1) our VR method does not alter the input space heavily; it
only adds noise within a limited range, which ensures that the original images
remain intact; (2) the more different the target domain is (e.g., GTSRB and
SVHN), the more pronounced the noise pattern will be; (3) on datasets that
prefer VR to be a narrow padding-sized watermark, SMM will convergence to a
similar situation, that is, the noise at the outer frame of the images is much
greater than that inside the images (e.g., UCF101, Food101, OxfordPets and
SUN397).