[table]capposition=top

# TransCL: Transformer Makes Strong and Flexible Compressive Learning

Chong Mou, Jian Zhang C. Mou and J. Zhang are with the School of Electronic
and Computer Engineering, Peking University Shenzhen Graduate School, Shenzhen
518055, China. Jian Zhang is also with the Peng Cheng Laboratory, Shenzhen,
China. (Corresponding author: Jian Zhang.)
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Manuscript received
August 26, 2021; revised December 10, 2021, March 24, 2022 and July 6, 2022;
accepted July 14, 2022. This work was supported by National Natural Science
Foundation of China (61902009).

###### Abstract

Compressive learning (CL) is an emerging framework that integrates signal
acquisition via compressed sensing (CS) and machine learning for inference
tasks directly on a small number of measurements. It can be a promising
alternative to classical image-domain methods and enjoys great advantages in
memory saving and computational efficiency. However, previous attempts on CL
are not only limited to a fixed CS ratio, which lacks flexibility, but also
limited to MNIST/CIFAR-like datasets and do not scale to complex real-world
high-resolution (HR) data or vision tasks. In this paper, a novel transformer-
based compressive learning framework on large-scale images with arbitrary CS
ratios, dubbed TransCL, is proposed. Specifically, TransCL first utilizes the
strategy of learnable block-based compressed sensing and proposes a flexible
linear projection strategy to enable CL to be performed on large-scale images
in an efficient block-by-block manner with arbitrary CS ratios. Then,
regarding CS measurements from all blocks as a sequence, a pure transformer-
based backbone is deployed to perform vision tasks with various task-oriented
heads. Our sufficient analysis presents that TransCL exhibits strong
resistance to interference and robust adaptability to arbitrary CS ratios.
Extensive experiments for complex HR data demonstrate that the proposed
TransCL can achieve state-of-the-art performance in image classification and
semantic segmentation tasks. In particular, TransCL with a CS ratio of $10\%$
can obtain almost the same performance as when operating directly on the
original data and can still obtain satisfying performance even with an
extremely low CS ratio of $1\%$. The source codes of our proposed TransCL is
available at https://github.com/MC-E/TransCL/.

###### Index Terms:

Compressed sensing, Compressive learning, Transformer, Image classification,
Semantic segmentation.

##  1 Introduction

Signal acquisition and processing methods have changed substantially over the
last several decades. Many methods have moved from the analog to the digital
domain, creating sensing systems that are more robust, flexible, and cost-
effective than their analog counterparts. However, in many important real-
world applications, the resulting Nyquist rate is so high that it is not
viable or even physically impossible. Compressed sensing (CS), built upon the
groundbreaking works by Candes et al. [1] and Donoho et al. [2], has become a
more efficient and hardware-friendly signal acquisition method and has been
receiving more and more attention. CS theory shows that when a signal has a
sparse representation on a certain basis, one can vastly reduce the number of
required samples—below the Nyquist rate and can still recover the signal
perfectly from its CS measurement. CS has been applied in many practical
applications, including but not limited to single-pixel imaging [3, 4],
accelerating magnetic resonance imaging (MRI) [5], wireless telemonitoring
[6], cognitive radio communication [7], and snap compressive imaging [8, 9].
Correspondingly, there exist several types of CS equipments, e.g., single-
pixel camera [3], flexible voxels camera [10], and P2C2 camera [11], which can
project images from the high-dimensional image domain to the low-dimensional
measurement domain.

Most of the recent works [12, 13, 14, 15, 16, 17, 18, 19] in the CS community
focus on how to reconstruct the original signal, and less attention has been
devoted to whether one can perform high-level vision tasks direct in the
measurement domain. However, in many applications, such as classification and
segmentation, we are not interested in obtaining a precise reconstruction of
the scene under view, but rather are only interested in the results of
inference tasks. There also exist coupling problems between the reconstruction
results and the inference model trained on natural images. Moreover, with the
development of deep learning, data privacy has gradually become a concern [20,
21]. Thus, in some specific applications, signal reconstruction is undesirable
since this step can potentially disclose private information. Directly
performing inference tasks in the measurement domain without knowing the
sampling matrix makes it almost inaccessible to original visual perception,
which is an appropriate scheme to protect data privacy during network training
and inference.

Calderbank et al. [22] and Davenport et al. [23] first construct the concept
of compressive learning (CL) in which the inference system is directly built
on the top of CS measurements without an explicit reconstruction step. Some
theoretical demonstrations of CL have been made in [24, 25]. Some early works
have been made in [26, 27, 28, 29]. The very recent researches (Tran et al.
[30, 31]) propose an efficient method to exploit the multidimensional property
of CL. Although there have been several novel attempts, some intractable
problems make existing compressive learning methods not yet comparable to the
image-domain methods. First, the information loss during the CS sampling
process due to the lax sparsity of natural images makes the performance of CL
significantly lower than image-domain methods. Second, a large-scale sampling
matrix and large matrix multiplication are needed to project real-world
images, making the sampling process complicated. The high complexity in CS of
high-resolution images is the reason of limiting existing CL methods [26, 27,
28, 29, 30, 31] to MNIST/CIFAR-like datasets [32, 33]. Third, different CS
ratios will generate measurement vectors with different lengths, leading to
the fact that most CL methods can only perform inference tasks with a fixed CS
ratio, limiting their practice in some real-world applications. In this paper,
we propose several solutions to solve these weaknesses, helping bridge the gap
between CL and image-domain methods.

In fact, an image and its CS measurement have a lot in common, and several
helpful priors in the image domain can also be applied in the measurement
domain. For instance, the non-local self-similarity prior, which assumes that
similar content would recur across the whole scene. This prior has been widely
used in the community of image reconstruction [34, 35]. However, how to
combine this effective prior naturally into CL to resist the information loss
has been hardly studied. Recently, Transformer (Vaswani et al. [36]) has
attracted more and more attention from the community of computer vision. This
family of networks [36, 37, 38, 39], i.e., transformer networks, originate
from machine translation and require serialized inputs, and are very good at
modeling long-range correlations in sequences. Thus, transformer can be
applied in CL to compensate for information loss through element-wise
correlations. In order to make complex real-world data tractable, we propose
to adopt the strategy of block-based compressed sensing (BCS) [40, 41] into
CL, which applies CS to image blocks through the same operator to generate a
sequence of CS measurements instead of an entire one. BCS is mostly suitable
for processing very large-scale images in CL, while requiring significantly
less memory to store the sensing matrix and provides a serialized input to the
transformer layer. Thus, a significant finding is that BCS and Transformer can
be well combined to solve the two main challenges faced by existing CL
methods, which greatly inspires our proposed TransCL framework. We further
propose a flexible linear projection to enable a single model to handle input
measurements with arbitrary CS ratios. In addition, our TransCL exploits a
learnable binary sampling matrix, which is more practical and hardware-
friendly.

The main contributions of this paper are summarized as follows:

  * •

We reformulate the CL problem from a sequence-to-sequence sensing and learning
perspective and propose a novel transformer-based CL framework, dubbed
TransCL. Our TransCL can be generalized to more complex high-level vision
tasks and real-world benchmarks, which helps bridge the gap between inference
tasks in the image and measurement domains.

  * •

We propose a flexible projection scheme to enable a single model to sample and
infer images with arbitrary CS ratios. In this way, our method can be trained
only once to deal with general cases.

  * •

We propose incorporating the strategy of learnable block-based compressed
sensing (BCS) and binary sampling matrix into CL, making the whole system more
memory-saving and hardware-friendly.

  * •

Extensive experiments demonstrate that our TransCL can achieve state-of-the-
art performance in various compressive learning tasks (i.e., image
classification and semantic segmentation), especially at extremely low CS
ratios (e.g., 1%). We further verify the robustness of our TransCL in the face
of disturbances.

Figure 1: Illustration of CS acquisition for a still image with a single-pixel
camera [2], which can be formulated as $\mathbf{y}=\mathbf{\Phi}\mathbf{x}$.

##  2 Related Work

Our proposed TransCL is closely related to existing compressed sensing
methods, compressive learning methods, and transformer networks. Thus, in what
follows, we give a brief review of these aspects and focus on the specific
methods most relevant to our own.

###  2.1 Compressed Sensing

Compressed sensing (CS) [2] is an emerging mathematical paradigm that permits,
under certain conditions, linear projection of a signal into a dimension much
lower than that of the original signal while allowing exact recovery of the
signal from its projection. That is, from much fewer acquired measurements
than determined by Nyquist sampling theory, CS theory demonstrates that a
signal can be reconstructed with high probability when it exhibits sparsity in
some transform domain. This novel acquisition strategy is much more hardware-
friendly and enables image or video acquisition with a sub-Nyquist sampling
rate. By leveraging the redundancy inherent to a signal, CS conducts sampling
and compression at the same time, which greatly alleviates the need for high
transmission bandwidth and large storage space, enabling low-cost on-sensor
data compression.

Concretely, suppose we have the CS measurement
$\mathbf{y}$$\in$$\mathbb{R}^{M}$ of the signal $\mathbf{x}\in\mathbb{R}^{N}$,
i.e., $\mathbf{y}=\mathbf{\Phi x}$, where $M\ll N$,
$\mathbf{\Phi}\in\mathbb{R}^{M\times N}$ is the sampling matrix, and CS ratio
is defined as $\gamma=\frac{M}{N}$. Because the number of unknowns is much
larger than the number of observations, reconstructing $\mathbf{x}$ from its
corresponding $\mathbf{y}$ is impossible in general; however, if $\mathbf{x}$
is known to be sufficiently sparse in some domain, the exact recovery of
$\mathbf{x}$ is possible——that is the fundamental tenet of CS theory. Fig. 1
illustrates the CS acquisition process of a still image with a single-pixel
camera.

Since the advent of the CS, most methods [12, 13, 14, 15] focus on the inverse
problem of CS, which aims to fully recover the signal from its CS measurement.
Given the linear measurement $\mathbf{y}$, traditional reconstruction methods
usually restore the original signal by solving the following optimization
problem:

$\mathop{min}\limits_{\mathbf{{x}}}\frac{1}{2}||\mathbf{\Phi}\mathbf{{x}}-\mathbf{y}||^{2}_{2}+\lambda||\psi(\mathbf{x})||_{1},$
(1)

where $\psi(\mathbf{x})$ represents the transformation of $\mathbf{x}$ with
respect to some transform operator $\psi(\cdot)$, and the sparsity of
$\psi(\mathbf{x})$ is encouraged by the $\ell_{1}$ norm with $\lambda$ being
the regularization parameter (generally pre-defined). However, in many high-
level applications, we are not interested in obtaining a precise
reconstruction of the scene under view, but rather are the results of specific
inference tasks.

###  2.2 Compressive Learning

As illustrated by the red dashed line in Fig. 2, compressive learning (CL)
aims to directly perform inference tasks in the measurement domain without
signal reconstruction. The concept of CL is first constructed in [23, 22].
This strategy bypasses the reconstruction process and greatly reduces the
amount of data. Thus, it is highly cost-effective in terms of storage memory
and computational complexity. Moreover, it can help to protect data privacy.
The theoretical basics of CL have been well established. The early research
[22] demonstrates that given certain conditions of the sampling matrix
$\mathbf{\Phi}$, the performance of a linear support vector machine (SVM)
trained on CS measurements is as good as the best linear threshold classifier
trained on the original signal. [24] proves that the Kullback-Leibler and
Chernoff distances between two probability density functions are preserved up
to a factor of $\frac{M}{N}$, where $M$ and $N$ represent the length of the CS
measurement and original signal, respectively. [25] exploits the performance
bound under perturbation bringing from the uncertainty of the measurement
matrix. Some early works [26, 27, 28, 29] study directly performing inference
tasks in the measurement domain. [42] applies CL to action recognition. With
the advances in computing hardware and stochastic optimization techniques,
[43, 44] propose to jointly optimize the sampling matrix with the whole
network in an end-to-end manner. Benefiting from the programmable CS methods
[45, 46], the learned sampling matrix has the opportunity to be deployed to
hardware for CS. The very recent researches [30, 31] propose an efficient
method to exploit the multidimensional property of CL. However, existing CL
methods are restricted to a small range of high-level vision tasks (e.g.,
image classification/recognition). These methods are usually difficult or
unstable [47] to handle large-scale images, and most works focus on small
datasets, such as MNIST [32] and CIFAR [33], not comparable to complex real-
world high-resolution data. Furthermore, these CL methods only handle
inference tasks with a fixed CS ratio without considering the variation of CS
ratio in practical applications.

Figure 2: Illustration of two ways of performing inference tasks (i.e., image
classification) on CS measurements. The blue dashed line presents the
conventional two-stage operation (reconstruction first and then inference in
the image domain), and the red dashed line represents the single-stage
compressive learning (CL) direct in the measurement domain. Figure 3:
Illustration of the self-attention (SA) mechanism [36], which is used to
construct long-range correlations among the input sequence. Figure 4: An
overview of our proposed transformer-based compressive learning (TransCL)
framework, which is composed of compressed sensing module (CSM), Transformer-
based Backbone (TB) and Task-oriented Head (TH).

(a) Transformer-based Backbone (TB)

(b) Task-oriented Head for Image Classification (THIC)

(c) Task-oriented Head for Semantic Segmentation (THSS)

Figure 5: Detailed architectures of transformer-based backbone (TB) and task-
oriented heads (TH) in our TransCL. (a) presents the details of our
transformer-based backbone (TB). (b) is the illustration of task-oriented head
for image classification (THIC). (c) presents the details of task-oriented
head for semantic segmentation (THSS), which is a plain network to perform
upsampling and predicting in a progressive manner.

###  2.3 Transformer

Transformer was first introduced in [36] for machine translation, which is a
self-attention mechanism to construct long-range correlations for elements in
different places. Given an input sequence $\mathbf{E}\in\mathbb{R}^{C\times
L}$ with the sequence length being $L$ and the number of channels being $C$, a
pure transformer layer has a global receptive field to build correlations for
information transfer. At the beginning of the transformer network, each
element of the input sequence is first embedded by a linear projection
$\mathbf{W}$$\in$$\mathbb{R}^{d\times C}$ to the length of $d$ and then added
by the position embedding $\mathbf{P}$$\in$$\mathbb{R}^{d\times L}$. Let
$\mathbf{Z}^{l}$$\in$$\mathbb{R}^{d\times L}$ represent the input of the
$l$-th transformer layer. The core component is a self-attention module with
multiple heads. As illustrated in Fig. 3, each head performs non-local
operation in a triplet of embedded features (query, key, value) computed from
the input as:

$query=\mathbf{W}_{Q}\mathbf{Z}^{l},\ key=\mathbf{W}_{K}\mathbf{Z}^{l},\
value=\mathbf{W}_{V}\mathbf{Z}^{l},$ (2)

where
$\mathbf{W}_{Q}$,$\mathbf{W}_{K}$,$\mathbf{W}_{V}$$\in$$\mathbb{R}^{d^{\prime}\times
d}$ are the learnable parameters of three linear projection layers. The self-
attention operation can be formally defined as:

$\text{SA}(\mathbf{Z}^{l})=\left(\mathbf{W}_{V}\mathbf{Z}^{l}\right)\text{Softmax}\left(\frac{(\mathbf{W}_{Q}\mathbf{Z}^{l})^{\top}\mathbf{W}_{K}\mathbf{Z}^{l}}{\sqrt{d}}\right).$
(3)

However, there exists a gap between $2D$ image and $1D$ sequence in applying
transformer to vision tasks. A straightforward method [48] is to flat an input
feature map $\mathbf{F}_{in}$$\in$$\mathbb{R}^{C\times H\times W}$ into a
sequence with the length of $HW$. Given the quadratic model complexity of the
transformer, it is not possible to handle the input sequence with high
dimensions. [49] proposed a solution to unfold an input image into a sequence
of image blocks with each block size being $B$$\times$$B$, significantly
reducing the sequence length to $\frac{HW}{B^{2}}$. This strategy can reduce
the computational complexity of the transformer layer. Since ViT was proposed
[49], there have been several attempts to adapt transformers towards vision
tasks, including image classification [50, 51], object detection [52, 53],
segmentation [54], human pose estimation [55], and image restoration [56, 57,
58]. However, this strategy has hardly been studied in the community of
compressive learning (CL) and compressed sensing (CS).

##  3 Methodology

In this section, we will elaborate on our proposed strong and flexible
transformer-based compressive learning framework, dubbed TransCL.

###  3.1 Framework

The global architecture of our TransCL is presented in Fig. 4, which is mainly
composed of a Compressed Sensing Module (CSM), a Transformer-based Backbone
(TB), and a Task-oriented Head (TH). Note that our TransCL targets performing
high-level vision tasks on real-world images, rectifying the information loss
in the measurement domain and handling input with arbitrary CS ratios. CSM is
designed to perform sensing and compressing in a block-by-block manner with
arbitrary CS ratios, and it is jointly optimized with the whole network. To
handle the serialized output of CSM and resist information loss, TB is
designed as a pure transformer architecture. A creative modulation is that TB
begins with a flexible linear projection (FLP) layer to project the input with
arbitrary CS ratios to a set of measurement embeddings with the same vector
length. Based on TB, Th utilizes the extracted features to perform specific
vision tasks. We will provide the details below.

###  3.2 Compressed Sensing Module

As illustrated in Fig. 4, CSM is a data acquisition step via CS, aiming to
compress the scene to a small number of measurements through a linear sampling
matrix $\mathbf{\Phi}$. In our TransCL, we set $\mathbf{\Phi}$ as a trainable
parameter, which is jointly optimized with the whole network. Based on these
basic settings, we carry out more elaborate designs to make CSM powerful and
flexible.

####  3.2.1 Efficient Block-based Arbitrary Sampling

Due to the high computational complexity in sampling large-scale images, most
existing CL methods are limited in MNIST/CIFAR-like datasets. To process high-
resolution images in real-world applications and provide serialized
measurements to the transformer-based backbone (TB), we propose to apply the
strategy of block-based compressed sensing (BCS) [40][41] to CL tasks. BCS is
able to sense and compress images in a block-by-block manner through the same
operator, which has the advantage of decreasing the bandwidth required to
transmit measurement vectors between system memory and the DMD array.
Specifically, given a high-resolution input image of size $H$$\times$$W$.
Instead of direct CS by a large sampling matrix
$\mathbf{\Phi}$$\in$$\mathbb{R}^{M\times N}$, where $M$=$\gamma N$ and
$N$=$HW$. As shown in Fig. 6, we first divide the image into $L$ non-
overlapped blocks with each block size being $B$$\times$$B$ and
$L$=$\frac{HW}{B^{2}}$. Denote $\mathbf{x}_{i}$$\in$$\mathbb{R}^{B^{2}}$ as
the vectorized representation of the $i$-th image block and define
$\mathbf{X}$$=$$[\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{L}]$. The
corresponding block-based sampling matrix is represented as
$\mathbf{\Phi}^{\gamma}_{B}$$\in$$\mathbb{R}^{M_{B}^{\gamma}\times B^{2}}$,
where $M_{B}^{\gamma}$$=$$\gamma B^{2}$. Then, the resulting CS measurement
$\mathbf{y}_{i}$$\in$$\mathbb{R}^{M_{B}^{\gamma}}$ is generated by
$\mathbf{y}_{i}$$=$$\mathbf{\Phi}_{B}^{\gamma}\mathbf{x}_{i}$. Finally, our
CSM generates a sequence of measurements:
$\mathbf{Y}$$=$$[\mathbf{y}_{1},\mathbf{y}_{2},...,\mathbf{y}_{L}]$. The
output of CSM is formulated as:

$\mathbf{Y}=\mathbf{\Phi}_{B}^{\gamma}\mathbf{X},$ (4)

where $\mathbf{Y}$$\in$$\mathbb{R}^{M_{B}^{\gamma}\times L}$. Note that BCS
only needs to store a block-based sampling matrix instead of a full one and
greatly reduces the computational burden of matrix multiplication through a
block-by-block strategy, which is quite memory efficient. In addition, BCS can
also be easily implemented on hardware by introducing a mask for each square
region to drive the DMD of the single-pixel camera to route light from each
region to the detector one by one [59] [60].

Furthermore, unlike existing methods [43, 44, 30, 31, 61] for jointly
optimizing the sampling matrix and inference model for a fixed CS ratio each
time, we propose a learnable and powerful CSM, which can generate a block-
based sampling matrix with arbitrary CS ratio $\gamma$ by training once. Our
proposed sampling scheme is presented in Fig. 7(a). As a reusable scheme, we
first define a learnable sampling base matrix (SBM) as $\mathbf{\Phi}$ of size
$B^{2}$$\times$$B^{2}$. Then, the sampling matrix $\mathbf{\Phi}_{B}^{\gamma}$
with any CS ratio $\gamma$ used in Eq. (4) can be directly constructed by
truncating the first $M_{B}^{\gamma}$ rows from SMB $\mathbf{\Phi}$, i.e.,
$\mathbf{\Phi}_{B}^{\gamma}$$=$$\mathbf{\Phi}[{:M_{B}^{\gamma}]}$. Hence, our
CSM only needs one learnable SBM to generate matrices with arbitrary sampling
ratios ($B^{2}$ in total), and it only needs to be trained once, without
introducing $B^{2}$ different sampling matrices as learnable parameters and
training for $B^{2}$ times.

####  3.2.2 Hardware-friendly Binary Sampling Design

Although float-point CS matrices have been widely used, the most efficient and
energy-saving hardware utilizes binary sampling matrix ($1$ or $-1$) in real-
world applications. This hardware-friendly choice has been studied in the
recent reconstruction work [61]. In this paper, we exploit this property in
the field of CL to further train a binary SBM $\mathbf{\Phi}$ in our CSM. We
denote $\widetilde{\mathbf{\Phi}}$ as the binary version of the float-point
SBM $\mathbf{\Phi}$, which is computed by
$\widetilde{\mathbf{\Phi}}=\mathcal{B}(\mathbf{\Phi})$, where
$\mathcal{B}(\cdot)$ is an element-wise operation defined as:

$\mathcal{B}(a)=\left\\{\begin{array}[]{ll}1,&if~{}~{}a\geq 0;\\\
-1,&if~{}~{}a<0.\end{array}\right.$ (5)

However, such a setting makes it difficult to calculate the gradient. Inspired
by Binarized neural networks (BNN) [62] and differentiable quantization [63],
we introduce a proxy function $\tilde{\mathcal{B}}(\cdot)$ to approximate
$\mathcal{B}(\cdot)$, which is defined as:

$\tilde{\mathcal{B}}\left(a\right)=\left\\{\begin{array}[]{ll}1,&if~{}~{}a>1;\\\
a,&if~{}~{}-1\leq a\leq 1;\\\ -1,&if~{}~{}a<-1.\end{array}\right.$ (6)

Thus, the gradient of $\mathcal{B}(\cdot)$, denoted by
$\mathcal{B}^{\prime}(\cdot)$, can be approximated by the gradient of
$\tilde{\mathcal{B}}(\cdot)$, denoted by
$\tilde{\mathcal{B}}^{\prime}(\cdot)$, that is:

$\tilde{\mathcal{B}}^{\prime}(a)=\left\\{\begin{array}[]{ll}1,&if~{}~{}-1\leq
a\leq 1;\\\ 0,&otherwise.\end{array}\right.$ (7)

Therefore, $\mathcal{B}(\cdot)$ is used in the forward propagation, while
$\tilde{\mathcal{B}}(\cdot)$ is used in back-propagation.

Figure 6: Illustration of compressed sensing module (CSM) in our TransCL.

(a) Sampling with
arbitrary CS ratio

(b) Projection with arbitrary CS ratio

Figure 7: Illustration of generating the sampling matrix
$\mathbf{\Phi}_{B}^{\gamma}$ and the projection matrix
$\mathbf{W}_{B}^{\gamma}$ with arbitrary CS ratio $\gamma$ from their
corresponding base matrices $\mathbf{\Phi}$ and $\mathbf{W}$, respectively.

###  3.3 Transformer-based Backbone

Fig. 5(a) presents the detailed architecture of our proposed transformer-based
backbone (TB), which is used to extract informative features from the
serialized measurements $\mathbf{Y}$ with arbitrary CS ratios. TB is composed
of a flexible linear projection (FLP) and a stack of transformer layers [36]
dubbed transformer encoder (TE).

Flexible Linear Projection. As shown in Fig. 5(a), our FLP is to project the
input measurements $\mathbf{Y}$ of size $M_{B}^{\gamma}$$\times$$L$ to the
measurement embedding $\mathbf{Q}$ of size $d$$\times$$L$, followed by adding
a set of learnable position embedding, denoted by $\mathbf{P}$ of size
$d$$\times$$L$. Note that different sampling matrix $\Phi_{B}^{\gamma}$ with
different $\gamma$ will produce different $\mathbf{Y}$ with different sizes.
To overcome the weakness of existing fixed linear projections that can not
handle the measurement vectors with variable lengths and analogous to SBM, we
similarly define a trainable projection base matrix (PBM) as $\mathbf{W}$ of
size $d$$\times$$B^{2}$. Then, corresponding to each
$\mathbf{\Phi}_{B}^{\gamma}$ with CS ratio $\gamma$, we adopt the first
$M_{B}^{\gamma}$ columns from PBM as the projection matrix
$\mathbf{W}_{B}^{\gamma}$, i.e.,
$\mathbf{W}_{B}^{\gamma}=\mathbf{W}[:,:M_{B}^{\gamma}]$, which is of size
$d$$\times$$M_{B}^{\gamma}$. Obviously, we only need to jointly train the PBM
$\mathbf{W}$ once, rather than train $B^{2}$ different projection matrices for
$B^{2}$ times, making our FLP much more flexible. Therefore, as illustrated in
Fig. 5(a), the output of our FLP layer, denoted by $\mathbf{Z}^{0}$, can be
formulated as:

$\mathbf{Z}^{0}=\mathbf{Q}+\mathbf{P}=\mathbf{W}_{B}^{\gamma}\mathbf{Y}+\mathbf{P},$
(8)

where $\mathbf{Z}^{0}$$\in$$\mathbb{R}^{d\times L}$. Note that an extra class
token exists in the image classification task concatenated with
$\mathbf{Z}^{0}$ going through the transformer encoder to collect the category
information for prediction. The class token is inherited from [37] and has
been widely used in recent transformer-based image classification researches
[49, 50, 51].

Transformer Encoder. Taking the embedding sequence $\mathbf{Z}^{0}$ as input,
the transformer encoder (TE), constructed by a stack of pure transformer
layers [36] as shown in Fig. 5(a), is deployed to extract deeper features from
the CS measurements. The core part of each transformer layer is a multiheaded
self-attention (MSA) block. Each head performs the operation defined in Eq.
(3), and their outputs are concatenated together. Multilayer perceptrons
(MLP), Layer Norm (LN), and residual connections are applied in the tail of
each transformer layer. Thus, the output of each transformer layer has the
following formulation:

$\begin{cases}&\widetilde{\mathbf{Z}}^{l}=\text{MSA}(\text{LN}(\mathbf{Z}^{l}));\\\
&\mathbf{Z}^{l+1}=\text{MLP}(\text{LN}(\widetilde{\mathbf{Z}}^{l}))+\widetilde{\mathbf{Z}}^{l},\
l=0,1,2,...,K-1,\\\ \end{cases}$ (9)

where $K$ is the number of transformer layers (12 by default), and
$\mathbf{Z}^{K}$ denotes the output of TE.

###  3.4 Task-oriented Head

Our TransCL focuses on two high-level vision tasks: image classification and
semantic segmentation. For each task, we adopt a specific task-oriented head
(TH) to process $\mathbf{Z}^{K}$.

####  3.4.1 Task-oriented Head for Image Classification

As illustrated in Fig. 5(b), the take-oriented head for image classification
(THIC) is a simple combination of linear projection and layer norm. The input
of THIC is the learnable class token (i.e., $\mathbf{Z}^{K}[:,0]$) with the
vector length being $768$, and then the classification head performs a
prediction on the top of the class token. Note that the class token only
exists in the image classification task.

####  3.4.2 Task-oriented Head for Semantic Segmentation

The take-oriented head for semantic segmentation (THSS) follows the same
architecture as the recent transformer-based method [54], which takes the
original image as input and is also the upper bound model of our TransCL in
this task. Since we obtain measurements in an image block of size
$16$$\times$$16$, the size of $\mathbf{Z}^{K}$ is
$\frac{W}{16}$$\times$$\frac{H}{16}$. Through THSS, $\mathbf{Z}^{K}$ is then
gradually upsampled to a segmentation map with the same size as the original
image. As illustrated in Fig. 5(c), THSS is essentially a simple progressive
upsampling model without any bells and whistles. Specifically, each upsampling
block is a triple combination of a $1\times 1$ convolutional layer, a
Synchronized BatchNorm, and a ReLU [64] activation function ending with a
bilinear upsampling operation. The simpleness of THSS can also better
demonstrate the effectiveness of our proposed TransCL.

Original

Reconstructed

Original

Reconstructed

Figure 8: Visualization of privacy protection. We present the original image
and reconstructed result generated by [61] without knowing the sampling
matrix.

###  3.5 Privacy Protection

It is worth emphasizing that our TransCL is superior in privacy protection. On
the one hand, the CS measurements have little visual information, which is an
inherent property of CS. Although there exist several CS reconstruction
methods, e.g., [12, 13, 61, 65], they all need to know the corresponding
sampling matrix, especially at low CS ratios. To verify it, we adopt the
recent [61] for reconstruction with the CS measurements generated by our
trained CSM rather than its own learned sampling matrix. The reconstructed
results are presented in Fig. 8, from which one can observe that it is almost
inaccessible to original images without the corresponding sampling matrix. On
the other hand, the transformer-based backbone (TB) in our TransCL can further
strengthen the ability of privacy protection due to the serialized
intermediate features, which are more abstract and invisible than CNN
counterparts. Thus, we believe that our TransCL can protect data privacy in
the process of data transmission and processing.

##  4 Experiment

To demonstrate the superiority of our proposed TransCL, we apply it to two
representative high-level vision tasks, i.e., image classification and
semantic segmentation. The experiment design comes from two factors. On the
one hand, studying several linear projection layers under traditional problem
settings with tiny images has little significance for promoting CL. The urgent
problem faced by CL is how to handle real-world benchmarks and perform
inference as flexibly as image-domain methods. On the other hand, though our
approach is a new formulation of CL under new problem settings, only comparing
our TransCL with image-domain methods on real-world benchmarks is inadequate.
We also need to compare our TransCL with existing CL methods (i.e., VCL [66]
and MCL [30]), which are difficult to handle large-scale images. Thus, in the
experiment, we focus on comparing our TransCL with the recent image-domain
methods to present that CL can also be applied to complex and real-world
benchmarks (e.g., ImageNet [67], ADE20K [68], PASCAL Context [69], and
Cityscapes [70]) without rigid restrictions from data scale and CS ratio.
Additionally, we also train and evaluate our TransCL on tiny datasets (e.g.,
CIFAR10 and CIFAR100 [33]) to directly compare with existing CL methods.

###  4.1 Model Details and Variants

In image classification, the transformer architecture is the same as the base
vision transformer (ViT-B), which contains $12$ transformer layers, with the
number of heads being $12$ in each layer. The dimension of the measurement
embedding $\mathbf{Q}$ is set as $768$. In semantic segmentation, we utilize
the same backbone (ViT-L) as the upper-bound model (SETR). It contains $24$
transformer layers, with the number of heads being $16$. The dimension of the
measurement embedding $\mathbf{Q}$ is set as $1024$. Several variants of our
proposed TransCL are defined with different names. For instance,
“TransCL-16-10” represents the variant with the block size of $16$$\times$$16$
and CS ratio of $10\%$. In our experiments, we evaluate our TransCL with both
fixed CS ratios and arbitrary CS ratios. The fixed CS ratios include $10\%$,
$5\%$, $2.5\%$, and $1\%$. We denote $arb$ as the suffix of the variant that
can handle input measurements with arbitrary CS ratios. The suffixes of block
size include $16$ and $32$.

###  4.2 Image Classification

Experiment Settings. Image classification is the most representative high-
level vision task. For this application, we apply our model to the ImageNet
2012 [67], CIFAR-10, and CIFAR-100 [33] datasets:

  * •

ImageNet 2012 contains $1.28$ million training images and $50,000$ validation
images from $1,000$ categories. The image resolution is set as
$384$$\times$$384$. It is a commonly used benchmark in the image domain.
However, limited by the computational cost, little CL method can be applied to
this dataset at such scale.

  * •

CIFAR-10 consists of $60,000$ $32$$\times$$32$ colour images in $10$ classes,
with $6,000$ images per class. There are $50,000$ training images and $10,000$
test images.

  * •

CIFAR-100 is just like the CIFAR-10, except it has $100$ classes. For each
class, there are $500$ training images and $100$ testing images.

Note that CIFAR-10 and CIFAR-100 are two datasets with tiny images. The
transformer architecture has difficulty in processing tiny input. ViT [49]
upsampled images from these two datasets and then performed inference.
Considering the upsampling will introduce additional information into CS
measurements, we interpolate the measurements $\mathbf{Y}$
($\mathbb{R}^{M_{B}^{\gamma}\times
L^{\prime}}$$\mapsto$$\mathbb{R}^{M_{B}^{\gamma}\times L}$) instead of the
input images. In this way, no extra information is involved in the
measurements. During training, we apply the same training settings as the
upper bound model (ViT-B [49]) of our TransCL. Specifically, we use SGD with
momentum being $0.9$ and the weight-decay being $10^{-4}$ as the optimizer.
The batch size is set as $256$, and we train our proposed TransCL on 4 NVIDIA
V100 GPUs. Realizing the importance of the pre-trained model in transformer-
based methods, we initialize the parameters in TB with the pre-trained model
of ViT-B. The CSM, TB, and TH in TransCL are jointly optimized by a cross-
entropy loss. Moreover, traditional ViT requires large-scale datasets (e.g.,
ImageNet-21K) for pre-training. Recently, some strategies were proposed to
build data-efficient ViT. For instance, the timm [71] presents a training
procedure to improve the top-1 accuracy of ViT-B from 77.91$\%$ to 79.35$\%$
with ImageNet-1K only. The recent DeiT [50] utilizes a more elaborate
optimization process and knowledge distillation strategies to further improve
the performance. Inspired by DeiT, in this paper, we also present a data-
efficient version (i.e., ), which is trained with knowledge distillation on
ImageNet-1K only. The distillation process and teacher network follow the same
design as DeiT.

TABLE I: Image classification performance on validation dataset of ImageNet. “Top-1 ACC” denotes the top-1 ($\%$) accuracy. Performances of different methods, model parameters, and the number of measurements for each image are reported.  Method | Meas. | Param./Flops | Top-1 ACC
---|---|---|---
ResNet-18[72] |  3$\times$50176 |  12M/1.9G | 69.83
ResNet-50[72] |  3$\times$50176 | 25M/4.2G | 76.20
ResNet-101[72] |  3$\times$50176 | 45M/7.8G | 77.41
ResNet-152[72] |  3$\times$50176 | 60M/11.58G | 78.33
ViT-B-16[49] |  3$\times$147456 | 86M/49.3G | 83.97
ViT-B-32[49] |  3$\times$147456 | 88M/12.3G | 81.28
PVT-Large[53] |  3$\times$50176 | 62M/9.9G | 81.71
DeiT-B $\uparrow 384$[50] |  3$\times$147456 | 86M/49.4G | 83.12
Twins[73] |  3$\times$50176 | 99M/14.8G | 83.70
VCL-T-10 |  3$\times$14745 | 589M/50.8 | 67.59
VCL-T-1 |  3$\times$1474 | 136M/49.6 | 59.89
MCL-T-10 |  3$\times$14770 | 87M/49.4 | 74.07
MCL-T-1 |  3$\times$1477 | 86M/49.3 | 68.92
TransCL-16,32-10 |  3$\times$14745 | 86,88M/49.3,12.3G | 83.86,81.82
TransCL-16,32-5 |  3$\times$7372 | 86,88M/49.3,12.3G | 83.05,81.53
TransCL-16,32-2.5 |  3$\times$3686 | 86,88M/49.3,12.3G | 81.34,80.50
TransCL-16,32-1 | 3$\times$1474 | 86,88M/49.3,12.3G | 78.86,78.00
TransCL-16-10 |  3$\times$14745 | 86M/49.3G | 83.29
TransCL-16-5 |  3$\times$7372 | 86M/49.3G | 82.46
TransCL-16-2.5 |  3$\times$3686 | 86M/49.3G | 81.24
TransCL-16-1 | 3$\times$1474 | 86M/49.3G | 78.58
TABLE II: Image classification performance on validation datasets of CIFAR-10 and CIFAR-100 [33]. Top-1 ($\%$) accuracy is presented below.  CS Ratio | Method | Backbone | Meas. |  CIFAR -10 |  CIFAR -100
---|---|---|---|---|---
$25\%$ |  VCL [66] | ResNet110 | 768 | 78.56 | 53.03
MCL [30] | ResNet110 | 760 | 89.22 | 67.21
VCL-T | ViT-B-32 | 768 | 93.80 | 78.65
MCL-T | ViT-B-32 | 760 | 94.51 | 78.60
TransCL-16 | ViT-B-16 | 760 | 94.35 | 81.45
TransCL-32 | ViT-B-32 | 760 | 95.18 | 82.33
$10\%$ |  VCL [66] | ResNet110 | 306 | 67.65 | 47.90
MCL [30] | ResNet110 | 308 | 84.74 | 60.30
MCLwP [31] |  AllCNN [74] | 308 | 85.84 | 59.83
VCL-T | ViT-B-32 | 306 | 89.22 | 70.22
MCL-T | ViT-B-32 | 308 | 90.16 | 67.53
TransCL-16 | ViT-B-16 | 306 | 90.66 | 72.80
| TransCL-32 | ViT-B-32 | 306 | 91.57 | 72.11
$1.8\%$ |  VCL [66] | ResNet110 | 54 | 61.96 | 41.03
MCL [30] | ResNet110 | 54 | 64.14 | 33.67
VCL-T | ViT-B-32 | 54 | 61.96 | 41.86
MCL-T | ViT-B-32 | 54 | 62.28 | 36.43
TransCL-16 | ViT-B-16 | 54 | 68.44 | 42.66
TransCL-32 | ViT-B-32 | 54 | 69.60 | 42.75
Figure 9: Image classification performance of a single model (TransCL-32-arb)
for handling input with arbitrary CS ratios. The first row presents the
performance comparison in the case of input with a fixed CS ratio and
arbitrary CS ratios. The second row is a partial enlargement of the region
labeled with a green box in the first row, presenting the high robustness of
our method in handling extremely low and arbitrary CS ratios.

Cityscapes PASCAL

ADE20K

SETR [54]

TransCL-16-10

TransCL-16-5

TransCL-16-2.5

TransCL-16-1

Figure 10: Visual comparison of semantic segmentation. We visualize the
segmentation result of samples from PASCAL Context [69], Cityscapes [70], and
ADE20K [68], which are presented in the first row, the second row, and the
third row, respectively.

ADE20K [68]

Cityscapes [70]

PASCAL Context [69]

Figure 11: Semantic segmentation performance of our TransCL with arbitrary CS
ratios on three benchmarks.

Comparison Results. The evaluation is conducted on the large-scale ImageNet
and tiny CIFAR-10/100 datasets. We compare our TransCL with some image-domain
methods, e.g., the well-known ResNet [72] and transformer-based methods [49,
53, 50, 73]. We also compare our method with existing CL methods, i.e., VCL
[66] and MCL [30, 31]. To fully demonstrate the superiority of our method, we
also replace the CNN-based backbone in VCL and MCL with ViT-B model, dubbed
VCL-T and MCL-T, respectively. We apply the same training settings to optimize
these two models. We report the comparison results of different methods on the
ImageNet validation set in Tab. 1. One can see that compared with image-domain
methods, our proposed TransCL achieves state-of-the-art performance while
utilizing significantly fewer measurements. Even if only $1\%$ measurements
are available, our TransCL can still achieve $78.86\%$ classification
accuracy. Furthermore, our data-efficient version (i.e., ) can achieve
comparable performance, trained on the ImageNet-1K only. VCL-T and MCL-T can
not perform well on such large-scale benchmarks, and they produce a lot of
extra computational complexity. More details can be found in Sec. 5.3. The
evaluation on tiny CIFAR-10/100 datasets is mainly used to compare our method
with existing CL methods. Note that the measurement results of VCL-T and MCL-T
are tensors with the same size ($32\times 32$) as the input, which is not
suitable as input of ViT. We utilize bilinear upsampling to resize them to
$384\times 384$ and then feed them to the ViT backbone. The results are
presented in Tab. 2, presenting that our TransCL significantly outperforms
recent top-performing CL methods [30, 31]. In addition, as the number of
categories increases or the CS ratio decreases, the margin in performance
gains of our proposed TransCL becomes greater. The results in Tab. 1 and Tab.
2 demonstrate that our proposed TransCL performs well on high-resolution
images with large visual redundancy and tiny images with intensive
information.

Performance with arbitrary CS ratios. Handling measurements with arbitrary CS
ratios is valuable and practical, but it has not been explored in existing CL
tasks. In this part, we study image classification performance in the case of
input measurements with arbitrary CS ratios. The classification performance is
presented in Fig. 9. We can find that our proposed TransCL can handle $1,024$
different CS ratios ($\gamma\in[\frac{1}{1024},1]$) with a single model while
maintaining stable performance. The enlarged local part in the second row of
Fig. 9 shows that the performance of our TransCL rapidly reaches the peak with
the increase of CS ratio and then becomes stable. In the extreme case where
the CS ratio is $\frac{1}{1024}$ (only about $148$ measurements are available
for each $384$$\times$$384$ image), our TransCL can still achieve $12\%$
classification accuracy of $1,000$ categories. In addition, compared with the
variants at fixed CS ratios, the performance loss is minimal, presenting the
strong robustness of our TransCL.

TABLE III: Quantitative comparison of semantic segmentation task on ADE20K [68] dataset. Performances of different methods and the number of measurements for each image are reported.  Methods (Backbone) | Meas. | Param./Flops | mIoU/Acc
---|---|---|---
PSP[75] (ResNet-269) |  $3$$\times$$223729$ |  73M/246G | 44.94/81.69
GFF[76] (ResNet-101) |  $3$$\times$$262144$ | 141M/811G | 45.33/-
APC[77] (ResNet-101) |  $3$$\times$$331776$ | 76M/357G | 45.38/-
Twins[73] (SVT-L) |  $3$$\times$$262144$ | 133M/297G | 48.80/-
SETR[54] (ViT-L-16) |  $3$$\times$$262144$ | 309M/316G |  50.28/83.46
Ours (TransCL-16-10) |  $3$$\times$$26214$ | 309M/316G |  50.00/83.51
Ours (TransCL-16-5) |  $3$$\times$$13107$ | 309M/316G | 49.96/83.44
Ours (TransCL-16-2.5) |  $3$$\times$$6553$ | 309M/316G | 48.60/83.13
Ours (TransCL-16-1) |  $\mathbf{3}$$\mathbf{\times}$$\mathbf{2621}$ | 309M/316G | 46.57/82.14
TABLE IV: Quantitative comparison of semantic segmentation task on Cityscapes [70] dataset. Performances of different methods and the number of measurements for each image are reported.  Methods (Backbone) | Meas. | Param./Flops | mIoU
---|---|---|---
PSP [75] (ResNet-101) |  3$\times$508369 |  73M/554G | 78.50
CCNet [78] (ResNet-101) |  3$\times$591361 |  71M/698G | 80.20
GFF [76] (ResNet-101) |  3$\times$746496 | 141M/2305G | 80.40
SETR [54] (ViT-L-16) |  3$\times$589825 | 309M/818G | 82.15
Ours (TransCL-16-10) |  3$\times$58982 | 309M/818G | 82.02
Ours (TransCL-16-5) |  3$\times$29491 | 309M/818G | 81.73
Ours (TransCL-16-2.5) |  3$\times$14745 | 309M/818G | 81.53
Ours (TransCL-16-1) | 3$\times$5898 | 309M/818G | 79.29
TABLE V: Quantitative comparison of semantic segmentation task on Pascal Context [69] dataset. Performances of different methods and the number of measurements for each image are reported.  Methods (Backbone) | Meas. | Param./Flops | mIoU
---|---|---|---
PSP [75] (ResNet-101) |  3$\times$223729 |  73M/246G | 47.80
GFF [76] (ResNet-101) |  3$\times$262144 | 141M/811G | 54.20
APC [77] (ResNet-101) |  3$\times$262144 | 76M/282G | 54.70
SETR [54] (ViT-L-16) |  3$\times$230400 | 309M/281G | 55.83
Ours (TransCL-16-10) |  3$\times$23040 | 309M/281G | 55.52
Ours (TransCL-16-5) |  3$\times$11520 | 309M/281G | 55.29
Ours (TransCL-16-2.5) |  3$\times$5760 | 309M/281G | 54.03
Ours (TransCL-16-1) | 3$\times$2304 | 309M/281G | 51.83

###  4.3 Semantic Segmentation

Experiment Settings. For the application of semantic segmentation, we conduct
experiments on the following three widely-used benchmarks:

  * •

Cityscapes [70] has 19 annotated object categories in street scenes from 50
different cities. It contains 5,000 annotated images in which $2,975$ images
for training, 500 images for validation, and 1,525 images for testing. The
images are in high resolution with the size being $2048\times 1024$.

  * •

ADE20K [68] is a challenging benchmark with 150 annotated object categories,
which has 20210 training pairs, 2000 validation pairs, and 3352 testing pairs.

  * •

PASCAL Context [69] has 60 annotated object categories (59 classes and 1
background). It has 4998 training pairs and 5105 validation pairs.

As mentioned above, SETR [54] is the upper-bound model of our TransCL in image
domain in this application. Thus, we apply the same training strategy as SETR.
Specifically, at the beginning of training, we utilize the pre-trained
parameters of ViT-L-16 [49] in the image classification task to initialize
parameters in TB. Then we use the default setting of public codebase
mmsegmentation [79] to train our TransCL. Concretely, the input image is
randomly resized with the ratio between $0.5$ and 2 and randomly cropped with
the size being 768, 512, and 480 for Cityscapes, ADE20K, and Pascal Context,
respectively. A random horizontal flipping is also applied during training. We
train the network on these three benchmarks with the batch size being 8, 16,
and 8, respectively. We adopt a polynomial learning rate decay schedule [75]
and employ SGD as the optimizer during training. The momentum and weight decay
are set to 0.9 and 0, respectively. We set the initial learning rate as 0.001
on ADE20K and Pascal Context and 0.01 on Cityscapes. The CSM, TB, and TH in
TransCL are jointly optimized by a pixel-wise cross-entropy loss.

Comparison Results. In this application, we focus on comparing our proposed
TransCL with its upper bound model [54] and some state-of-the-art methods [77,
76, 78, 75, 73] in image domain. The quantitative comparison of semantic
segmentation on ADE20k [68], Cityscapes [70], and Pascal Context [69] are
presented in Tab. III, Tab. IV, and Tab. V, respectively. Clearly, our method
can achieve comparable performance to SETR [54] and outperform some state-of-
the-art image-domain methods with significantly fewer measurements. The
visualization comparison on three benchmarks is presented in Fig. 10,
presenting the attractive semantic segmentation results of our methods.
Specifically, in scenarios with few objects (e.g., the first row and second
row of Fig. 10), our TransCL can achieve comparable performance to [54] with
much fewer measurements. As presented in the third row of Fig. 10, our TransCL
maintains superior performance in the challenging ADE20K benchmark, which
contains 150 annotated object categories.

Performance with Arbitrary CS Ratios. In semantic segmentation, we also study
the performance of our TransCL in the challenging setting with arbitrary CS
ratios. The block size is set as $16$$\times$$16$. Thus, our TransCL can
handle 256 different CS ratios ($\gamma$$\in$$[\frac{1}{256},1]$) with a
single model. We present the segmentation performance in Fig. 11, presenting
that the performance rapidly improves with the increase of CS ratio and then
becomes stable. There is only a slight performance decrease compared with the
model trained with fixed CS ratios. Note that in the challenging ADE20K [68],
our proposed method can still maintain stable performance with arbitrary CS
ratios. In the extreme case with the CS ratio being $\frac{1}{256}$, our
proposed TransCL can still achieve satisfactory results on these three
benchmarks. These results fully demonstrate that our proposed TransCL can
perform complex high-level vision tasks in the measurement domain with state-
of-the-art performance and robustly handle arbitrary CS ratios with a single
model.

##  5 Ablation Study and Discussions

In this section, we focus on studying the effectiveness of different
components in our TransCL and presenting some superior properties and
interesting findings of our TransCL.

Figure 12: Image classification performance comparison between our proposed
TransCL and its CNN counterpart (ResCL) at different CS ratios and the
performance changes compared with their upper bound model. The results reflect
the advantages of our proposed TransCL in performance and stability.

###  5.1 Comparison with CNN Counterpart

Non-local self-similarity is an effective prior in compensating for the
information loss and resisting inference due to its robust capacity in
constructing long-range correlations. To demonstrate its effectiveness in CL
tasks, we compare our TransCL with its CNN counterpart. Specifically, we
replace the transformer-based backbone (TB) in TransCL with the very deep
ResNet-152 [72] (represented as ResCL). We evaluate the performance of our
TransCL and ResCL under various conditions in the image classification task.
Note that the serialized input is intractable to the CNN architecture. Thus,
we combine the input sequence into a square tensor in which each vectorized
measurement is located in the position of the original image block.

####  5.1.1 Performance and Stability

The comparison is conducted on ImageNet 2012 [67] dataset with the CS ratios
being $1\%$, $2.5\%$, $5\%$, and $10\%$. We report the top-1 ($\%$)
classification accuracy of our TransCL and ResCL at different CS ratios and
present the performance changes ($\%$) compared with their upper bound models
(ViT-B [49] and ResNet-152) in the first row and second row of Fig. 12,
respectively. Clearly, our TransCL enjoys advantages in both performance and
stability. Specifically, our proposed TransCL performs better than ResCL in
all evaluation points, and the margin becomes larger as the decrease of CS
ratio. The stability of our TransCL is reflected in two aspects, i.e., the
performance decays more slowly and is closer to the upper bound model.

####  5.1.2 Resisting Noise Perturbation

In the process of CS, there usually exists noise perturbation from the natural
environment and hardware. [25] exploited the performance bound of CL under the
noise perturbation. In this part, we study the capacity of our TransCL and
ResCL to resist the noise perturbation. Following [25], we add random Gaussian
noise $\mathbf{n}$$\sim$$\mathcal{N}(\mu,\sigma^{2})$ to original images. The
CS process is correspondingly expressed as:

$\mathbf{y}_{i}=\mathbf{\Phi}_{B}^{\gamma}\mathbf{x}_{i}+\mathbf{\Phi}_{B}^{\gamma}\mathbf{n}.$
(10)

In the experiment, we utilize the zero-mean Gaussian noise and set the CS
ratio as $10\%$. Note that all methods are trained with clean images, and we
only add the noisy signal in the evaluation. We evaluate the performance under
noise perturbation with different noise levels on the ImageNet validation set
and present the comparison result in Fig. 13. Obviously, our proposed TransCL
performs better than ResCL, demonstrating the effectiveness of the
transformer-based backbone. We can also find that TransCL-32 performs better
than TransCL-16, which can be interpreted by the fact that the TransCL-32 has
a larger receptive field of matching patches leading to more robust long-range
correlations to resist noisy signals.

Figure 13: The comparison of ability to resist noise perturbation of our
TransCL and its CNN counterpart (ResCL). We present the top-1 ($\%$) accuracy
on ImageNet validation set with the Gaussian noise level
$\sigma\in\\{0,10,20,...,60\\}$.

(a) Original Image

(b) Random PatchDrop

(c) Random PatchShuffle

Figure 14: Visualization of disturbing patterns, including random PatchDrop and random PatchShuffle. Figure 15: The comparison of resisting random PatchDrop of our TransCL and its CNN counterpart (ResCL) on ImageNet validation set with various packet losses. TABLE VI: Quantitative comparison of two-stage method and our single-stage TransCL on image classification task. We present the top-1 ($\%$) accuracy on ImageNet validation set in this table. Type | Method | CS ratio
---|---|---
$10\%$ | $5\%$ | $2.5\%$ | $1\%$
Two-stage |  |  OPINE-Net+[61]
---
ViT-B-32[49]
80.65 | 77.00 | 70.89 | 55.57
Single-stage |  | TransCL-32
---
(Ours)
81.82 | 81.53 | 80.50 | 78.00
Figure 16: Top-1 ($\%$) accuracy over ImageNet validation set with shuffled image patches. We compare our TransCL with its CNN counterpart (ResCL). TABLE VII: Comparison of complexity of various strategies to infer a $384\times 384$ image and the top-1 ($\%$) accuracy on ImageNet [67] validation set. The additional complexity produced in measurement reconstruction stage is highlighted in red. Methods | ResCL-152 | ResCL-101 | ResCL-50 | DensCL-201 | DensCL-121 | Two Stages | TransCL-16 | TransCL-32
---|---|---|---|---|---|---|---|---
Speed (ms/image) | 16.9 | 11.9 | 7.5 | 11.9 | 8.4 |  7.4$+$25.2 | 30.15 | $\mathbf{7.4}$
Peak Memory (GB) | 2.2 | 1.9 | 1.5 | 2.3 | 1.7 | $\max(1.5,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}6.8})$ | 3.2 | 1.5
Param. (M) | 60 | 45 | 25 | 20 | 8 |  88$+$0.6 | 86 | 88
Flops (G) | 30.1 | 23.0 | 12.1 | 12.8 | 8.4 |  12.3$+$112 | 49.3 | 12.3
Accuracy ($\%$) | 75.22 | 71.77 | 70.19 | 72.05 | 69.23 | 80.65 | 83.36 | 81.82

####  5.1.3 Resisting Random PatchDrop and PatchShuffle

To further demonstrate the superiority of our TransCL in resisting information
loss and external interference, we deploy two types of interference, i.e.,
Random PatchDrop and Random PatchShuffle, which are only added during testing.
As shown in Fig. 14(b), for Random PatchDrop, we randomly mask some square
regions of the block-based compressed sensing (BCS) process with a certain
proportion to simulate the packet loss during data transmission. The
evaluation results are presented in Fig. 15, presenting that our proposed
TransCL performs better than the ResCL with a large margin. This benefits from
the fact that transformer can fully utilize the non-local property in
sequence, rather than operating within a local region. For Random
PatchShuffle, we eliminate the structural information within images by
defining a shuffling operation on input image patches. One sample of this
operation is presented in Fig. 14(c), whose spatial structure is badly
damaged. This setting can simulate data transmission in the wrong sequence.
The evaluation results are presented in Fig. 16, from which one can observe
that the resistance ability of our TransCL to this interferential factor is
obviously better than ResCL.

###  5.2 Comparison with Two-stage Method

As illustrated in Fig. 2, a straightforward method to solve high-level vision
tasks with CS measurements is reconstructing first and then inferring.
However, this strategy potentially discloses private information and has
computational redundancy, i.e., in many applications, we are not interested in
obtaining a precise reconstruction of the scene under view, but rather are
only interested in the results of inference tasks. In this part, we provide
experimental results to demonstrate that the reconstruction step is almost
unnecessary, and our proposed TransCL can achieve better performance with a
single stage. In terms of the implementation of the two-stage method, we apply
the top-performing method [61] as the reconstruction module. Then we use
ViT-B-32 [49] to perform image classification based on the reconstruction
output. We set the CS ratio as $10\%$, $5\%$, $2.5\%$, and $1\%$. The
comparison result on ImageNet validation set is presented in Tab. VI. One can
see that the two-stage method has satisfied performance with high CS ratios,
but the performance declines sharply with the decrease of CS ratio. It
indicates that there exits serious coupling problem between the reconstruction
model and inference model, i.e., the inference model trained with natural
images can not adapt the input from CS restoration results. Therefore, for a
satisfactory result, we need jointly optimize a specific restoration model and
an inference model at each CS ratio, which is not feasible in practical
applications. By contrast, our proposed TransCL can achieve attractive and
stable performance without coupling problem while also having the advantage of
privacy protection.

TABLE VIII: Quantitative comparison of our compressive learning method with traditional data compression approaches on image classification task. We present the top-1 ($\%$) accuracy on ImageNet validation set. Sampling ratio | $10\%$ | $5\%$ | $2.5\%$ | $1\%$
---|---|---|---|---
|  SVD+ViT-B-32[49]
---
(w/o re-training)
68.69 | 42.02 | 12.08 | 2.7
|  Bicubic+ViT-B-32[49]
---
(w/o re-training)
74.00 | 68.24 | 60.12 | 42.51
Bicubic+ViT-B-32[49] | 81.00 | 80.03 | 78.48 | 73.91
TransCL-32 | 81.82 | 81.53 | 80.50 | 78.00

###  5.3 Complexity Analysis

As illustrated in Fig. 2, since no signal reconstruction is required, CL is an
efficient way to perform high-level vision tasks in the measurement domain and
should runs faster than two-stage methods (reconstruct and then infer).
Additionally, the transformer-based backbone is efficient in data processing,
which has been studied in ViT [49] (please refer to Fig.11 in [49]). Thus, in
this part, we focus on analyzing the model complexity of CL and two-stage
strategy, and we compare our proposed TransCL with some CNN counterparts,
i.e., ResCL and DensCL. Note that the implementation of two-stage method is
the same as Tab. VI. In the experiment, we set the image size and CS ratio as
$384$$\times$$384$ and $10\%$, respectively. The evaluation results of
inference speed, peak memory, parameters, and Flops are presented in Tab. VII,
which are all evaluated on an NVIDIA TESLA T4 GPU. We also present the top-1
accuracy of each strategy on ImageNet validation set in this table. We can
find that compared with some commonly used CNN-based backbones, our TransCL
has advantages in terms of inference speed and Flops, and the efficiency is
more obvious when compared with the two-stage strategy. Specifically, the time
complexity and Flops of our TransCL-32 are significantly lower than the
ResNet-152 and ResNet-101 and achieve comparable inference speed with the tiny
ResNet-50. This property mainly benefits from the efficiency of dot products
on GPU. In the two-stage strategy, the restoration model produces $3.4\times$
time complexity, $4.3\times$ peak memory, and $9.1\times$ flops even with much
fewer parameters. This is because the reconstruction process is a pixel-to-
pixel projection without changing spatial size. Thus, given a $384\times 384$
input, the reconstruction model demands a high computational burden. Due to
the four times length of the input sequence, TransCL-16 has higher model
complexity than TransCL-32. The performance gains are also obvious compared
with other strategies.

We present the model complexity of different CL methods to measure a
$384\times 384$ image with the CS ratio being $1\%$ in Tab. IX. We can find
that VCL and MCL produce a lot of extra computational complexity in handling
large-scale images. Such additional model complexity is unnecessary and makes
them difficult to deal with real-world benchmarks.

TABLE IX: Complexity comparison of various CL methods to sample a $384\times 384$ color image, with the CS ratio being $1\%$. | VCL [66] | MCL [30] | TransCL (Ours)
---|---|---|---
Parameters | $2.0\times 10^{9}$ | $1.1\times 10^{5}$ | $\mathbf{2.7\times 10^{3}}$
Flops | $1.3\times 10^{9}$ | $4.6\times 10^{7}$ | $\mathbf{2.6\times 10^{6}}$

###  5.4 Comparison with Traditional Data Compression Methods

In this part, we want to show that learning in the measurement domain better
preserves image information than the conventional data compression approaches.
We compare our TransCL with the spatial downsampling and SVD [80] methods.
Specifically, for spatial downsampling, we utilize the bicubic algorithm to
downsample the input image with the same ratio as the compressed sensing
operation in our TransCL. Then, we utilize bilinear interpolation to resize
the downsampled image to the original scale. To make a fair comparison, we
also re-train a specific ViT-B model on ImageNet 2012 dataset for each
downsampling factor, and the training strategy is the same as that we use to
train our TransCL. For SVD compression, we decompose the input image by the
SVD algorithm and then retain a specific amount of data according to the
sampling ratio. Concretely, given an input image
$\mathbf{I}\in\mathbb{R}^{w\times h}$, the SVD algorithm decomposes
$\mathbf{I}$ as $\mathbf{I}=\mathbf{u}\mathbf{s}\mathbf{v}$, where
$\mathbf{u}\in\mathbb{R}^{w\times w}$ and $\mathbf{v}\in\mathbb{R}^{h\times
h}$ are two orthogonal matrixes. $\mathbf{s}\in\mathbb{R}^{w\times h}$ is a
diagonal matrix. For data compression, we can retain $k$ columns in
$\mathbf{u}$ and $k$ rows in $\mathbf{v}$ according to the singular value in
$\mathbf{s}$. The rebuilding result is defined as
$\mathbf{I}=\mathbf{u}_{w\times k}\mathbf{s}_{k\times k}\mathbf{v}_{k\times
h}$. The compression ratio is computed as $r=\frac{w\times k+h\times
k+k}{w\times h}$. Because the SVD algorithm runs slowly on large-scale data,
we only provide the performance of SVD without re-training.

The comparison results are presented in Tab. VIII. We can find that the
special downsampling and SVD methods are not as efficient as compressive
learning. Even though the joint optimization improves the performance of
spatial downsampling, our method is still significantly better than the
downsampling operation, and the superiority becomes more obvious with the
decrease of sampling ratio. Furthermore, the spatial downsampling does not
have the property of privacy protection.

TABLE X: Comparison of the float-point sampling matrix and the binary sampling matrix. The result presents that our TransCL has the hardware-friendly property. The image classification task is conducted on TransCL-32, and the semantic segmentation task is conducted on TransCL-16. | CS Ratio | $1\%$ | $2.5\%$ | $5\%$ | $10\%$
---|---|---|---|---|---
Classification (Top-1 accuracy) | Float-point | 78.00 | 80.50 | 81.53 | 81.82
Binary | 76.12 | 78.90 | 80.11 | 80.91
| $\Delta$ | -1.88 | -1.6 | -1.42 | -0.09
Segmentation (mIoU) | Float-point | 50.83 | 54.03 | 55.29 | 55.52
Binary | 49.80 | 53.76 | 54.87 | 55.47
| $\Delta$ | -1.03 | -0.27 | -0.42 | -0.05

Learned float-point Fixed Gauss

Learned binary

Spatial view

Distribution

Orthogonality

Figure 17: Visualization of distribution and orthogonality of fixed Gaussian
matrix (the first row), learned float-point matrix (the second row), and
learned binary matrix (the third row).

###  5.5 Sampling Matrix

####  5.5.1 Performance of Binary Sampling Matrix

In this part, we make a performance comparison between learned float-point
sampling and learned binary sampling matrix. The implementation is illustrated
in Sec. 3.2 and the experiments are conducted on image classification and
semantic segmentation tasks with ImageNet 2012 [67] and Pascal Context [69]
datasets, respectively. As shown in Tab. X, our proposed TransCL can achieve
attractive performance with the binary sampling matrix, and there is only a
slight performance reduction compared with using the float-point sampling
matrix.

####  5.5.2 Interesting Findings

There are also some interesting findings of the sampling matrix. We take the
CS ratio of 0.1 as an example. We reshape one row in a fixed random Gaussian
matrix, our learned float-point matrix, and our learned binary matrix into
$B$$\times$$B$ and visualize their spatial property in the first column of
Fig. 17. Moreover, we visualize the distribution and orthogonality
$\mathbf{\Phi}_{B}^{\gamma}(\mathbf{\Phi}_{B}^{\gamma})^{\top}$ of the
sampling matrix in the second column and the third column of Fig. 17,
respectively. One can see that the learned float-point sampling matrix obeys
the Gaussian distribution, which is similar to the commonly used fixed random
Gaussian matrix in many previous works [81, 12, 82]. The learned binary matrix
has an even distribution at the points of $-1$ and $1$. Fig. 17 also presents
that both float-point and binary sampling matrix is approximately orthogonal,
i.e.,
$\mathbf{\Phi}_{B}^{\gamma}(\mathbf{\Phi}_{B}^{\gamma})^{\top}$$\approx$$\lambda\mathbf{I}$,
where $\mathbf{I}$$\in$$\mathbb{R}^{M_{B}^{\gamma}\times M_{B}^{\gamma}}$ is
the identity matrix. This property is also consistent with the commonly used
fixed random Gaussian matrix. Note that these properties are all learned
without specific constraints. These similar characters naturally lead to the
following inference: the learned sampling matrix retains the same properties
as the fixed random Gaussian matrix, such as RIP [83].

Feature representation in image domain (by ViT-B-32 [49])

Feature representation in measurement domain (by our TransCL-32)

Figure 18: Visualization of feature representation in measurement domain and
image domain. Each figure shows the feature representation after dimension
reduction using the T-SNE method [84].

###  5.6 Discussion: Feature Representation in Measurement Domain and Image
Domain

We generally assume that a large margin between measurement domain and image
domain exists, and the performance of existing CL methods is not comparable to
the image-domain methods. In this paper, we propose a novel CL method to
bridge this gap. To further demonstrate our motivation, we visualize the
feature representations from the measurement domain and the image domain,
respectively. Concretely, we randomly select $50$ categories from $1000$
categories of the ImageNet validation set, each contains $50$ samples. Then we
feed these sets of images to pre-trained TransCL-32-10 and ViT-B-32 [49], and
we collect the features extracted by these two models. Since the class
information is stored in the class token, we utilize T-SNE [84] to reduce the
dimensionality of the class token feature of each sample to $2$ dimensions for
illustration purposes. The results from our TransCL and ViT-B are shown in
Fig. 18. We can find that the feature representations generated by our TransCL
and ViT-B have good category discrimination, which demonstrates the
superiority of our TransCL of better preserving image information from a few
measurements. Fig. 18 also presents a high similarity of the feature
representations from the measurement domain and image domain. Thus, these two
domains are interrelated, and we can bridge the gap through specific designs.

##  6 Conclusion

In this paper, we delve into the high-level visual applications of compressed
sensing (CS) beyond the signal reconstruction, presenting that CS
reconstruction is almost unnecessary in high-level vision tasks. Technically,
we offer a completely new sequence-to-sequence perspective to compressive
learning (CL) by applying the block-based compressed sensing (BCS) strategy,
and we propose a novel transformer-based CL framework, dubbed TransCL. The
proposed TransCL extends the concept of CL to more complex high-level vision
tasks and larger datasets with real-world scale, which offers a fresh and
successful instance and bridges the gap between inference tasks in the image
domain and those in the measurement domain. Furthermore, we study several
useful properties of our TransCL in terms of handling arbitrary CS ratios with
a single model (Fig. 9 and Fig. 11), good generalization to binary sampling
(Tab. X), robustness under different degradation factors (Sec. 5.1), privacy
protection (Fig. 8) and effective data compression (Tab. VIII). Extensive
experiments and theoretical analysis demonstrate that our proposed TransCL has
these superior properties and can achieve state-of-the-art performance in
image classification and semantic segmentation tasks, even at extremely low CS
ratios. Our future work will focus on designing more efficient sequence-to-
sequence CL methods and establishing real-world systems integrated with the
hardware.

## References

  * [1] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” _IEEE Transactions on Information Theory_ , vol. 52, pp. 489–509, 2006.
  * [2] D. L. Donoho, “Compressed sensing,” _IEEE Transactions on Information Theory_ , vol. 52, pp. 1289–1306, 2006.
  * [3] M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” _IEEE Signal Processing Magazine_ , vol. 25, pp. 83–91, 2008.
  * [4] F. Rousset, N. Ducros, A. Farina, G. Valentini, C. D’Andrea, and F. Peyrin, “Adaptive basis scan by wavelet prediction for single-pixel imaging,” _IEEE Transactions on Computational Imaging_ , vol. 3, pp. 36–46, 2016.
  * [5] M. Lustig, D. Donoho, and J. M. Pauly, “Sparse mri: The application of compressed sensing for rapid mr imaging,” _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 58, pp. 1182–1195, 2007.
  * [6] Z. Zhang, T.-P. Jung, S. Makeig, and B. D. Rao, “Compressed sensing for energy-efficient wireless telemonitoring of noninvasive fetal ecg via block sparse bayesian learning,” _IEEE Transactions on Biomedical Engineering_ , vol. 60, pp. 300–309, 2012.
  * [7] S. K. Sharma, E. Lagunas, S. Chatzinotas, and B. Ottersten, “Application of compressive sensing in cognitive radio communications: A survey,” _IEEE Communications Surveys Tutorials_ , vol. 18, pp. 1838–1860, 2016.
  * [8] Z. Wang, H. Zhang, Z. Cheng, B. Chen, and X. Yuan, “Metasci: Scalable and adaptive reconstruction for video compressive sensing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 2083–2092.
  * [9] Z. Wu, J. Zhang, and C. Mou, “Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging,” in _Proceedings of International Conference on Computer Vision (ICCV)_ , 2021, pp. 4892–4901.
  * [10] M. Gupta, A. Agrawal, A. Veeraraghavan, and S. G. Narasimhan, “Flexible voxels for motion-aware videography,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2010, pp. 100–114.
  * [11] D. Reddy, A. Veeraraghavan, and R. Chellappa, “P2c2: Programmable pixel compressive camera for high speed imaging,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2011, pp. 329–336.
  * [12] W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang, “Compressive sensing via nonlocal low-rank regularization,” _IEEE Transactions on Image Processing_ , vol. 23, pp. 3618–3632, 2014.
  * [13] M. Iliadis, L. Spinoulas, and A. K. Katsaggelos, “Deep fully-connected networks for video compressive sensing,” _Digital Signal Processing_ , vol. 72, pp. 9–18, 2018.
  * [14] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok, “Reconnet: Non-iterative reconstruction of images from compressively sensed measurements,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 449–458.
  * [15] S. Mun and J. E. Fowler, “Block compressed sensing of images using directional transforms,” in _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , 2009, pp. 3021–3024.
  * [16] C. Zhao, S. Ma, and W. Gao, “Image compressive-sensing recovery using structured laplacian sparsity in dct domain and multi-hypothesis prediction,” in _Proceedings of IEEE International Conference on Multimedia and Expo (ICME)_ , 2014, pp. 1–6.
  * [17] C. Zhao, S. Ma, J. Zhang, R. Xiong, and W. Gao, “Video compressive sensing reconstruction via reweighted residual sparsity,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 27, pp. 1182–1195, 2016\.
  * [18] D. You, J. Zhang, J. Xie, B. Chen, and S. Ma, “COAST: COntrollable Arbitrary-Sampling NeTwork for Compressive Sensing,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 6066–6080, 2021.
  * [19] J. Song, B. Chen, and J. Zhang, “Memory-Augmented Deep Unfolding Network for Compressive Sensing,” in _Proceedings of ACM International Conference on Multimedia (ACM MM)_ , 2021.
  * [20] P. Mohassel and Y. Zhang, “Secureml: A system for scalable privacy-preserving machine learning,” in _Proceedings of the IEEE Symposium on Security and Privacy (SP)_ , 2017, pp. 19–38.
  * [21] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing, “Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy,” in _International Conference on Machine Learning (ICML)_ , 2016, pp. 201–210.
  * [22] R. Calderbank, S. Jafarpour, and R. Schapire, “Compressed learning: Universal sparse dimensionality reduction and learning in the measurement domain,” _Preprint_ , 2009. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.481.8129&rep=rep1&type=pdf
  * [23] M. A. Davenport, M. F. Duarte, M. B. Wakin, J. N. Laska, D. Takhar, K. F. Kelly, and R. G. Baraniuk, “The smashed filter for compressive classification and target recognition,” in _Proceedings of SPIE Computational Imaging_ , 2007, pp. 142–153.
  * [24] T. Wimalajeewa, H. Chen, and P. K. Varshney, “Performance limits of compressive sensing-based signal classification,” _IEEE Transactions on Signal Processing_ , vol. 60, pp. 2758–2770, 2012.
  * [25] Y. Cui, W. Xu, Y. Wang, J. Lin, and L. Lu, “Performance bounds of compressive classification under perturbation,” _Signal Processing_ , vol. 180, p. 107855, 2021.
  * [26] H. Reboredo, F. Renna, R. Calderbank, and M. R. Rodrigues, “Projections designs for compressive classification,” in _Proceedings of the IEEE Global Conference on Signal and Information Processing_ , 2013, pp. 1029–1032.
  * [27] J. Haupt, R. Castro, R. Nowak, G. Fudge, and A. Yeh, “Compressive sampling for signal classification,” in _Proceedings of the Fortieth Asilomar Conference on Signals, Systems and Computers_ , 2006, pp. 1430–1434.
  * [28] E. Zisselman, A. Adler, and M. Elad, “Compressed learning for image classification: A deep neural network approach,” in _Processing, Analyzing and Learning of Images, Shapes, and Forms: Part 1_ , ser. Handbook of Numerical Analysis, 2018, pp. 3–17.
  * [29] S. Lohit, K. Kulkarni, P. Turaga, J. Wang, and A. C. Sankaranarayanan, “Reconstruction-free inference on compressive measurements,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2015, pp. 16–24.
  * [30] D. T. Tran, M. Yamaç, A. Degerli, M. Gabbouj, and A. Iosifidis, “Multilinear compressive learning,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, pp. 1512–1524, 2021.
  * [31] D. T. Tran, M. Gabbouj, and A. Iosifidis, “Multilinear compressive learning with prior knowledge,” _arXiv preprint arXiv:2002.07203_ , 2020.
  * [32] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, pp. 2278–2324, 1998.
  * [33] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf
  * [34] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2005, pp. 60–65.
  * [35] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” _IEEE Transactions on Image Processing_ , vol. 16, pp. 2080–2095, 2007.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [37] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2019, pp. 4171–4186.
  * [38] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://www.cs.ubc.ca/amuham01/LING530/papers/radford2018improving.pdf
  * [39] Z. Li, Z. Zhang, H. Zhao, R. Wang, K. Chen, M. Utiyama, and E. Sumita, “Text compression-aided transformer encoding,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, pp. 3840–3857, 2021.
  * [40] L. Gan, “Block compressed sensing of natural images,” in _Proceedings of the International Conference on Digital Signal Processing_ , 2007, pp. 403–406.
  * [41] X. Gao, J. Zhang, W. Che, X. Fan, and D. Zhao, “Block-based compressive sensing coding of natural images by local structural measurement matrix,” in _Proceedings of the IEEE 2015 Data Compression Conference (DCC)_ , 2015, pp. 133–142.
  * [42] K. Kulkarni and P. Turaga, “Reconstruction-free action inference from compressive imagers,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , vol. 38, pp. 772–784, 2015.
  * [43] S. Lohit, K. Kulkarni, and P. Turaga, “Direct inference on compressive measurements using convolutional neural networks,” in _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , 2016, pp. 1913–1917.
  * [44] A. Adler, M. Elad, and M. Zibulevsky, “Compressed learning: A deep neural network approach,” _arXiv preprint arXiv:1610.09615_ , 2016.
  * [45] Y. Oike and A. El Gamal, “Cmos image sensor with per-column $\sigma$ $\delta$ adc and programmable compressed sensing,” _IEEE Journal of Solid-State Circuits_ , vol. 48, pp. 318–328, 2012.
  * [46] P. S. Gupta and G. S. Choi, “Programmable compressed sensing using simple deterministic sensing matrices,” in _Proceedings of the Optoelectronic Imaging and Multimedia Technology V_ , 2018, pp. 74–85.
  * [47] D. T. Tran, M. Gabbouj, and A. Iosifidis, “Performance indicator in multilinear compressive learning,” in _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_ , 2020, pp. 1822–1828.
  * [48] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7794–7803.
  * [49] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * [50] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021, pp. 10 347–10 357.
  * [51] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” in _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 2021, pp. 15 908–15 919.
  * [52] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020, pp. 213–229.
  * [53] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2021, pp. 568–578.
  * [54] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 6881–6890.
  * [55] S. Yang, Z. Quan, M. Nie, and W. Yang, “Transpose: Keypoint localization via transformer,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2021, pp. 11 802–11 812.
  * [56] C. Mou, J. Zhang, X. Fan, H. Liu, and R. Wang, “Cola-net: Collaborative attention network for image restoration,” _IEEE Transactions on Multimedia_ , 2021.
  * [57] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture transformer network for image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5791–5800.
  * [58] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 12 299–12 310.
  * [59] S. Mun, _Block compressed sensing of images and video_. Mississippi State University, 2012.
  * [60] I. Noor and E. L. Jacobs, “Adaptive compressive sensing algorithm for video acquisition using a single-pixel camera,” _Journal of Electronic Imaging_ , vol. 22, p. 021013, 2013.
  * [61] J. Zhang, C. Zhao, and W. Gao, “Optimization-inspired compact deep compressive sensing,” _IEEE Journal of Selected Topics in Signal Processing (JSTSP)_ , vol. 14, pp. 765–774, 2020.
  * [62] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks,” in _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 2016, pp. 4114–4122.
  * [63] M. Li, W. Zuo, S. Gu, D. Zhao, and D. Zhang, “Learning convolutional networks for content-weighted image compression,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 3214–3223.
  * [64] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2010.
  * [65] J. Zhang and B. Ghanem, “Ista-net: Interpretable optimization-inspired deep network for image compressive sensing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1828–1837.
  * [66] E. Zisselman, A. Adler, and M. Elad, “Compressed learning for image classification: A deep neural network approach,” in _Handbook of Numerical Analysis_ , 2018, pp. 3–17.
  * [67] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision_ , vol. 115, pp. 211–252, 2015.
  * [68] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ade20k dataset,” _International Journal of Computer Vision (IJCV)_ , vol. 127, pp. 302–321, 2019.
  * [69] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille, “The role of context for object detection and semantic segmentation in the wild,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 891–898.
  * [70] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 3213–3223.
  * [71] R. Wightman, “Pytorch image models,” https://github.com/rwightman/pytorch-image-models, 2019.
  * [72] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [73] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision transformers,” in _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 2021, pp. 9355–9366.
  * [74] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” _arXiv preprint arXiv:1412.6806_ , 2014.
  * [75] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2881–2890.
  * [76] X. Li, H. Zhao, L. Han, Y. Tong, S. Tan, and K. Yang, “Gated fully fusion for semantic segmentation,” in _Proceedings of the AAAI conference on artificial intelligence (AAAI)_ , 2020, pp. 11 418–11 425.
  * [77] J. He, Z. Deng, L. Zhou, Y. Wang, and Y. Qiao, “Adaptive pyramid context network for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 7519–7528.
  * [78] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, “Ccnet: Criss-cross attention for semantic segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision (CVPR)_ , 2019, pp. 603–612.
  * [79] M. Contributors, “MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark,” https://github.com/open-mmlab/mmsegmentation, 2020.
  * [80] Y. Koren, “Factorization meets the neighborhood: a multifaceted collaborative filtering model,” in _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2008, pp. 426–434.
  * [81] J. Zhang, D. Zhao, C. Zhao, R. Xiong, S. Ma, and W. Gao, “Image compressive sensing recovery via collaborative sparsity,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 2, pp. 380–391, 2012.
  * [82] J. Zhang, D. Zhao, and W. Gao, “Group-based sparse representation for image restoration,” _IEEE Transactions on Image Processing (TIP)_ , vol. 23, pp. 3336–3351, 2014.
  * [83] E. J. Candes, “The restricted isometry property and its implications for compressed sensing,” _Comptes Rendus Mathematique_ , vol. 346, no. 9-10, pp. 589–592, 2008.
  * [84] P. G. Poličar, M. Stražar, and B. Zupan, “OpenTSNE: a modular python library for t-SNE dimensionality reduction and embedding,” _BioRxiv_ , p. 731877, 2019.

|  Chong Mou is currently pursuing the M.S. degree at the School of Electronic
and Computer Engineering, Peking University Shenzhen Graduate School,
Shenzhen, China. He received the B.S. degree at South China University of
Technology, in 2020. His research interests include image denoising, feature
attention, pattern recognition and deep learning.
---|---
|  Jian Zhang (M’14) received the B.S. degree from the Department of
Mathematics, Harbin Institute of Technology (HIT), Harbin, China, in 2007, and
received his M.Eng. and Ph.D. degrees from the School of Computer Science and
Technology, HIT, in 2009 and 2014, respectively. From 2014 to 2018, he worked
as a postdoctoral researcher at Peking University (PKU), Hong Kong University
of Science and Technology (HKUST), and King Abdullah University of Science and
Technology (KAUST). Currently, he is an Assistant Professor with the School of
Electronic and Computer Engineering, Peking University Shenzhen Graduate
School, Shenzhen, China. His research interests include intelligent multimedia
processing, deep learning and optimization. He has published over 90 technical
articles in refereed international journals and proceedings. He received the
Best Paper Award at the 2011 IEEE Visual Communications and Image Processing
(VCIP) and was a co-recipient of the Best Paper Award of 2018 IEEE MultiMedia.
---|---