# Fast Hyperspectral Image Recovery via Non-iterative Fusion of Dual-Camera
Compressive Hyperspectral Imaging

Wei He, Naoto Yokoya, and Xin Yuan Manuscript received March 26, 2019, and
accepted June 13, 2019. This work was supported by the Japan Society for the
Promotion of Science (KAKENHI 19K20308; KAKENHI 18K18067). (_Corresponding
author: wei.he@riken.jp_)W. He and N. Yokoya are with the RIKEN Center for
Advanced Intelligence Project, RIKEN, 103-0027, Tokyo, Japan (e-mail:
<EMAIL_ADDRESS>naoto.yokoya@riken.jp). N. Yokoya is also with the University
of Tokyo, Tokyo, Japan. X. Yuan is with Nokia Bell Labs, USA (xyuan@bell-
labs.com). Color versions of one or more of the figures in this letter are
available online at http://ieeexplore.ieee.org. Digital Object Identifier XXXX

###### Abstract

Coded aperture snapshot spectral imaging (CASSI) is a promising technique to
capture the three-dimensional hyperspectral image (HSI) using a single coded
two-dimensional (2D) measurement, in which algorithms are used to perform the
inverse problem. Due to the ill-posed nature, various regularizers have been
exploited to reconstruct the 3D data from the 2D measurement. Unfortunately,
the accuracy and computational complexity are unsatisfied. One feasible
solution is to utilize additional information such as the RGB measurement in
CASSI. Considering the combined CASSI and RGB measurement, in this paper, we
propose a new fusion model for the HSI reconstruction. We investigate the
spectral low-rank property of HSI composed of a spectral basis and spatial
coefficients. Specifically, the RGB measurement is utilized to estimate the
coefficients, meanwhile the CASSI measurement is adopted to provide the
orthogonal spectral basis. We further propose a patch processing strategy to
enhance the spectral low-rank property of HSI. The proposed model neither
requires non-local processing or iteration, nor the spectral sensing matrix of
the RGB detector. Extensive experiments on both simulated and real HSI dataset
demonstrate that our proposed method outperforms previous state-of-the-art not
only in quality but also speeds up the reconstruction more than 5000 times.

###### Index Terms:

Hyperspectral imaging, reconstruction, CASSI, RGB, spectral low-rank, Fusion.

##  I Introduction

Hyperspectral image (HSI), a three-dimensional (3D) spatial-spectral cube,
which is rich in spectral information, has been widely used in applications
such as face recognition [1], remote sensing [2], food surveillance [3] and so
on. Typically, a spectrometer is used, which scans a 1D-line or 2D plane to
capture a full 3D image [4]. Though it can achieve a high spectral resolution
such as 10 nm or less [5], it is time-consuming and unsuitable for dynamic
scenes [6]. Therefore, it is necessary to improve the efficiency of
hyperspectral imaging for the dynamic applications.

Inspired by compressive sensing [7], several compressive spectral imaging
(CSI) techniques have been well developed [8, 9, 10] to improve the imaging
efficiency. Specifically, a single or few 2D snapshot compressed measurements
are captured by the CSI system to reconstruct the 3D HSI data-cube using
inverse algorithms. Among these CSI systems, the coded aperture snapshot
spectral imager (CASSI) [9] has attracted much attention, which utilizes a
coded aperture and one or two dispersive elements to modulate the optical
field from a scene. A detector captures a 2D, multiplexed projection of the 3D
spatial-spectral data-cube representing the scene [9]. Following this,
different variants of CASSI have been developed, such as dual dispersive CASSI
(DDCASSI) [11], single disperser CASSI (SDCASSI) [9], spatial-spectral encoded
compressive hyperspectral imager (SSCSI) [12], dual-coded hyperspectral imager
(DCSI) [13], colored coded aperture spectral camera imager (CCASSI) [14] and
so on. For more details, we recommend to the review works [15, 16, 17]. In a
nutshell, CSI is a system composed of a hardware setup plus a reconstruction
algorithm. In this paper, we focus on the algorithm. In particular, we develop
an efficient HSI recovery method via non-iterative fusion of CASSI and the RGB
measurements.

In order to solve the inverse problem in CSI, i.e., retrieving the 3D HSI from
the coded 2D measurement, various regularized methods, including sparse
representation [13, 12, 18, 19, 20], total variation (TV) [21, 22], and non-
local low-rank regularization [23, 6, 24] have been developed. Recently, deep
learning methods based on the convolutional neural network (CNN) have also
been utilized to explore the prior knowledge from coded image to the
reconstructed HSI [25, 26]. However, the quality of the reconstructed HSI is
still limited, due to the fact that CASSI is too undersampled by compressing
HSI from tens to hundreds of spectral bands to one single compressed
measurement. In addition, CNN based methods [25, 26] depend on the quality of
the training samples, and lose the flexibility [27]. In short, the main
factors of the reconstruction algorithms for CASSI are the accuracy, speed and
flexibility [27]. However, due to the severe ill-posed nature of CASSI,
existing algorithms cannot meet all these three criteria. One solution from
the hardware side to alleviate this challenge is to capture complementary
measurements such as panchromatic or RGB images with CASSI [28, 24, 29]. The
related reconstruction algorithms for these hybrid systems [21, 6, 24, 28, 29]
usually regard the complementary measurements as an addition side information
to boost the quality of reconstructed HSI. In particular, these algorithm
compose a larger sensing matrix by stacking the complementary sensing matrix
and CASSI sensing matrix. That is to say, the computational algorithms to
reconstruct HSIs from CASSI and complementary measurements will cost much more
time compared to the reconstruction of single CASSI.

Bearing the above concerns in mind, in this paper, we aim to develop an
efficient HSI reconstruction algorithm from CASSI and complementary
measurements. Different from the previous works, which utilize iteration
methods to update the reconstructed HSIs step-by-step to explore the spatial-
spectral priors of the full HSIs [23, 6, 24, 29], our proposed method is
inspired by the fact that HSIs are assumed to underlie a low dimensional
spectral subspace, which has been widely used in different applications [30,
31, 32]. In detail, by taking the advantages of both measurements in the
hybrid system, we propose to compute the spatial coefficients from the RGB
measurement, and optimize the orthogonal spectral basis from the CASSI
measurement, and finally reconstruct the HSI from the product of these two
components. We depict the proposed method in Fig. 1. To make the algorithm
efficient, we further propose a non-iterative optimization, which will be
described in Section III-C. In addition, to reduce the rank of the spatial
coefficients, we segment the HSI into spatially overlapping patches, and
process each patch separately.

###  I-A Contributions of This Paper

The main advantages of the proposed method compared with the previous
algorithms [21, 6, 24, 25, 33, 28, 29] can be summarized as follows:

  * •

Fast: our proposed two-stage fusion method of CASSI and RGB measurements, does
not require regularization or iteration. Therefore, it is efficient to finish
the HSI reconstruction within seconds, which is $5000\times$ faster compared
to DeSCI [23] and DLTR [24].

  * •

Flexible: our model does not need to know the spectral response (sensing
matrix) of the RGB detector in advance. What we need is the alignment of CASSI
and RGB measurements. It is thus flexible to apply our model to different
applications.

  * •

High accuracy: the proposed method is evaluated on extensive simulated and
real data experiments. It is reported to achieve the best results compared to
other state-of-the-art methods [23, 24].

###  I-B Paper Organization and Notations

The remainder of this paper is organized as follows. Section II introduces the
related works of computational imaging reconstruction. Section III introduces
the dual-camera compressive hyperspectral imaging system, and presents the
proposed fusion model and related solutions. Section IV illustrates the
experimental results and Section V presents the detailed discussion of our
method, which is followed by the conclusions in Section VI.

In this paper, tensors of order $3$ are denoted by boldface Euler script
letters, e.g., $\boldsymbol{\mathcal{X}}\in\mathbb{R}^{M\times N\times B}$.
Scalars are denoted by normal lowercase letters or uppercase letters, e.g.,
$x,X\in\mathbb{R}$. $X(i,j,k)$ denotes the element of tensor
$\boldsymbol{\mathcal{X}}$ in position $(i,j,k)$. Vectors are denoted by
boldface lowercase letters, e.g., $\mathbf{x}\in\mathbb{R}^{M}$. Matrices are
denoted by boldface capital letters, e.g., $\mathbf{X}\in\mathbb{R}^{M\times
N}$. The mode-$n$ unfolding [34] of tensor
$\boldsymbol{\mathcal{X}}\in\mathbb{R}^{I\times J\times K}$ is denoted by
$\mathbf{X}_{(n)}\in\mathbb{R}^{B\times{MN}}$. The definition of mode-$n$
unfolding is denoted as $\text{fold}_{n}(\cdot)$, $i.e.,$ for a tensor
$\boldsymbol{\mathcal{X}}$, we have
$\text{fold}_{n}(\mathbf{X}_{(n)})=\boldsymbol{\mathcal{X}}$.

Figure 1: Illustration showing the proposed fusion model.

##  II Related Works

In this section, we briefly introduce the related reconstruction algorithms
from CASSI measurements.

To reconstruct HSI from the coded image from CASSI measurements, a two-step
iterative shrinkage/thresholding (TwIST) algorithm was firstly utilized to
undertake the reconstruction task [21, 35]. As we know, the HSI reconstruction
for the highly compressed CASSI measurements is an under-estimated problem.
Furthermore, the sensing matrix in CASSI is specifically designed and brings
challenge for the reconstruction of HSI [23]. Therefore, different kinds of
HSI regularization are introduced into the reconstruction framework to
increase the quality of reconstructed HSI. At the beginning, sparse
regularization with TI-haar basis [11], wavelet basis [9] and learned over-
complete dictionary [13, 12, 18, 19, 20] have been utilized to explore the
spatial-spectral sparsity of HSI. To exploit the spatial smoothness of the
HSI, total variation (TV) has been utilized into the reconstruction framework
[21, 22]. In addition, low-rank matrix/tensor decomposition has also been
introduced to explore the spatial-spectral low-rank property [36, 37, 38, 39]
and non-local spatial low-rank property of HSI [23, 6, 40, 24]. As the
development of different state-of-the-art HSI regularizers, although the
quality of reconstructed HSIs has increased significantly, they also meet new
problems. As reported in [23], the proposed non-local regularized method DeSCI
costs more than $3$ hours to reconstruct a HSI of size $512\times 512\times
31$, preventing the real-time acquisition of HSI. Different regularizers are
suitable for different scenario of HSI, which also result in another problem
that it is very hard to choose the optimal regularizer for the reconstruction
of HSI.

Very recently, deep learning has also been utilized to the reconstruction of
CASSI [41]. [42] proposed to use convolutional auto-encoder (AE) to learn a
nonlinear sparse representation for CASSI. [33] introduced the CNN to train
the reconstruction model, and then tested on the coded image. After that,
different CNN architectures have been introduced to the reconstruction of
coded image via CASSI [25, 26]. The CNN based methods utilize numerous
external dataset to train the model, and then apply the trained model to the
testing stage. Frankly speaking, the training stage is time-consuming, however
the test stage is fast. From this perspective, CNN based methods can alleviate
the problems in the previous regularizer based methods, and learn the prior
knowledge from the training dataset. However, these end-to-end deep learning
methods lose the flexibility [27]. Since each trained model is allocated to
one specific sensing matrix, as the sensing matrix changes, these methods need
to train a new model, which is again time-consuming. Furthermore, CNN based
methods are out of work when the training samples are insufficient.

Although regularization based methods improve the quality of the reconstructed
HSI from CASSI measurements, the performance faces the upper bound due the
limited measurements. The complex regularizers further bring the heavy
computation burden for the reconstruction task. Therefore, some researchers
try to reconstruct the HSI from much more measurements, i.e., CASSI
measurements with complementary measurements (panchromatic, or RGB). Several
improved imagers have been proposed, including multi-frame CASSI [16], dual-
camera CASSI with a panchromatic measurement [21, 6, 24], and dual-camera
CASSI with an RGB measurement [28, 29]. However, the previous works [21, 6,
24, 28, 29] regard the complementary measurements as an addition to CASSI, and
compose a larger sensing matrix by stacking the complementary sensing matrix
and CASSI sensing matrix, which leads to even longer running time. To
reconstruct HSI from CASSI and the complementary measurements, various
regularizers have been investigated to restrict full HSI, $i.e.,$ TV [21, 29],
non-local sparse representation [6], and non-local low-rank matrix/tensor
factorization [23, 24].

Compared to CASSI reconstruction, the HSI reconstruction from CASSI and the
complementary measurements can achieve higher accuracy, since the additional
measurements of complementary image. However, during the optimization, the new
sensing matrix (by stacking the complementary sensing matrix and CASSI sensing
matrix) has also increased to several times larger than that of CASSI,
resulting in the unacceptable time cost of the HSI reconstruction.
Furthermore, the optimization needs to know the complementary sensing matrix
in advance, which requires additional cross-sensor calibration in the real
applications.

Figure 2: Illustration showing the DCCHI system.

##  III Proposed Fusion Method

In this section, we firstly introduce the dual-camera compressive
hyperspectral imaging (DCCHI) system. Subsequently, we present the proposed
fusion model of CASSI and RGB measurements with non-iterative optimization
strategy. Finally, we analyze the advantages of the proposed model compared to
the previous works.

###  III-A Dual-camera Compressive Hyperspectral Imaging

The dual-camera compressive hyperspectral imaging system was firstly proposed
by [21, 28]. As illustrated in Fig. 2, the incident light firstly goes through
the beam splitter, and is equally partitioned to two directions. The first
partition of the incident light is captured by the CASSI, which is composed of
objective lens, dispersive prism, mask, relay lens, dispersive prism, and
finally the light is sensed by a gray-scale camera to a coded image. We adopt
$\boldsymbol{\mathcal{X}}\in\mathbb{R}^{M\times N\times B}$ to represent the
original HSI, with $M,N$ denoting the two spatial size and $B$ denoting the
spectral channels. Therefore, the compressive measurement at position $(i,j)$
via CASSI can be represented as

$Y(i,j)=\sum_{k=1}^{B}X(i,j,k)\odot C(i,j,k),$ (1)

where $\mathbf{Y}\in\mathbb{R}^{M\times N}$ is the coded image, $\odot$ is the
Hadamard (element-wise) product, and
$\boldsymbol{\mathcal{C}}\in\mathbb{R}^{M\times N\times B}$ denotes the mask.
The illustration of CASSI from 3D scene to 2D coded image is presented in Fig
3. As a linear transformation, (1) can be reformulated as the following [23,
43]

$\mathbf{y}=\mathbf{\Phi}^{c}\mathbf{x},$ (2)

where $\mathbf{\Phi}^{c}\in\mathbb{R}^{MN\times MNB}$ is the sensing matrix
from $\boldsymbol{\mathcal{C}}$, $\mathbf{y},\mathbf{x}$ are the vectorization
of $\mathbf{Y},\boldsymbol{\mathcal{X}}$ respectively.

Recalling Fig. 2, the second partition of the incident light is captured by
the RGB camera and the obtained measurements for each position $(i,j)$ is

$\mathbf{Z}_{ij:}=\mathbf{A}^{\top}\mathbf{X}_{ij:}.$ (3)

Here, $\boldsymbol{\mathcal{Z}}\in\mathbb{R}^{M\times N\times 3}$ is the
measured RGB image, and $\mathbf{A}\in\mathbb{R}^{B\times 3}$ is the spectral
response function of RGB detector [32]. $\mathbf{Z}_{ij:}\in\mathbb{R}^{B}$
and $\mathbf{X}_{ij:}\in\mathbb{R}^{3}$ represent the vectorization of
$\mathbf{Z}(i,j,:)$ and $\mathbf{X}(i,j,:)$, respectively. Similar to (2), (3)
can be also reformulated as the linear case

$\mathbf{z}=\mathbf{\Phi}^{r}\mathbf{x},$ (4)

where $\mathbf{\Phi}^{r}\in\mathbb{R}^{3MN\times MNB}$ is the spectral sensing
matrix from $\mathbf{A}$, $\mathbf{z}$ is the vectorization of
$\boldsymbol{\mathcal{Z}}$.

Figure 3: Illustration of CASSI from 3D scene to 2D coded image.

###  III-B Proposed Observation Model

To reconstruct the HSI from the CASSI and RGB measurements, previous works
[21, 6, 24, 25, 28, 29] stack $\mathbf{y}$ and $\mathbf{z}$, meanwhile
$\mathbf{\Phi}^{c}$ and $\mathbf{\Phi}^{r}$ from (2) and (4) to build the
following framework

$\left[\begin{array}[]{c}\mathbf{y}\\\
\mathbf{z}\end{array}\right]=\left[\begin{array}[]{c}\mathbf{\Phi}^{c}\\\
\mathbf{\Phi}^{r}\end{array}\right]\mathbf{x}.$ (5)

Therefore, (5) can be regarded as an extension framework as that of CASSI
reconstruction (1). The optimization of (5) with proper regularizers [21, 29,
6, 23, 24] always cost huge computational time.

To avoid the huge computation, different from (5), we assume that the HSIs can
be decomposed to the orthogonal spectral basis and related spatial
coefficients. We separately compute the coefficients from the RGB measurement,
and the orthogonal spectral basis from the CASSI measurement, and finally fuse
the two elements to reconstruct the HSIs. In detail, HSIs are assumed to lie
in an approximate low-dimensional spectral subspace. Specifically, the
spectral low-rank representation of HSIs can be formulated as

$\mathbf{X}=\mathbf{E}\mathbf{W},$ (6)

where we define $\mathbf{X}:=\mathbf{X}_{(3)}$ without losing generality,
$\mathbf{E}\in\mathbb{R}^{B\times k}$ and $\mathbf{W}\in\mathbb{R}^{k\times
MN}$ are the orthogonal spectral basis and coefficients, respectively. $k$ is
the rank of $\mathbf{X}$.

Our major contribution is to embed (6) into (3) and (1) to obtain a new
observation model from CASSI and RGB measurements

$\displaystyle\mathbf{Z}$ $\displaystyle=$
$\displaystyle(\mathbf{A}^{\top}\mathbf{E})\mathbf{W},$ (7)
$\displaystyle\mathbf{y}$ $\displaystyle=$
$\displaystyle\mathbf{\Phi}^{\mathbf{W}}\mathbf{e},$ (8)

where we again define $\mathbf{Z}:=\mathbf{Z}_{(3)}$.
$\mathbf{\Phi}^{\mathbf{W}}$ is the composition of $\boldsymbol{\mathcal{C}}$
and $\mathbf{W}$, and $\mathbf{e}$ is the vectorization of $\mathbf{E}$. We
detailed introduce how to obtain $\mathbf{\Phi}^{\mathbf{W}}$ from known
$\boldsymbol{\mathcal{C}}$ and $\mathbf{W}$. From (1), we can observe that the
sensor mask $\boldsymbol{\mathcal{C}}$ of CASSI measures the HSI
$\boldsymbol{\mathcal{X}}$ pixel by pixel. Therefore, (1) can be reformulated
as [44]

$\displaystyle Y(i,j)$
$\displaystyle=\mathbf{C}_{ij:}^{\top}\mathbf{E}\mathbf{W}_{ij:}$ (9)
$\displaystyle=\\{\mathbf{W}_{ij:}^{\top}\otimes\mathbf{C}_{ij:}^{\top}\\}\mathbf{e},$

where $\mathbf{C}_{ij:}$ represents the vectorization of $\mathbf{C}(i,j,:)$,
$\mathbf{W}_{ij:}=\mathbf{W}(:,i+(j-1)\times N)$, and $\otimes$ is the
Kronecker operator. By stacking the measurements of whole pixels together, we
can obtain $\mathbf{\Phi}^{\mathbf{W}}\in\mathbb{R}^{MN\times kB}$ in (8) as

$\mathbf{\Phi}^{\mathbf{W}}=[\mathbf{W}_{11:}^{\top}\otimes\mathbf{C}_{11:}^{\top};\ldots;\mathbf{W}_{MN:}^{\top}\otimes\mathbf{C}_{MN:}^{\top}].$
(10)

The objective of (7), (8) is to reconstruct the $\boldsymbol{\mathcal{X}}$
from the measurements $\mathbf{Y}$ and $\boldsymbol{\mathcal{Z}}$ via CASSI
and RGB camera, respectively. Using this new observation model, we can
efficiently reconstruct the HSIs using non-iterative fusion method.

###  III-C Proposed Fusion Model

According to the observation model (7), (8), we try to estimate $\mathbf{W}$
and $\mathbf{E}$ separately, and then reconstruct the HSI. As illustrated in
Fig. 1, we optimize the coefficients $\mathbf{W}$ from the RGB measurements
$\boldsymbol{\mathcal{Z}}$, and then obtain the orthogonal spectral basis
$\mathbf{E}$ from CASSI measurements $\mathbf{Y}$. Finally, we fuse
$\mathbf{E}$ and $\mathbf{W}$ to reconstruct the HSI. The proposed fusion
model can be formulated as

$\displaystyle\mathbf{W}$ $\displaystyle=$
$\displaystyle\underset{\mathbf{W}}{\arg\min}\|\mathbf{Z}-(\mathbf{A}^{\top}\mathbf{E})\mathbf{W}\|_{F}^{2},$
(11) $\displaystyle\mathbf{e}$ $\displaystyle=$
$\displaystyle\underset{\mathbf{e}}{\arg\min}\|\mathbf{y}-\mathbf{\Phi}^{\mathbf{W}}\mathbf{e}\|_{F}^{2}.$
(12)

From (11), we can easily optimize $\mathbf{W}$ via singular value
decomposition (SVD) [31]. Subsequently, we can adopt (10) to compose
$\mathbf{\Phi}^{\mathbf{W}}\in\mathbb{R}^{MN\times kB}$. Since $MN\gg kB$,
$\mathbf{\Phi}^{\mathbf{W}}$ can be regarded as a full-column rank matrix.
Therefore, the optimization of $\mathbf{e}$ can be deduced as

$\mathbf{e}=\left[(\mathbf{\Phi}^{\mathbf{W}})^{\top}(\mathbf{\Phi}^{\mathbf{W}})\right]^{-1}(\mathbf{\Phi}^{\mathbf{W}})^{\top}\mathbf{y}.$
(13)

Then, we reshape $\mathbf{e}$ to the matrix version $\mathbf{E}$, and
reconstruct HSI via $\mathbf{X}=\mathbf{E}\mathbf{W}$. The final 3D HSI can be
obtained via the folding operator
$\boldsymbol{\mathcal{X}}=\text{fold}_{3}(\mathbf{X})$.

Compared to previous works [23, 6, 24, 29] which try to reconstruct the full
HSIs, our proposed observation model (7), (8) is to decompose the original
$\boldsymbol{\mathcal{X}}$ into two very low-dimensional components, and
propose to reconstruct the low-dimensional components instead of high
dimensional $\boldsymbol{\mathcal{X}}$. The spectral low-rank property has
been embedded into our observation model, and the ill condition of HSI
reconstruction is significantly alleviated. This advantage of our proposed
observation model assures the success of non-iterative optimization with no
regularization, further saves huge computational time. Furthermore, different
from precious works (5), the spectral sensing matrix of RGB detector in our
model is absorbed into the estimation of orthogonal spectral basis, and we do
not need to know the spectral sensing matrix in advance.

###  III-D Analysis and Improvements

Fusion model choice: From the proposed observation model (7), (8), we can
formulate a generalized fusion model as follows

$\displaystyle\\{\mathbf{E},\mathbf{W}\\}=\underset{\mathbf{E},\mathbf{W}}{\arg\min}$
$\displaystyle\|\mathbf{Z}-(\mathbf{A}^{\top}\mathbf{E})\mathbf{W}\|_{F}^{2}+\|\mathbf{y}-\mathbf{\Phi}^{\mathbf{W}}\mathbf{e}\|_{F}^{2}$
$\displaystyle+\lambda_{1}\|\mathbf{W}\|_{reg1}+\lambda_{2}\|\mathbf{E}\|_{reg2}$
(14)

to jointly update $\mathbf{E}$ and $\mathbf{W}$. Here, $\|\mathbf{W}\|_{reg1}$
and $\|\mathbf{E}\|_{reg2}$ mean the regularizers on $\mathbf{W}$ and
$\mathbf{E}$, separately; $\lambda_{1}$ and $\lambda_{2}$ are the parameters
to balance the contributions from each regularizers. Typically, TV [21, 29],
non-local sparse representation [6], and non-local low-rank matrix/tensor
factorization [23, 31] can be utilized to regularize $\mathbf{W}$ and spectral
smoothness can be utilized to restrict $\mathbf{E}$. However, we utilize the
proposed fusion model (11), (12) rather than (III-D) to reconstruct the HSI
due to the following reasons. $i$) Our proposed model do not require the
additional regularizers on $\mathbf{W}$ and $\mathbf{E}$, which significantly
reduces the computation time; meanwhile we don’t need to know the spectral
sensing matrix $\mathbf{A}$ in advance, which increases the applicability in
different settings. $ii$) As reported in the experimental section, our
proposed fusion model has achieved the state-of-the-art reconstruction results
compared to existing non-training based methods. $iii$) Theoretical guarantee
can be weakly obtained to our proposed fusion model.

###### Proposition 1

(conditioned exact reconstruction) If we have the conditions: 1) the spectral
sensing matrix of RGB is of full-column rank; 2) the composed
$\mathbf{\Phi}^{\mathbf{W}}$ is of full-column rank; and 3) $k\leqslant 3$.
Then $\boldsymbol{\mathcal{X}}$ is the exact reconstruction from the
measurements $\mathbf{Y}$ and $\boldsymbol{\mathcal{Z}}$ via
$\text{fold}_{3}(\mathbf{\mathbf{E}\mathbf{W}})$, with $\mathbf{E}$ from SVD,
and $\mathbf{W}$ from (13).

The proof is provided in the Appendix. In Proposition 1, condition 1) is
related to the design of RGB camera. As reported in [45], the spectral sensing
matrix in Nikon D700 is of full-column rank. That is to say, condition 1) is
easy to obtain. Furthermore, since $MN\gg kB$, $\mathbf{\Phi}^{\mathbf{W}}$
can be regarded as a full-column rank matrix with specific pattern
$\mathbf{\Phi}$. The only problem of condition is that $k\leqslant 3$, which
is not the real case in most of HSIs. We will propose an approach to meet this
condition and thus an improvement of the fusion algorithm in the next
subsection.

Figure 4: (a) 3D patch extracted from the Toy dataset; (b) singular values in
logarithm of reshaped patches V.S global image. We select 100 patches and
average the singular values in logarithm.

Patch based fusion algorithm: As analyzed before, the condition of $k\leqslant
3$ is unreasonable. Generally speaking, for a HSI scenario, the rank is always
larger that $3$ [30, 31]. However, from another perspective, the HSIs usually
have the piece-wise smoothness property [23, 39]. That is to say, the pixel
signatures from a local spatial patch have a higher probability of being
similar. In Fig. 4, we illustrate the comparison of the singular values in
logarithm between the extracted patch images and the global Toy image. We
choose 100 patch images of size $100\times 100\times 31$ and plot the average
singular values. From the figure, it can be seen that the average singular
values of patch images drop much faster than the ones from the global image,
indicating that local patch analysis can significantly increase the spectral
low-rank property [46]. To utilize this finding, we propose to segment the
HSIs to overlapping patches, and reconstruct each patch separately (and in
parallel). Finally, the whole reconstructed patches are composed to the final
reconstructed HSI. The proposed patch based fusion algorithm is presented in
Algorithm 1. What we need to mention is that we should obey $mn>kB$, where
$m,n$ are the patch spatial size, to make sure $\mathbf{\Phi}^{\mathbf{W}}$
satisfy the full-column rank condition.

The patch based fusion can enhance the spectral low-rank property and increase
the HSI reconstruction performance, as analyzed in Table VI. However, the
condition of $k\leqslant 3$ is not always strictly satisfied, as presented in
Fig. 4. Furthermore, the patch based fusion also increases the computational
time. This means that the proposed patch based fusion algorithm still has room
for improvement. Another strategy to alleviate this condition is to explore
priors with regularization. However, as we mentioned before, the
regularizations will significantly improve the computational burden. We leave
this question, i.e., the condition of $k\leqslant 3$, as future work.

Algorithm 1 Patch based fusion

0: $\mathbf{Y}$ measured via CASSI, $\boldsymbol{\mathcal{Z}}$ measured via
RGB camera, CASSI sensing matrix $\mathbf{\Phi}$, rank $k$.

1: Initialization: segment $\mathbf{Y}$, $\boldsymbol{\mathcal{Z}}$ and
$\mathbf{\Phi}$ into aligned overlapping patches of size $m\times n$, $m\times
n\times 3$, and $m\times n\times B$ respectively.

2: for each aligned patch do

3: A). Update the coefficients via SVD.

4: B). Update the orthogonal spectral basis via (13).

5: C). Reconstruct each patch of size $m\times n\times B$.

6: end for

7: return Aggregate all the patches into the final $\mathcal{X}$;

Orthogonal spectral basis estimation improvement: The optimization of
$\mathbf{E}$ via (13) simply utilized the measurement
$\boldsymbol{\mathcal{C}}$, ignoring the RGB sensor matrix $\mathbf{A}$.
Similar to (9), the RGB measurement can be also formulated as the pixel-by-
pixel version

$\displaystyle Z(i,j,k)$
$\displaystyle=(\mathbf{C}(:,k))^{\top}\mathbf{E}\mathbf{W}_{ij:}$ (15)
$\displaystyle=\\{\mathbf{W}_{ij:}^{\top}\otimes(\mathbf{C}(:,k))^{\top}\\}\mathbf{e}.$

So we can obtain $\mathbf{\Phi}_{RGB}^{\mathbf{W}}$ in the similar way of
(10), and update $\mathbf{e}$ via

$\mathbf{e}=\underset{\mathbf{e}}{\arg\min}\left\|[\mathbf{y};\mathbf{z}]-[\mathbf{\Phi}^{\mathbf{W}};\mathbf{\Phi}_{RGB}^{\mathbf{W}}]\mathbf{e}\right\|_{F}^{2}$
(16)

instead of (12). In Section V-D, we compare the results obtained by (12) and
(16), respectively. As reported, optimizing $\mathbf{E}$ via (16) results in
the limited improvement of accuracy, unfortunately, increasing almost
$4\times$ more computation time. We thus recommend to use (12) to update the
orthogonal spectral basis $\mathbf{E}$.

TABLE I: The size of the dataset used in the experiments. Image name | HSI | CASSI | RGB
---|---|---|---
CAVE |  512$\times$512$\times$31 |  512$\times$512 |  512$\times$512$\times$3
ICVL |  512$\times$512$\times$31 |  512$\times$512 |  512$\times$512$\times$3
RS |  400$\times$200$\times$128 |  400$\times$200 |  400$\times$200$\times$3
Bird |  1021$\times$703$\times$24 |  1021$\times$703 |  1021$\times$703$\times$3
TABLE II: Quantitative evaluation of CAVE data experiments for different HSI reconstruction methods from CASSI and RGB measurements. CAVE | method | balloons | beads | cd | toy | clay | cloth | egyptian | face | beers | food | Average
---|---|---|---|---|---|---|---|---|---|---|---|---
M-PSNR | TV | 31.10 | 20.82 | 28.73 | 25.93 | 27.75 | 22.98 | 33.70 | 31.86 | 30.07 | 28.89 | 28.18
DeSCI | 32.14 | 21.76 | 29.04 | 28.70 | 28.15 | 25.30 | 36.10 | 33.49 | 31.28 | 29.78 | 29.57
TV-RGB | 36.75 | 26.67 | 31.68 | 29.19 | 36.42 | 25.45 | 37.49 | 35.82 | 35.13 | 34.27 | 32.89
DeSCI-RGB | 40.35 | 30.54 | 32.94 | 34.36 | 39.93 | 30.27 | 42.23 | 39.78 | 39.28 | 37.67 | 36.74
DLTR | 41.35 | 28.66 | 32.26 | 33.12 | 39.68 | 29.00 | 42.13 | 43.63 | 41.29 | 37.82 | 36.89
Fusion | 39.50 | 32.30 | 29.25 | 40.02 | 38.94 | 38.34 | 48.40 | 43.10 | 41.01 | 38.86 | 38.97
PFusion | 48.50 | 32.91 | 32.95 | 44.33 | 49.24 | 39.28 | 50.93 | 46.40 | 46.49 | 43.12 | 43.42
M-SSIM | TV | 0.952 | 0.659 | 0.922 | 0.880 | 0.845 | 0.660 | 0.948 | 0.956 | 0.946 | 0.852 | 0.862
DeSCI | 0.963 | 0.727 | 0.939 | 0.925 | 0.861 | 0.819 | 0.968 | 0.971 | 0.960 | 0.878 | 0.901
TV-RGB | 0.964 | 0.878 | 0.908 | 0.944 | 0.911 | 0.783 | 0.980 | 0.971 | 0.946 | 0.913 | 0.920
DeSCI-RGB | 0.991 | 0.942 | 0.962 | 0.981 | 0.954 | 0.923 | 0.994 | 0.990 | 0.991 | 0.956 | 0.968
DLTR | 0.985 | 0.903 | 0.975 | 0.971 | 0.973 | 0.900 | 0.981 | 0.992 | 0.992 | 0.977 | 0.965
Fusion | 0.985 | 0.933 | 0.916 | 0.968 | 0.934 | 0.975 | 0.983 | 0.981 | 0.995 | 0.952 | 0.962
PFusion | 0.995 | 0.944 | 0.971 | 0.989 | 0.994 | 0.977 | 0.996 | 0.994 | 0.997 | 0.980 | 0.984
MSA | TV | 10.21 | 25.35 | 13.01 | 12.49 | 17.15 | 16.30 | 14.21 | 10.29 | 4.80 | 16.12 | 13.99
DeSCI | 9.79 | 25.28 | 11.08 | 12.06 | 15.94 | 14.48 | 13.43 | 9.49 | 4.72 | 15.09 | 13.14
TV-RGB | 6.63 | 13.78 | 12.40 | 10.36 | 14.24 | 14.22 | 11.33 | 8.83 | 3.71 | 13.40 | 10.89
DeSCI-RGB | 4.64 | 9.96 | 7.44 | 7.23 | 9.40 | 10.16 | 8.23 | 6.60 | 2.60 | 9.57 | 7.58
DLTR | 6.24 | 13.79 | 8.03 | 8.67 | 15.38 | 9.23 | 13.27 | 7.10 | 1.27 | 6.43 | 8.94
Fusion | 9.91 | 17.65 | 28.94 | 15.90 | 29.15 | 5.84 | 23.38 | 15.58 | 2.19 | 23.99 | 17.25
PFusion | 4.54 | 14.59 | 7.26 | 6.81 | 7.99 | 5.07 | 9.25 | 6.37 | 1.21 | 10.77 | 7.39

(a) Original

(b) CASSI

(c) RGB

(d) TV

(e) DeSCI

(f) TV-RGB

(g) DeSCI-RGB

(h) DLTR

(i) Fusion

(j) PFusion

Figure 5: Reconstructed results of different methods on CAVE-Toy image. The
color images are composed of bands 31, 11, 6 for red, green and blue,
respectively.

##  IV Experimental Results

In this section, we adopted simulated and real dataset experiments to
demonstrate the advantage of the proposed methods on the reconstruction of
CASSI and RGB measurements compared to other state-of-the-art methods. We
employ the following methods for comparison: CASSI reconstruction methods,
i.e. GAP-TV [22], DeSCI [23]111https://github.com/liuyang12/DeSCI and CASSI &
RGB reconstruction methods, i.e. TV-RGB [29], DeSCI-RGB222We implement the
algorithm by using (5) instead of (2). and DLTR [24]333Thank Dr. S. Zhang for
providing the experimental results.. The proposed algorithm for the global
reconstruction is marked as ‘Fusion’, meanwhile the proposed patch based
fusion algorithm is denoted as ‘PFusion’.

In the simulated experiments, the peak signal-to-noise ratio (PSNR),
structural similarity (SSIM) and the mean of the spectral angle (MSA) [25, 31]
between the reconstructed and original images are used to evaluate the
results. Since HSIs have multi spectral bands, we calculate the mean PSNR and
mean SSIM of each band, and then average them [25, 31], which are denoted as
‘M-PSNR’ and ‘M-SSIM’, respectively. The whole experiments (except DLTR) are
programmed in Matlab R2017b on a laptop with CPU Core i7-8750H 16G memory
444DLTR was programmed on a laptop with CPU Core i7-6700 64G memory. TV-RGB
and DeSCI-RGB dealing with the global HSIs are out of memory on the laptop.
Therefore, as the same of ‘PFusion’, we segment the HSIs to overlapping
patches and adopt TV-RGB and DeSCI-RGB to process small patches separately.

###  IV-A Simulated Experiments On CAVE

The first test dateset is CAVE555http://www1.cs.columbia.edu/CAVE/databases/,
which contains $32$ HSI scenes. We select the first $10$ images for the
experiments. We adopted the CASSI sensing matrix from [23] to simulate the
CASSI measurements, meanwhile spectral sensing matrix of Nikon D700 camera
[45] to simulate the RGB measurements. The size of original HSI, CASSI
measurements and RGB measurements are presented in Table I. For our proposed
methods, we choose $k=3$ and $m=n=100$.

(a) beads

(b) food

Figure 6: The absolute difference errors (smaller is better) between the means
of the whole original signatures and the one obtained by different
reconstruction methods on CAVE images.

Quantitative comparison. We present the M-PSNR, M-SSIM and MSA values of
reconstructed images via different methods in Table II. Our proposed PFusion
method achieves the best values in average of ten different HSI scenes of all
these evaluation metrics, conforming the advantage of our proposed method.
Compared to CASSI reconstruction methods TV and DeSCI, the reconstruction from
CASSI and RGB measurements can significantly improve the quality of the
reconstructed image. This benefit is from the additional RGB measurements.
DeSCI-RGB and DLTR can achieve satisfied results, indicating the efficiency of
non-local low-rank processing. Our proposed method can achieve higher MPSNR
values compared to DeSCI-RGB, meanwhile lower MSA values. This is mainly
because our proposed Fusion can take full advantage of RGB measurements to
enhance the spatial characteristic, unfortunately fails to take the full
features of global spectral property within rank $k=3$. Therefore, the
spectral distortion of our proposed Fusion is severe. Our proposed PFusion,
which processes overlapping patches instead of global image, can significantly
improve the reconstructed spectral quality with smaller MSA values.
Furthermore, the M-PSNR and M-SSIM values obtained by PFusion are further
improved compared to those of Fusion.

Visual comparison. To further illustrate the efficiency of different methods,
we present the color images (composed of bands 31, 21 and 6 [31]) of different
reconstruction methods on Toy image in Fig. 5. From the figure, it can be
observed that the RGB measurements are different from the original images
composed of bands 31, 11 and 6. TV based method results in the blurred
results. DeSCI can produce clearer image, and DeSCI-RGB and DLTR can further
improve the results. However, the spectral distortion is also obvious, as some
undesired color artifacts appeared in the results of DeSCI-RGB and DLTR. Our
proposed Fusion and PFusion can produce the most similar results compared to
the original image presented in Fig. 5(a).

To further check the reconstructed spectrum, we also present the absolute
values between the means of the whole original signatures and the one obtained
by different reconstruction methods. We choose two images, $i.e.,$ Beads and
food for illustration. From Fig. 6, we can obtain that Fusion method can
achieve similar results compared to DeSCI-RGB and DLTR. Our proposed PFusion
can significantly improve the spectral quality compared to Fusion, indicating
the significant advantage of our proposed PFusion to process overlapping
patches instead of the global image.

###  IV-B Simulated Experiments On ICVL

TABLE III: Quantitative evaluation of ICVL data experiments for different HSI reconstruction methods from CASSI and RGB measurements. ICVL | method | 4cam | BGU | IDS | Ist | prk | Average
---|---|---|---|---|---|---|---
M-PSNR | TV | 31.11 | 28.93 | 28.39 | 34.85 | 32.42 | 31.14
DeSCI | 33.17 | 31.38 | 30.46 | 35.44 | 34.24 | 32.94
TV-RGB | 35.20 | 33.80 | 32.67 | 38.34 | 37.67 | 35.54
DeSCI-RGB | 40.39 | 40.30 | 37.91 | 39.93 | 43.89 | 40.48
DLTR | 41.28 | 39.12 | 36.33 | 39.98 | 39.01 | 38.04
Fusion | 43.65 | 44.91 | 43.19 | 37.13 | 46.95 | 43.17
Pfusion | 48.76 | 48.94 | 45.07 | 41.91 | 49.87 | 46.91
M-SSIM | TV | 0.912 | 0.858 | 0.912 | 0.967 | 0.907 | 0.911
DeSCI | 0.953 | 0.927 | 0.951 | 0.974 | 0.940 | 0.949
TV-RGB | 0.946 | 0.931 | 0.928 | 0.982 | 0.961 | 0.950
DeSCI-RGB | 0.992 | 0.990 | 0.992 | 0.990 | 0.994 | 0.991
DLTR | 0.986 | 0.979 | 0.983 | 0.991 | 0.969 | 0.975
Fusion | 0.993 | 0.992 | 0.993 | 0.985 | 0.993 | 0.991
Pfusion | 0.995 | 0.996 | 0.995 | 0.992 | 0.996 | 0.995
MSA | TV | 4.36 | 6.50 | 4.34 | 5.08 | 5.84 | 5.23
DeSCI | 3.87 | 5.57 | 3.94 | 4.59 | 4.76 | 4.55
TV-RGB | 3.12 | 4.41 | 2.92 | 2.92 | 3.87 | 3.45
DeSCI-RGB | 1.93 | 2.28 | 1.81 | 1.97 | 1.94 | 1.99
DLTR | 1.23 | 2.27 | 0.96 | 1.25 | 2.97 | 1.74
Fusion | 1.91 | 2.28 | 0.94 | 5.20 | 2.49 | 2.56
Pfusion | 0.82 | 1.16 | 0.64 | 1.94 | 1.21 | 1.15

(a) Original

(b) CASSI

(c) RGB

(d) TV

(e) DeSCI

(f) TV-RGB

(g) DeSCI-RGB

(h) DLTR

(i) Fusion

(j) PFusion

Figure 7: Reconstructed results of different methods on ICVL-BGU image. The
color images are composed of bands 31, 11, 6 for red, green, and blue,
respectively.

(a) BGU

(b) IDS

Figure 8: The absolute difference errors between the mean of the whole original signatures and the one obtained by different reconstruction methods on ICVL images. TABLE IV: Quantitative evaluation of EO-1 data experiments for different HSI reconstruction methods from CASSI and RGB measurements. Method | TV | DeSCI | TV-RGB | DeSCI-RGB | DLTR | Fusion | PFusion
---|---|---|---|---|---|---|---
M-PSNR | 26.64 | 28.96 | 29.91 | 37.01 | 35.56 | 41.8 | 43.34
M-SSIM | 0.839 | 0.948 | 0.931 | 0.977 | 0.963 | 0.983 | 0.984
MSA | 4.929 | 4.253 | 3.864 | 1.6 | 1.78 | 1.03 | 0.85

(a) Original

(b) CASSI

(c) RGB

(d) TV

(e) DeSCI

(f) TV-RGB

(g) DeSCI-RGB

(h) DLTR

(i) Fusion

(j) PFusion

Figure 9: Reconstructed results of different methods on EO-1 image. The color
images are composed of bands 100, 60, 10 for red, green, and blue,
respectively.

(a) point 1

(b) point 2

(c) point 3

Figure 10: The absolute difference errors between the original signature and
the one obtained by different reconstruction methods on EO-1 image.

(a) CASSI

(b) RGB

(c) TV

(d) DeSCI

(e) Fusion

(f) PFusion

Figure 11: Reconstructed results of different methods on real Bird image. The
color images are composed of bands 20, 10, 6 for red, green, and blue,
respectively.

(a) (100,100)

(b) (350,350)

(c) (700,700)

Figure 12: The absolute difference errors between the original signature and
the one obtained by different reconstruction methods on real Bird image.

The second test dataset is ICVL666http://icvl.cs.bgu.ac.il/hyperspectral/. We
simulate the CASSI measurements and RGB measurements the same as that of CAVE
dataset. We select $5$ images from ICVL dataset for the experiments. The
original image from ICVL is of size $1392\times 1304\times 31$ and we reshape
it to the size of $512\times 512\times 31$. For the proposed methods, we still
choose $k=3$ and $m=n=100$.

Quantitative comparison. Table III presents the quantitative comparison
results of reconstructed images via different methods on $5$ ICVL images.
Again, the reconstruction methods from only the CASSI measurements $i.e.,$ TV
and DeSCI achieve the worst values of M-PSNR, M-SSIM and MSA. The combined
measurements of CASSI and RGB can significantly improve the reconstruction
results, just as TV-RGB. DeSCI-RGB and DLTR further improve the results.
DeSCI-RGB can achieve higher M-PSNR and M-SSIM values, meanwhile DLTR obtains
higher MSA values. Our proposed PFusion significantly improves the spectral
quality compared to Fusion, and again achieves the best quantitative values
across different ICVL images, further indicating the advantage of our proposed
strategy to process the reconstruction of CASSI and RGB measurements.

Visual comparison. Fig. 7 presents the color images (composed of bands 31, 11
and 6) of different reconstruction methods on ICVL BGU image. It can be
clearly observed that RGB measurements can provide the spatial structure
information, but totally different spectral information. The reconstruction
result of TV from the CASSI measurements is blurred. DeSCI and TV-RGB can
improve the results, but they are still blurred. DeSCI-RGB and DLTR further
improve the results. However, as presented in the enlarged rectangle of Fig.
7(h), DLTR loses the color of the board. To sum up, the proposed Fusion and
PFusion produce the best results, with much more detailed information compared
to the original image. Fig. 8 illustrates the absolute values between the mean
of the original signatures and the one obtained by different reconstruction
methods of $3$ signatures from three different ICVL images. Compared to DeSCI-
RGB and DLTR, the advantage of the proposed Fusion is limited to preserve the
spectral information. However, the proposed PFusion can produce the smallest
absolute difference errors, indicating the best performance in the
preservation of HSI spectrum.

###  IV-C Simulated Experiments On Remote Sensing

We further conduct the remote sensing compressive reconstruction experiments
to testify the efficiency of the proposed methods. The HSI is from the earth-
observing One (EO-1) Hyperion sensor
777https://archive.usgs.gov/archive/sites/eo1.usgs.gov/index.html. Similar to
[23], we adopt CASSI to compress the HSI, meanwhile the spectral sensing
matrix to obtain RGB measurements. The size of HSI, CASSI measurements and RGB
measurements are presented in Table I.

Quantitative comparison. We adopt Table IV to present the quantitative
comparison results of reconstructed images via different methods on the EO-1
image. From the table, it can be observed that the proposed PFusion obtains
the best accuracy in three different metrics. TV-RGB, DeSCI-RGB and DLTR
achieves better results than that of TV and DeSCI, indicating that the
reconstruction results from the combined CASSI and RGB measurements are better
those of only CASSI measurements.

Visual comparison. Fig. 9 presents the color images (composed of bands 100, 60
and 10) of different reconstruction methods on the EO-1 image. We can observe
that our proposed Fusion and PFusion outperform others to preserve the details
of the reconstructed image. Other comparison methods produce the blurred
results. Fig. 10 shows the absolute difference errors between the original
signature and the ones obtained by different reconstruction methods on the
EO-1 image. Our proposed PFusion again beats other comparison methods, and
achieves the smallest absolute difference errors. That is to say, our proposed
Fusion and PFusion can preserve the spatial details, meanwhile, the
overlapping patches processing of PFusion can significantly improve the
capability to preserve the spectral information.

###  IV-D Real Experiments On Bird

In this section, we apply the proposed method to the HSI reconstruction of the
real snapshot-hyperspectral compressive imaging data. The real Bird data is
from CASSI system [9], which has been widely used for the real experiments
analysis [23, 28]. The size of reconstructed Bird image, CASSI and RGB
measurements are presented in Table I. Since we do not know the spectral
sensing matrix from HSI to RGB measurements, we cannot implement TV-RGB,
DeSCI-RGB and DLTR for comparison. So we simply compare our proposed methods
to TV and DeSCI888The results of TV and DeSCI are provided by Dr. Y. Liu at
website https://github.com/liuyang12/DeSCI.

Visual comparison. Fig. 11 illustrates the color images (composed of bands 20,
10 and 6) of different reconstruction methods on the real bird data. We may
find that the measured RGB image contains abundance spatial information.
However, the reconstructed images via TV and DeSCI lose the spatial details.
By contrast, our proposed Fusion and PFusion are capable of preserving the
spatial information from RGB measurements. Fig. 12 presents the signatures of
three points obtained by different methods. It can be observed that our
proposed PFusion are closest to the reference signatures. It indicates that
our proposed PFusion can preserve the spectral information, therefore, again
demonstrating the advantage of our proposed method.

##  V Ablation Study and Discussion

###  V-A Spectral Sensing Matrix Analysis

We first analyze the proposed methods on different spectral sensing matrices.
The spectral sensing matrix of RGB detector is adopted to measure the RGB
image from HSI, and the RGB measurement is adopted to provide the
coefficients, as illustrated in Fig. 1. Fig. 13 presents different kinds of
designed spectral sensing matrices, including (a) provided by Nikon D700
camera, (b) average of the bands, and (c) random selection of three single
bands. Table V shows the average quantitative evaluation results of proposed
methods with different sensor matrices on 10 CAVE HSIs. It can be clearly
observed that the proposed Fusion and PFusion can produce almost similar
results with spectral sensing matrices provided by Nikon D700 camera and
average design strategy. On the other hand, the random single band selection
strategy produces the worse results. This comparison of different spectral
sensing matrices guides us to design the RGB measurements efficiently. This
suggests that the RGB measurement should try to cover all bands information.
In this case, the coefficients estimated from the measured RGB image will
provide more useful information for the subsequent HSI reconstruction.

(a)

(b)

(c)

Figure 13: Different spectral sensing matrix to measure RGB from HSI. (a) Provided by Nikon D700 camera, (b) average based design, and (c) single band selection. TABLE V: Quantitative evaluation of CAVE data experiments for proposed methods with different sensor matrices. sensing matrix | Fusion | PFusion
---|---|---
(a) | (b) | (c) | (a) | (b) | (c)
MPSNR | 38.97 | 38.47 | 35.32 | 43.42 | 43.39 | 37.76
MSSIM | 0.962 | 0.955 | 0.906 | 0.984 | 0.984 | 0.955
MSA | 17.25 | 16.06 | 16.86 | 7.39 | 6.16 | 7.61

###  V-B Rank $k$ Analysis

(a) MPSNR

(b) MSSIM

(c) MSA

Figure 14: Changes of quantitative evaluation results obtained by the proposed
methods with different rank $k$. We test on 10 CAVE HSIs.

Subsequently, we analyze the influence of rank $k$. Note that, we assume that
HSIs lie in an approximate low-dimensional spectral subspace, and therefore, a
larger rank $k$ will assure more precise low-rank approximation for the
original HSI as presented in (6). However, in our fusion model (7) and (11),
the rank $k$ is bounded by the size of spectral sensing matrix $\mathbf{A}$.
In the proposed methods, we set $k=3$ (3 is the column size of $\mathbf{A}$)
to explore more information of HSI. In this section, we change the size of
spectral sensing matrix $\mathbf{A}$, and the measured RGB image is changed
from 1-band panchromatic image to 5-band multispectral image, and therein the
rank $k$ changes from 1 to 5. Fig. 14 presents the quantitative evaluation
results obtained by the proposed methods with different rank $k$. As
illustrated, with the increase of rank $k$, the quantitative evaluation
results increase significantly. This is reasonable, since more measurements
suggest higher reconstruction accuracy. From the figure, we can observe that
the complementary 3-band RGB measurements can produce much higher HSI
reconstruction results compared to complementary panchromatic measurements. By
contrast, the improvement from 3-band RGB to multispectral measurements with
more than 3-band is limited. What’s more, the multispectral measurements will
bring huge measured burden for the hardware. Therefore, to realize our
proposed methods on the real hardware, we recommend the complementary 3-band
RGB measurements.

###  V-C Patch Spatial Size Analysis

TABLE VI: Changes of quantitative evaluation results obtained by the proposed PFusion with patch size $m$. We test on on 10 CAVE HSIs. patch size | 20 | 40 | 60 | 100 | 160 | 320 | 512
---|---|---|---|---|---|---|---
MPSNR | 41.72 | 43.17 | 43.41 | 43.42 | 42.31 | 40.03 | 38.97
MSSIM | 0.988 | 0.989 | 0.985 | 0.984 | 0.979 | 0.967 | 0.962
MSA | 6.72 | 6.98 | 7.29 | 7.39 | 9.94 | 15.82 | 17.25

As analyzed in Section III-D and the experimental results presented in Section
IV, we can conclude that the proposed PFusion with patch processing can
significantly improve the reconstruction results compared to Fusion. For our
proposed PFusion on the 10 CAVE images, we change the patch size $m$ (we set
$m=n$) from $20$ to $512$, and present the quantitative evaluation results in
table VI. It can be observed that our proposed PFusion is robust when the
patch size changes from $40$ to $100$. On the other hand, when the patch size
becomes larger than $320$, the performance decreases significantly. This is
mainly because the rank $k=3$ is not enough to describe the spectral
information of such big patches. Meanwhile, the smaller patch size will
decrease the quality of orthogonal spectral basis estimation, as analyzed in
Section III-D. It should be noticed that the patch size should be also related
to the complexity of the image. If the image is complex and contains multiple
materials, the patch size should be smaller, and vice versa. In the whole
experiments, we fix the patch size as 100.

###  V-D Improved Spectral Basis Estimation

We present an improved orthogonal spectral basis estimation method in (16).
Compared to original (12), the improved (16) combines CASSI and RGB
measurements to estimate the orthogonal spectral basis. Table VII illustrates
the quantitative evaluation results obtained by PFusion and the improved
method (16) on different datasets. We can clearly observe that the improved
method can slightly ($\leq$0.16dB in M-PSNR) enhance the results. However, it
brings more than three times additional computational burden compared to the
original case (12). It is in fact a trade-off between the accuracy and speed
of the HSI reconstruction.

TABLE VII: Quantitative evaluation results obtained by PFusion and the improved on different dataset. | CAVE | ICVL | RS
---|---|---|---
PFusion | Improved | PFusion | Improved | PFusion | Improved
M-PSNR | 43.42 | 43.46 | 46.91 | 47.07 | 43.34 | 43.38
M-SSIM | 0.984 | 0.984 | 0.995 | 0.995 | 0.984 | 0.985
MSA | 7.39 | 7.27 | 1.15 | 1.16 | 0.85 | 0.81
Time | 1.8 | 9.2 | 1.8 | 9.2 | 5.1 | 18.2

###  V-E Computational Efficiency

TABLE VIII: Running time (in seconds) of different methods on different dataset. Method | TV | DeSCI | TV-RGB | DeSCI-RGB | DLTR | Fusion | PFusion
---|---|---|---|---|---|---|---
CAVE | 251 | 3787 | 523 | 12544 | 52585 | 1.3 | 1.8
ICVL | 250 | 3776 | 511 | 12549 | 48572 | 1.3 | 1.8
RS | 1123 | 8288 | 1478 | 18297 | 79234 | 3.3 | 5.1
Bird | 472 | 18690 | * | * | * | 2.8 | 3.9

Finally, we analyze the computational efficiency of the proposed methods. The
cost time of TV and DeSCI methods on real Bird image are provided by [23].
Table VIII presents the running time of different methods on different
dataset. TV-RGB and DeSCI-RGB cost much more time compared to TV and DeSCI due
to the additional measurements (5). Non-local low-rank related methods DeSCI-
RGB and DLTR 999The cost time was provided by Dr. S. Zhang can obtain the
satisfied reconstruction results, however the computational burden is also
huge. In particular, our proposed methods can reconstruct the HSI within
several seconds, more than 5000 times faster than that of non-local related
methods DeSCI-RGB and DLTR. This demonstrates a huge advantage of our proposed
model to decompose the full HSI into orthogonal spectral basis and spatial
coefficients, and then reconstruct the two smaller components separately.
Compared to Fusion, PFusion needs additional computational time. However,
PFusion significantly improves the reconstruction accuracy, as presented in
Table VI. To conclude, our proposed model (7), (8) is proved to be efficient
to obtain the best accuracy, meanwhile much less computational time.

##  VI CONCLUSION

In this study, we have proposed a new model to reconstruct the HSI from CASSI
and RGB measurements. We explore the spectral low-rank property of HSI, and
decompose it to the orthogonal spectral basis and spatial coefficients. The
RGB measurements can provide the estimation of coefficients, meanwhile CASSI
measurements are adopted to estimate the orthogonal spectral basis. Compared
to the previous works that try to reconstruct full HSIs with different
regularizers, our proposed Fusion and PFusion methods do not require non-local
processing or iteration, which can save huge computational time. Furthermore,
our proposed methods does not require the spectral sensing matrix in advance.
The experiments on three simulated HSI datasets and one real dataset
demonstrated that our proposed Fusion and PFusion can reconstruct the HSI with
the highest accuracy using far less computational time. In summary, our
proposed methods can reconstruct the HSIs in a fast, flexible and high
accuracy manner. As illustrated in (III-D), we did not add any additional
regularizer on our model, to alleviate the computational burden. However, as
presented in Section III-D, our model still has room for improvement to meet
the condition of $k\leqslant 3$. We hope that the proposed optimization can be
used as a baseline, and regard the improvement of our proposed model as future
work.

##  VII Appendix

###  VII-A Proposition 1

###### Proof VII.1

The proof can be decomposed to two steps. First, the original HSI can be
uniquely decomposed to $\mathbf{X}=\mathbf{E}\mathbf{W}$, where $\mathbf{W}$
is computed via SVD on $\mathbf{Z}$. Using SVD, we have
$\mathbf{Z}=(\mathbf{A}^{\top}\mathbf{X})=\mathbf{F}\mathbf{W}$, where
$\mathbf{F}\in\mathbb{R}^{3\times K}$. Since $k\leqslant 3$ and $\mathbf{A}$
is of full-column rank, we can conclude that $\mathbf{F}$ is of full-column
rank, and $\mathbf{W}$ is of full-row rank. If we have another
$\mathbf{E}^{{}^{\prime}}$ such that
$\mathbf{X}=\mathbf{E}^{{}^{\prime}}\mathbf{W}$. We can obtain
$(\mathbf{E}-\mathbf{E}^{{}^{\prime}})\mathbf{W}=\mathbf{0}$. Since
$\mathbf{W}$ is of full-row rank, we can conclude that
$\mathbf{E}=\mathbf{E}^{{}^{\prime}}$. Therefore, the decomposition
$\mathbf{X}=\mathbf{E}\mathbf{W}$ is unique.

Second, $\mathbf{E}$ can be uniquely obtained via (13). We have obtained the
full-row rank $\mathbf{W}$ via SVD on $\mathbf{Z}$. Since the full-row rank
property of $\mathbf{W}$, and the specific design of CASSI mask
$\boldsymbol{\mathcal{C}}$ as in (10), we have the condition that
$\mathbf{\Phi}^{\mathbf{W}}$ is of full-column rank. Therefore, the solution
of (12) is an overdetermined equation problem. That is to say, the obtained
$\mathbf{E}$ from (12) is unique.

Thus, we obtain the proposition.

## References

  * [1] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Face recognition in hyperspectral images,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 25, no. 12, pp. 1552–1560, Dec 2003.
  * [2] D. Stein, S. Beaven, L. Hoff, E. Winter, A. Schaum, and A. Stocker, “Anomaly detection from hyperspectral imagery,” _IEEE Signal Process. Mag._ , vol. 19, no. 1, pp. 58–69, 2002.
  * [3] J. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader, and J. Chanussot, “Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches,” _IEEE J. Sel.Topics Appl. Earth Observ. Remote Sens._ , vol. 5, no. 2, pp. 354–379, Apr. 2012.
  * [4] G. A. Shaw and H.-h. K. Burke, “Spectral imaging for remote sensing,” _Lincoln Laboratory Journal_ , vol. 14, no. 1, pp. 3–28, 2003.
  * [5] R. O. Green, M. L. Eastwood, C. M. Sarture, T. G. Chrien, M. Aronsson, B. J. Chippendale, J. A. Faust, B. E. Pavri, C. J. Chovit, M. Solis, M. R. Olah, and O. Williams, “Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (aviris),” _Remote Sens. Environ._ , vol. 65, no. 3, pp. 227–248, Sep. 1998.
  * [6] L. Wang, Z. Xiong, G. Shi, F. Wu, and W. Zeng, “Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 10, pp. 2104–2111, Oct. 2016.
  * [7] D. L. Donoho, “Compressed sensing,” _IEEE Transactions on Information Theory_ , vol. 52, no. 4, pp. 1289–1306, 2006.
  * [8] M. Descour and E. Dereniak, “Computed-tomography imaging spectrometer: experimental calibration and reconstruction results,” _Applied optics_ , vol. 34, no. 22, pp. 4817–4826, 1995.
  * [9] A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” _Applied optics_ , vol. 47, no. 10, pp. B44–B51, 2008.
  * [10] B. K. Ford, M. R. Descour, and R. M. Lynch, “Large-image-format computed tomography imaging spectrometer for fluorescence microscopy,” _Optics Express_ , vol. 9, no. 9, pp. 444–453, 2001.
  * [11] M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual-disperser architecture,” _Optics express_ , vol. 15, no. 21, pp. 14 013–14 027, 2007\.
  * [12] X. Lin, Y. Liu, J. Wu, and Q. Dai, “Spatial-spectral encoded compressive hyperspectral imaging,” _ACM Transactions on Graphics (TOG)_ , vol. 33, no. 6, pp. 1–11, 2014.
  * [13] X. Lin, G. Wetzstein, Y. Liu, and Q. Dai, “Dual-coded compressive hyperspectral imaging,” _Optics letters_ , vol. 39, no. 7, pp. 2044–2047, 2014.
  * [14] H. Arguello and G. R. Arce, “Colored coded aperture design by concentration of measure in compressive spectral imaging,” _IEEE Transactions on Image Processing_ , vol. 23, no. 4, pp. 1896–1908, 2014.
  * [15] R. M. Willett, R. F. Marcia, and J. M. Nichols, “Compressed sensing for practical optical imaging systems: a tutorial,” _Optical Engineering_ , vol. 50, no. 7, p. 072601, 2011.
  * [16] X. Cao, T. Yue, X. Lin, S. Lin, X. Yuan, Q. Dai, L. Carin, and D. J. Brady, “Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world,” _IEEE Signal Processing Magazine_ , vol. 33, no. 5, pp. 95–108, 2016.
  * [17] G. Arce, D. Brady, L. Carin, H. Arguello, and D. Kittle, “Compressive coded aperture spectral imaging: An introduction,” _IEEE Signal Process Mag._ , vol. 31, no. 1, pp. 105–115, 2013.
  * [18] B. Arad and O. Ben-Shahar, “Sparse recovery of hyperspectral signal from natural rgb images,” in _European Conference on Computer Vision_. Springer, pp. 19–34.
  * [19] Z. Xing, M. Zhou, A. Castrodad, G. Sapiro, and L. Carin, “Dictionary learning for noisy and incomplete hyperspectral images,” _SIAM Journal on Imaging Sciences_ , vol. 5, no. 1, pp. 33–56, 2012.
  * [20] L. Zhang, W. Wei, Y. Zhang, C. Shen, A. Van, and Q. Shi, “Dictionary learning for promoting structured sparsity in hyperspectral compressive sensing,” _IEEE Trans. Geosci. Remote Sens._ , vol. 54, no. 12, pp. 7223–7235, Dec. 2016.
  * [21] L. Wang, Z. Xiong, D. Gao, G. Shi, and F. Wu, “Dual-camera design for coded aperture snapshot spectral imaging,” _Applied optics_ , vol. 54, no. 4, pp. 848–858, 2015.
  * [22] X. Yuan, “Generalized alternating projection based total variation minimization for compressive sensing,” in _IEEE International Conference on Image Processing (ICIP)_. IEEE, Sep. 2016, pp. 2539–2543.
  * [23] Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, “Rank minimization for snapshot compressive imaging,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 12, pp. 2990–3006, Dec 2019\.
  * [24] S. Zhang, L. Wang, Y. Fu, X. Zhong, and H. Huang, “Computational hyperspectral imaging based on dimension-discriminative low-rank tensor recovery,” in _The IEEE International Conference on Computer Vision (ICCV)_ , October 2019\.
  * [25] T. Zhang, Y. Fu, L. Wang, and H. Huang, “Hyperspectral image reconstruction using deep external and internal learning,” in _The IEEE International Conference on Computer Vision (ICCV)_ , October 2019.
  * [26] X. Miao, X. Yuan, Y. Pu, and V. Athitsos, “$\lambda$-net: Reconstruct hyperspectral images from a snapshot measurement,” in _The IEEE International Conference on Computer Vision (ICCV)_ , October 2019.
  * [27] M. Qiao, Z. Meng, J. Ma, and X. Yuan, “Deep learning for video compressive sensing,” _APL Photonics_ , vol. 5, no. 3, p. 030801, 2020.
  * [28] X. Yuan, T.-H. Tsai, R. Zhu, P. Llull, D. Brady, and L. Carin, “Compressive hyperspectral imaging with side information,” _IEEE Journal of selected topics in Signal Processing_ , vol. 9, no. 6, pp. 964–976, 2015.
  * [29] L. Wang, Z. Xiong, G. Shi, W. Zeng, and F. Wu, “Compressive hyperspectral imaging with complementary rgb measurements,” in _2016 Visual Communications and Image Processing (VCIP)_. IEEE, pp. 1–4.
  * [30] J. M. Bioucas-Dias and J. M. Nascimento, “Hyperspectral subspace identification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 46, no. 8, pp. 2435–2445, Aug. 2008.
  * [31] W. He, Q. Yao, C. Li, N. Yokoya, and Q. Zhao, “Non-local meets global: An integrated paradigm for hyperspectral denoising,” in _CVPR_ , Jun. 2019.
  * [32] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 50, no. 2, pp. 528–537, 2012\.
  * [33] Y. Zhao, H. Guo, Z. Ma, X. Cao, T. Yue, and X. Hu, “Hyperspectral imaging with random printed mask,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [34] T. Kolda and B. Bader, “Tensor decompositions and applications,” _SIAM review_ , vol. 51, no. 3, pp. 455–500, 2009.
  * [35] Y. August, C. Vachman, Y. Rivenson, and A. Stern, “Compressive hyperspectral imaging by random separable projections in both the spatial and the spectral domains,” _Applied optics_ , vol. 52, no. 10, pp. D46–D54, 2013.
  * [36] M. Golbabaee and P. Vandergheynst, “Joint trace/tv norm minimization: A new efficient approach for spectral compressive imaging,” in _2012 19th ICIP_. IEEE, 2012, pp. 933–936.
  * [37] Y. Wang, L. Lin, Q. Zhao, T. Yue, D. Meng, and Y. Leung, “Compressive sensing of hyperspectral images via joint tensor tucker decomposition and weighted total variation regularization,” _IEEE Geos. and Remote Sens.Lett._ , vol. 14, no. 12, pp. 2457–2461, 2017.
  * [38] F. Jiang, X.-Y. Liu, H. Lu, and R. Shen, “Efficient multi-dimensional tensor sparse coding using t-linear combination,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [39] Y. Fu, Y. Zheng, I. Sato, and Y. Sato, “Exploiting spectral-spatial correlation for coded hyperspectral image restoration,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3727–3736.
  * [40] J. Xue, Y. Zhao, W. Liao, and J. C. Chan, “Nonlocal tensor sparse representation and low-rank regularization for hyperspectral image compressive sensing reconstruction,” _Remote Sens._ , vol. 11, no. 2, p. 193, 2019.
  * [41] Z. Meng, J. Ma, and X. Yuan, “End-to-end low cost compressive spectral imaging with spatial-spectral self-attention,” in _European Conference on Computer Vision_. Springer, 2020, pp. 187–204.
  * [42] I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, “High-quality hyperspectral reconstruction using a spectral prior,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 6, pp. 1–13, 2017.
  * [43] L. Wang, C. Sun, Y. Fu, M. H. Kim, and H. Huang, “Hyperspectral image reconstruction using a deep spatial-spectral prior,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8032–8041.
  * [44] M. Golbabaee, S. Arberet, and P. Vandergheynst, “Compressive source separation: Theory and methods for hyperspectral imaging,” _IEEE Transactions on Image Processing_ , vol. 22, no. 12, pp. 5096–5110, 2013.
  * [45] Y. Qu, H. Qi, and C. Kwan, “Unsupervised sparse dirichlet-net for hyperspectral image super-resolution,” in _Proc. IEEE Conf. CVPR_ , 2018, pp. 2511–2520.
  * [46] W. He, H. Zhang, L. Zhang, and H. Shen, “Hyperspectral image denoising via noise-adjusted iterative low-rank matrix approximation,” _IEEE J. Sel.Topics Appl. Earth Observ. Remote Sens._ , vol. 8, no. 6, pp. 3050–3061, 2015\.