11institutetext: Montefiore Institute of Electrical Engineering and Computer
Science, University of Liège 22institutetext: Space sciences, Technologies and
Astrophysics Research (STAR) Institute, University of Liège

# A deep learning approach for focal-plane wavefront sensing
using vortex phase diversity

M. Quesnel 1122 G. Orban de Xivry 22 G. Louppe 11 O. Absil, F.R.S.-FNRS Senior
Research Associate22

###### Abstract

Context. The performance of high-contrast imaging instruments is limited by
wavefront errors, in particular by non-common path aberrations (NCPA). Focal-
plane wavefront sensing (FPWFS) is appropriate to handle NCPA because it
measures the aberration where it matters the most, i.e., at the science focal
plane. Phase retrieval from focal-plane images results nonetheless in a sign
ambiguity for even modes of the pupil-plane phase.

Aims. The phase diversity methods currently used to solve the sign ambiguity
tend to reduce the science duty cycle, i.e., the fraction of observing time
dedicated to science. In this work, we explore how we can combine the phase
diversity provided by a vortex coronagraph with modern deep learning
techniques to perform efficient FPWFS without losing observing time.

Methods. We apply the state-of-the-art convolutional neural network
EfficientNet-B4 to infer phase aberrations from simulated focal-plane images.
The two cases of scalar and vector vortex coronagraphs (SVC and VVC) are
considered, respectively using a single post-coronagraphic PSF or two PSFs
obtained by splitting the circular polarization states.

Results. The sign ambiguity is properly lifted in both cases even at low
signal-to-noise ratios (S/N). Using either the SVC or the VVC, we reach very
similar performance compared to using phase diversity with a defocused PSF,
except for high levels of aberrations where the SVC slightly underperforms
compared to the other approaches. The models finally show great robustness
when trained on data with a wide range of wavefront errors and noise levels.

Conclusions. The proposed FPWFS technique provides a 100% science duty cycle
for instruments using a vortex coronagraph and does not require any additional
hardware in the case of the SVC.

###### Key Words.:

Techniques: high angular resolution – Techniques: image processing

##  1 Introduction

Because of the small angular separation and high contrast between planetary
companions and their parent star, exoplanet imaging is particularly
challenging. Although these constraints can be addressed with specific
instruments such as coronagraphs, residual wavefront aberrations still
represent an inherent obstacle for detecting the majority of exoplanets. To a
large extent, these residuals originate from non-common path aberrations
(NCPA) between the scientific and wavefront sensing arms. Focal-plane
wavefront sensing (FPWFS) is an approach that has the advantage of taking into
account NCPA by probing their signature in the focal-plane images (Jovanovic
et al. 2018) while offering high sensitivity.

Estimating phase aberrations from the sole scientific images is not trivial,
since the relationship between focal-plane intensities and pupil-plane phase
is non-linear and degenerate (Guyon 2018). Numerical methods have been
developed for FPWFS, in particular iterative algorithms (Fienup 1982), the
most standard one being the Gerchberg-Saxton algorithm (Gerchberg 1972). More
recent techniques have been proposed for various applications (see Jovanovic
et al. 2018, for a review), including recently the use of deep learning
techniques for FPWFS (Paine & Fienup 2018; Andersen et al. 2019, 2020; Orban
de Xivry et al. 2021). All of these approaches have to deal with one important
hindrance: for a centrosymmetric pupil, two different phase distributions in
the input pupil plane can produce the same PSF. This ambiguity, also called
twin-image problem (e.g., Guizar-Sicairos & Fienup 2012), is typically solved
with phase diversity using for instance an additional defocused PSF (Gonsalves
1982), or an asymmetric pupil mask (Martinache 2013). This however reduces the
science duty cycle because some observing time, and/or part of the science
beam, is to be dedicated exclusively to wavefront measurements.

Based on the properties of the vector vortex coronagraph (VVC, Mawet et al.
2005), a Nijboer-Zernike phase retrieval approach tailored to the post-VVC PSF
was formulated in Riaud et al. (2012a, b). They proposed to split the two
circular polarization states to exploit the phase diversity introduced by the
two opposite topological charges associated with the VVC. A similar approach
was more recently used by Bos et al. (2019) in the case of the grating-vector
apodizing phase plate, although it also required an asymmetric pupil to fully
lift the sign ambiguity.

Here, we revisit the problem of phase retrieval behind a vortex coronagraph
using deep learning techniques. Unlike an analytical approach, which could
show limitations regarding its formulation, deep learning models can be
trained regardless of the instruments and observing conditions. First, in
Sect. 2, we argue that a scalar vortex coronagraph (SVC) has the potential to
yield comparable residual phase errors to the dual-polarization VVC
implementation, using a single post-coronagraphic PSF instead of two. In Sect.
3, we present our deep learning approach, based on convolutional neural
networks (CNN), which have the advantage of being easy to implement and
flexible regarding the considered instrument, and were already shown in our
previous works to be capable of reaching fundamental noise limits (Quesnel et
al. 2020; Orban de Xivry et al. 2021). Finally, in Sect. 4, we provide
quantitative results on simulated data. We compare the performance of our
vortex phase diversity method to a classical approach, and assess the
robustness of the models, notably in the presence of representative
atmospheric turbulence residuals.

##  2 Vortex phase diversity

###  2.1 Vortex coronagraphs

The vortex coronagraph (VC), introduced by Mawet et al. (2005), is a
transparent focal plane mask that diffracts on-axis light outside of the pupil
area. A Lyot stop placed in a downstream pupil plane allows this diffracted
light to be blocked, enabling high contrast observations. Because of wavefront
aberrations, some incoming light from the star is however not blocked. Indeed,
the VC only removes the Airy disk, and speckles will still appear in the focal
plane.

There are two different flavours of vortex coronagraphs: vectorial (VVC) and
scalar (SVC). The VVC applies a geometrical phase ramp to the incoming
wavefront with a transmission $t=\exp(\pm j\,l_{p}\,\theta)$, where $l_{p}$ is
the topological charge and $\theta$ is the azimuthal coordinate. Conjugated
phase ramps are applied to each circular polarization state, producing a
different signature in the focal plane for each (Riaud et al. 2012a). Here we
focus on a topological charge $|l_{p}|=2$, which is the most commonly used
design so far (Mawet et al. 2009; Absil et al. 2016), but the following
developments would also hold for any even topological charge. Unlike the VVC,
the SVC uses longitudinal phase delays (Ruane et al. 2019; Desai et al. 2021),
and thereby applies the same phase ramp (e.g., $l_{p}=+2$) to both
polarization states. The focal-plane signature behind an SVC corresponds to
the one obtained with a single polarization state using the VVC.

###  2.2 Sign ambiguity and phase diversity

In FPWFS, the Fourier relationship between the PSF and the pupil-plane phase
causes a sign ambiguity for Zernike modes of even radial order (e.g., defocus,
astigmatism):

$|\mathcal{F}(E_{\rm even}(x))|^{2}=|\mathcal{F}(E_{\rm even}^{*}(-x))|^{2},$
(1)

where $E_{\rm even}=\text{exp}(-j\phi_{\rm even}(x))$ is the pupil-plane
electric field composed of even Zernike modes only, $E_{\rm even}^{*}$ is its
conjugate, and $\mathcal{F}(.)$ is the Fourier transform operator. This sign
ambiguity is a strong limitation for FPWFS using a single in-focus image. A
fair amount of FPWFS methods have been developed to solve the twin-image
problem. The most standard one is to use an additional known defocus together
with the in-focus image. An illustration of this ambiguity can be found in
Fig. 1, where we generated two phase maps with opposite signs for their even
Zernike modes. After propagation through a VVC, the in-focus PSFs are the same
in both cases (Fig. 1(d) and 1(f)), showcasing the twin-image problem. The
out-of-focus PSFs, however, are different (Fig. 1(e) and 1(g)) because the
added defocus has the same sign in both cases, which allows the ambiguity to
be lifted.

Now, if the two orthogonal circular polarization states are split downstream
of the VVC to separate the conjugated phase ramps ($l_{p}=-2$ and $+2$), or if
the case of the SVC is considered, the in-focus PSFs are not identical any
more (Fig. 1(h) and 1(j), or Fig. 1(i) and 1(k)). The resulting PSFs are in
actually switched between the two circular polarization states. This indicates
that the sign ambiguity can potentially be lifted when using either the two
PSFs obtained from the separate circular polarization states, or the single
PSF behind the SVC independently of the polarization state. This illustrates
the fact the VC provides an azimuthal phase diversity, which can be used
instead of the radial phase diversity provided by an additional defocus (Riaud
et al. 2012a). In the case of the SVC, the sign ambiguity would then be lifted
similarly to using only an out-of-focus PSF in classical phase diversity
(e.g., Lamb et al. 2021).

(a) entrance pupil

(b) Phase map $\phi$

(c) Phase map $\phi^{\prime}(\phi^{\prime}_{even}=-\phi_{even})$

(d) $l_{p}=\pm 2$

(e) $l_{p}=\pm 2$; out

(f) $l_{p}=\pm 2$

(g) $l_{p}=\pm 2$; out

(h) $l_{p}$ = +2

(i) $l_{p}$ = -2

(j) $l_{p}$ = +2

(k) $l_{p}$ = -2

Figure 1: Comparison of simulated PSFs for two conjugated phase maps $\phi$
(left) and $\phi^{\prime}$ (right): for $\phi^{\prime}$ we set opposite
Zernike coefficients to those of $\phi$ only for the even modes, with a total
of 18 modes starting from defocus. (a): Entrance annular pupil. (b, c): phase
maps with opposite even modes and identical odd modes. (d, e): in-focus and
out-of-focus PSFs obtained from propagating (b) with both opposite topological
charges $l_{p}$ together. (f, g): the same but propagating c instead. (h, i):
in-focus PSFs obtained from (b) with each $l_{p}$ used separately. (j, k): the
same but propagating (c) instead.

##  3 Deep learning approach

###  3.1 Data generation

In our simulations, we consider an annular entrance pupil with a diameter of 8
m and a central obstruction of 30%. An observed bandwidth of 0.2 $\mu$m is
defined around 2.2 $\mu$m (K band), by simulating a total of 5 wavelengths. A
pixel scale of $0.25\,\lambda/D/$pix is set with a detector containing
64$\times$64 pixels, giving a field-of-view of $16\,\lambda/D$. The most
relevant simulation parameters are listed in Table 1. Based on these
parameters, we generate the phase aberrations using annular Zernike
polynomials, which make up an orthonormal basis on the input pupil:

$\phi(x,y)=\sum_{i=1}^{N_{\rm modes}}c_{i}\,Z_{i}(x,y),$ (2)

where $\phi$ is the complete phase map, $Z_{i}$ the Zernike polynomials,
$c_{i}$ the corresponding coefficients, $(x,y)$ the pupil plane coordinates,
and $N_{\rm modes}$ the number of modes considered.

The generated data sets are composed of 18 or 88 Zernike modes, up to the 5th
and 12th radial orders respectively, excluding the piston, tip and tilt modes.
The set of Zernike coefficients for each sample is first randomly generated
within the range $[-1,1]$ before each coefficient is divided by its
corresponding radial order to approximate a $1/f^{2}$ power spectral density
profile, typically encountered with good quality optics (Dohlen et al. 2011).
Low and high aberration levels, represented by wavefront error (WFE)
distributions centered at 70 and 350 nm rms, respectively, are considered by
normalizing the Zernike coefficients accordingly. An example of such
distribution can be seen in Orban de Xivry et al. (2021). For classical phase
diversity, the additional defocus is set to $\lambda$ / 5 = 440 nm. In our
case, this amount of diversity is close to the optimal value in terms of phase
retrieval performance. The defocus is added in the entrance pupil plane, as if
done by the DM, which means that the resulting defocused PSFs contain more
flux than the in-focus PSFs as the coronagraphic performance of the VC is
degraded.

To increase the representativeness of our simulations, and to test the
robustness of our approach, we add atmospheric turbulence residuals to the
phase maps. A state-of-the-art extreme adaptive optics (AO) is simulated using
the COMPASS library (Ferreira et al. 2018), assuming a loop frequency of
3.5kHz, 2-frame delay, a 50$\times$50 deformable mirror (i.e., 2040
modes/valid actuators), a pyramid sensor with $5\lambda$/D of modulation
(without noise). This yields a Strehl ratio of about 98% at 2.2 $\mu$m,
corresponding to a wavefront error (WFE) of about 50 nm rms. We sample the AO
residuals at 10 Hz and we use a sequence of 10 consecutive phase screens by
summing up the corresponding PSFs. We therefore simulate a 1-s exposure in the
presence of a given amount of static NCPA. Results with data containing these
AO residuals are shown in Sect. 4.3.

To simulate a PSF obtained behind a VVC, we perform two propagations, one with
$l_{p}=+2$ and the other with $l_{p}=-2$, to consider each circular
polarization state. The downstream Lyot stop blocks 2% of the outer pupil
area. The resulting PSFs are then either summed up to reproduce the non-
polarized case, or they are kept separate to consider the dual-polarization
case. To simulate the SVC, only one such PSF is taken. The optical propagation
is handled by the HEEPS package111https://github.com/vortex-exoplanet/HEEPS
(Carlomagno et al. 2020), which makes use of PROPER (Krist 2007). Examples of
generated phase maps and PSFs can be found in Fig. 1. We then add photon noise
to our PSFs, so that the signal-to-noise ratio (S/N) is defined as
$S/N=\sqrt{N_{\rm ph}}$, where $N_{\rm ph}$ is the number of photons. A
square-root stretching operation is applied to the PSFs to help the CNN
identify the speckle patterns. Finally, we normalize the PSFs with a min-max
scaling to obtain flux in the range [0,1], which ensures the CNN is fed with
same-scale quantities.

Table 1: Data generation parameters Parameter | Value
---|---
Central obstruction | 30 %
Topological charge | 2
Pixel scale |  $0.25\,\lambda/D$ / px
Field-of-view | $16\,\lambda/D$

###  3.2 Model architecture

We build deep neural network models whose goal is to map the Zernike
coefficients of phase aberrations $\phi$ from a given PSF $I$, i.e., to
approximate a non-linear function $f$ such that $\phi\approx f(I)$. CNNs have
proven to be very well suited for image analysis, with numerous applications
for both classification and regression tasks. CNN-based architectures have
been developing very quickly in recent years, with performance still improving
greatly. We have therefore used a state-of-the-art deep CNN called
EfficientNet (Tan & Le 2019). This type of architecture stands out from other
ones by using a new scaling technique: all dimensions of the CNN (depth,
width, and resolution) are scaled by the same compound coefficient $\Phi$, and
the parameters are inferred from the original model or baseline
EfficientNet-B0 ($\Phi=0$). There are thus different models available, and we
choose to use EfficientNet-B4, for which we obtain the best trade-off between
model performance and runtime. EfficientNet-B4 has a total of $1.9\times
10^{7}$ parameters and 4.2 $\times 10^{9}$ FLOPS. It has about the same number
of parameters as the ResNet-50 architecture, which was used in Quesnel et al.
(2020) and Orban de Xivry et al. (2021).

###  3.3 Model training

For a given training, a data set composed of $10^{5}$ PSFs (or PSF pairs for
the cases with two input channels) is randomly split into training (90%) and
validation (10%) sets. Each sample also contains the true NCPA phase maps as
labels, while the AO phase screens are not given. Batches composed of 64 data
samples are then consecutively fed to the neural network. We define the loss
function as the root-mean-square error of the phase residuals. Weight updates
based on the loss are handled by the Adam optimizer (Kingma & Ba 2017). To
improve the performance, we set a penalty on the loss (“weight decay”) of
$10^{-7}$ for the low aberration regime and $10^{-6}$ for the higher
aberration regime. We also set an initial learning rate of $10^{-3}$ that is
decreased by a factor of two as soon as the validation loss reaches a plateau
over 15 epochs. This results in sudden loss drops, allowing the performance to
be greatly improved. Pretrained models on ImageNet are used to initialize the
weights. The training of the model is stopped if no improvement of the
validation loss is observed over 25 epochs. This results in training
procedures lasting between 50 and 250 epochs.

##  4 Results

Figure 2: Root-mean-squared error (RMSE) per Zernike mode, following the Noll
convention, starting from the defocus mode. Four cases are compared (see Table
2 for notations), using a single in-focus post-VVC PSF without splitting the
polarization states (light blue), two post-VVC PSFs with additional defocus
(dark blue), the two post-VVC PSFs associated with each polarization states
(red), and a single PSF after the SVC (orange). The RMSE of the input phase
maps is represented in black and the even modes are indicated by the green
areas. Left: Input WFE of 70 nm distributed over 18 modes. Right: Input WFE of
350 nm distributed over 88 modes. In both examples, the S/N in the entrance
pupil plane is equal to 100.

We compare the capacity of different configurations to lift the sign ambiguity
as well as their performance. The designation of these configurations,
together with some of their parameters, can be found in Table 2: we consider
the cases of the VVC with or without classical phase diversity (“VVC [in,
out]-focus” and “VVC in-focus” respectively), which are compared to the new
approaches presented in this paper (“VVC dual-polar” and “SVC”). The non-
coronagraphic case (“no vortex [in, out]-focus”) is evaluated as well. We also
investigate the possibility to work with differential PSFs obtained by
subtracting the separate circular polarization states (“VVC dual-polar; diff
PSFs”). In the last part of this section, we add atmospheric turbulence
residuals and we assess the robustness of the models regarding variations in
the S/N levels, input wavefront errors, and Zernike polynomial orders.

Table 2: Configurations considered for phase retrieval Designation | $l_{p}$ | Defocus | Inputs
---|---|---|---
VVC in-focus |  $\pm$2 | no | 1
VVC [in, out]-focus |  $\pm$2 | yes | 2
VVC dual-polar | [+2, -2] | no | 2
SVC | +2 | no | 1
no vortex [in, out]-focus | 0 | yes | 2
VVC dual-polar; diff PSFs | [+2, -2] | no | 1

###  4.1 Phase sign determination

To determine whether the models predict the correct sign, we look at the
performance per Zernike mode. The metric used is the root-mean-squared error
(RMSE) per mode:

$\sigma_{z}=\sqrt{\frac{1}{N_{\rm test}}\,\sum_{i}^{N_{\rm
test}}(\hat{c}_{i}-c_{i})^{2}}\>,$ (3)

where $N_{\rm test}$ is the number of test samples, while $\hat{c}$ and $c$
are the estimated and true Zernike coefficients, respectively.

In Fig. 2 we compare the performance per mode between four cases for two
spatial frequency contents and levels of aberrations. A network using only in-
focus PSFs in the non-polarized case with the VVC yields no correction for
even Zernike modes, because the model tends to predict zero for the
coefficients facing the ambiguity (due to the $l_{2}$-norm training loss). For
odd modes, the model is able to provide some correction, even though its
quality is limited by the loss function, which does not discriminate between
even and odd modes. Adding defocused PSFs as input solves the problem as
expected (Quesnel et al. 2020). In the dual-polarization case, a network using
either one or both circular polarization states separately as input (SVC and
VVC respectively) also yields good performance for even modes as well as for
odd modes. This indicates that the sign ambiguity is lifted properly with
these two approaches.

It is noteworthy that the performance depends marginally on the Zernike mode:
the error tends to increase for larger angular azimuthal orders, at a given
radial order. Our interpretation is that since the phase information is of
higher spatial frequency and located closer to the edge of the pupil in these
cases, it is more difficult for the CNN model to identify those features.

###  4.2 Performance compared to classical phase diversity

Figure 3: Performance in terms of rms WFE on the phase residuals at different
S/N levels. Each point corresponds to a model trained and evaluated on the
indicated S/N (six S/N are considered, and slight horizontal shifts are
applied to be able to discern each point.) The same colors in Fig. 2 are used,
with the addition of the performance with classical imaging (green), using
differential post-VVC PSFs (violet), as well as the theoretical limit (black
dashed line). The median values are represented and the error bars correspond
to the 2-98th percentiles. The S/N indicated are the ones at the entrance
pupil plane, i.e the flux suppression induced by the vortex mask is taken into
account. Left: Input WFE of 70 nm distributed over 18 modes. Right: Input WFE
of 350 nm distributed over 88 modes.

We now compare our method to the classical phase diversity approach in terms
of overall phase retrieval performance. The rms WFE on the phase residuals is
used as a metric and it is defined for each test sample as:

$\sigma_{\phi}=\sqrt{\frac{1}{N_{\rm pix}}\sum_{i}^{N_{\rm
pix}}(\hat{\phi}_{i}-\phi_{i})^{2}}$ (4)

where $N_{\rm pix}$ is the number of pixels, while $\hat{\phi}$ and $\phi$ are
the estimated and true pupil phases, respectively.

In our simulations, we consider the fact that the vortex coronagraphs block
out most of the starlight, and that for a given stellar magnitude, the
resulting flux in the detector plane is reduced. The flux is also equally
split between each PSF for all the cases having two channels, while for the
configurations having a single one, the PSF receives the total remaining flux
behind the vortex mask. The performance of the trained models at different S/N
levels defined in the entrance pupil plane is shown in Fig. 3. In our case,
S/N between $10^{1}$ and ${3\times 10^{3}}$ correspond to stars of apparent
magnitudes in the range from 18.6 to 6.2.222with an integration time of 1s, a
transmission and quantum efficiency equal to 50%, a telescope diameter of 8 m,
and a filter bandwidth of 50 nm. For a median input WFE of 70 nm with 18 modes
(Fig. 3, left), the simulated performance is almost identical for the
classical, SVC and VVC dual-polarization approaches, even though the
additional defocus increases the overall S/N at the focal plane for the
classical method. For a median input WFE of 350 nm with 88 modes (Fig. 3,
right), the phase residuals are distinctly higher for all the configurations,
and a plateau is reached for S/N above 1000. We can especially notice that the
sole PSF behind the SVC somewhat limits the performance in this case. Our main
hypothesis for this discrepancy is that, in a high aberration regime, the
effects of the non-linear nature of the problem are greater. The extra
information given by having two input channels is therefore favorable and
makes the models easier to train. In general, it is more difficult to train
data sets containing strong aberrations, and this can typically be improved by
using more data (e.g., $5\times 10^{5}$ samples, see Orban de Xivry et al.
2021), more complex architectures (e.g., EfficientNet-B6), and/or stronger
weight decay.

We also consider the possible presence of planetary companions in the detected
images. This additional, off-axis source of light is largely unaffected by the
vortex phase ramp and therefore adds the same signature in both circular
polarization states. This additional light source may bias the phase retrieval
process, and lead to unwanted planetary signal subtraction. A possible
workaround is to subtract one polarization image from the other, in an attempt
to remove the signature of any off-axis light source. We thus assess the phase
retrieval capabilities using the difference between both polarization states
after the VVC. The results are shown in Fig. 3, to be compared with the other
configurations. We only obtain a marginal increase of the error at high S/N,
especially in the low aberration regime, which can be explained by the loss of
information produced by subtracting one PSF from the other.

The performance of the various configurations are finally compared to the
theoretical limit in Fig. 3. This limit is discussed in Orban de Xivry et al.
(2021) for non-coronagraphic imaging. For both the non-coronagraphic and
vortex imaging cases, the residual errors reach the fundamental limit in the
low aberration regime (Fig. 3, left). In a higher aberration regime, the
performance does not reach the fundamental limit, and the gap increases toward
higher S/N (Fig. 3, right). This can be improved with more robust training as
explained above. One can note that the residual errors are constrained by the
WFE distribution in the data toward lower S/N, while the theoretical limit is
independent of the input WFE distribution and continues to increase for lower
S/N, thus yielding residual WFE below the limit.

###  4.3 Model robustness

Figure 4: Phase prediction errors at different S/N levels, presented the same
way as in Fig. 3, this time including also atmospheric turbulence residuals in
the PSFs during both training and evaluation.

Figure 5: Examples of PSFs at different S/N levels (defined in the entrance
pupil plane) for $l_{p}=+2$. The resulting S/N in the detector place is
reduced due to the extinction factor introduced by the coronagraph and by the
beam splitting between the two polarization channels. The level of NCPA is
equal to 70 nm rms distributed over 18 modes (top) and 350 nm rms over 88
modes (bottom). AO residuals are also present: each PSF is the result of
combining 10 PSFs containing each a different AO residual phase screen. Figure
6: Performance with altered S/N levels during evaluation, for three models
trained on data with a median rms WFE of 70 nm over 18 modes, with an S/N of
30 (purple), 1000 (blue), and with S/N uniformly distributed over the entire
S/N range (green). Each point is obtained from an evaluation batch composed of
1000 samples (the median value together with the 2-98th percentiles are
shown).

To test how the method handles more realistic ground-based observations, we
add atmospheric turbulence residuals on top of the NCPAs, as described in
Sect. 3.1. This represents an additional source of noise since the AO
residuals are not included in the labels for training. Examples of input PSFs
at the different flux levels can be found in Fig. 5. The performance now
starts to reach a plateau of a few nm rms in the low aberration regime at high
S/N (Fig. 4, left), due to the presence of these atmospheric turbulence
residuals. In the high NCPA regime (Fig. 4, right) the AO residuals become
however negligible and the performance is almost identical to the case without
turbulence (Fig. 3, right).

We finally study the robustness of the models regarding variation in the data
during evaluation. First, we may encounter different flux levels than
considered during training. In Fig. 6, we illustrate how models in the VVC
dual-polar configuration trained on data containing 70 nm rms behave in such
conditions. Whether the training S/N is low or high, models show good
robustness to other flux levels only within a limited range, outside of which
the performance is strongly degraded. If a more robust model is required, it
is also possible to train with various flux levels. We investigate this by
using a training dataset covering the entire test S/N range, without
increasing its size. The median performance is much more consistent at every
S/N, although the variation of residual error between samples is greater, and
a small degradation can naturally be seen compared to using identical training
and testing S/N (as shown in Fig. 4).

Figure 7: Performance with different input WFE levels defined during
evaluation, for models trained on data with a median rms WFE of 70 nm over 18
modes (blue), and 350 nm rms over 88 modes (red). Models are also trained on
data following a uniform distribution covering the whole input WFE range,
using both spatial frequency regimes (cyan and orange). The S/N is 1000 and
each dataset contains 105 samples.

We also study the change in performance when evaluating outside the input WFE
training range. Fig. 7 shows the robustness of models trained on the two
aberration regimes studied in this paper. Data containing more aberrations
rapidly deteriorates the reconstruction. The models perform better when
evaluated at lower aberration levels, but have limited performance when
trained in the high aberration regime. To overcome these limitations, we train
two models over the entire test WFE range, for each of the Zernike mode
contents considered in the paper. Such models show excellent robustness, with
minimal degradation compared to models having identical training and testing
WFE distributions. This suggests that these models could be robustly used in
closed-loop operations, even with the aberration level decreasing with time.
Regarding the varying spatial power spectral density of the wavefront, the
residuals are generally constant along the Zernike modes, as seen in Fig. 2.
When giving the reconstructed PSFs as input to the same trained model, we have
observed that most residual RMS WFE stay below 10 nm for a model trained on 70
nm rms as input and an S/N of 1000. A thorough analysis of a closed-loop
application will be the subject of future work when testing the algorithm in
the lab or on-sky.

Figure 8: Robustness on higher-order aberrations. Top: Performance per Zernike
mode on test data following the distribution of the training data (red),
adding to the test data 70 modes containing 35 nm rms (purple) and 70 nm rms
(blue) of NCPAs. Bottom: Example of post-VVC PSFs ($l_{p}=+2$) for each case.

It can also be expected to have observations containing higher-order NCPAs (on
top of the changing atmospheric residuals) than considered during training.
For a model trained on 18 modes at 70 nm rms (S/N = 1000), we add 70 higher-
order Zernike modes in the test data. We observe in Fig. 8 a moderate
degradation for the 18 modes when increasing the wavefront error contained in
these additional modes, because the central PSF signature is mostly preserved.

##  5 Conclusions

We have investigated in this paper a new way to perform focal-plane wavefront
sensing using vortex coronagraphs. Based on a deep learning approach and
considering simulated data, we leverage the modulation introduced by the
vortex coronagraph (either scalar, or vectorial after splitting the circular
polarization states) to lift the sign ambiguity and perform FPWFS for various
S/N, input WFE, and spatial frequency contents. The dual-polarization method
with the VVC offers a very similar performance to the classical phase
diversity method using additional defocused PSFs, even though the level of
light is largely reduced after filtering by the VVC. For instance, considering
a star of magnitude 6.2 observed at a wavelength of 2200 nm, we obtain a
residual of 0.73 nm rms from an input WFE of 70 nm rms. In the case of the
SVC, which provides a single focal-plane image, a loss in performance is
observed only for high aberration levels. For bright stars, and with higher
order and higher levels of aberrations, the CNN training is generally
challenging, and the performance reaches a plateau of approximately 20 nm rms.
In such circumstances, more training data, larger/deeper CNN architectures,
and/or regularization techniques could further improve the phase retrieval
accuracy. Atmospheric turbulence residuals that are expected in ground-based
data only produce minor degradation in performance in a low NCPA regime, and
should not be a concern in practice. We also showed that models trained on
data containing particularly wide WFE and S/N distributions provide very
satisfactory robustness.

Potential applications of the proposed method could rely on including a
polarizing beam splitter downstream of the VVC to collect both circular
polarization states separately, either on a single or on two distinct sensors.
Since our simulation-based FPWFS experiments work well even with a single
image obtained behind an SVC, it appears that this flavour of vortex
coronagraph offers an interesting alternative, notably because it would work
without any additional optical components.

Deep learning models offer a flexible framework and fast inference speeds,
which are appreciable features for on-sky applications. The requirement on
speed is however not very stringent as we expect the lifetime of NCPAs that
produce quasi-static speckles to be in the order of minutes. But on-sky
applications will naturally come with their own challenges and discrepancies
unpredicted by simulations. To account for the difference between simulations
and real data, transfer learning techniques can be used to efficiently fine-
tune the models before observations. Finally, it is difficult to obtain
reliable and very precise NCPA labels for model training. Employing
unsupervised learning techniques, e.g. autoencoder-based architectures, is
another interesting approach that we are considering for future developments.

###### Acknowledgements.

This research made use of PyTorch (Paszke et al. 2019) and the following
implementation of EfficientNet: https://github.com/lukemelas/EfficientNet-
PyTorch. The HEEPS (Carlomagno et al. 2020) and PROPER (Krist 2007) open-
source optical propagation Python packages were used for data generation. This
project has received funding from the European Research Council (ERC) under
the European Union’s Horizon 2020 research and innovation programme (grant
agreement No 819155), and from the Wallonia-Brussels Federation (grant for
Concerted Research Actions).

## References

  * Absil et al. (2016) Absil, O., Mawet, D., Karlsson, M., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9908, Ground-based and Airborne Instrumentation for Astronomy VI, ed. C. J. Evans, L. Simard, & H. Takami, 99080Q
  * Andersen et al. (2019) Andersen, T., Owner-Petersen, M., & Enmark, A. 2019, Optics Letters, 44, 4618
  * Andersen et al. (2020) Andersen, T., Owner-Petersen, M., & Enmark, A. 2020, Journal of Astronomical Telescopes, Instruments, and Systems, 6, 034002
  * Bos et al. (2019) Bos, S. P., Doelman, D. S., Lozi, J., et al. 2019, A&A, 632, A48
  * Carlomagno et al. (2020) Carlomagno, B., Delacroix, C., Absil, O., et al. 2020, Journal of Astronomical Telescopes, Instruments, and Systems, 6, 035005
  * Desai et al. (2021) Desai, N., Llop-Sayson, J., Jovanovic, N., et al. 2021, in Techniques and Instrumentation for Detection of Exoplanets X, ed. S. B. Shaklan & G. J. Ruane, Vol. 11823, International Society for Optics and Photonics (SPIE), 238 – 246
  * Dohlen et al. (2011) Dohlen, K., Wildi, F. P., Puget, P., Mouillet, D., & Beuzit, J.-L. 2011, in Second International Conference on Adaptive Optics for Extremely Large Telescopes. Online at ¡A href=“http://ao4elt2.lesia.obspm.fr”¿http://ao4elt2.lesia.obspm.fr¡/A, 75
  * Ferreira et al. (2018) Ferreira, F., Gratadour, D., Sevin, A., & Doucet, N. 2018, 2018 International Conference on High Performance Computing & Simulation (HPCS), 180
  * Fienup (1982) Fienup, J. 1982, Applied optics, 21, 2758
  * Gerchberg (1972) Gerchberg, R. W. 1972, Optik, 35, 237
  * Gonsalves (1982) Gonsalves, R. A. 1982, Optical Engineering, 21, 829
  * Guizar-Sicairos & Fienup (2012) Guizar-Sicairos, M. & Fienup, J. R. 2012, J. Opt. Soc. Am. A, 29, 2367
  * Guyon (2018) Guyon, O. 2018, ARA&A, 56, 315
  * Jovanovic et al. (2018) Jovanovic, N., Absil, O., Baudoz, P., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10703, Adaptive Optics Systems VI, ed. L. M. Close, L. Schreiber, & D. Schmidt, 107031U
  * Kingma & Ba (2017) Kingma, D. P. & Ba, J. 2017, Adam: A Method for Stochastic Optimization
  * Krist (2007) Krist, J. E. 2007, in Optical Modeling and Performance Predictions III, ed. M. A. Kahan, Vol. 6675, International Society for Optics and Photonics (SPIE), 250 – 258
  * Lamb et al. (2021) Lamb, M. P., Correia, C., Sivanandam, S., Swanson, R., & Zavyalova, P. 2021, MNRAS, 505, 3347
  * Martinache (2013) Martinache, F. 2013, Publications of the Astronomical Society of the Pacific, 125, 422
  * Mawet et al. (2005) Mawet, D., Riaud, P., Absil, O., & Surdej, J. 2005, ApJ, 633, 1191
  * Mawet et al. (2009) Mawet, D., Serabyn, E., Liewer, K., et al. 2009, The Astrophysical Journal, 709
  * Orban de Xivry et al. (2021) Orban de Xivry, G., Quesnel, M., Vanberg, P. O., Absil, O., & Louppe, G. 2021, MNRAS, 505, 5702
  * Paine & Fienup (2018) Paine, S. W. & Fienup, J. R. 2018, Optics Letters, 43, 1235
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., et al. 2019, in Advances in Neural Information Processing Systems, ed. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett, Vol. 32 (Curran Associates, Inc.), 8026–8037
  * Quesnel et al. (2020) Quesnel, M., Orban de Xivry, G., Louppe, G., & Absil, O. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11448, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 114481G
  * Riaud et al. (2012a) Riaud, P., Mawet, D., & Magette, A. 2012a, A&A, 545, A151
  * Riaud et al. (2012b) Riaud, P., Mawet, D., & Magette, A. 2012b, A&A, 545, A150
  * Ruane et al. (2019) Ruane, G., Mawet, D., Riggs, A. E., & Serabyn, E. 2019, in Techniques and Instrumentation for Detection of Exoplanets IX, ed. S. B. Shaklan, Vol. 11117, International Society for Optics and Photonics (SPIE), 454 – 469
  * Tan & Le (2019) Tan, M. & Le, Q. 2019, in Proceedings of Machine Learning Research, Vol. 97, Proceedings of the 36th International Conference on Machine Learning, ed. K. Chaudhuri & R. Salakhutdinov (PMLR), 6105–6114


[table]capposition=top

# Multi-View Near-field Imaging in NLOS with Non-Reconfigurable EM Skins
††thanks: This work was partially supported by the European Union under the
Italian National Recovery and Resilience Plan (NRRP) of NextGenerationEU,
partnership on “Telecommunications of the Future” (PE00000001 - program
“RESTART”).

Davide Tornielli Bellini, Dario Tagliaferri, Marouan Mizmizi, Stefano
Tebaldini and Umberto Spagnolini
Politecnico di Milano, Milan, Italy
E-mail<EMAIL_ADDRESS>
###### Abstract

This paper deals with radar imaging in non-line of sight (NLOS) with the aid
of non-reconfigurable electromagnetic skins (NR-EMSs). NR-EMSs are passive
metasurfaces whose reflection properties are defined during the manufacturing
process, and represent a low-cost alternative to reconfigurable intelligent
surfaces to implement advanced wave manipulations. We propose and discuss a
multi-view near-field radar imaging system where a moving source progressively
illuminates different portions of the NR-EMS, whereby each portion (module) is
purposely phase-configured to focus the impinging radiation over a desired
NLOS area of interest. The source, e.g., a radar-equipped vehicle, synthesizes
a wide aperture that maps onto the NR-EMS, allowing NLOS imaging with enhanced
resolution compared to the standalone radar capabilities. Simulation results
show the feasibility and benefits of such an imaging approach and shed light
on a possible practical application of metasurfaces for sensing.

###### Index Terms:

Radar imaging, multi-view, NLOS, near-field, EM skins, synthetic aperture

##  I Introduction

Radar sensing technology has witnessed remarkable progress in recent years,
driven by the constant pursuit of improving overall performance. Among the
innovative approaches, the integration of metasurfaces into radar systems has
emerged as a promising frontier. Metasurfaces are planar or conformal
metamaterial surfaces whose EM properties can be can be engineered to
implement advanced wave manipulation (anomalous reflection, focusing, beam
shaping, etc. [1]). When manufactured as reflectarrays, metasurfaces are made
by a very large number of sub-wavelength elements (meta-atoms), whose
scattering phases are designed to accomplish the aforementioned
functionalities. If the phase across the metasurface is time-varying, we refer
to reconfigurable intelligent surfaces (RISs), and their application to
communication and localization systems has been the focus of the vast majority
of research works. In [2], for instance, the authors propose to use a RIS to
assist radar in non-line-of-sight (NLOS) conditions. The work [3], instead,
uses a RIS for NLOS radar localization of targets. The authors of [4] analyze
the fundamental position and orientation error bounds on a RIS-aided
localization system, proposing a suitable phase design approach. Recently, the
research interest shifted towards large metasurfaces (whose size is comparable
with the propagation distance), to exploit the additional degrees of freedom
brought by near-field operation, namely the propagation condition in which the
wavefront across the metasurface is non-planar [5]. Examples of works on RIS-
aided near-field localization are in [6, 7].

Differently from localization, whose aim is to infer meaningful parameters of
a set of desired targets (e.g., position, orientation, speed, etc.), imaging
refers to the generation of a reflectivity map of the environment, from which
to infer the presence of targets via detection, prior to localization. Imaging
performance is not assessed in terms of estimation accuracy but rather
resolution, i.e., the capability of distinguishing two closely spaced targets.
Currently, very few works address the usage of RISs (and metasurfaces in
general) for imaging (e.g., see [8]), with specific use cases mostly limited
to biomedical imaging [9]. Other relevant works are [10, 11, 12]. The authors
of [10] address the imaging problem as an inverse scattering reconstruction in
far-field, considering distributed antenna systems. Work [11], instead,
addresses the near-field by employing the RIS as an active reflector,
amplifying and reflecting the impinging signals from a large holographic
aperture. Recently, the authors of [12] tackled the imaging with RISs in both
LOS and NLOS scenarios, designing the illumination pattern and the RIS
coefficients to minimize the reconstruction error of the reflectivity map in a
desired area of interest.

While the aforementioned works address the usage of RISs, whose phase pattern
can be dynamically changed in time according to special needs, relevant
research has been produced on non-reconfigurable metasurfaces, also called
non-reconfigurable EM skins (NR-EMSs), whose wave manipulation capabilities
are defined in the design process [13, 14]. NR-EMSs offer limited flexibility
compared to RISs, but they come with orders-of-magnitude lower manufacturing
costs, allowing potential mass-production of large devices [13, 14]. NR-EMS
with conformal design have been targeted in our previous works for vehicular
communication systems [15], while few others addressed the potential of SP-EMS
for localization (see [16, 17]).

Contribution: This paper delves in the usage of NR-EMSs for radar imaging in
NLOS in a way that multiples modules generates an artificial sweeping useful
for imaging. The selected use case is the high-resolution environment mapping
for autonomous or assisted driving, where a radar-equipped traveling vehicle
exploits an NR-EMS purposely deployed on a building to illuminate a desired
area in NLOS (”see around the corner”), and gathers the echoes undergoing
double bounce with the NR-EMS. The peculiarity of such a system is that the
vehicle, while moving, progressively illuminates different portions of the
metasurface, namely modules, each suitably designed to focus the impinging
radiation on a desired spot in NLOS. The vehicle generates a synthetic
aperture that maps into a multi-view imaging system through the NR-EMS. The
imaging resolution is dictated by the equivalent aperture on the NR-EMS,
rather than the radar physical aperture. As the former can be very large, the
proposed system allows high-resolution near-field imaging with progressive
low-resolution far-field acquisitions. We discuss the peculiar system design,
outlining the related requirements. Simulation results confirm the feasibility
of our imaging system and shed light on a promising usage of NR-EMSs for
sensing.

Organization: The paper is organized as follows: Sect. II describes the role
of NR-EMSs in multi-view radar imaging, Sect. III outlines the system model,
Sect. IV details the design of the system and related trade-offs while Sect. V
reports the image synthesis technique and the simulation results. Finally,
Sect. VI concludes the paper.

Notation: The following notation is adopted in the paper: bold upper- and
lower-case letters describe matrices and column vectors. Vector norm is
denoted with $\|\mathbf{a}\|$. Vector transposition is $\mathbf{a}^{T}$. With
$a\sim\mathcal{CN}(\mu,\sigma^{2})$ we denote a multi-variate circularly
complex Gaussian random variable $a$ with mean $\mu$ and variance
$\sigma^{2}$. $\mathbb{E}[\cdot]$ is the expectation operator, while
$\mathbb{R}$, $\mathbb{C}$ stand for the set of real and complex numbers,
respectively. $\delta_{n}$ is the Kronecker delta function.

##  II NR-EMSs in Multi-view Radar Systems

Let us consider the 2D scenario depicted in Fig. 1a, where a source equipped
with a radar, located in $\mathbf{s}=[s_{x},s_{y}]^{T}$, aims at creating an
image of an area of interest where a target is located in
$\mathbf{r}=[r_{x},\,r_{y}]^{T}$. We assume that the area where the target is
present is in NLOS, thus the image of the environment is enabled by the
presence of a reflection plane deployed along $x$. The source is equipped with
a radar with aperture $A$, emitting a signal along direction $\psi$ (pointing
towards the reflection plane) over a narrow beamwidth $\Delta\psi\propto
A^{-1}$. If the reflection plane is a mirror, the target is illuminated under
specular reflection, i.e., for $\theta_{i}=\theta_{o}$, where
$\theta_{i}=\pi-\psi$ and $\theta_{o}$ are the incidence and reflection angles
onto the reflection plane. The point $\mathbf{p}$ of specular reflection that
allows imaging the target satisfies $s_{y}/D_{i}=r_{y}/D_{o}$ where
$D_{i}=\|\mathbf{p}-\mathbf{s}\|$, $D_{o}=\|\mathbf{r}-\mathbf{p}\|$ denote
the source-plane and plane-target distances, respectively. The resolution of
the corresponding image of the target $\mathbf{r}$ is ruled by the radar’s
physical aperture $A$, thus reducing the beamwidth $\Delta\psi\propto A^{-1}$
yields better resolution.

(a)

(b)

Figure 1: NLOS imaging with (a) reflection plane (mirror) and fixed source
position, (b) NR-EMS and moving source.

Since increasing the radar’s physical aperture $A$ leads to a non-negligible
additional cost for the hardware platform, we opt for a different solution to
enhance the imaging resolution. Let us consider the case in which the radar
moves along $x$ with velocity $v$ (see Fig. 1b). The radar’s position in space
is now time-varying, $\mathbf{s}(\tau)$, where $\tau$ denotes the slow time.
We assume to fix the pointing angle $\psi$ onto the reflection plane, thus the
incident angle $\theta_{i}$ is fixed accordingly and the corresponding point
on the plane is denoted by
$\mathbf{p}(\tau)=[s_{x}(\tau)+D_{i}\cos\psi,0]^{T}$. We have a multi-view
radar system if the reflection plane behaves like an anomalous mirror for the
impinging radiation, namely it changes the reflection angle such that the
signal emitted by the radar along its motion $\mathbf{s}(\tau)$,
$\tau=0,...,T$ is focused on the target in $\mathbf{r}$. The reflection angle
at time $\tau$ shall be set as follows:

$\theta_{o}(\tau)=\arccos\left(\frac{r_{y}}{\|\mathbf{r}-\mathbf{p}(\tau)\|}\right)$
(1)

and the dependence on time can be related to a specific position onto the
anomalous mirror through $\mathbf{p}(\tau)$. The system is depicted in Fig.
1b. Now, with the aid of the anomalous mirror, the moving radar progressively
increases the spatial diversity in the observation of the target, implement a
synthetic aperture that enhances both the resolution of the image of
$\mathbf{r}$ as well as the SNR. Indeed, the image resolution is now ruled by
the effective aperture onto the anomalous mirror:

$A^{\prime}\simeq vT+\frac{D_{i}\Delta\psi}{\sin\psi}\gg A,$ (2)

typically much larger that the aperture provided by the radar [18]. To cater
to arbitrary geometries, we opt for the usage of a NR-EMS as the required
anomalous mirror deployed on the reflection plane, whose design is detailed in
the following.

##  III System Model

The system model stems from Section II. The radar-equipped source moves along
$x$, taking positions located in
$\mathbf{s}(\tau)=[s_{x}(0)+v\tau,s_{y}]^{T}$, along the slow time interval
$\tau\in[0,T]$. The radar periodically emits the pass-band signal

$h(t)=g(t)e^{j2\pi f_{0}t}$ (3)

where $t$ is the fast time, $g(t)$ is the base-band range-compressed pulse of
bandwidth $B$, centered around carrier frequency $f_{0}$. Tx signal is emitted
every pulse repetition interval $\Delta\tau$, that is herein neglected for the
seek of simplicity in the notation. The radar is characterized by a linear
aperture $A$ along $y$ (i.e., orthogonal to the direction of motion), that
allows beamforming the pass-band signal (3) towards direction $\psi$. The
beamwidth on the azimuth plane (i.e., $xy$ plane) is
$\Delta\psi\simeq\lambda_{0}/(A\cos\psi)$, where $\lambda_{0}=c/f_{0}$ is the
carrier wavelength. The anomalous reflection is by a planar NR-EMS, made by
$N$ $\lambda_{0}/4$-spaced elements along the $x$ axis , whose center is
located in the origin of the global coordinate system111The 2D system model
outlined in this paper is instrumental to describe the NR-EMS-aided radar
imaging system design on the azimuth plane. In practice, planar NR-EMSs
deployed on the $xz$ plane are required to guarantee a sufficient SNR. The 3D
modeling would complicate the exposition and it is therefore not treated
herein, but its development follows from the present with due adaptations. .

At time instant $\tau$ the radar illuminates a portion of the NR-EMS made by

$N_{rad}\approx\frac{D_{i}\,\Delta\psi}{d\sin\psi}<N$ (4)

elements, for $n=n_{0}(\tau)-N_{rad}/2,...,n_{0}(\tau)+N_{rad}/2-1$, where
$n_{0}(\tau)$ is the element index corresponding at the maximum pointing at
time $\tau$, whose location in global coordinates is $\mathbf{x}_{0}(\tau)$.
The general model of the received signal at the radar in $\mathbf{s}(\tau)$
due to scattering from target $\mathbf{r}$ through a double reflection from
the NR-EMS is reported in (5).

$\begin{split}y(t,\tau)=\sum_{n=n_{0}(\tau)-\frac{N_{rad}}{2}}^{n_{0}(\tau)+\frac{N_{rad}}{2}-1}\sum_{n^{\prime}=n_{0}(\tau)-\frac{N_{rad}}{2}}^{n_{0}(\tau)+\frac{N_{rad}}{2}-1}\rho_{nn^{\prime}}(\tau)\,e^{j\phi_{n}}e^{j\phi_{n^{\prime}}}\,g(t-\Delta_{nn^{\prime}}(\tau))\,e^{-j2\pi
f_{0}\Delta_{nn^{\prime}}(\tau)}+w(t,\tau)\end{split}$ (5)

$\begin{split}y(t,\tau)\overset{(a,b)}{\simeq}\rho(\tau)\,g\left(t-\frac{2\left[D_{i}+D_{o}(\tau)\right]}{c}\right)&e^{-j\frac{4\pi}{\lambda_{0}}\left[D_{i}+D_{o}(\tau)\right]}\times\\\
&\sum_{n=n_{0}(\tau)-\frac{N_{rad}}{2}}^{n_{0}(\tau)+\frac{N_{rad}}{2}-1}\sum_{n^{\prime}=n_{0}(\tau)-\frac{N_{rad}}{2}}^{n_{0}(\tau)+\frac{N_{rad}}{2}-1}e^{j\phi_{n}}e^{j\phi_{n^{\prime}}}e^{-j\frac{2\pi
d}{\lambda_{0}}(n+n^{\prime})\left[\sin\theta_{i}-\sin\theta_{o}(\tau)\right]}+w(t,\tau)\end{split}$
(6)

The two-way propagation delay between the source $\mathbf{s}(\tau)$ and target
$\mathbf{r}$ is

$\Delta_{nn^{\prime}}(\tau)=\frac{D_{\mathbf{s}n}(\tau)+D_{n\mathbf{r}}+D_{\mathbf{r}n^{\prime}}+D_{n^{\prime}\mathbf{s}}(\tau)}{c}$
(7)

accounting for propagation as forward scattering from the $n$th element and
backward scattering from the $n^{\prime}$th, $n^{\prime}\neq n$ in general,
$D_{\mathbf{s}n}(\tau)=\|\mathbf{x}_{n}-\mathbf{s}(\tau)\|$ being the distance
between $\mathbf{s}(\tau)$ and the $n$th element in $\mathbf{x}_{n}$ (and
similar definition for other terms), while $\phi_{n}$ is the NR-EMS phase at
the $n$th element. The back-scattering power, whose expression is not reported
for brevity, is inversely proportional to the product of the four travelled
distances and proportional to the directivity of the radar, thus
$|\rho_{nn^{\prime}}(\tau)|^{2}\propto
D^{-2}_{\mathbf{s}n}(\tau)D^{-2}_{n\mathbf{r}}D^{-2}_{\mathbf{r}n^{\prime}}D^{-2}_{n^{\prime}\mathbf{s}}(\tau)\Delta\psi^{-4}$.
Term $w(t,\tau)\in\mathcal{CN}(0,\sigma_{w}^{2})$ is the additive white
Gaussian noise corrupting the signal. Model (5) describes a generic near-field
and spatially wideband system, where the wavefront across the illuminated NR-
EMS portion is curved and $\Delta_{nn^{\prime}}(\tau)>1/B$ in general.

###  III-A Simplified Narrow-Beam Model

The usage of a comparatively narrow beam $\Delta\psi$ at the radar allows
approximating the model of the Rx signal, yielding some useful
simplifications. We have: $(a)$ far-field when the wavefront across the
portion of illuminated metasurface is approximated as planar and $(b)$
spatially narrowband operation when the residual delays across the portion of
illuminated metasurface do not practically affect the envelope of the Rx base-
band pulse. Far-field implies

$\begin{split}\Delta_{nn^{\prime}}(\tau)\simeq&\frac{2D_{i}}{c}+\frac{2D_{o}(\tau)}{c}+(n+n^{\prime})\frac{d}{c}[\sin\theta_{i}-\sin\theta_{o}(\tau)]\end{split}$
(8)

where $D_{i}=\|\mathbf{x}_{0}(\tau)-\mathbf{s}(\tau)\|$ and
$D_{o}(\tau)=\|\mathbf{r}-\mathbf{x}_{0}(\tau)\|$, and
$|\rho(\tau)|^{2}\propto D^{-4}_{i}D^{-4}_{o}(\tau)\Delta\psi^{-4}$. Spatially
narrowband operation holds when

$\frac{1}{B}\gg
N_{rad}\frac{d}{c}\cos\psi=\frac{D_{i}}{c}\Delta\psi\cot\psi,\,\,\,\,\psi\in(0,\pi/2]$
(9)

and it yields to this delayed echo:

$g(t-\Delta_{nn^{\prime}}(\tau))\simeq
g\left(t-\frac{2\left[D_{i}+D_{o}(\tau)\right]}{c}\right).$ (10)

The signal model under approximations $(a)$ and $(b)$ is reported in (6).
While the far-field condition usually applies for beams $\Delta\psi$ in the
order of some degrees and distances of few to tens of meters (as in the
considered use case), the spatially narrowband operation is strongly dependent
on the employed bandwidth. For instance, for $\psi=30$ deg, $D_{i}=10$ m, we
obtain that $B<100$ MHz for $\Delta\psi=10$ deg, and $B<1$ GHz for
$\Delta\psi=1$ deg, the latter being the current limit for off-the-shelf
automotive radars in terms of angular resolution [19].

##  IV System Design

The design of the system has the ultimate goal of enabling high-resolution
imaging in NLOS conditions. If the radar employs a very large beamwidth,
illuminating the whole NR-EMS, the NR-EMS phase profile is optimal only for a
single position of the source and the target, as double focusing is required:

$\phi_{n}=\frac{2\pi}{\lambda_{0}}\left(\big{\|}\mathbf{x}_{n}-\mathbf{s}\big{\|}+\big{\|}\mathbf{r}-\mathbf{x}_{n}\big{\|}\right).$
(11)

Thus, the coverage is limited to a very small region in space, unless RISs are
employed. In the following, we detail the system design using NR-EMSs under
the narrow-beam approximation in Sect. III-A.

###  IV-A Modular NR-EMS

The NR-EMS shall fulfill two goals, namely (i) increasing the effective
aperture of the radar, thus the resolution of the image, and (ii) ensuring the
full coverage of the NLOS area, i.e., any target shall be detectable within
NLOS region. In this setting, the NR-EMS can be designed by composition of
multiple equally sized modules, each made by $N^{\prime}\ll N$ elements (along
$x$) and pre-configured with a linear phase profile on each. The latter eases
the design and manufacturing of each module, and the optimal non-linear phase
profile across the NR-EMS (for near-field focusing) is obtained by suitable
configuration of multiple adjacent modules. Let us consider $L$ NR-EMS modules
of $N^{\prime}$ elements each. In order to enable the coherent combination of
echoes from a target located in $\mathbf{r}$, (i) the source must implement a
synthetic aperture of duration $T\geq\frac{(LN^{\prime}-N_{rad})d}{v}$
(assuming a narrow beamwidth, i.e., $N_{rad}<<LN^{\prime}$) and (ii) the $L$
modules must focus the impinging radiation from the location
$\mathbf{s}(\tau)$ of the moving source to the location of the target,
yielding a phase profile

$\phi_{n,\ell}=\frac{2\pi
d}{\lambda_{0}}n\left[\sin\left(\pi-\psi\right)-\sin\left(\arccos{\left(\frac{r_{y}}{\|\mathbf{r}-\mathbf{x}_{0,\ell}\|}\right)}\right)\right]$
(12)

for $n=n_{0,\ell}-N^{\prime}/2,...,n_{0,\ell}+N^{\prime}/2-1$,
$\ell=-L/2,...,L/2-1$, where $\mathbf{x}_{0,\ell}$ is the phase center of the
$\ell$th module—identified by element $n_{0,\ell}$—illuminated by the moving
source at time $\tau=\ell(N^{\prime}d/v)$. With this configuration, the
effective aperture of the radar system is (2), approximately equal to
$A^{\prime}\simeq LN^{\prime}d$. Notice that this system allows near-field
sensing by employing multiple NR-EMS modules configured for far-field,
progressively approximating the curved wavefront with local planar ones.

Figure 2: Modular NR-EMS for imaging in NLOS. The NR-EMS is made of multiple
modules, organized in clusters. Each cluster focus the impinging radiation on
a fixed anchor point, and each module implements a linear phase gradient.

With $L$ modules configured as (12), the impinging radiation from the radar is
focused on a single point $\mathbf{r}$ in space, thus the system has enhanced
resolution but not enough coverage. For the coverage of the entire NLOS area,
it is necessary to consider multiple clusters of $L$ modules, properly
cascaded along $x$, each configured to focus on a different anchor point
$\mathbf{r}^{*}$. By having $K$ clusters, a reasonable choice to allow the
high-resolution imaging of the environment is to select the anchor points
$\\{\mathbf{r}_{k}^{*}\\}_{k=1}^{K}$ over the iso-range curve at distance $R$
from the center of the NR-EMS. The result is depicted in Fig. 2. The phase
gradient at the $\ell$th module of the $k$th cluster is obtained by (12)
plugging $\mathbf{r}=\mathbf{r}^{*}_{k}$. The complete image of the NLOS area
is synthesized by illuminating all the $K$ clusters through a synthetic
aperture of duration $T\simeq KLN^{\prime}/v$.

###  IV-B Selection of the radar beamwidth

The choice of the radar beamwidth $\Delta\psi$ is a degree of freedom for the
proposed system. Employing a wide radar beam, such that $N_{\mathrm{rad}}\gg
N^{\prime}$, allows illuminating multiple modules at the same time instant
$\tau$. Once the NR-EMS is deployed, a wide radar beamwidth relaxes the
hardware requirements (and the cost) of the device, but it yields a lower SNR
at the target’s location compared to a narrow beamwidth. Possibly, if
$N_{\mathrm{rad}}\simeq N$ (all the metasurface is illuminated in one shot),
there is no need of a synthetic aperture. However, the SNR at the target’s
location diminishes by widening the radar beamwidth $\Delta\psi$, as can be
devised by inspection of the CRB on bistatic range estimation (see [20] for
further details and the analytical derivations)222The bistatic range
considered in the CRB is the distance between the source (at the center of its
synthetic aperture) to the phase center of the metasurface, plus the distance
with the target.. It can be shown that the relative SNR loss due to beam
widening amounts to
$\left(\frac{\Delta\psi_{\mathrm{wide}}}{\Delta\psi_{\mathrm{narrow}}}\right)^{3}$,
pushing for the usage of the phased-array radars with the narrowest possible
beamwidth compatible with the hardware availability.

Besides the radar beamwidth, for $N_{\mathrm{rad}}<N$ it is necessary that the
radar implements a further synthetic aperture to illuminate all the
metasurface (all the $K$ clusters), to form the image without spatial
ambiguities (ghost targets). Those arise from partial illumination of multiple
clusters that focus the radiation over different anchor points. If no
synthetic aperture is implemented (thus no coherent combination of echoes from
all clusters), it is possible that ghost targets appear in the final image.
Differently, by progressively illuminating the whole metasurface, ghost
targets disappear.

##  V Image Synthesis and Simulation Results

###  V-A Image Synthesis

The image is synthesized by back-projection (BP) in time, under narrow
beamwidth assumption (6). The BP technique for image synthesis is based on the
wavefield migration integral extensively used in remote sensing to find the
location of one or more targets [21]. It describes the spatio-temporal matched
filtering (correlation) between the Rx signal and the model signal that would
be received by back-scattering from a target in a given position in space
(typically chosen over a pre-defined grid). The complex image value for a
point $\mathbf{r}$ in the NLOS area can be computed as follows [21]333Image
(13) can be also cast as an operation that provides the likelihood of the
targets’ positions in space, from which it follows their estimation by maximum
likelihood approaches as described in [22].:

$\begin{split}I(\mathbf{r})\overset{(a,b)}{\simeq}\sum_{\tau}y\left(t=\frac{2[D_{i}+D_{o}(\mathbf{r},\tau)]}{c},\tau\right)e^{j\frac{4\pi}{\lambda_{0}}[D_{i}+D_{o}(\mathbf{r},\tau)]}\end{split}$
(13)

where $D_{i}=\|\mathbf{x}_{0}(\tau)-\mathbf{s}(\tau)\|=s_{y}/\sin\psi$ is the
forward path toward the metasurface (that is constant over $\tau$) while
$D_{o}(\mathbf{r},\tau)=\|\mathbf{r}-\mathbf{x}_{0}(\tau)\|$ is the distance
between the illuminated point at time $\tau$, i.e., $\mathbf{x}_{0}(\tau)$ and
point $\mathbf{r}$. BP for narrow beamwidths and spatially wideband operation
(over single slow-time snapshots) requires the relative position w.r.t.
metasurface ($s_{y}$) as well as the pointing angle $\psi$.

###  V-B Simulation results

Figure 3: Sketch of the simulation environment with moving radar. Figure 4: 2D
map of the SNR (in dB) in the NLOS area (image not in scale). The SNR is $>0$
dB in 98% of the area. Figure 5: Focused image in the considered NLOS area,
showing few point targets. The inset is a zoom on a selected target (images
not in scale).

The system setup is skecthed in Fig. 3. The simulation parameters are: carrier
frequency $f_{0}=77$ GHz, employed bandwidth $B=1$ GHz, radar beamwidth along
the azimuth plane ($xy$) $\Delta\psi=1$ deg, Tx power $-10$ dBm. The NLOS area
to be monitored is $10\times 40$ m2. The NR-EMS is made by $K=4$ clusters of
$L=5$ modules, each of size $0.1$ m along $x$ (horizontal) and $0.5$ m along
$z$ (vertical). We assume that the radar’s physical aperture is 1D, as for a
uniform linear array, thus the elevation beamwidth is large444As for typical
automotive radars, achieving high azimuth resolution and no (or poor)
elevation resolution [19].. In this latter setting, we exploit the vertical
direction on the reflection plane to deploy additional NR-EMS clusters (up to
12) to increase the number of anchor points on the NLOS covered area. The
results are discussed in the following.

Fig. 4 reports the achieved SNR map (in dB scale) in the NLOS area. The 12
anchor points are selected to be deployed at range distances of $15,25,35$ m
from the phase center of the NR-EMS. The synthetic aperture is $2$ m such that
to illuminate all the NR-EMS clusters. The SNR map is built by deploying
regularly spaced fictitious targets with an RCS of $0.1$ m2 and evaluate the
SNR after image formation. In the considered settings, the proposed system is
able to achieve an SNR above 0 dB in $\simeq 98\%$ of the NLOS area, allowing
the detection of targets notwithstanding the severe path-loss induced by the
double reflection on the NR-EMS. It is worth remarking that such an SNR can be
attained with very large metasurfaces, that can be practically manufactured
only as NR-EMS.

The second remarkable result, reported in Fig. 5, shows instead a focused
radar image of a few randomly deployed point targets in the NLOS area, each
with an RCS of $0.5$ m2. The image is normalized to the maximum value and
reported in dB scale. Remarkably, we have that (i) both targets in near and
far range are detectable (as the SNR after image formation is comparably
large) and (ii) each point target is imaged with a resolution of few to tens
of centimeters, with little variation in space. The latter result should be
compared with the radar’s resolution capabilities without the NR-EMS focusing
effect. For instance, for the target in the inset of Fig. 5, located at approx
$D_{o}=20$ m from the NR-EMS and $D_{i}=10$ m from the source, the native
cross-range resolution of the radar (approximately equivalent to the
resolution along $x$) would be $\delta x\simeq(D_{i}+D_{o})\Delta\psi=50$ cm,
while drops to $\delta x\simeq 10$ cm thanks to the NR-EMS. The range
resolution (circa $\delta y$) is instead ruled by the employed bandwidth for
the considered point $\delta y\simeq c/2B=15$ cm, with a slight improvement
when using the NR-EMS thanks to the near-field effect. The results show the
feasibility of the proposed imaging system with NR-EMS, and shed light on a
promising usage of passive metasurfaces for NLOS sensing.

##  VI Conclusions

This work proposes and discusses a novel radar system exploiting a modular
low-cost NR-EMS for NLOS imaging of a desired area in the environment, with
application to automotive and road safety. The proposed system leverages the
vehicle’s motion to synthesize an aperture on the NR-EMS that allows for
increasing the physical radar’s aperture, enabling high-resolution imaging in
NLOS over a desired area of interest. Design peculiarities are discussed and
simulation results are presented to validate the idea. Next investigations
will involve generalization to integrated communication and sensing over NR-
EMS, and further simulations for more realistic settings with possibly
dedicated experimentation.

## References

  * [1] N. Yu, P. Genevet, M. A. Kats, F. Aieta, J.-P. Tetienne, F. Capasso, and Z. Gaburro, “Light propagation with phase discontinuities: Generalized laws of reflection and refraction,” _Science_ , vol. 334, no. 6054, pp. 333–337, 2011. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.1210713
  * [2] S. Buzzi, E. Grossi, M. Lops, and L. Venturino, “Foundations of mimo radar detection aided by reconfigurable intelligent surfaces,” _IEEE Transactions on Signal Processing_ , vol. 70, pp. 1749–1763, 2022.
  * [3] A. Aubry, A. De Maio, and M. Rosamilia, “Ris-aided radar sensing in n-los environment,” in _2021 IEEE 8th International Workshop on Metrology for AeroSpace (MetroAeroSpace)_ , 2021, pp. 277–282.
  * [4] A. Elzanaty, A. Guerra, F. Guidi, and M.-S. Alouini, “Reconfigurable intelligent surfaces for localization: Position and orientation error bounds,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 5386–5402, 2021.
  * [5] Z. Wang, Z. Liu, Y. Shen, A. Conti, and M. Z. Win, “Source localization with intelligent surfaces,” in _ICC 2022 - IEEE International Conference on Communications_ , 2022, pp. 895–900.
  * [6] D. Dardari, N. Decarli, A. Guerra, and F. Guidi, “Los/nlos near-field localization with a large reconfigurable intelligent surface,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 6, pp. 4282–4294, 2022\.
  * [7] M. Luan, B. Wang, Y. Zhao, Z. Feng, and F. Hu, “Phase design and near-field target localization for ris-assisted regional localization system,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 2, pp. 1766–1777, 2022.
  * [8] G. Lerosey, J. de Rosny, A. Tourin, A. Derode, G. Montaldo, and M. Fink, “Time reversal of electromagnetic waves,” _Phys. Rev. Lett._ , vol. 92, p. 193904, May 2004. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.92.193904
  * [9] N. Ghavami, E. Razzicchia, O. Karadima, P. Lu, W. Guo, I. Sotiriou, E. Kallos, G. Palikaras, and P. Kosmas, “The use of metasurfaces to enhance microwave imaging: Experimental validation for tomographic and radar-based algorithms,” _IEEE Open Journal of Antennas and Propagation_ , vol. 3, pp. 89–100, 2022.
  * [10] Y. Tao and Z. Zhang, “Distributed computational imaging with reconfigurable intelligent surface,” in _2020 International Conference on Wireless Communications and Signal Processing (WCSP)_ , 2020, pp. 448–454.
  * [11] Y. Jiang, F. Gao, Y. Liu, S. Jin, and T. Cui, “Near field computational imaging with ris generated virtual masks,” 2023.
  * [12] G. Torcolacci, A. Guerra, H. Zhang, F. Guidi, Q. Yang, Y. C. Eldar, and D. Dardari, “Holographic imaging with xl-mimo and ris: Illumination and reflection design,” 2023.
  * [13] G. Oliveri, F. Zardi, P. Rocca, M. Salucci, and A. Massa, “Constrained design of passive static em skins,” _IEEE Transactions on Antennas and Propagation_ , vol. 71, no. 2, pp. 1528–1538, 2023.
  * [14] G. Oliveri, P. Rocca, M. Salucci, and A. Massa, “Holographic smart em skins for advanced beam power shaping in next generation wireless environments,” _IEEE Journal on Multiscale and Multiphysics Computational Techniques_ , vol. 6, pp. 171–182, 2021.
  * [15] M. Mizmizi, R. A. Ayoubi, D. Tagliaferri, K. Dong, G. G. Gentili, and U. Spagnolini, “Conformal metasurfaces: a novel solution for vehicular communications,” _IEEE Transactions on Wireless Communications_ , pp. 1–1, 2022.
  * [16] D. Tagliaferri, M. Mizmizi, G. Oliveri, U. Spagnolini, and A. Massa, “Reconfigurable and static em skins on vehicles for localization,” 2023.
  * [17] M. Lotti and D. Dardari, “Metaprism-aided nlos target localization,” in _2023 31st European Signal Processing Conference (EUSIPCO)_ , 2023, pp. 895–899.
  * [18] G. Lerosey and M. Fink, “Wavefront shaping for wireless communications in complex media: From time reversal to reconfigurable intelligent surfaces,” _Proceedings of the IEEE_ , vol. 110, no. 9, pp. 1210–1226, 2022.
  * [19] T. Instruments, “Imaging radar using cascaded mmwave sensor reference design,” available at: https://www.ti.com/lit/ug/tiduen5a/tiduen5a.pdf.
  * [20] A. Liu, Z. Huang, M. Li, Y. Wan, W. Li, T. X. Han, C. Liu, R. Du, D. K. P. Tan, J. Lu, Y. Shen, F. Colone, and K. Chetty, “A survey on fundamental limits of integrated sensing and communication,” _IEEE Communications Surveys & Tutorials_, vol. 24, no. 2, pp. 994–1034, 2022.
  * [21] S. Tebaldini, M. Manzoni, D. Tagliaferri, M. Rizzi, A. V. Monti-Guarnieri, C. M. Prati, U. Spagnolini, M. Nicoli, I. Russo, and C. Mazzucco, “Sensing the urban environment by automotive sar imaging: Potentials and challenges,” _Remote Sensing_ , vol. 14, no. 15, 2022. [Online]. Available: https://www.mdpi.com/2072-4292/14/15/3602
  * [22] J. Gunther, R. West, N. Crookston, and T. Moon, “Maximum likelihood synthetic aperture radar image formation for highly nonlinear flight tracks,” in _2011 Digital Signal Processing and Signal Processing Education Meeting (DSP/SPE)_ , 2011, pp. 449–454.


# Exploring Driving-aware Salient Object Detection via Knowledge Transfer

###### Abstract

Recently, general salient object detection (SOD) has made great progress with
the rapid development of deep neural networks. However, task-aware SOD has
hardly been studied due to the lack of task-specific datasets. In this paper,
we construct a driving task-oriented dataset where pixel-level masks of
salient objects have been annotated. Comparing with general SOD datasets, we
find that the cross-domain knowledge difference and task-specific scene gap
are two main challenges to focus the salient objects when driving. Inspired by
these findings, we proposed a baseline model for the driving task-aware SOD
via a knowledge transfer convolutional neural network. In this network, we
construct an attention-based knowledge transfer module to make up the
knowledge difference. In addition, an efficient boundary-aware feature
decoding module is introduced to perform fine feature decoding for objects in
the complex task-specific scenes. The whole network integrates the knowledge
transfer and feature decoding modules in a progressive manner. Experiments
show that the proposed dataset is very challenging, and the proposed method
outperforms 12 state-of-the-art methods on the dataset, which facilitates the
development of task-aware SOD.

Index Terms—  Salient object detection, driving-aware saliency, knowledge
transfer

††footnotetext: * Correspondence should be addressed to Changqun Xia and Jia
Li. The dataset, code and models are available on http://cvteam.net.

##  1 Introduction

Image-base salient object detection (SOD) aims to detect and segment objects
that capture human visual attention, which is a preliminary step for
subsequent vision tasks such as object recognition and tracking. Over the past
decades, many benchmark datasets [1, 2, 3] have been constructed for the study
of SOD. These datasets include natural images with simple and complex scenes,
which usually don’t focus on specific tasks. Such datasets can be referred to
as general SOD datasets (or conventional SOD datasets). Based on these
datasets, many SOD methods [4, 5, 6, 7] have been proposed and achieve great
performance. Correspondingly, this kind of task is called general SOD (or
conventional SOD). However, in real world, what we usually need is to deal
with specific application scenarios (_e.g._ , automatic driving and medical
image processing). For these scenarios, the analysis of salient objects plays
an important role for assisting subsequent high-level visual tasks and
understanding these scenes. As a distinction, this kind of task can be named
as task-aware SOD.

Fig. 1: Comparisons of the conventional and task-specific SOD dataset. Images
and ground-truth masks of the (a) conventional dataset and (b) task-specific
dataset are from DUT-OMRON [2] and CitySaliency, respectively.

Recently some task-specific datasets have been collected. These datasets
contain images of specific tasks, such as driving, webpage-related behavior
and game. For example, Alletto et al. [8] proposed a publicly available actual
driving dataset DR(eye)VE with task-specific fixation maps. Based on these
datasets, task-aware fixation prediction (FP) has made progress. However,
these still lack task-specific datasets for SOD, which handers the development
of task-aware SOD.

Toward this end, we propose a new driving task-oriented dataset (denoted as
CitySaliency), which contains 3,475 images with pixel-level annotation based
on the common driving task. Some representative examples are shown in the
task-specific dataset of Fig. 1. In constructing this dataset, we first
collect these images from the driving scene. Given these images, we ask 25
volunteers to view these images by simulating

driving task to collect eye-tracking data and human fixations. Based on these
datas, we compute the saliency value of each object with the help of semantic
annotations to determine ground-truth masks of salient objects. Finally, the
images and masks together form the task-specific dataset. Comparing
CitySaliency with conventional datasets, we find there exist two main
challenges: 1) cross-domain knowledge difference. As shown in Fig. 1, cars and
persons in conventional datasets are the focus, while more attention in
CitySaliency is paid to drivable roads while ignoring vehicles (first two
rows) and pedestrians (last two rows of Fig. 1) on both sides of roads. The
knowledge difference about different task-related targets brings resistance to
SOD. 2) task-specific scene gap. Based on the statistical analysis, we find
that CitySaliency has three main characteristics including more discrete
distribution, more salient objects and more area occupation, which reflects
the complexity of scenes in CitySaliency. These two problems make it difficult
to deal with task-aware SOD.

To address these problems, we propose a baseline model for the driving task-
aware SOD via a knowledge transfer convolutional neural network. In the
network, we put forward an attention-based knowledge transfer module, which is
used to transfer the knowledge from general domain to task domain to make up
the knowledge difference, assisting the network to find the salient objects.
In addition, we introduce an efficient boundary-aware feature decoding module
to fine decode features for lots of existing salient objects with structures
in complex task-specific scenes. The two modules are integrated in a
progressive manner to deal with the task-aware SOD. Experiments show the
proposed method outperforms 12 state-of-the-art methods on CitySaliency, which
demonstrates the effectiveness of the proposed knowledge transfer network.

The main contributions of this paper include: 1) we propose a new task-
specific SOD dataset, which can be used to boost the development of task-aware
SOD, 2) we propose a baseline model to deal with the difficulties in the
proposed dataset and it consistently outperforms 12 state-of-the-art
algorithms, 3) we provide a comprehensive benchmark of state-of-the-art
methods and the proposed method on CitySaliency, which reveals the key
challenges of the proposed dataset, and validates the usefulness of the
proposed method.

##  2 A New Dataset for Task-aware SOD

We propose a new driving task-oriented dataset (named as CitySaliency) for
task-aware SOD, which focuses on the common urban driving task. In this
section, we will introduce the details in constructing the dataset.

###  2.1 Data Collection

The proposed dataset is built on Cityscapes [9], which is a complex urban
driving dataset for instance-level semantic labeling. Cityscapes consists of
5,000 images with pixel-level semantic annotation (as shown in Fig. 2(a)(b)),
which is split into 2,975 image for training, 500 image for validating and
1,525 for testing (note the annotation of testing set is withheld for official
benchmarking purposes). In constructing CitySaliency, we combine the training
and validation sets (3,475 images) as a raw data source.

###  2.2 Psychophysical Experiments

To annotate salient objects in complex urban driving scenes, there exists a
tough problem: there may exist several candidate objects while different
annotator may have different biases in determining which objects are salient.
To alleviate this influence, we conduct psychophysical experiments to collect
human fixations as auxiliary information, as done in [10].

In the psychophysical experiments, 25 subjects with normal vision and never
seeing collected images participate in. During the experiment, we show the
images on a 22-inch display screen with a resolution of 1680x1050 and install
eye-tracking apparatus (SMI RED 500) with a sampling rate of 500HZ to record
various eye movements including fixation, saccade and blink. After the data
collection, various information of human eye movement is recorded. Based on
the fixation data, we can compute the fixation density map for each image to
annotate the ground-truth salient regions as in [10]. The generated fixation
ground truth as displayed in Fig. 2(c).

Fig. 2: Process of annotating CitySaliency. For (a) an image, we get (c) the
human fixations by psychophysical experiments. With (b) semantic annotation
from Cityscapes [9] and (c), we generate (d) the ground truth of salient
objects. Fig. 3: AAMs of two conventional datasets and CitySaliency.

###  2.3 Generation of Salient Object Ground Truth

Since Cityscapes provides pixel-level and instance-level semantic labeling,
location of objects in CitySaliency is known. We represent the fixation
density map of an image at time $t$ as $S_{t}$. Based on the location of
objects and the computed fixation density map, we can compute the saliency
score $S(\mathcal{O})$ for each object $\mathcal{O}\in\mathcal{I}_{t}$ from
the global perspective:

$\begin{split}&S(\mathcal{O})=\\\
&\frac{\sum_{\mathcal{I}_{t}\in\mathcal{V}}\text{I}(\mathcal{O}\in\mathcal{I}_{t})\cdot\left((1+\frac{1}{\|\mathcal{O}\|})\sum_{p\in\mathcal{O}}S_{t}(p)\right)}{\sum_{\mathcal{I}_{t}\in\mathcal{V}}\text{I}(\mathcal{O}\in\mathcal{I}_{t})},\end{split}$
(1)

where $\|\mathcal{O}\|$ is the number of pixels in object $\mathcal{O}$. In
Eq. (1), saliency of an object is defined as the sum of its total fixation
density and its average fixation density. After that, we select objects with
saliency scores above an empirical threshold of top 80% of the maximum
saliency score. In this manner, we get several salient objects with pixel-wise
instance-level annotation and different saliency scores as shown in Fig. 2(d).

Fig. 4: Histograms of the number of salient objects. Fig. 5: Histograms of the
area of salient objects.

Finally, we obtain CitySaliency with fixation annotation and pixel-wise
instance-level salient object annotation. According to the division of
Cityscapes, corresponding 2,975 images are for training and 500 images are for
testing. For the task of SOD, we make salient objects with same saliency
scores _i.e._ , 1 to generate binary masks since we don’t need to distinguish
different instances. In this paper, we only use the binary masks of
CitySaliency except for special instructions.

Fig. 6: The framework of the baseline. We first extract the general knowledge
by common SOD methods, and then the extracted knowledge is transferred to the
task-specific knowledge by an attention-based knowledge transfer module (AKT)
to deal with the cross-domain knowledge difference. After that, the task-
specific knowledge is decoded by a boundary-aware feature decoding module
(BFD) in a progressive manner to detect task-specific salient objects.

###  2.4 Analysis

Comparing CitySaliency with conventional datasets, we can find there usually
exist different task-related targets as displayed in Fig. 1. In addition to
the respective of tasks, we provide comparisons for the perspective of scenes
by presenting the annotation maps (AAMs) of two conventional image-based SOD
datasets (ECSSD [1] and DUT-OMRON [2]) and CitySaliency, as shown in Fig. 3,
which reflects the distribution of salient objects in the dataset. From Fig.
3, we can see that conventional datasets are usually center-biased, while the
CitySaliency is more discrete. In addition, the histograms of number and area
of salient objects are displayed in Fig. 4 and 5. We can observe that there
are usually more salient objects and there is more area occupation in
CitySaliency.

From these observations, we can find two main difficulties _i.e._ , the cross-
domain knowledge difference and task-specific scene gap (including more
discrete distribution, more salient objects and more area occupation), which
makes it difficult to deal with the task-aware SOD for conventional SOD
methods.

##  3 A Baseline Model

To address these difficulties (_i.e._ , the cross-domain knowledge difference
and task-specific scene gap), we propose a baseline model for driving task-
aware SOD via a knowledge transfer network. The framework of our approach is
shown in Fig. 6.

###  3.1 General Knowledge Extraction

To extract the general knowledge, we construct the general subnetwork taking
ResNet-50 [11] with feature pyramid as the feature extractor, which is
modified by removing the last two layers for the pixel-level prediction task.
As shown in Fig. 6, the feature extractor has five residual stages denoted as
$\mathcal{E}_{\mathcal{G}}^{(i)}(\pi_{\mathcal{G}}^{(i)}),i=\\{1,\dots,5\\}$,
where $\pi_{\mathcal{G}}^{(i)}$ is the set of parameters of
$\mathcal{E}_{\mathcal{G}}^{(i)}$. In addition, to obtain larger feature maps
and receptive fields, we set the strides of last residual stages
$\mathcal{E}_{\mathcal{G}}^{(5)}$ to 1 and dilation rates of last two stages
to 2 and 4. The subnetwork is trained by the conventional datasets and learns
the general knowledge (_i.e._ , the output of
$\mathcal{E}_{\mathcal{G}}^{(i)}$).

###  3.2 Attention-based Knowledge Transfer

To deal with the knowledge difference between conventional datasets and
CitySaliency, we propose an attention-based knowledge transfer module (AKT),
which transfers the extracted general features in general subnetwork to task-
specific features in task-specific subnetwork.

As shown in Fig. 7, AKT is based on attention mechanism to purify general
knowledge. For the sake of simplification, we denote the input convolutional
features as $\text{F}_{\mathcal{G}}\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times C}$, which is the output of
$\mathcal{E}_{\mathcal{G}}^{(i)},i={1,\dots,5}$. To purify the features, we
can compute an attention map as follows:

$\text{A}_{{\mathcal{P}}}=\zeta_{s}(\text{F}_{{\mathcal{G}}})\otimes\zeta_{c}(GAP(\text{F}_{{\mathcal{G}}})),$
(2)

where $\zeta_{s}(\cdot)$ and $\zeta_{c}(\cdot)$ denotes the Softmax operation
on the spatial and channel dimension respectively, $GAP(\cdot)$ means global
average pooling, and $\otimes$ and $\oplus$ represents element-wise product
and summation. Based on the attention map $\text{A}_{{\mathcal{P}}}$ from
general knowledge, we use a residual module to learn the cross-domain
knowledge difference from the general domain to task domain and the learned
shift is added to the task-specific features
$\text{F}_{\mathcal{T}}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C}$
produced by task-specific subnetwork. As shown in Fig. 6, the task-specific
subnetwork has the same feature extractor as general subnetwork and its
residual stages are denoted as
$\mathcal{E}_{\mathcal{S}}^{(i)}(\pi_{\mathcal{S}}^{(i)}),i=\\{1,\dots,5\\}$,
where $\pi_{\mathcal{S}}^{(i)}$ is the set of parameters of
$\mathcal{E}_{\mathcal{S}}^{(i)}$.

Fig. 7: The structure of AKT.

For knowledge transfer, we put AKT after the output of
$\mathcal{E}_{\mathcal{G}}^{(i)}$ to learn the knowledge difference, which are
added to $\mathcal{E}_{\mathcal{S}}^{(i)}$ as the input of
$\mathcal{E}_{\mathcal{S}}^{(i+1)},\forall i=1,\dots,4$. For the convenient
representation, we denote the AKT behind $\mathcal{E}_{\mathcal{G}}^{(i)}$ as
$\phi_{\mathcal{A}}^{(i)}(\pi_{\mathcal{A}}^{(i)})$, where
$\pi_{\mathcal{A}}^{(i)}$ is the set of parameters of
$\phi_{\mathcal{A}}^{(i)}$. In this manner, the knowledge difference of
conventional datasets and CitySaliency can be learned and used to make up for
task-aware SOD, assisting to find salient objects.

###  3.3 Boundary-aware Feature Decoding

To deal with the complex scenes in the task domain, we introduce an effective
boundary-ware feature decoding module (BFD) to fine decode features of lots of
salient objects and their structures. Inspired by [7], we make the decoding
process pay attention to the boundaries and interiors to fine decode the
features from the feature extractor of task-specific subnetwork. During
decoding, the features from feature extractor
$\mathcal{E}_{\mathcal{T}}^{(i)}(\pi_{\mathcal{T}}^{(i)})$ and the feature
from the finer decoding layer are integrated to form the features, named as
$\text{F}_{\mathcal{D}}\in\mathbb{R}^{H^{\prime\prime}\times
W^{\prime\prime}\times C^{\prime}}$, which is the input of BFD.

Fig. 8: The structure of BFD.

As shown in Fig. 8, BFD has three parallel branches, _i.e._ , boundary branch
($\phi_{\mathcal{B}}(\pi_{\mathcal{B}})$), transition branch
($\phi_{\mathcal{T}}(\pi_{\mathcal{T}})$) and interior branch
($\phi_{\mathcal{I}}(\pi_{\mathcal{I}})$), where $\pi_{\mathcal{B}}$,
$\pi_{\mathcal{T}}$ and $\pi_{\mathcal{I}}$ are the set of parameters of three
branches, respectively. In the boundary branch, two convolution and one
transposed layers are stacked. In the transition branch, an integrated
successive dilation module from [7] is used to integrate various scale
context. The interior branch has the similar structure with the transition
branch. Let $\textbf{F}_{\mathcal{T}}$ be the fused features, which is
obtained by the fusion of three maps according to the boundary confidence map
$\textbf{M}_{\mathcal{B}}=\sigma(\phi_{\mathcal{B}})$ and the interior
confidence map $\textbf{M}_{\mathcal{I}}=\sigma(\phi_{\mathcal{I}})$. Finally,
we obtain the prediction of salient objects as
$\textbf{M}_{0}=\sigma(\textbf{F}_{\mathcal{T}})$.

We put the BFD after the each decoding stage of task-specific subnetwork as
shown in Fig. 6. For the sake of simplification, we add the index for each
confidence maps as $\textbf{M}_{\mathcal{B}}^{(i)}$,
$\textbf{M}_{\mathcal{T}}^{(i)}$ and
$\textbf{M}_{\mathcal{I}}^{(i)},i={1,\dots,5}$. Therefore, the overall
learning objective can be formulated as:

$\begin{split}\min_{\mathbb{P}}\sum_{i=1}^{5}L(\textbf{M}_{0}^{(i)},G_{0})+L(\textbf{M}^{(i)}_{B},G_{\mathcal{B}})+L(\textbf{M}^{(i)}_{I},G_{\mathcal{I}}),\end{split}$
(3)

where $\mathbb{P}$ is the set of
$\\{\pi^{(i)}_{\mathcal{A}},\pi^{(i)}_{\mathcal{S}},\pi^{(i)}_{\mathcal{B}},\pi^{(i)}_{\mathcal{T}},\pi^{(i)}_{\mathcal{I}}\\}^{5}_{i=1}$
for convenience of presentation, $L(\cdot,\cdot)$ is the binary cross-entropy
loss, and $G_{0},G_{\mathcal{B}},G_{\mathcal{I}}$ represents the ground-truth
masks of salient objects, boundaries and interiors, respectively.

##  4 Experiments

###  4.1 Settings

####  4.1.1 Evaluation Metrics

We choose mean absolute error (MAE), weighted F-measure score
($F^{w}_{\beta}$), F-measure score ($F_{\beta}$) and S-measure ($S_{m}$) to
evaluate methods on CitySaliency.

####  4.1.2 Training and Inference

We use stochastic gradient descent algorithm to train our network. The network
is trained in two stages as follow. 1) We first train the general subnetwork.
In the optimization process, parameters of the feature extractor are
initialized by pre-trained ResNet-50 model with learning rate of $1\times
10^{-3}$, weight decay of $5\times 10^{-4}$ and momentum of 0.9. And learning
rates of rest layers are set to 10 times larger. The general subnetwork is
trained on DUTS [3] to learn general knowledge. Training images are resized to
the resolution of $512\times 256$, and applied horizontal flipping. The
training process takes 50,000 iterations with mini-batch of 4. 2) We fix the
general subnetwork to train the rest network (including AKT and task-specific
subnetwork) with the same setting as general subnetwork and the optimization
objective is Eq. (3). We use CitySaliency to training the network. The
training process takes 200,000 iterations. During inference, one image is
directly fed into the network to produce the saliency map at the output of
first stage in the task-specific subnetwork.

Fig. 9: Examples of the state-of-the-art methods on CitySaliency without fine-
tuning. GT means ground truth.

###  4.2 Model Benchmarking on CitySaliency

To analyze the proposed dataset, we first benchmark 12 state-of-the-art model
performance in Tab. 1 before fine-tuning on CitySaliency dataset. Some
examples are shown in Fig. 9. From the quantitative and qualitative
evaluation, we can find that it is difficult for these models to find correct
salient objects and to restore their fine structures.

Beyond the comparisons without fine-tuning, we fine-tune these models and the
proposed method on CitySaliency. The performance comparisons of these models
are listed in Tab. 1 and shown in Fig. 10. we can find that the performance of
these methods is all improved, which validates the knowledge difference of
conventional datasets and CitySaliency. Also, our method consistently
outperforms other state-of-the-art methods on four metrics, even if the dense
CRF is used on R3Net and DSS. We can also find the proposed method has better
locations and structures of salient objects.

From these experiments, we can believe that task-specific CitySaliency is a
challenging dataset, and the proposed model with knowledge transfer and
feature decoding are useful for solving the task-aware SOD.

Fig. 10: Representative examples of the state-of-the-art methods and our approach after being fine-tuned. Table 1: Performance of 13 state-of-the-art models before and after fine-tuning on CitySaliency. The best two results are in red and blue. “†” means utilizing dense CRF [12]. | Before |  |  |  | After |  |  |
---|---|---|---|---|---|---|---|---
| MAE | $F^{w}_{\beta}$ | $F_{\beta}$ | $S_{m}$ | MAE | $F^{w}_{\beta}$ | $F_{\beta}$ | $S_{m}$
UCF[13] | 0.428 | 0.208 | 0.291 | 0.361 | 0.344 | 0.468 | 0.515 | 0.583
NLDF[14] | 0.369 | 0.084 | 0.226 | 0.332 | 0.172 | 0.676 | 0.749 | 0.744
Amulet[4] | 0.416 | 0.175 | 0.274 | 0.333 | 0.170 | 0.671 | 0.766 | 0.737
FSN[15] | 0.365 | 0.085 | 0.230 | 0.339 | 0.146 | 0.725 | 0.780 | 0.755
SRM[3] | 0.358 | 0.099 | 0.299 | 0.349 | 0.171 | 0.639 | 0.759 | 0.721
RAS[16] | 0.380 | 0.075 | 0.174 | 0.314 | 0.147 | 0.716 | 0.797 | 0.771
PiCANet[5] | 0.372 | 0.112 | 0.259 | 0.340 | 0.189 | 0.656 | 0.768 | 0.749
R3Net†[17] | 0.392 | 0.074 | 0.153 | 0.292 | 0.140 | 0.741 | 0.805 | 0.760
DGRL[6] | 0.360 | 0.086 | 0.401 | 0.342 | 0.146 | 0.701 | 0.725 | 0.725
RFCN[18] | 0.364 | 0.090 | 0.229 | 0.342 | 0.168 | 0.672 | 0.766 | 0.747
DSS†[12] | 0.386 | 0.080 | 0.185 | 0.315 | 0.155 | 0.697 | 0.790 | 0.767
BANet[7] | 0.375 | 0.112 | 0.258 | 0.331 | 0.144 | 0.735 | 0.805 | 0.771
Ours | - | - | - | - | 0.133 | 0.751 | 0.811 | 0.785

###  4.3 Performance Analysis of the Baseline Model

To investigate the effectiveness of the proposed domain-aware network, we
conduct ablation experiments by introducing four different models. The first
setting is only the task-specific subnetwork without AKT and BFD, which is
regraded as “Baseline”. To explore the effectiveness of AKT, we add the second
(named as “Baseline + PT”, pre-trained on DUTS-TR and then fine-tuned on
CitySaliency) and third (“Baseline + AKT”, with AKT) models. In addition, we
add the fourth model arming “Baseline” with BFD as “Baseline + BFD”. We also
list the proposed model as “Ours”.

The comparisons of above models are listed in Tab. 2. By comparing first three
rows, we can observe the pre-trained operation on conventional datasets is
useful for task-aware SOD, and AKT is more effective to utilize the general
knowledge from conventional datasets. In addition, we can find “Baseline +
BFD” also has an obvious improvement compared with “Baseline”, which indicates
the effectiveness of BFD. Moreover, the combination of the attention-based
knowledge transfer and boundary-aware feature decoding achieve a better
performance, which verifies the usefulness of the proposed baseline model. As
a conclusion, this model verifies the feasibility of solving driving-aware SOD
and push the future research.

Table 2: Performance of ablation experiments. |  MAE $\downarrow$ |  $F^{w}_{\beta}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{m}$ $\uparrow$
---|---|---|---|---
Baseline | 0.149 | 0.721 | 0.796 | 0.770
Baseline + PT | 0.143 | 0.726 | 0.797 | 0.772
Baseline + AKT | 0.141 | 0.735 | 0.810 | 0.782
Baseline + BFD | 0.140 | 0.737 | 0.810 | 0.780
Ours | 0.133 | 0.751 | 0.811 | 0.785

##  5 Conclusion

In this paper, we construct a new driving task-specific salient object
detection (SOD) dataset in urban driving to explore SOD in special tasks. By
analyzing this dataset, we find two main difficulties, which prevent the
development of task-aware SOD. Inspired by the difficulties, a baseline model
via the knowledge transfer network is proposed. This model is organized in a
progressive manner, utilizing the attention-based knowledge transfer module
and boundary-aware feature decoding module to deal with the existing
difficulties. In addition, we provide a comprehensive benchmark of the state-
of-the-art methods and our baseline model on the proposed dataset, which shows
the challenges of the task-specific dataset, and validates the effectiveness
of the proposed model. This work pays more attention to the salient features
of objects in the driving scene and does not focus on the dynamic attribute
analysis, which will be explored more with the help of time domain signals in
the feature.

##  6 Acknowledgement

This work is partially supported by the National Natural Science Foundation of
China under the Grant 61922006, and Baidu academic collaboration program.

## References

  * [1] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia,  “Hierarchical saliency detection,”  in CVPR, 2013.
  * [2] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang,  “Saliency detection via graph-based manifold ranking,”  in CVPR, 2013.
  * [3] Tiantian Wang, Ali Borji, Lihe Zhang, Pingping Zhang, and Huchuan Lu,  “A stagewise refinement model for detecting salient objects in images,”  in ICCV, 2017.
  * [4] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan,  “Amulet: Aggregating multi-level convolutional features for salient object detection,”  in ICCV, 2017.
  * [5] Nian Liu, Junwei Han, and Ming-Hsuan Yang,  “Picanet: Learning pixel-wise contextual attention for saliency detection,”  in CVPR, 2018.
  * [6] Tiantian Wang, Lihe Zhang, Shuo Wang, Huchuan Lu, Gang Yang, Xiang Ruan, and Ali Borji,  “Detect globally, refine locally: A novel approach to saliency detection,”  in CVPR, 2018.
  * [7] Jinming Su, Jia Li, Yu Zhang, Changqun Xia, and Yonghong Tian,  “Selectivity or invariance: Boundary-aware salient object detection,”  in ICCV, 2019.
  * [8] Stefano Alletto, Andrea Palazzi, Francesco Solera, Simone Calderara, and Rita Cucchiara,  “Dr (eye) ve: a dataset for attention-based tasks with applications to autonomous and assisted driving,”  in CVPRW, 2016.
  * [9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele,  “The cityscapes dataset for semantic urban scene understanding,”  in CVPR, 2016.
  * [10] Jia Li, Changqun Xia, and Xiaowu Chen,  “A benchmark dataset and saliency-guided stacked autoencoders for video-based salient object detection,”  IEEE TIP, vol. 27, no. 1, pp. 349–364, 2017.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016.
  * [12] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr,  “Deeply supervised salient object detection with short connections,”  IEEE TPAMI, 2019.
  * [13] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Baocai Yin,  “Learning uncertain convolutional features for accurate saliency detection,”  in ICCV, 2017.
  * [14] Zhiming Luo, Akshaya Kumar Mishra, Andrew Achkar, Justin A Eichel, Shaozi Li, and Pierre-Marc Jodoin,  “Non-local deep features for salient object detection.,”  in CVPR, 2017.
  * [15] Xiaowu Chen, Anlin Zheng, Jia Li, and Feng Lu,  “Look, perceive and segment: Finding the salient objects in images via two-stream fixation-semantic cnns,”  in ICCV, 2017.
  * [16] Shuhan Chen, Xiuli Tan, Ben Wang, and Xuelong Hu,  “Reverse attention for salient object detection,”  in ECCV, 2018.
  * [17] Zijun Deng, Xiaowei Hu, Lei Zhu, Xuemiao Xu, Jing Qin, Guoqiang Han, and Pheng-Ann Heng,  “R3net: Recurrent residual refinement network for saliency detection,”  in IJCAI, 2018.
  * [18] Wenguan Wang, Jianbing Shen, Xingping Dong, and Ali Borji,  “Salient object detection driven by fixation prediction,”  in CVPR, 2018.