# Deep-learning-driven end-to-end metalens imaging

Joonhyuk Seo  Department of Artificial Intelligence, Hanyang University,
Seoul, 04763, Republic of Korea  Jaegang Jo  Department of Electronic
Engineering, Hanyang University, Seoul, 04763, Republic of Korea  Joohoon Kim
Department of Mechanical Engineering, Pohang University of Science and
Technology (POSTECH), Pohang, 37673, Republic of Korea  Joonho Kang
Department of Electronic Engineering, Hanyang University, Seoul, 04763,
Republic of Korea  Chanik Kang  Department of Artificial Intelligence, Hanyang
University, Seoul, 04763, Republic of Korea  Seongwon Moon  Department of
Mechanical Engineering, Pohang University of Science and Technology (POSTECH),
Pohang, 37673, Republic of Korea  Eunji Lee  Department of Chemical
Engineering, Pohang University of Science and Technology (POSTECH), Pohang,
37673, Republic of Korea  Jehyeong Hong  Department of Electronic Engineering,
Hanyang University, Seoul, 04763, Republic of Korea  Junsuk Rho  Department of
Mechanical Engineering, Pohang University of Science and Technology (POSTECH),
Pohang, 37673, Republic of Korea  Department of Chemical Engineering, Pohang
University of Science and Technology (POSTECH), Pohang, 37673, Republic of
Korea  POSCO-POSTECH-RIST Convergence Research Center for Flat Optics and
Metaphotonics, Pohang, 37673, Republic of Korea  National Institute of
Nanomaterials Technology (NINT), Pohang, 37673, Republic of Korea  These
authors are corresponding authors.  Haejun Chung  Department of Artificial
Intelligence, Hanyang University, Seoul, 04763, Republic of Korea  Department
of Electronic Engineering, Hanyang University, Seoul, 04763, Republic of Korea
Division of Nanoscale Semiconductor Engineering, Hanyang University, Seoul,
04763, Republic of Korea  Department of Semiconductor Engineering, Hanyang
University, Seoul, 04763, Republic of Korea  These authors are corresponding
authors.

###### Abstract

Recent advances in metasurface lenses (metalenses) have shown great potential
for opening a new era in compact imaging, photography, light detection and
ranging (LiDAR), and virtual reality/augmented reality (VR/AR) applications.
However, the fundamental trade-off between broadband focusing efficiency and
operating bandwidth limits the performance of broadband metalenses, resulting
in chromatic aberration, angular aberration, and a relatively low efficiency.
In this study, a deep-learning-based image restoration framework is proposed
to overcome these limitations and realize end-to-end metalens imaging, thereby
achieving aberration-free full-color imaging for mass-produced metalenses with
10-mm diameter. Neural-network-assisted metalens imaging achieved a high
resolution comparable to that of the ground truth image.

##  1 Introduction

Metalenses, which are ultrathin film lenses composed of subwavelength
structures, have been highlighted as the technology that would overcome the
limitations of chromatic aberration [1, 2], shadowing effects [3, 4], bulky
systems [5, 6], and high manufacturing costs [7, 8] associated with
conventional lenses. In a recent study, deep-ultraviolet immersion lithography
was combined with wafer-scale nano-imprint lithography to mass-produce low-
cost and high-throughput large-aperture metalenses, contributing to their
commercialization [9]. Metalenses also have great potential to realize ultra-
light and ultra-compact imaging systems [10, 1] because of diffraction-limited
focusing within subwavelength thicknesses, even in high-NA regimes [11].
Consequently, metalenses are considered ideal replacements for conventional
optical elements in optical sensors [12], smartphone cameras [13, 14],
unmanned aerial vehicle cameras [15, 16] and Augmented Reality/Virtual Reality
(AR/VR) devices [17, 18]. However, recent studies have suggested that large-
area broadband metalenses suffer from a fundamental trade-off between
broadband focusing efficiency and diameter [19, 20]. In addition, meta-atom-
based metalenses have a narrow Field of View (FoV) because of the angular
dispersion of the meta-atoms [21], where meta-atoms are periodically aligned
metasurface bases that simplify the design process of metasurfaces while
maintaining the predicted efficiency. Consequently, at present, reported
broadband metalenses exhibit chromatic aberration [22, 3] or low focusing
efficiency over a large bandwidth [1, 6], which hinders the commercialization
of metalens-based compact imaging.

In this study, we propose a deep neural network (DNN)-based image-
reconstruction framework customized for a metalens imaging system (referred to
as DRMI) to overcome the aforementioned physical constraints by learning image
deformities elicited by the largest mass-produced metalens (a 10-mm diameter)
[9]. Unlike conventional image restoration tasks [23, 24], our mass-produced
metalens simultaneously undergoes significant chromatic and angular
aberrations. Therefore, restoring the images captured by the metalens
(“metalens images”) to their clean state is immensely challenging. DRMI
identifies and restores the aberration problems of the metalens by a deep
learning approach. Specifically, we collect hundreds of blurred images
captured by the metalens imaging system shown in Fig. 1(a) before training. We
then train DRMI using the captured images exhibiting the physical defects of
the metalens, whereby the quality of the image created by the compact metalens
imaging system was restored with great success. DRMI, shown in Fig. 1(e),
consists of a Nonlinear Activation Free Network (NAFNet) as the baseline and
an adversarial learning scheme in the frequency domain (details in the Methods
section). By applying DRMI to the mass-produced metalens, we construct a
hybrid imaging system that achieves high-quality compact imaging. The system
is scalable to larger apertures and different wavelengths, thereby providing
the ultimate solution for a novel miniaturized imaging scheme.

Figure 1: Proposed image restoration framework. (a) The metalens imaging
system comprises a metalens, left-handed circular polarization (LCP) and
right-handed circular polarization (RCP) filters, and a camera. The metalens
imaging system acquires the training dataset by capturing the ground truth
images displayed on an $None$ 4K LCD monitor. (b-c) Photographs of fabricated
mass-produced 10-mm-diameter metalenses on $None$ glass wafer. (d) Scanning
Electron Microscopy (SEM) image showing the meta-atoms that compose the
metalens. The metalenses were fabricated by atomic layer deposition of
titanium dioxide thin film following nano-imprint lithography. The scalebar is
3 . (e) Proposed image restoration model. The model consists of a Nonlinear
Activation Free Network (NAFNet) as the baseline, applies random cropping and
position embedding to the input data using coordinate information of the
cropped patches. To address the underconstrained problem of restoring degraded
images to latent sharp images, adversarial learning in the frequency domain is
applied through the fast Fourier transform($\mathcal{F}$). $\hat{x}$ and $x$
denote the reconstructed and ground truth image, respectively. The details are
in the Methods section.

##  2 Results

###  2.1 Metalens imaging system

Our metalens imaging system suffers from severe wavelength-dependent blurring
due to the chromatic aberration originating from the meta-atoms of the
metalens. As shown in Fig. 1(d), we use a metalens comprising an array of
nano-structures with arbitrary rotational angles, referred to as a
Pancharatnam–Berry (PB) phase-based metalens. PB-phase-based metalenses can
achieve diffraction-limited focusing [5, 9]; however, the dispersion of meta-
atom can potentially cause chromatic aberrations similar to diffractive lenses
[25]. Substantial efforts have been made to achieve achromatic metalenses
through dispersion engineering of meta-atoms [6, 1], adjoint optimization [26,
27], and many other methods [28, 29]. However, the resulting metalenses still
suffer from relatively low efficiency compared to single-frequency metalenses.
Our mass-produced metalens has relatively high efficiency (55.6%) at 532 nm
wavelength but exhibited severe chromatic aberration, resulting in wavelength-
dependent focal lengths of 29.0, 24.5, and 20.5 mm at wavelengths of 450 nm
(blue), 532 nm (green), and 635 nm (red), respectively (Fig. 2(a)) [9]. The
blurring of the imaging system further depends on the viewing angle because of
the angular aberration that originates both from the Seidel aberrations [4]
and angular dispersion of the meta-atom [21].

The chromatic and angular aberrations in our metalens imaging system were
quantitatively analyzed by measuring the Point Spread Functions (PSF), which
is the 2D intensity distribution obtained in response to a single point light
source [30]. The PSF is a quintessential metric for evaluating the quality of
an imaging system because it is directly related to image formation [31]. We
also calculated Modulation Transfer Function (MTF) from the measured PSF to
describe the imaging quality in terms of resolution and contrast [30] (details
about the PSF measurement and MTF calculation are provided in Supplementary
Section 1.).

Before measuring the PSF, the distance between the metalens and the image
sensor was set considering Transverse Axial Chromatic aberration (TAC), which
can be expressed as

$\text{TAC}=|f-f_{0}|\frac{D}{2f_{0}}$ (1)

where $f$ is the focal length of the incident light with a single wavelength;
$f_{0}$ is the distance between the metalens and image sensor; and $D$ is the
diameter of the metalens. When $f$ differs from $f_{0}$, the incident light
forms a top-hat-like PSF profile with a radius equal to that of TAC [25]. The
reduction in the overall TAC in the visible band is important for high-quality
metalens imaging because the blur of the image intensifies proportionally with
the increase in TAC. To reduce overall TAC, we chose $f_{0}$ as the focal
length of green light (24.5 mm). The longitudinal chromatic aberrations of red
and blue lights, the difference between $f$ and $f_{0}$ for red and blue
lights, are 4.0 and 4.5 mm, respectively.

Figure 2(e) shows the PSFs for various viewing angles ($None$, $None$, and
$None$). The PSF with zero viewing angle ($None$) shows a non-ideal profile,
where the blue and red channels have wider distributions than the green
channel, resulting in severe blurring in the red and blue channels. In
addition, the non-uniformity of the metasurface during the fabrication process
[9] may result in the PSFs with complex profiles not matching the PSFs from
the Rayleigh-Sommerfeld diffraction formula [32], which further complicates
the image restoration task.

Figures 2(b-d) show the MTFs of the metalens for various viewing angles and
RGB color channels. The MTF of the green channel is higher than that of the
red and blue channels at all spatial frequencies and viewing angles.
Furthermore, the MTFs of the red and green channels decrease with the viewing
angle, but the MTF of the blue channel increases with the viewing angle,
resulting in complex and combined aberrations.

Figures 2(f) and (g) show the metalens image, the corresponding ground truth
image, and the subset images depicting the red, green, and blue color
channels. The red and blue channels of the metalens image are severely blurred
from TAC, making it difficult to recognize any objects. The green channel of
the metalens image shows a relatively higher resolution at the center, which
gradually decreases as the viewing angle increases (e.g., the outer region of
the image) due to PSF deformation at the higher viewing angle.

Figure 2: (a) Focal lengths of the metalens for wavelengths of 450 (blue), 532
(green), and 635 (red) nm. The dashed line indicates the linear fitting
result. The change in focal length with respect to wavelength implies a
significant chromatic aberration. (b-d) MTFs for various viewing angles of
red, green, and blue channels, respectively. The MTF of the green channel is
higher than those of the red and blue channels for all spatial-frequencies and
viewing angles. (e) PSFs with various viewing angles ($None$, $None$, $None$,
$None$). The measured PSFs show angle-dependent complex profiles arising from
both Seidel aberrations and non-uniformity of the metalens fabrication. The
PSF profile is widely spread in the red and blue channels compared to the
green, implying a significant chromatic aberration of the metalens. (f) The
metalens image (left) and its subset images showing red, green, and blue color
channels. (g) Corresponding ground truth image (left) and its subset images
showing red, green, and blue color channels. The red and blue color channels
of the metalens image are severely blurred by TAC.

###  2.2 Overview of the metalens image reconstruction process

Computational image reconstructions have been widely used for non-ideal images
such as noisy [33] and blurred [34] images. Classical image reconstruction
methods achieve higher resolution by relying on linear deconvolution methods,
such as applying the Wiener filter [35]. Deconvolution, an inverse
convolution, enables the reconstitution of the original image from an image
convolved with PSF. The performance of the deconvolution process depends on
two factors: the space invariance of PSF across the FOV and the low condition
number for the inverse of the PSF [36]. Wiener filters exhibit limited
reconstruction quality for imaging systems with space-variant PSFs, such as
metalens imaging systems [28] and under display cameras [31].

Another restoration approach is DNN-based image reconstruction. The DNN-based
reconstruction models [23, 24] demonstrate superior performance compared to
traditional approaches in specialized tasks such as denoising [33], de-
blurring [34], super-resolution [37], and light-enhancement [38]. Moreover,
they are applicable to imaging systems with complex and combined degradations
such as under-display cameras [31], single-lens cameras [39], and the
$360^{\circ}$ FOV panoramic camera [40]. However, conventional DNN approaches
are impossible for learning position-variant image degradations (e.g.,
metalens imaging) because these methods train models with randomly cropped
patches from full-resolution images, which results in the complete loss of
position-dependent information.

Thus, we propose an end-to-end image restoration model customized for the
metalens imaging system to address non-uniform aberration over wavelength and
viewing angle. Unlike images commonly used in typical image restoration tasks
[41, 42], our metalens images exibit more intense blur and significant color
distortion. Consequently, the restoration of metalens images is a severely
ill-posed inverse problem. To address this critically underconstrained
problem, we employ strong regularization (prior), i.e., model the distribution
of sharp data, performing adversarial learning in Fourier space to train the
data distribution for prior modeling. Therefore the proposed model $f(y)$ is
trained by minimizing

$L(x,y,f)=E(x,f(y))+\lambda\Phi(x)$ (2)

where $E(x,f(y))$ is the image fidelity term that approximates the metalens
image $y$ by the ground truth image $x$, and $\Phi(x)$ is the prior term that
limits the space of $x$. Subsequently, we apply positional embedding to learn
the angular aberration of metalens imaging. Because the proposed method
utilizes information on the absolute coordinates of randomly cropped patches,
the model effectively trains the highly space-variant degradations.

| Image quality metric | Frequency measurement
---|---|---
Model | PSNR | SSIM | LPIPS | MAE | CS
Metalens image | 14.722/1.328 | 0.464/0.160 | 0.788/0.112 | 3.281/1.089 | 0.018/0.006
MIRNetV2 | 18.507/1.893 | 0.591/0.135 | 0.559/0.098 | 2.240/0.900 | 0.069/0.020
SFNet | 18.223/1.727 | 0.602/0.129 | 0.519/0.095 | 2.194/0.837 | 0.088/0.028
HINet | 21.364/2.333 | 0.677/0.117 | 0.456/0.097 | 1.851/0.800 | 0.143/0.045
NAFNet | 21.689/2.382 | 0.676/0.116 | 0.440/0.097 | 1.817/0.801 | 0.138/0.040
DRMI | 22.095/2.423 | 0.692/0.109 | 0.432/0.096 | 1.759/0.779 | 0.155/0.048
Table 1:  Comparison of quantitative results from various models (Best). All
quantitative results in this table were obtained using the test set of images
(n=70). The values are represented as mean and standard deviation Figure 3:
Image quality of the proposed model compared to that of the baseline model and
the metalens images. (a-e) Results of PSNR, SSIM, LPIPS in RGB space and
cosine similarity, MAE of the magnitudes in Fourier space calculated by
comparing the metalens image and the image reconstructed by DRMI with the
ground truth image. (f-j) Restoration results of the baseline model and the
image restored by DRMI. In (a-j), the red horizontal lines in each box
represent the median, and the boxes extend from the first to the third
quartile. The whiskers span 1.5 times the interquartile range of the first and
third quartiles. A statistical hypothesis test was performed through a two-
sided paired t-test on the performance difference between the metalens image
and the image reconstructed by DRMI (significance level $P=10^{-4}$, (a)
$1.055*10^{-39}$, (b) $3.886*10^{-35}$, (c) $1.363*10^{-48}$, (d)
$2.311*10^{-35}$, (e) $2.150*10^{-38}$, (f) $2.671*10^{-11}$, (g)
$1.909*10^{-14}$, (h) $6.277*10^{-8}$, (i) $2.041*10^{-14}$, (j)
$6.369*10^{-14}$). The metrics were evaluated on a test set with 70 data
points. Qualitative results of the (k) ground truth images, (l) metalens
images, and (m) images reconstructed by DRMI. The central (red) and outer
(yellow) regions of the images are zoomed in to assess the restoration of the
spatially variant degradations. The central regions of the metalens images
show the color and detail of the images well, and in the outer region of the
image the degradation due to oblique incidence is alleviated (enlarged).

###  2.3 Assessment of DRMI-based metalens image reconstruction

Predicting detailed information of latent sharp images and achieving high
fidelity are important factors for image restoration. To compare the quality
of images created by DRMI with those of the metalens and the baseline model,
we restore the metalens images (n=70) to their cleaned state and analyze the
degree of improvement. We quantify the image quality with peak signal-to-noise
ratio (PSNR) and structural similarity index measure (SSIM) for fidelity
assessment of the restored images. We also utilize a perceptual index, learned
perceptual image patch similarity (LPIPS), because the former two are
insufficient for evaluating structured outputs [43]. Figures 3(a-c), (f-h) and
Table 1 present the DRMI’s PSNR, SSIM, and LPIPS results, respectively. The
DRMI images have significantly better quality compared to the metalens images
and also showed superior performance compared to the baseline model, NAFNet.
DRMI improved the image quality by 7.37 dB for PSNR, 22.8%p for SSIM, and
35.6%p for LPIPS for the metalens images and also showed a 0.40 dB gain in
PSNR, 1.6%p increase in SSIM, and 0.8%p increase in LPIPS compared with the
baseline model. We also compare DRMI with state-of-the-art (SOTA) models,
including restoration models for natural images (SFNet [44], MIRNetv2 [45],
HINet [46]). As shown in Table 1, DRMI surpasses these SOTA models by a
substantial margin in terms of PSNR, SSIM, and LPIPS. This suggests that DRMI
is more suitable for the metalens image restoration task than conventional
models designed for restoring natural images, such as those in the DIV2K
dataset [41].

The measured MTF of the metalens in Fig. 2(b-d) exhibits intense degradation
at high spatial frequencies; therefore, it is crucial to restore the spatial-
frequency information during the metalens image restoration task. Spatial-
frequency includes both magnitude and phase information; the latter is
considered more important in signal processing [47]. We utilize two metrics to
evaluate the magnitude and phase of the spatial-frequency of the reconstructed
images. The first is the mean absolute error (MAE) between the magnitude of
the reconstructed and ground truth images, and the second is the cosine
similarity (CS) that finds the similarity between the phases of the ground
truth and reconstructed images. To quantify these two metrics, we apply fast
Fourier transform (FFT) to images restored by different models. The calculated
MAE and CS indicate an exceptional improvement in the image quality as shown
in Fig. 3(d-e), (i-j) and Table 1. In addition, the qualitative results in
Fig. 3(k-m) show the competitive qualities of the DRMI images compared to the
ground truth. Metalens images show significant chromatic aberration, which
makes red and blue appear significantly more blurred than green. Additionally,
the loss of spatial-frequency information results in a loss of the original
fine details of the original clean images. In particular, the metalens images
show more intense blur in the outer region (yellow box) than in the central
area (red box), as shown in Fig. 3(k-m). In the outer area of the original
clean image containing sharp details, severe blurring causes a significant
loss in detail, and a specific color dominates. However, the outer and central
regions of the images reconstructed using DRMI are nearly indistinguishable
from the ground truth. DRMI also effectively restores details that have been
lost due to chromatic aberration. The results indicate that DRMI can
effectively overcome the complicated degradation caused by a highly
inconsistent PSF, thereby improving the imaging performance for diverse
scenes.

We also employ Variance of Color Difference (VCD) [48], a metric that
quantifies the distance between the green channel and other channels, to
measure the degree of restoration regarding the chromatic aberration. To
calculate the VCD values, we choose monochromatic 1951 U.S. Air Force
resolution test chart images (‘USAF images’), which do not have any grey tones
but have conspicuously high contrast. Figures 4(a) and (b) show that the USAF
images captured with the metalens exhibit severe blurring of the stripe
pattern in the central area, which corresponds to the highest resolution.
Strong color distortion also appears in regions where green color is heavily
emphasized instead of white and black. The baseline model restores the edges
of the degraded USAF images well; however, the color distortion caused by
chromatic aberration is not fully corrected. Specifically, the outer region in
Fig. 4(c) is represented in green color instead of white. By contrast, DRMI
effectively restores the edges of the USAF images and successfully corrects
color distortion in both the central and outer regions (Fig. 4(e) and (f)). As
a consequence, DRMI achieves the highest VCD values, indicating that DRMI
successfully restores the distorted colors in the metalens images.

Figure 4: Comparison of chromatic aberration in USAF images (white, black).
The central region corresponding to the highest resolution is enlarged to
allow the evaluation of fine details corrupted by chromatic aberration. (a-b)
The USAF images were captured by the metalens. Strong color distortion appears
in regions with heavy green color emphasis. (c-d) USAF images restored using
the baseline model. The outer region of the white USAF image appears green
instead of white, indicating that chromatic aberration is not fully corrected.
(e-f) USAF images restored using DRMI show the highest VCD values and
successful correction of the color distortion. Figure 5: Object detection
tasks using pre-trained SSD model on the original images (a, d), the metalens
images (b, e), and the images restored by DRMI (c, f). The pre-trained SSD
model does not detect any objects in metalens images; however, it successfully
captures multiple classes and objects in images restored by DRMI.

DRMI improves the performance of downstream applications such as object
detection. To validate the performance of DRMI in object detection on the
restored images, we first capture with the metalens each image in the entire
PASCAL VOC2007 dataset (n=4,952) [49] displayed on the LCD monitor. Then, we
employ Single Shot multibox Detector (SSD) [50] pre-trained for the PASCAL
VOC2007. Figure 5 shows examples of object detection using SSD. The detector
predicts the entire region (red box) as an object because it cannot identify
the detailed features in the metalens images (Fig. 5(b)). On the other hand,
the detector accurately predicts the bounding boxes at the desired objects in
the DRMI images because compared to the original PASCAL VOC2007, the quality
of the DRMI images is competitive (Fig. 5(a) and (c)). We introduce the mean
average precision (AP) as an evaluation metric. DRMI’s
$\mathrm{AP},\mathrm{AP_{50}}$ on the restored images are approximately 34%p
and 56%p higher than those on the metalens images. They are approximately 86%
and 88% of the $\mathrm{AP},\mathrm{AP_{50}}$ on the ground truth (Table S1).

##  3 Discussion

In this study, we demonstrated a metalens imaging system assisted by DNN image
reconstruction, thereby producing an image visually indistinguishable from the
ground truth image. The metalens that we used had severe chromatic and angular
aberrations, resulting in a widely distributed and viewing angle-dependent
PSF. To overcome these complex aberrations, we utilized an adversarial
learning scheme in the Fourier space and positional embedding, which are
widely used in conventional image reconstructions. Positional embedding
enables the restoration network to effectively alleviate viewing-angle-
dependent PSFs. The adversarial learning scheme in Fourier space contributes
to restoring high-spatial-frequency information. DRMI achieved significantly
higher performance metrics, PSNR, SSIM, and LPIPS, than the baseline model,
NAFNet. Furthermore, the DRMI-restored images display a higher cosine
similarity to ground truth images than the images restored by the baseline
model in terms of magnitude and phase distribution in Fourier space. Overall,
DRMI has great potential for restoring complex and combined aberrations in
singlet metalens imaging systems. In addition, we conducted an object-
detection test to demonstrate the capability of DRMI in downstream
applications, which turned out to be a success with an accuracy similar to
that of the ground truth.

The proposed image restoration framework achieves aberration-free full-color
imaging in mass-produced metalenses with 10-mm diameter. Thus, DRMI may pave
the way for the commercialization of aberration-free ultra-compact imaging
systems. DRMI is also applicable to refractive and meta-optics hybrid systems
[51], near-infrared cameras [12, 52], polarization-imaging [53], and others
[54, 55].

##  4 Methods

###  4.1 Optical setup and data acquisition

####  4.1.1 Metalens imaging system

As shown in Fig. 1(a), the metalens imaging system comprised a mass-produced
metalens [9], a commercial camera (Basler acA5472-17uc), and Left-hand
Circular polarization (LCP) and Right-hand Circular Polarization (RCP) filters
(Edmund Optics CP42HE and CP42HER). The metalens was sliced from a metalens
array on a glass wafer, mounted on a 3D-printed lens holder, and covered with
LCP and RCP filters. The metalens was designed to transmit LCP incidence
light, therefore, LCP filter is essential to eliminate unnecessary incidence
light. The RCP filter faced the camera and absorbed the transmitted LCP light
because the metalens converts the incidence LCP light into the RCP. The
optical cage system aligned all the optical components with the optical axis
of the metalens directed toward the center of the image sensor (details on the
metalens are provided in Supplementary sections 2 and 3).

####  4.1.2 Data acquisition

The metalens imaging system acquired the training data by capturing the ground
truth images displayed on the $None$ 4K LCD monitor (Samsung KU85UA7050F). The
metalens imaging system was placed 2.6 away from the monitor with the optical
axis perpendicular to the monitor and pointing toward the center of the
monitor. We utilized DIV2K dataset[41] for training, examples of which are
shown in Fig. 3. This dataset contains 2K resolution images of various objects
that provide diversity in the environment. The 698 images of the dataset were
cropped in a 16:9 ratio and resized to $1920\times 1080$ resolution before
displaying on the monitor. The ground truth images for training were obtained
by cropping the center of the $1920\times 1080$ resolution images with
$1280\times 800$ resolution, because the ground truth images should fit the
FOV of the metalens imaging system.

The positions of the objects in both the metalens image and the corresponding
ground truth image were matched for effective training. Raw metalens images
with $5472\times 3648$ resolution were rotated, cropped to $5328\times 3328$
resolution, and resized to $1280\times 800$ resolution to match the
corresponding ground truth images. The rotation angle and cropping parameters
were optimized to maximize the SSIM between the metalens images and the
corresponding ground truth images.

###  4.2 Network Architecture

The network architecture of DRMI employs NAFNet [23], which is computationally
efficient and achieves the SOTA results in the field of image restoration. We
use randomly cropped patches from full-resolution (meaning $1280\times 800$
resolution in this work) images as training data and the full-resolution
images themselves as testing data during inference. However, we observe a
severe statistical difference in the inference process of the full-resolution
image, as shown in Fig. S1. Thus, we apply test-time local converter (TLC)
[56] in test phase to overcome this inconsistency, which provides a
significant performance improvement. The detailed experimental results are
presented in Table S2.

The metalens we used suffers from severe chromatic and angular aberrations,
resulting in serious information loss in the metalens images. Therefore, we
train the model with respect to the dataset distribution of latent clean
images to efficiently restore a wide range of spatial-frequencies and
constrain the space of the latent ground truth images. Because generative
models can learn complex, high-dimensional data distributions from a given
dataset [57], we utilize an adversarial learning scheme to learn the
distribution of latent sharp images. We initially applied adversarial learning
in the RGB space, but found that conspicuous pattern artifacts occurred in
both RGB and Fourier spaces (Fig. S2). Thus, we convert the RGB space data to
Fourier space data to prevent pattern artifacts while reflecting spatial-
frequency information. These Fourier space data are then used as the input
data for the discriminator.

The training loss is composed of two distinct terms; (1) PSNR loss
$L_{\mathrm{PSNR}}$ between the reconstructed image $\hat{x}$ and the ground
truth image $x$; (2) Adversarial loss $L_{a}$ between $\mathcal{F}(\hat{x})$
and $\mathcal{F}(x)$. The PSNR loss signifies the image fidelity loss, and the
adversarial loss indicates the prior regularization loss. Therefore, the total
loss function $L_{\mathrm{Total}}$ is:

$L_{\mathrm{Total}}=L_{\mathrm{PSNR}}+\lambda L_{\mathrm{a}}$ (3)

where $\lambda$ is a hyperparameter for balancing $L_{\mathrm{PSNR}}$ and
$L_{\mathrm{a}}$. $L_{\mathrm{PSNR}}$ is calculated as follows:

$L_{\mathrm{PSNR}}(\hat{x},x)=-10\log{R^{2}\above
1.0pt\mathrm{MSE}(\hat{x},x)}$ (4)

where $\hat{x},x$ and $R$ denote the ground truth image, reconstructed image,
and the maximum signal value of the ground truth image, respectively. MSE is
the distance between the ground truth and reconstructed image and is
formulated as $\mathrm{MSE}(\hat{x},x)={1\above
1.0ptN}\sum_{n=1}^{N}(\hat{x}_{n}-x_{n})^{2}$.

For adversarial learning, we construct an additional discriminator and apply
spectral normalization [58] for training stability. In addition, we employ the
GAN training scheme based on hinge loss [59] for enhanced stability of
adversarial training. The adversarial loss $L_{a}$ of the discriminator (D)
and the image restoration model (G) is:

$L^{\mathrm{D}}_{a}=\mathbb{E}_{x}[\max(0,1-D(\mathcal{F}(x)))]+\mathbb{E}_{\hat{x}}[\max(0,1+D(\mathcal{F}(\hat{x})))]$
(5) $L^{\mathrm{G}}_{a}=-\mathbb{E}_{\hat{x}}[D(\mathcal{F}(\hat{x}))]$ (6)

where $\mathcal{F}$ refers to FFT. Here $\mathbb{E}_{x}[\cdot]$ and
$\mathbb{E}_{\hat{x}}[\cdot]$ are operators that denote the calculation of the
mean of the ground truth and reconstructed images in the given minibatch,
respectively. The image restoration model and discriminator each try to
minimize $L^{\mathrm{G}}_{a}$ and $L^{\mathrm{D}}_{a}$, respectively.

Figure 3(l) shows that the degradation in the outer region of the metalens
image is more severe than in the central region because of the angular
aberration. This implies that positional information is crucial for
understanding the degradation of the metalens imaging system. However, the
training method makes it impossible for the model to learn positional
information because our neural network learns through random patches during
training and restores full-resolution images during inference. To solve this
problem, we take the coordinate values of each pixel of the patches, based on
the coordinates of a full resolution image, and map them through a $1\times 1$
convolutional layer to transform them into proper space when generating random
patches for a full-resolution image. The processed coordinate information is
concatenated with metalens images corresponding to the information. The
concatenated data is used as input data.

###  4.3 Training Details

As mentioned above, we used patches randomly cropped from full-resolution
images for training. Larger receptive fields provide richer semantic
information but also increase training time and computational complexity.
Thus, we set the patch size to $256\times 256$ and the batch size to 16 to
balance performance and training time in the proposed model. In addition,
horizontal flips, vertical flips and transpose transformations were randomly
applied, then coordinate information of the patches was loaded under these
settings.

The model used in this paper is divided into two parts. The first is the image
restoration model; NAFNet was used as a baseline model, and the width of the
starting layer of the network was set to 32, which increases by twofold as a
block gets a level deeper. The encoder and decoder of the network were
composed of four blocks. We also adopted TLC in test phase to solve the train-
test inconsistency. The number of input and output channels of $1\times 1$
convolutional layer which processes coordinate information are both set to 2.
The second part is the discriminator. We set the discriminator’s width to 64,
and all layers had the same width. The discriminator was composed of five
blocks. Moreover, spectral normalization was applied [58] to stabilize
learning.

The training consists of two stages. In the first stage, the metalens images
were restored to clean images using the image restoration model, and in the
second stage the restored and ground truth images were expressed as data in
the spatial-frequency domain through fast Fourier transform (FFT), and
adversarial learning was performed using the discriminator. Because the
spatial-frequency domain data converted through FFT are complex (real and
imaginary), we expressed their real and imaginary parts as a two-dimensional
vector, thereby expressing the data in the spatial-frequency domain as real
vectors to be used as input into the discriminator. During the training, the
number of iterations was set to 300,000. In the image restoration model, AdamW
was used as the optimizer with the learning rate initially set to $3\times
10^{-4}$ and gradually decreased to $10^{-7}$ following the cosine annealing
schedule; the betas were $[0.9,0.9]$. For the discriminator optimizer, we used
Adam with the learning rate set to $3\times 10^{-4}$, the same as the
restoration model, but the betas were set to $[0.0,0.9]$; NVIDIA RTX 4090 24GB
was used as a computational resource for this training.

###  4.4 Performance Metrics

####  4.4.1 PSNR and SSIM

We used PSNR and SSIM, representative image quality metrics, to evaluate the
images reconstructed by the proposed model. The PSNR is :

$\mathrm{PSNR}=10\log_{10}({R^{2}\above 1.0pt\mathrm{MSE}}).$ (7)

Since MSE is a metric indicating the difference of pixel values, higher PSNR
values indicate a better restoration performance. $R$ is the maximum signal
value of the ground truth image. However, PSNR is only an approximation to the
human visual perception of the reconstruction quality because MSE is the
pixel-level intensity difference between the reconstructed and ground truth
images. Therefore, we use SSIM to accurately evaluate the visually perceived
quality of the restored images. SSIM utilizes three elements in its
evaluation: luminance, contrast, and structure between the reconstructed and
ground truth images.

####  4.4.2 LPIPS

To evaluate the perceptual quality of given images, we used LPIPS. LPIPS is
calculated by inputting two images into a pre-trained network (e.g., AlexNet)
and calculating the similarity between the resulting features. Higher LPIPS
values indicate better perceptual quality.

####  4.4.3 MAE and cosine similarity

When transforming data from RGB space into Fourier space, the transformed data
are represented as two-dimensional vectors in complex space. The complex
vectors can then be completely expressed in magnitude and phase. First, we
used MAE to evaluate the magnitude difference in Fourier space. Second, the
cosine similarity of two vectors was used to evaluate the phase difference.

####  4.4.4 VCD

We use VCD to evaluate the degree of chromatic aberration. $VCD_{X}$ is the
difference between the X channel and green channel values.

$\mathrm{VCD}_{X}={\sum^{M}_{i=0}\sum^{N}_{j=0}[X(i,j)-G(i,j)]^{2}\above
1.0ptMN}$ (8)

The VCD is a mean of $VCD_{R}$ and $VCD_{B}$:

$\mathrm{VCD}={\mathrm{VCD}_{R}+\mathrm{VCD}_{B}\above 1.0pt2}$ (9)

Since VCD is the difference across the three channels, USAF images with black
and white color only are used to evaluate chromatic aberration precisely.

####  4.4.5 Average precision

The mean average precision (AP) is a standard metric for object detection. AP
is measured based on a preset Intersection over Union (IoU) threshold. The AP
is calculated in multiple scales using IoU thresholds ranging from 0.5 to
0.95. The terms “$\mathrm{AP_{50}}$” and “$\mathrm{AP_{75}}$” each represent
the results of measurements with the IoU threshold set to 0.5 and 0.75,
respectively.

###  4.5 Statistics Details

Statistical hypothesis testing was performed using the statistical functions
of the SciPy library in Python using 70 test images. The two-sided paired
t-tests were used to compare the performances of models at the $P<10^{-4}$
significance level.

##  5 Data Availability

The metalens and ground truth image dataset can be accessed from the GitHub
repository at https://github.com/yhy258/EIDL_DRMI and Figshare repository at
https://doi.org/10.6084/m9.figshare.24634740.v1 [60]. USAF images taken with
the metalens, reconstructed with the baseline model, and reconstructed with
DRMI are also available in the GitHub repository. The test set of PASCAL
VOC2007 dataset can be found at
http://host.robots.ox.ac.uk/pascal/VOC/voc2007/.

##  6 Code Availability

The code in this study is available in the GitHub repository at
https://github.com/yhy258/EIDL_DRMI. The pre-trained models are also available
in the same repository. We used the publicly available SSD code
(https://github.com/amdegroot/ssd.pytorch) for object detection using the
PyTorch library.

## References

  * [1] Shuming Wang et al.  “A broadband achromatic metalens in the visible”  In _Nature Nanotechnology_ 13.3 Nature Publishing Group UK London, 2018, pp. 227–232
  * [2] Wei Ting Chen et al.  “A broadband achromatic metalens for focusing and imaging in the visible”  In _Nature Nanotechnology_ 13.3 Nature Publishing Group UK London, 2018, pp. 220–226
  * [3] Chun-Yuan Fan, Chia-Ping Lin and Guo-Dung J Su  “Ultrawide-angle and high-efficiency metalens in hexagonal arrangement”  In _Scientific Reports_ 10.1 Nature Publishing Group UK London, 2020, pp. 15677
  * [4] Fan Yang et al.  “Wide field-of-view metalens: a tutorial”  In _Advanced Photonics_ 5.3 Society of Photo-Optical Instrumentation Engineers, 2023, pp. 033001
  * [5] Mohammadreza Khorasaninejad et al.  “Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging”  In _Science_ 352.6290 American Association for the Advancement of Science, 2016, pp. 1190–1194
  * [6] Sajan Shrestha et al.  “Broadband achromatic dielectric metalenses”  In _Light: Science & Applications_ 7.1 Nature Publishing Group, 2018, pp. 85
  * [7] Joohoon Kim et al.  “One-step printable platform for high-efficiency metasurfaces down to the deep-ultraviolet region”  In _Light: Science & Applications_ 12.1 Nature Publishing Group UK London, 2023, pp. 68
  * [8] Joohoon Kim et al.  “Metasurface holography reaching the highest efficiency limit in the visible via one-step nanoparticle-embedded-resin printing”  In _Laser & Photonics Reviews_ 16.8 Wiley Online Library, 2022, pp. 2200098
  * [9] Joohoon Kim et al.  “Scalable manufacturing of high-index atomic layer–polymer hybrid metasurfaces for metaphotonics in the visible”  In _Nature Materials_ 22.4 Nature Publishing Group UK London, 2023, pp. 474–481
  * [10] You Zhou, Hanyu Zheng, Ivan I Kravchenko and Jason Valentine  “Flat optics for image differentiation”  In _Nature Photonics_ 14.5 Nature Publishing Group UK London, 2020, pp. 316–323
  * [11] Haowen Liang et al.  “Ultrahigh numerical aperture metalens at visible wavelengths”  In _Nano Letters_ 18.7 ACS Publications, 2018, pp. 4460–4466
  * [12] Mikhail Y Shalaginov et al.  “Single-element diffraction-limited fisheye metalens”  In _Nano Letters_ 20.10 ACS Publications, 2020, pp. 7429–7437
  * [13] Augusto Martins et al.  “On metalenses with arbitrarily wide field of view”  In _ACS Photonics_ 7.8 ACS Publications, 2020, pp. 2073–2079
  * [14] Feilou Wang et al.  “Visible achromatic metalens design based on artificial neural network”  In _Advanced Optical Materials_ 10.3 Wiley Online Library, 2022, pp. 2101842
  * [15] Mu Ku Chen et al.  “Meta-lens in the sky”  In _IEEE Access_ 10 IEEE, 2022, pp. 46552–46557
  * [16] Sun-Je Kim et al.  “Dielectric metalens: properties and three-dimensional imaging applications”  In _Sensors_ 21.13 MDPI, 2021, pp. 4584
  * [17] Zhaoyi Li et al.  “Meta-optics achieves RGB-achromatic focusing for virtual reality”  In _Science Advances_ 7.5 American Association for the Advancement of Science, 2021, pp. eabe4458
  * [18] Zhaoyi Li et al.  “Inverse design enables large-scale high-performance meta-optics reshaping virtual reality”  In _Nature Communications_ 13.1 Nature Publishing Group UK London, 2022, pp. 2409
  * [19] Owen Miller and Zeyu Kuang  “Fundamental limits for large-area meta-optics”  In _Freeform Optics_ , 2021, pp. JTu6A.6  Optica Publishing Group
  * [20] Seyedeh Mahsa Kamali, Ehsan Arbabi, Amir Arbabi and Andrei Faraon  “A review of dielectric optical metasurfaces for wavefront control”  In _Nanophotonics_ 7.6 De Gruyter, 2018, pp. 1041–1068
  * [21] Sunae So, Jungho Mun, Junghyun Park and Junsuk Rho  “Revisiting the design strategies for metasurfaces: Fundamental physics, optimization, and beyond”  In _Advanced Materials_ Wiley Online Library, 2022, pp. 2206399
  * [22] Federico Presutti and Francesco Monticone  “Focusing on bandwidth: achromatic metalens limits”  In _Optica_ 7.6 Optica Publishing Group, 2020, pp. 624–631
  * [23] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang and Jian Sun  “Simple baselines for image restoration”  In _European Conference on Computer Vision_ , 2022, pp. 17–33  Springer
  * [24] Syed Waqas Zamir et al.  “Restormer: Efficient transformer for high-resolution image restoration”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5728–5739
  * [25] Jacob Engelberg and Uriel Levy  “Optimizing the spectral range of diffractive metalenses for polychromatic imaging applications”  In _Optics Express_ 25.18 Optica Publishing Group, 2017, pp. 21637–21651
  * [26] Haejun Chung and Owen D Miller  “High-NA achromatic metalenses by inverse design”  In _Optics Express_ 28.5 Optica Publishing Group, 2020, pp. 6945–6965
  * [27] Zin Lin and Steven G Johnson  “Overlapping domains for topology optimization of large-area metasurfaces”  In _Optics express_ 27.22 Optica Publishing Group, 2019, pp. 32445–32453
  * [28] Luocheng Huang, James Whitehead, Shane Colburn and Arka Majumdar  “Design and analysis of extended depth of focus metalenses for achromatic computational imaging”  In _Photonics Research_ 8.10 Optica Publishing Group, 2020, pp. 1613–1623
  * [29] Mohammadreza Khorasaninejad et al.  “Achromatic metalens over 60 nm bandwidth in the visible and metalens with reverse chromatic dispersion”  In _Nano Letters_ 17.3 ACS Publications, 2017, pp. 1819–1824
  * [30] Joseph W Goodman  “Introduction to Fourier optics”  RobertsCompany publishers, 2005
  * [31] Yuqian Zhou et al.  “Image restoration for under-display camera”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9179–9188
  * [32] Fabin Shen and Anbo Wang  “Fast-Fourier-transform based numerical integration method for the Rayleigh-Sommerfeld diffraction formula”  In _Applied Optics_ 45.6 Optica Publishing Group, 2006, pp. 1102–1110
  * [33] Yawei Li et al.  “NTIRE 2023 challenge on image denoising: Methods and results”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1904–1920
  * [34] Seungjun Nah et al.  “NTIRE 2021 challenge on image deblurring”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 149–165
  * [35] J Scott Goldstein, Irving S Reed and Louis L Scharf  “A multistage representation of the Wiener filter based on orthogonal projections”  In _IEEE Transactions on Information Theory_ 44.7 IEEE, 1998, pp. 2943–2959
  * [36] Michael T Heath  “Scientific computing: an introductory survey”  SIAM, 2018
  * [37] Ren Yang et al.  “NTIRE 2022 challenge on super-resolution and quality enhancement of compressed video: Dataset, methods and results”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1221–1238
  * [38] Wenhui Wu et al.  “URetinex-Net: Retinex-based deep unfolding network for low-light image enhancement”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5901–5910
  * [39] Xiu Li et al.  “Universal and flexible optical aberration correction using deep-prior based deconvolution”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2613–2621
  * [40] Qi Jiang et al.  “Annular computational imaging: Capture clear panoramic images through simple lens”  In _IEEE Transactions on Computational Imaging_ 8 IEEE, 2022, pp. 1250–1264
  * [41] Eirikur Agustsson and Radu Timofte  “NTIRE 2017 challenge on single image super-resolution: Dataset and study”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition workshops_ , 2017, pp. 126–135
  * [42] Seungjun Nah, Tae Hyun Kim and Kyoung Mu Lee  “Deep multi-scale convolutional neural network for dynamic scene deblurring”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3883–3891
  * [43] Richard Zhang et al.  “The unreasonable effectiveness of deep features as a perceptual metric”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 586–595
  * [44] Yuning Cui et al.  “Selective Frequency Network for Image Restoration”  In _The Eleventh International Conference on Learning Representations_ , 2022
  * [45] Syed Waqas Zamir et al.  “Learning enriched features for fast image restoration and enhancement”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45.2 IEEE, 2022, pp. 1934–1948
  * [46] Liangyu Chen et al.  “HINet: Half Instance Normalization network for image restoration”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 182–192
  * [47] Alan V Oppenheim and Jae S Lim  “The importance of phase in signals”  In _Proceedings of the IEEE_ 69.5 IEEE, 1981, pp. 529–541
  * [48] Joonyoung Chang, Hee Kang and Moon Gi Kang  “Correction of axial and lateral chromatic aberration with false color filtering”  In _IEEE Transactions on Image Processing_ 22.3 IEEE, 2012, pp. 1186–1198
  * [49] Mark Everingham et al.  “The PASCAL visual object classes (VOC) challenge”  In _International Journal of Computer Vision_ 88 Springer, 2010, pp. 303–338
  * [50] Wei Liu et al.  “SSD: Single shot multibox detector”  In _European conference on computer vision_ , 2016, pp. 21–37  Springer
  * [51] Rajath Sawant et al.  “Aberration-corrected large-scale hybrid metalenses”  In _Optica_ 8.11 Optica Publishing Group, 2021, pp. 1405–1411
  * [52] Gwanho Yoon et al.  “Printable nanocomposite metalens for high-contrast near-infrared imaging”  In _ACS Nano_ 15.1 ACS Publications, 2021, pp. 698–706
  * [53] M Khorasaninejad et al.  “Multispectral chiral imaging with a metalens”  In _Nano Letters_ 16.7 ACS Publications, 2016, pp. 4595–4600
  * [54] Beibei Xu et al.  “Metalens-integrated compact imaging devices for wide-field microscopy”  In _Advanced Photonics_ 2.6 Society of Photo-Optical Instrumentation Engineers, 2020, pp. 066004
  * [55] Xue Jiang et al.  “All-dielectric metalens for terahertz wave imaging”  In _Optics Express_ 26.11 Optica Publishing Group, 2018, pp. 14132–14142
  * [56] Xiaojie Chu, Liangyu Chen, Chengpeng Chen and Xin Lu  “Improving image restoration by revisiting global information aggregation”  In _European Conference on Computer Vision_ , 2022, pp. 53–71  Springer
  * [57] Lars Ruthotto and Eldad Haber  “An introduction to deep generative modeling”  In _GAMM-Mitteilungen_ 44.2 Wiley Online Library, 2021, pp. e202100008
  * [58] Takeru Miyato, Toshiki Kataoka, Masanori Koyama and Yuichi Yoshida  “Spectral Normalization for Generative Adversarial Networks”  In _International Conference on Learning Representations_ , 2018
  * [59] Jae Hyun Lim and Jong Chul Ye  “Geometric GAN”  In _arXiv preprint arXiv:1705.02894_ , 2017
  * [60] Joonhyuk Seo  “Dataset for DRMI”, 2023  DOI: 10.6084/m9.figshare.24634740.v1