# Toward Flare-Free Images: A Survey

Yousef Kotp, Marwan Torki
Computer and Systems Engineering Department, Alexandria University, Egypt.

###### Abstract

Lens flare is a common image artifact that can significantly degrade image
quality and affect the performance of computer vision systems due to a strong
light source pointing at the camera. This survey provides a comprehensive
overview of the multifaceted domain of lens flare, encompassing its underlying
physics, influencing factors, types, and characteristics. It delves into the
complex optics of flare formation, arising from factors like internal
reflection, scattering, diffraction, and dispersion within the camera lens
system. The diverse categories of flare are explored, including scattering,
reflective, glare, orb, and starburst types. Key properties such as shape,
color, and localization are analyzed. The numerous factors impacting flare
appearance are discussed, spanning light source attributes, lens features,
camera settings, and scene content. The survey extensively covers the wide
range of methods proposed for flare removal, including hardware optimization
strategies, classical image processing techniques, and learning-based methods
using deep learning. It not only describes pioneering flare datasets created
for training and evaluation purposes but also how they were created. Commonly
employed performance metrics such as PSNR, SSIM, and LPIPS are explored.
Challenges posed by flare’s complex and data-dependent characteristics are
highlighted. The survey provides insights into best practices, limitations,
and promising future directions for flare removal research. Reviewing the
state-of-the-art enables an in-depth understanding of the inherent
complexities of the flare phenomenon and the capabilities of existing
solutions. This can inform and inspire new innovations for handling lens flare
artifacts and improving visual quality across various applications.

###### Index Terms:

Flare Removal, Flare Removal Survey, Single Image Flare Removal, Lens Flare
Removal, Glare, low-level computer vision, Image Restoration.

##  I Introduction

LENS flare [1, 2, 3], also known as image flare is a common problem that
occurs when a camera lens is pointed at a strong light source. It can manifest
as ghosting, blooming, or other artifacts that can degrade the image quality.
It poses a significant challenge in both photography and computer vision
domains. It occurs when intense light sources, such as the sun or powerful
artificial lights, directly enter a camera’s lens. Instead of faithfully
capturing the intended scene, the camera ends up with unwanted artifacts in
the form of blurry and multicolored spots or streaks scattered across the
image. These artifacts not only obscure the image’s content but also degrade
its overall visual quality. The presence of lens flare can be particularly
problematic for computer vision systems [4, 5], as these unexpected
aberrations can fool algorithms designed for image analysis and
interpretation. Consequently, addressing lens flare has become a crucial
preprocessing step in computer vision applications, paving the way for more
accurate and reliable results in tasks such as stereo matching [6, 7], optical
flow estimation [8, 9, 10], semantic segmentation [11, 12, 13, 14], and object
detection [15, 16, 17, 18]. The survey’s contributions can be summarized as
follows:

Figure 1: Examples for flare-corrupted images.

(a) Scattering Flare

(b) Reflective Flare

(c) Veiling Glare

(d) Lens Starburst

(e) Lens Orbs

Figure 2: Example for different types of flares. In (a), we can see that
scattering flare manifests in the form of a line or radial pattern emerging
from the light source, we can also deduce that the effect of scattering flare
can be magnified during nighttime. (b) shows the effect of reflective flare in
the image which is usually an aperture-shaped polygon, its color can also vary
depending on the type and color of lens coating which was used. From (c), we
can notice what glare represents, which is induced by excessive and
uncontrolled brightness or intensity of light that can cause discomfort and
reduced visibility. It occurs when there is a significant contrast between a
bright light source and the surrounding environment, often resulting in a
dazzling or blinding effect. It is worth noting that scattering flare usually
includes glare around the light source. In (d), we can clearly notice the
star-shaped pattern around the light source, hence the name starburst. (e)
represents lens orbs, which manifest in the form of ghost-like shapes within
the image. These ghostly manifestations often depend on particulate matter
within the lens optical system. They can introduce dreamy quality to the
photograph depending on the scene context.

  * •

We provide a comprehensive analysis of the physics, optics, and factors behind
lens flare formation. In addition to this, we categorize different flare types
according to their patterns.

  * •

We discuss different challenges posed by the complex, data-dependent nature of
lens flares that make them difficult to model and remove effectively.

  * •

We extensively review different methods for lens flare removal, covering
hardware-based strategies, classical image processing techniques, and
learning-based methods using deep neural networks.

  * •

We review pioneering flare datasets and how they were created for training and
evaluation purposes.

  * •

We provide a comparative summary of the performance of the reviewed flare
removal methods on popular benchmarks.

  * •

We highlight promising future research directions such as flare removal in
videos, real-time processing, improved datasets, modular approaches, daytime-
to-nighttime translation, and multi-stage methods.

The paper is structured as follows: Section II delves into the physics of lens
flare, covering its causes, types, and influencing factors. Section III
explores hardware-based approaches for flare mitigation, while Section IV
investigates computational methods for flare removal. In section V, state-of-
the-art learning-based approaches using deep learning are discussed. Section
VI details the creation of flare removal datasets, and section VII provides an
extensive quantitative analysis of the discussed methods. Finally, in section
VIII, current limitations and future research directions in the field of flare
removal are addressed.

Figure 3: Causes of two main types of flare: scattering and reflective.
Scattering flare happens due to the dispersion of light within the camera’s
optical system, often resulting from refractions of incoming light to
unintended positions on the camera sensor. On the other hand, reflective flare
occurs when intense light sources, such as the sun or bright artificial
lights, directly illuminate the camera lens, causing reflections at various
lens elements. It can also happen due to scratching the lens surface. Those
internal reflections result in unwanted polygon-shaped artifacts. It is worth
noting that this figure contains only one lens for simplicity, in a typical
camera system, multiple lenses are commonly employed. From [19].

##  II Physics of Lens Flare

When you first capture an image [20, 21], light from the scene travels through
the camera’s lens system and eventually reaches the camera’s image sensor. The
primary function of the camera lens is to bend and focus incoming light onto
the image sensor to create a sharp and well-defined image. However, lens flare
occurs when certain conditions are met during this light propagation. In
reality, actual camera lenses tend to disperse and bounce light in unintended
directions, giving rise to unwanted flare artifacts. You can see an example of
these flare artifacts in Figure 1. The portions of light that get scattered
represent only a small fraction of the total incoming light rays.
Consequently, in most photos, this flare is so minor that it goes unnoticed.
However, things change when a powerful light source, like the sun, is
significantly brighter than the rest of the scene being captured. In such
cases, even the small portion of light that gets scattered by this intense
source can create visible artifacts on other parts of the image. The
particular patterns created by the scattering of light due to dust and
scratches result in distinct visual characteristics. The detailed factors
contributing to the flare are discussed in section II-B.

One of the other fundamental physical principles behind lens flare is the
phenomenon of internal reflection within the camera lens. A lens consists of
multiple optical elements, including curved glass or plastic surfaces. Some of
the light rays are inevitably scattered and reflected within the lens
elements. This happens due to many factors as the incident angle at which the
intense light enters the camera lens is a crucial factor. Light rays can
bounce back and forth between the lens elements, creating a series of
reflections and scattering. In addition to this, the design of the lens
including the number and arrangement of lens elements, can influence the
likelihood of lens flare. Complex lens designs with multiple elements may be
more prone to internal reflections.

The shape and position of the aperture blades within the lens can also affect
the appearance of lens flare. The aperture creates multiple points of
reflection for incoming light, potentially resulting in polygonal or
starburst-like flare patterns. External factors such as humidity and airborne
particles in the atmosphere can scatter and diffract light too, adding to the
complexity of the flare pattern. These factors can contribute to the colorful
and unpredictable nature of lens flare. The lens flare artifacts that we
observe in images are a consequence of these internal reflections and
scattering processes. The reflections and scattering create bright spots,
streaks, polygons, or other patterns in the image, which can obscure the
original scene’s details and reduce image contrast.

Figure 3 demonstrates a perfect example of how flare can occur in photography
[19]. Flare arises from two primary causes as discussed previously. One is due
to light dispersion within the camera’s optical system, typically resulting
from the refraction of incoming light to unintended positions on the camera
sensor (scattering flare). This is exemplified by the orange light ray in the
figure. The other cause is when strong light sources, such as the sun or
bright artificial lights, directly illuminate the camera lens, leading to
reflections at various lens elements (reflective flare), as shown by the
yellow light ray. Those internal reflections produce undesirable polygon-
shaped artifacts.

###  II-A Flare Types

Lens flares manifest in images in a variety of shapes and patterns based on
the conditions of light interaction within the lens system as mentioned in the
previous subsection. They can be broadly categorized into the following types
as shown by Figure 2:

Scattering Flare: [22, 23, 24, 25] These appear as radiating streaks spreading
outward from bright light sources. They are caused by light scattering off
defects, scratches, or dirt particles on the lens surface. The streak patterns
tend to remain fixed relative to the light source when the camera moves.

Reflective Flare [26, 27, 28, 29] These are caused by internal reflections
between the lens elements. They often appear as a string of polygonal shapes
such as circles, ovals, or hexagons. In contrast to scattering flares, these
shapes tend to move in the direction opposite to the direction of light source
movement.

Veiling Glare: [30, 24] This causes an overall foggy or hazy region around
bright light sources. It is caused by the forward scattering of light within
the lens system and leads to reduced local image contrast. Most of the
published research considers glare a part of the scattering flare.

Lens Starburst: [31] These manifest as bright streaks radiating from light
sources forming a star-like pattern. They are caused by diffraction and
scattering from the aperture blades. Since they exhibit nearly the same streak
pattern as scattering flare, researchers consider lens starburst a part of the
scattering flare.

Lens Orbs: These appear as out-of-focus blobs that do not move with camera
motion. They are mainly caused by particulate matter within the lens system.

Due to the fixed location of lens orbs, they can be removed easily, that’s why
nearly all of the papers that are published nowadays address only two types of
flare: scattering and reflective. Researchers also implicitly consider both
glare and lens starbursts as part of the scattering flare, as mentioned
previously.

###  II-B Factors Affecting The Flare

Several interconnected factors can contribute to the occurrence and
characteristics of lens flare. Some of these are related to the camera
capturing the scene, and others are related to the environment regardless of
the camera. We can mainly narrow the factors affecting the flare to the
following:

Light Source Properties: The intensity, color, shape, and positioning of the
light source impacts flare patterns. Brighter light causes more prominent
flares. Colored lights lead to tinted flare streaks. The shape and orientation
of extended sources like neon signs determine flare streak geometry.

Lens Properties: The number of lens elements, their curvature, spacing, and
anti-reflective treatments influence internal reflection paths that cause
flares. These treatments are explained in Section III. Also, the lens aperture
shape drives patterns like polygonal shapes. Lens defects, scratches, and dust
lead to scattering artifacts.

Camera Settings: Exposure time, aperture size, and ISO sensitivity affect
flare prominence. Longer exposures make flares more noticeable. Higher
f-numbers increase diffraction causing starburst patterns.

Scene Content: Flare visibility depends on the scene. Flares are more obvious
against dark regions as in nighttime photography compared to textured
backgrounds. The presence of other bright regions can mask flare patterns.

Shooting Conditions: Camera angle relative to the light source and distance
impacts flare shapes and locations. Environmental factors like humidity affect
dust buildup on lenses.

##  III Hardware-Based Approaches

Mitigating flare through hardware optimization aims to prevent flare by
reducing the need for post-processing. It involves the use of physical
components and mechanisms within the camera and lens system to reduce or
eliminate the unwanted effects of lens flare. The following are some hardware-
based methods to reduce the flare.

###  III-A Anti Reflective Coating

Figure 4: Light passing through both mediums ends up being reflected twice.
First when it hits the surface of the AR coat, and second when it hits the
surface of the object. Thus we have two distinct rays R1 and R2 traveling in
the same direction away from the object. Interference from two similar waves
traveling in the same direction is a regular wave – that depends solely on the
phase difference of the two waves. The phase difference is caused by the fact
that ray R1 travels a distance shorter by 2w than ray R2 where w is the width
of the coating layer. Different phase differences produce different merged
waves. When the two waves are separated by half the wavelength. The resultant
wave has zero amplitude. At every point, the two waves cancel each other.
Therefore, if we give the lens a coat of transparent paint of a width of a
quarter of the wavelength of the light, then R2 will be out of phase by half a
wavelength and therefore produce destructive interference with R1. From [32].
Figure 5: Destructive interference is produced when 2 waves are out of phase
by half a wavelength.

Anti-reflective coating [33], often referred to as AR coating, is a thin layer
of special material applied to the surfaces of optical components like lenses
and camera filters to reduce unwanted reflections and glare. The purpose of AR
coating is to enhance the transmission of light through the lens or filter
while minimizing the amount of light that is bounced back. With nothing added
to the surface of a glass lens, about 4% of the light hitting it gets
reflected back and is lost due to differences in the refractive indices
between the air and the material of the surface. This reflected light can
cause flare and reduce the overall image quality. A camera lens with only a
single element has two air-glass surfaces, so the reflection losses double.
Modern lenses often have up to 20 elements in them, so uncoated surfaces would
result in huge light losses.

The mechanism behind AR coatings involves the principle of destructive
interference [32] as shown in Figure 5. By applying a carefully calculated and
precisely engineered thin layer of material onto the lens surface, the coating
thickness is adjusted to create a phase difference in the reflected light
waves. When the reflected light waves interact with each other, they cancel
out and reduce the overall reflection. The physics behind AR coating is
explained clearly in Figure 4.

This results in improved image contrast allowing for better differentiation
between light and dark areas. In addition to this, it reduces glare and
reflections which means that more of the intended light reaches the imaging
sensor, resulting in sharper and clearer images. In low-light situations, AR-
coated lenses can capture more of the available light, allowing for better
performance without introducing excessive glare. The wavelength in visible
light is around 500nm, and lens coatings are typically 100nm to 250nm thin
layers. To put this into perspective, an average human hair is about a
thousand times thicker. However, the thickness of this coating can only be
optimized for particular wavelengths and angles of incidence and therefore
cannot be perfect. Additionally, adding an AR coating to all optical surfaces
is expensive, and may interfere with other coatings as anti-scratch and anti-
fingerprint.

###  III-B Lens Hoods

Figure 6: Illustration of how lens hoods help in blocking unwanted light rays
that are causing flare artifact.

A lens hood is a simple but essential accessory for cameras and lenses,
designed to enhance the quality of photographs. It is a tubular or petal-
shaped attachment that is mounted on the front of a camera lens. It blocks and
shades the lens from unwanted light sources to capture the intended scene.

A lens hood works by creating a physical barrier around the front of the lens.
This barrier prevents light from entering the lens at extreme angles or from
off-axis sources. Essentially, it shades the lens from these unwanted light
rays as shown in Figure 6. By doing so, the lens hood helps maintain the
intended exposure, enhances color accuracy, and minimizes the risk of lens
flare by improving contrast, and reducing glare. Lens hood design is carefully
crafted to match the lens’s field of view and focal length. Some lens hoods
have a round, cylindrical shape, while others have a more distinct petal
shape, depending on the lens’s aspect ratio and the potential for vignetting
(darkening of the corners) in the image. The shape and length of the hood are
optimized to effectively block out unwanted light while avoiding any
interference with the image frame.

Lens hoods are especially useful in outdoor photography, where you often
encounter strong and directional sunlight. They also prove beneficial when
shooting in scenarios with artificial light sources. However, it’s important
to note that not all situations require a lens hood. In some low-light or
controlled lighting conditions, using a lens hood may not be necessary and
could even become impractical.

###  III-C Limitations

Hardware-based approaches for mitigating flare, especially in optical systems
like cameras and lenses, come with several limitations. Firstly, they can
significantly increase the cost of optical systems, which might not be
practical for consumer-grade devices. Additionally, the complexity of
implementing these solutions, such as specialized lens coatings or the
addition of baffles and shades, can make optical systems bulkier and more
challenging to manufacture.

Once integrated into a device, these hardware solutions lack flexibility,
making it difficult to adapt to changing environmental conditions or different
photography scenarios. Furthermore, while aiming to reduce flare, hardware-
based solutions may inadvertently affect overall image quality. Striking the
right balance between flare reduction and image quality can be a challenge.
Moreover, some hardware solutions are specific to certain types of flare and
may not be effective against all forms of flare, limiting their applicability.
Maintenance and durability are also concerns, as components like lens coatings
can wear off over time or be damaged. Therefore, designers and engineers need
to carefully consider these limitations and weigh them against the benefits
when choosing flare mitigation strategies.

##  IV Computational-Based Approaches

Unlike hardware solutions that often come with limitations and can add
complexity and cost, computational methods can be fine-tuned and customized to
specific scenarios and environmental conditions. They offer the ability to
detect and mitigate flare, making them suitable for dynamic situations.
Additionally, computational approaches are implemented in software, reducing
the need for additional hardware components and allowing for easy updates and
adjustments. The following are some of the computational-based approaches for
mitigating the flare effect.

###  IV-A Deconvolution

Deconvolution operation has been used to remove flare, mainly in X-ray imaging
[34]. Seibert et al. [3] assumed that the removal of veiling glare can be done
by knowing the point intensifier spread function (PSF) and applying
mathematical deconvolution. Parameters of PSF are represented by $\rho$ and
$k$. The parameter $\rho$ indicates the fraction of light strongly scattered
in the image intensifier, and $k$ is a measure of the mean travel distance of
the scattered photons. Both of those parameters can be determined from a
least-square fit of measured contrast ratios versus lead disk diameter from
acquired images. However, the derivation and parameterization of the veiling
glare PSF in this case is based on the assumption of circular symmetry and
spatial invariance of the image intensifier PSF, which is not true for most
cases of the flare. It has been shown by experimental results [35] that the
analytical form of the PSF is given as

$h(r)=(1-\rho)\frac{\delta(r)}{r}+\frac{\rho}{2kr}e^{-r/k}$ (1)

Values of $\rho$ and $k$ can be experimentally determined for the system as
discussed before. $\delta(r)$ in the above equation is the Dirac delta
function representing a direct mapping term (no veiling glare) which was
presented by Paul Dirac in a 1927 paper [36]. When those parameters are
substituted into Equation 1, they give the system PSF as a function of radial
distance where the relation between undergraded image U(r,$\theta$) and the
actual detected image V(r,$\theta$) is given by

$V(r,\theta)=[h**U]_{r,\theta}$ (2)

Where $**$ represents two-dimensional convolution in the spatial domain, which
can be represented in the frequency domain

$F_{2}[V]=F_{2}[U]F_{2}[h]$ (3)

Where $F_{2}$ is a 2D Fast Fourier Transform (FFT) operation [37, 38, 39]. By
solving for $F_{2}[U]$ in Equation 3 and inverse transforming the result

$U=F^{-1}_{2}\\{F_{2}[V](\frac{1}{F_{2}[h]})\\}$ (4)

Where $F^{-1}_{2}$ is Inverse Fourier Transform. Transforming the PSF is
analytically performed by taking advantage of the circular symmetry using the
zero-order Bessel function

$\hat{H}(f)=2\pi\int_{0}^{\infty}h(r)J_{o}(2\pi rf)rdr$ (5)

Where $\hat{H}$ is the frequency domain representation of h, meaning that:

$\hat{H}(f)=F_{2}[h]$ (6)

Substituting Equation 1 into Equation 5, followed by integration and solution,
the results are in the frequency domain PSF

$\hat{H}(f)=\pi\frac{\rho}{\sqrt{1+(2\pi kf)^{2}}}+(1-\rho)$ (7)

The inverse frequency filter is obtained by inverting the previous equation

$\hat{H}^{-1}(f)=\frac{1}{\pi}\\{\frac{\sqrt{1+(2\pi
kf)^{2}}}{\rho+(1-\rho)\sqrt{1+(2\pi kf)^{2}}}\\}$ (8)

By multiplying the filter on a point-by-point basis with the frequency domain
degraded image, it gives us a product having lower frequencies attenuated. An
inverse 2D FFT and image scaling factors are applied to produce an approximate
estimate of the original undergraded image in the spatial domain.

###  IV-B Automated Lens Flare Removal

Algorithm 1 Exemplar-based Inpainting [40].

Extract the manually selected initial front $\delta\ohm^{0}$

while Not Done do

Identify the fill front $\delta\ohm^{0}$

if $\ohm^{t}=\phi$ then

EXIT

end if

Computer priorities $P(p)$ where $\forall p\in\delta\ohm^{t}$

Find the patch $\psi_{\hat{p}}$ with the maximum priority
($\hat{p}=arg\,max_{p\in\delta\ohm^{t}}P(p)$)

Find the exemplar $\psi_{\hat{q}}\in\phi$ that minimizes
$d(\psi_{\hat{p}},\psi_{\hat{q}})$

Copy image data from $\psi_{\hat{q}}$ to $\psi_{\hat{p}}$ $\forall
p\in\psi_{\hat{p}}\cap\ohm$

Update C(p) $\forall p\in\psi_{\hat{p}}\cap\ohm$

end while

Floris Chabert [26] introduced automated detection of the flares using a
single input image, particularly reflective flare. This involved a custom blob
detection algorithm based on a concept used in OpenCV [41] tuned for the
specific lens flare that we want to detect and a hybrid inpainting method
called exemplar-based inpainting [40].

The first stage, which is the detection algorithm, includes 5 steps:

Multiple Thresholding: The image is converted to grayscale and binarized using
a range of thresholds [42, 43].

Contour Detection: For each binary image, we then find the contours using a
border following the method in [44].

Blob Merging: The center of each blob is then computed and blobs from the
different binary images are merged depending on their distance and similarity.
We finally obtained a set of potential flare candidates.

Flare Candidates Filtering: The flare candidates are pruned using various
metrics whose parameters have been tuned using a set of images so as to be
robust while avoiding false positives. Those metrics include circularity of
the blob, convexity, inertia, and area.

Flare Mask Computation: Finally the mask selecting the flares is computed for
the next step

For the second stage, which is recovery using exemplar-based inpainting whose
pseudocode is shown in Algorithm 1, a window around the flare is selected,
then the following algorithm is executed until all the pixels have been
recovered:

Identify Fill Front: We first find the contour of the region we want to fill.

Identify Priority Patches: Patches on the fill front are assigned priorities
as to privilege patches that continue strong edges and are surrounded by high-
confidence pixels.

Find Best Exemplar: By priority order, we then search the window for known
patches that minimize the error.

Fill Region using Exemplar Patch: We finally select pixels from the best patch
to fill the masked pixels in the current patch to recover.

This approach is only limited to specific types of flare making it hard to
generalize for different type of flare that appears everywhere. In addition to
this, it can mistaken any saturated blob and mark it as a flare.

###  IV-C Limitations

Computational-based approaches may not be suitable for all scenarios,
particularly when dealing with diverse and unpredictable forms of flare. These
methods are often tailored to specific types of flare and may struggle to
address the wide range of potential artifacts and aberrations that can occur
in optical systems since we can’t estimate the luminous intensity, chromatic,
and mechanical properties of each camera. Moreover, computational approaches
can sometimes misinterpret saturated areas in an image as flare, potentially
leading to false positives and unintended alterations. In situations where the
nature of flare is highly variable and not easily characterized, alternative
solutions, such as learning-based approaches, may be more appropriate to
ensure accurate and reliable flare mitigation.

##  V Learning-Based Approaches

###  V-A Overview of Deep Neural Networks Architectures

####  V-A1 Convolutional Neural Network

Figure 7: Architecture of convolutional neural networks. From [45].

Convolutional Neural Networks (CNNs) [46, 47, 48, 49, 50] stand as one of the
most successful and widely adopted architectures within the realm of deep
learning, particularly in the field of computer vision. Their inception traces
back to Fukushima’s groundbreaking work on the Neocognitron [51] in 1980,
which is a self-organizing neural network model for a mechanism of pattern
recognition unaffected by shift in position. This foundational concept paved
the way for the evolution of CNNs, eventually leading to their presence in
various applications. For instance, Waibel et al. [52] introduced CNNs
featuring shared weights among temporal receptive fields and employed
backpropagation training for phoneme recognition. Also the first work on
modern CNN was by Yann LeCun et al. in their paper “Gradient-Based Learning
Applied to Document Recognition” [53] which demonstrated that a CNN model that
aggregates simpler features into progressively more complicated features can
be successfully used for handwritten character recognition [54]. CNNs have
revolutionized the field of computer vision by enabling automated feature
extraction and hierarchical learning, making them indispensable tools for a
wide range of applications.

CNNs typically comprise three primary types of layers as shown in Figure 7,
each serving a distinct purpose. Firstly, there are the convolutional layers
where kernels (or filters) are employed to extract features from the input
data. Subsequently, nonlinear layers come into play, applying activation
functions to the extracted feature maps [55, 56, 57]. This nonlinearity
enables the network to model complex, nonlinear functions effectively.
Secondly, pooling layers [58] contribute by summarizing information within a
small neighborhood of a feature map, often using statistical measures like
mean or max, to reduce the spatial resolution. Lastly, the standard neural
network [59, 60] is employed which is mainly responsible for classification.

One of the distinguishing characteristics of CNNs is their local connectivity,
where each unit within a layer receives weighted inputs from a confined region
known as the receptive field. By stacking layers to form multi-resolution
pyramids, higher-level layers are capable of learning features from
increasingly broader receptive fields. This hierarchical approach aids in
recognizing complex patterns and structures within the data.

One of the key computational advantages offered by CNNs is weight sharing. In
a given layer, all receptive fields share the same set of weights. This
ingenious design results in a considerably lower number of parameters compared
to normal fully-connected neural networks. Noteworthy CNN architectures that
have gained prominence include AlexNet [61], VGGNet [62], ResNet [63],
GoogLeNet [64], MobileNet [65], and DenseNet [66].

####  V-A2 Encoder-Decoder Architecture

Figure 8: Architecture of a simple encoder-decoder model. From [67]

Encoder-decoder models represent a family of architectures designed to map
data from an input domain to an output domain using a two-stage
transformation. This process involves two key components: the encoder and the
decoder as shown in Figure 8. The encoder, denoted as an encoding function,
transforms input data into a compressed, latent-space representation, usually
named ’z’. This serves as a feature vector capturing essential semantic
information. In simpler terms, it extracts meaningful features from the input
that are crucial for predicting the output. On the other hand, the decoder
learns using this representation to predict the output. This enables efficient
information extraction and transfer. These models find extensive application
in tasks like image-to-image translation where you need to map one image from
one domain to another one (e.g., daytime to nighttime). Also, sequence-to-
sequence tasks like machine translation in natural language processing (NLP)
can benefit a lot from this architecture. Badrinarayanan et al. introduced
SegNet [68] which is based on a deep convolutional encoder-decoder
architecture for image segmentation which has proven its accuracy at that
time.

Encoder-decoder models are frequently trained by minimizing a reconstruction
loss, typically denoted as $L(y,\hat{y})$ This loss function quantifies the
dissimilarities between the ground-truth output $y$ and the reconstructed
output $\hat{y}$. Depending on the specific task, the output can take various
forms. For instance, it might represent an enhanced version of an image such
as image deraining [69], image dehazing [70], image denoising [71], or super-
resolution [72]. It can also be a segmentation map that highlights different
regions within an image.

There is a variation for this architecture named Auto-encoders [73] where the
input and output are identical, meaning they aim to reconstruct the input
itself. This makes them particularly useful for tasks like feature learning,
dimensionality reduction [74], and anomaly detection [75] due to their ability
to learn meaningful representations from data.

####  V-A3 Generative Adversarial Networks (GANs)

Figure 9: Architecture of a Generative Adversarial Network. From [76]. Figure
10: Architecture of CycleGAN. From [77].

Generative Adversarial Networks [78], or GANs for short, are a class of deep
learning models used in unsupervised machine learning. They were introduced by
Ian Goodfellow and his colleagues in 2014. GANs consist of two neural
networks, the generator, and the discriminator, that are trained together
through adversarial training as shown in Figure 9. The primary goal of GANs is
to generate new data samples that are similar to a given dataset. GANs have
been used for a wide range of applications, including image generation, style
transfer, and data augmentation.

It can be represented by two networks. The first one is the generator network
that takes a random noise vector $z$ as input and maps it to a data sample $x$
that should resemble the real data distribution.

$x=G(z)$

The second is the discriminator network, on the other hand, it aims to
distinguish between real data samples ($x$) and generated data samples
($G(z)$) where $D(x)$ represents the probability that $x$ is a real data
sample.

The training process involves a minimax game, where the generator aims to
maximize the probability that the discriminator misclassifies the generated
samples, while the discriminator aims to correctly classify real and generated
samples. The objective function for GANs can be mathematically defined as:

$\min_{G}\max_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log
D(x)]+\mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z)))]$ (1)

Where:

  * •

$G$ represents the generator network.

  * •

$D$ represents the discriminator network.

  * •

$x$ represents real data samples.

  * •

$z$ represents random noise vectors sampled from a prior distribution
$p_{z}(z)$.

  * •

$\mathbb{E}$ represents the expectation operator.

  * •

$p_{\text{data}}(x)$ represents the distribution of real data.

In practice, the previous objective function may not provide enough gradient
for effectively training the generator especially initially when D can easily
discriminate fake samples from real ones. Instead of minimizing
$\mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z)))]$, we can maximize
$\mathbb{E}_{z\sim p_{z}(z)}[\log(D(G(z)))]$. So now, the new objective
function is as follows

$\max_{G}\max_{D}\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log
D(x)]+\mathbb{E}_{z\sim p_{z}(z)}[\log(D(G(z)))]$ (2)

There is another variation of GANs which is CycleGAN [79]. It is mainly
designed for unpaired image-to-image translation. It was introduced by Zhu et
al. in 2017. Unlike traditional GANs that require paired training data (e.g.,
the input image and corresponding target image), CycleGAN can learn
translations between two domains using unpaired data. This makes it
particularly useful for tasks like style transfer [80], where there is no one-
to-one correspondence between input and target data.

CycleGAN also uses generator-discriminator architecture. However, as shown in
Figure 10, it uses two generators and two discriminators:

1- Generator $G_{A}$ for Domain $A$. It takes an image from domain $B$ and
generates a corresponding image in domain $A$.

$A=G_{A}(B)$

2- Generator $G_{B}$ for Domain $B$. It takes an image from domain $A$ and
generates a corresponding image in domain $B$.

$B=G_{B}(A)$

3- Discriminator $D_{A}$. It distinguishes between real images from domain $A$
and generated images $G_{A}(B)$. where $D_{A}(X)$ represents the probability
that $X$ is a real image from domain $A$.

4- Discriminator $D_{B}$. It distinguishes between real images from domain $B$
and generated images $G_{B}(A)$ where $D_{B}(X)$ represents the probability
that $X$ is a real image from domain $B$.

The objective functions for CycleGAN involve both adversarial losses and cycle
consistency losses, ensuring that the generated images not only fool the
discriminators but also map back to the original domain correctly.

Cycle-consistency loss for $A\rightarrow B\rightarrow A$:

$L_{\text{cycle}}(A,G_{A},G_{B})=\mathbb{E}_{x\sim
p_{\text{data}}(A)}[||G_{A}(G_{B}(A))-A||_{1}]$ (3)

Cycle-consistency loss for $B\rightarrow A\rightarrow B$:

$L_{\text{cycle}}(B,G_{B},G_{A})=\mathbb{E}_{y\sim
p_{\text{data}}(B)}[||G_{B}(G_{A}(B))-B||_{1}]$ (4)

Adversarial losses for $A$ and $B$ can be defined similarly to the traditional
GAN loss as in Equation 1 or Equation 2.

####  V-A4 U-Net

Figure 11: Architecture of U-Net. From [81]. Figure 12: (a) Encoder-decoder
subnetwork in MPRNet. (b) Illustration of the original resolution block (ORB)
in the original-resolution subnetwork. Each ORB contains multiple channel
attention blocks. GAP represents global average pooling [82]. (c) Cross-stage
feature fusion between stage 1 and stage 2. (d) Cross-stage feature fusion
between stage 2 and the last stage. From [83].

U-Net [84] is a popular neural network architecture widely used in image
processing tasks, especially in image restoration and image segmentation. It
was mainly introduced by Olaf Ronneberger et al. for biomedical image
segmentation. It is known for its effectiveness in these domains due to its
unique design, which resembles the letter ”U”, hence the name U-Net.

In image restoration tasks, like denoising, deblurring, flare removal, or
super-resolution, U-Net operates by taking a noisy or degraded image as input
and aims to produce a clean and enhanced version as output. It accomplishes
this by employing a contracting path (encoder) on one side of the ”U” and an
expansive path (decoder) on the other as shown in Figure 11.

The contracting path is responsible for capturing contextual information. It
involves a series of convolutional layers followed by downsampling operations
(typically max-pooling) that progressively reduce the spatial resolution of
the input image. This path enables the network to learn high-level features
and understand the global context of the image.

The expansive path, on the other hand, is tasked with restoring fine-grained
details. It consists of upsampling layers, which increase the spatial
resolution, and convolutional layers which help refine the output.
Importantly, this path also incorporates skip connections [63], which connect
layers from the contracting path to corresponding layers in the expansive
path. These skip connections are a key feature of U-Net and allow the network
to access detailed information from earlier stages of processing. This helps
in reconstructing complex textures and features that might be lost during
downsampling.

The intuition behind U-Net lies in its ability to capture both high-level
context information and fine-grained details in images. The contracting path
effectively learns to recognize global image features and object boundaries.
The expansive path, with the help of skip connections, then refines the output
while preserving the spatial details. This combination of capturing context
and details is pivotal in achieving accurate pixel-wise operations whether
restoration or segmentation. This is crucial for tasks where preserving the
structure of objects in the image is essential.

####  V-A5 MPRNet

Figure 13: Architecture of multi-stage architecture for progressive image
restoration. Earlier stages employ encoder-decoders to extract multi-scale
contextualized features, while the last stage operates at the original image
resolution to generate spatially accurate outputs. A supervised attention
module is added between every two stages that learn to refine features of one
stage before passing them to the next stage. Dotted pink arrows represent the
cross-stage feature fusion mechanism. From [83]. Figure 14: Supervised
attention module employed by MPRNet. From [83].

Multi-Stage Progressive Restoration Network [83] is a proposed deep learning
network for image restoration that comprises three stages that work
sequentially to progressively enhance images. The first two stages employ
encoder-decoder subnetworks based on the standard U-Net [84], enabling them to
grasp broad contextual information through large receptive fields. MPRNet also
employ channel attention blocks (CABs) [85] to extract features at each scale.
The feature maps at U-Net skip connections are also processed with the CAB.
Finally, instead of using transposed convolution [86] for increasing the
spatial resolution of features in the decoder, bilinear upsampling is used
followed by a convolution layer. This helps reduce checkerboard artifacts [86]
in the output image that often arises due to the transposed convolution.

Figure 15: Architecture of Half Instance Normalization Network [87]. The
encoder of each subnetwork contains Half Instance Normalization Blocks (HIN
Block). For simplicity, only 3 layers of HIN Block are shown in the figure,
and HINet has a total of 5 layers. CSFF and SAM modules are adopted as in
MPRNet [83]. From [87].

It’s important to note that image restoration is a task that demands precise
pixel-to-pixel correspondence between the input and output images. Therefore,
the final stage utilizes a subnetwork that operates at the original input
image resolution, ensuring the preservation of fine details in the output
image.

Rather than simply stacking multiple stages, MPRNet incorporates a supervised
attention module between each pair of consecutive stages as shown in Figure
14. This module plays a crucial role in rescaling the feature maps from the
previous stage based on the guidance of ground-truth images before passing
them to the subsequent stage. Additionally, it introduces a cross-stage
feature fusion mechanism that leverages the intermediate multi-scale
contextualized features from earlier subnetworks to consolidate the
intermediate features of the latter subnetwork. This cross-stage feature
fusion is illustrated in Figure 12 (c) and (d). In order to preserve fine
details from the input image to the output image, the original-resolution
subnetwork (ORSNet) in the last stage is used where it does not employ any
downsampling operation and generates spatially enriched high-resolution
features. It consists of multiple original resolution blocks (ORBs) as shown
in Figure 12 (b), each of which further contains CABs.

While MPRNet comprises multiple stages as shown in Figure 13, it’s worth
noting that each stage has access to the input image. Following the approach
of recent restoration methods [88, 89], they implemented a multi-patch
hierarchy on the input image, splitting it into non-overlapping patches: four
for stage 1, two for stage 2, and retaining the original image for the last
stage.

Figure 16: Architecture of Restormer for high-resolution image restoration.
Restormer consists of a multi-scale hierarchical design incorporating
efficient Transformer blocks. The core modules of the Transformer block are:
(a) multi-Dconv head transposed attention (MDTA) that performs (spatially
enriched) query-key feature interaction across channels rather than the
spatial dimension, and (b) Gated-Dconv feed-forward network (GDFN) that
performs controlled feature transformation, i.e., to allow useful information
to propagate further. From [90].

At any given stage S, instead of directly predicting a fully restored image
$X_{S}$, MPRNet predicts a residual image $R_{S}$. This residual image $R_{S}$
is then added to the degraded input image I to obtain the final restored
image: $X_{S}=I+R_{S}$. We can optimize MPRNet end-to-end using the following
loss function:

$L=\sum_{S=1}^{3}[L_{\text{char}}(X_{S},Y)+\lambda L_{\text{edge}}(X_{S},Y)],$
(1)

Here, Y represents the ground-truth image, and we utilize the Charbonnier loss
[91], denoted as $L_{\text{char}}$, defined as:

$L_{\text{char}}=\sqrt{(X_{S}-Y)^{2}+\epsilon^{2}},$ (2)

For practical reasons, usually $\epsilon$ is set to $10^{-3}$ for all
experiments. Additionally, edge loss is used, $L_{\text{edge}}$, defined as:

$L_{\text{edge}}=\sqrt{(\Delta(X_{S})-\Delta(Y))^{2}+\epsilon^{2}},$ (3)

where $\Delta$ represents the Laplacian operator. The parameter $\lambda$ in
the loss function controls the relative importance of the two loss terms and
is set to 0.05, as in previous work (Multi-Scale Progressive Fusion Network
for Single Image Deraining [92]).

Figure 17: HINet’s Half Instance Normalization Block (HIN Block) and ResBlock
in details. From [87].

####  V-A6 HINet

Half Instance Normalization Network [87] consists of two subnetworks, each of
which is based on U-Net [84] as shown in Figure 15. In the encoder component,
they designed Half Instance Normalization Blocks to extract features in each
scale and double the channels of features when downsampling. In the decoder
component, they used ResBlocks [63] to extract high-level features, and fuse
features from the encoder component to compensate for the loss of information
caused by resampling. As for ResBlock, leaky ReLU [93] was used with a
negative slope equal to 0.2, and batch normalization was removed.

To connect the two subnetworks, the cross-stage feature fusion (CSFF) module
and supervised attention module (SAM) were used, these two modules were
introduced by Zamir et al. [94] and are the same ones used in MPRNet [83]. As
for the CSFF module, a 3 × 3 convolution was used to transform the features
from one stage to the next stage for aggregation, which helps to enrich the
multi-scale features of the next stage. As for SAM, the 1 × 1 convolutions in
the original module were replaced with 3 × 3 convolutions, and bias was added
in each convolution. By introducing SAM, the useful features at the current
stage can propagate to the next stage and the less informative ones will be
suppressed by the attention masks. Then we optimize HINet end-to-end as
follows:

$\text{Loss}=-\sum_{i=1}^{2}\text{PSNR}((R_{i}+X_{i}),Y)$ (1)

Where

  * •

$X_{i}\in\mathbb{R}^{N\times C\times H\times W}$ denote the input of
subnetwork $i$, where $N$ is the batch size of data, $C$ is the number of
channels, and $H$ and $W$ are spatial dimensions.

  * •

$R_{i}\in\mathbb{R}^{N\times C\times H\times W}$ denotes the final prediction
of subnetwork $i$.

  * •

$Y\in\mathbb{R}^{N\times C\times H\times W}$ is the ground truth in each
stage.

Because of variance of small image patches differ a lot among mini-batches and
the different formulations of training and testing [95], Batch Normalization
(BN) [96] is not commonly used in low-level tasks as deraining [97],
deblurring [98], and super-resolution [99]. Instead, Instance Normalization
(IN) [100] is used which keeps the same normalization procedure consistent in
both training and inference. Further, IN re-calibrates the mean and variance
of features without the influence of batch dimension, which can keep more
scale information than BN. IN is used to build Half Instance Normalization
Block (HIN block). By introducing the HIN block, the modeling capacity of
HINet is improved. Moreover, the extra parameters and computational costs
introduced by IN can be ignored. The complete structure of the HIN block can
be shown in Figure 17.

Figure 18:  (a) Overview of the Uformer structure. (b) LeWin Transformer
block. (c) Illustration of how the modulators modulate the W-MSAs in each
LeWin Transformer block which is named MW-MSA in (b). From [101].

####  V-A7 Restormer

The Restormer [90] architecture is designed for efficient image restoration
tasks, especially for handling high-resolution images. It is based on
Transformer architecture [102] which was first developed for sequence
processing in natural language tasks. It has been adapted in numerous vision
tasks such as image recognition [103, 89], segmentation [104, 105, 106],
object detection [107, 108, 109]. The Vision Transformers [110, 111, 112]
decompose an image into a sequence of patches (local windows) and learn their
mutual relationships. The distinguishing feature of these models is the strong
capability to learn long-range dependencies between image patch sequences and
adaptability to given input content. Due to these characteristics, Transformer
models have also been studied for low-level vision problems. To handle large-
resolution images, the researchers for Restormer incorporated key innovations
in their multi-scale hierarchical Transformer model.

The Restormer pipeline begins with a degraded input image (I) of dimensions
H×W×3. Initially, a convolutional layer is applied to create low-level feature
embeddings ($F_{0}$) with dimensions H×W×C, where C represents the number of
channels. These shallow features are then processed through a 4-level
symmetric encoder-decoder, resulting in deep features ($F_{d}$) with
dimensions H×W×2C. Each encoder-decoder level contains multiple Transformer
blocks, with the number of blocks increasing from top to bottom levels. This
hierarchical design allows for efficient spatial size reduction and channel
capacity expansion. The decoder takes low-resolution latent features ($F_{l}$)
as input and progressively restores high-resolution representations.

The Restormer model employs pixel-unshuffle and pixel-shuffle operations [113]
for feature downsampling and upsampling, respectively. Skip connections [84]
connect encoder features with decoder features to assist the recovery process.
Furthermore, a 1×1 convolution reduces channels at all levels except the top
one. Transformer blocks aggregate low-level encoder features with high-level
decoder features at level 1, preserving fine structural and textural details
in the restored images. Deep features ($F_{d}$) undergo further refinement at
high spatial resolution, contributing to quality improvements.

In the final stage, a convolutional layer is applied to refined features,
generating a residual image (R) with dimensions H×W×3. This residual image is
added to the degraded input image (I) to obtain the restored image
($\hat{I}=I+R$). The core components of the Restormer Transformer block
include the following as shown in Figure 16:

Multi-Dconv Head Transposed Attention (MDTA): MDTA addresses the computational
complexity of self-attention layers in Transformers [110, 112]. Unlike
conventional self-attention, which has quadratic complexity with input
resolution, MDTA operates linearly by computing cross-covariance across
channels. Depth-wise convolutions emphasize local context before calculating
feature covariance, resulting in a global attention map that implicitly
encodes global context.

Gated-Dconv Feed-Forward Network (GDFN): GDFN enhances representation learning
in the feed-forward network [114]. It incorporates a gating mechanism and
depth-wise convolutions [115]. The gating mechanism involves element-wise
products of linear transformations, one of which employs the GELU non-
linearity [116]. Depth-wise convolutions [115] capture information from
neighboring pixel positions, aiding in learning local image structures for
effective restoration.

Figure 19: Locally-enhanced feed-forward network employed by the Uformer. From
[101]. Figure 20: Wu et al.’s approach includes three steps: 1) Training input
by composing a flare-free image and a flare image. 2) A CNN-based network was
trained to restore the flare-free scene without the light source 3) After
prediction, the input light source was blended back into the output image.
From [117].

Progressive Learning: To address the challenge of training on fixed-size image
patches, Restormer adopts progressive learning. In early epochs, the model
trains on smaller image patches and gradually transitions to larger patches in
later epochs. This strategy allows the network to encode global image
statistics, improving performance on full-resolution images during testing.

####  V-A8 Uformer

Uformer [101] is a hierarchical network designed to address the challenges of
image restoration tasks efficiently. Uformer employs a U-shaped [84, 118]
hierarchical network structure with skip connections between the encoder and
decoder components. Given an input degraded image (I) with dimensions 3×H×W,
Uformer initially applies a 3×3 convolutional layer with LeakyReLU activation
[93] to extract low-level features ($X_{0}$) with dimensions C×H×W. It follows
a U-shaped architecture, containing K encoder stages, each incorporating LeWin
Transformer blocks and a down-sampling layer. These stages hierarchically
reduce spatial size while expanding channel capacity. A bottleneck stage at
the end of the encoder captures longer dependencies due to the hierarchical
structure.

For feature reconstruction, K decoder stages with up-sampling layers and LeWin
Transformer blocks, similar to the encoder, are employed. Transposed
convolution is used for upsampling [86]. Features from the decoder are
concatenated with corresponding encoder features through skip-connections [63]
before input to LeWin Transformer blocks for image restoration. After the K
decoder stages, a 3×3 convolutional layer is applied to produce a residual
image (R) of dimensions 3×H×W. The restored image is obtained by adding $R$ to
the degraded input image ($\hat{I}=I+R$). Uformer is trained using the
Charbonnier loss too [91, 119], which measures the difference between the
ground-truth image ($I$) and the restored image ($\hat{I}$).

Uformer faces two significant challenges when applying the Transformer
architecture to image restoration. Firstly, standard Transformers compute
global self-attention across all tokens, leading to high computational costs
for high-resolution feature maps [102, 110, 111]. Secondly, local context
information is crucial for image restoration, and previous Transformers show
limitations in capturing local dependencies [120, 121]. To address these
issues, Uformer introduces the locally enhanced Window (LeWin) Transformer
block as shown in Figure 19. This block combines self-attention for long-range
dependencies and convolution for local context. The block consists of two core
components: Window-based Multi-head Self-Attention (W-MSA) [108, 122] and
Locally-enhanced Feed-Forward Network (LeFF) [123, 120, 122]. W-MSA performs
self-attention within non-overlapping local windows, significantly reducing
computational costs compared to global self-attention. LeFF includes depth-
wise convolutions [115] to capture local information essential for image
restoration.

Uformer incorporates a multi-scale restoration modulator to enhance its
capability to handle various image perturbations. Different types of image
degradation, such as blur or noise, have distinct patterns requiring specific
restoration approaches. The modulator introduces minimal additional parameters
and computational cost. In each LeWin Transformer block of the decoder, a
modulator is applied as a learnable tensor with a shape of $M\times M\times
C$, where M is the window size, and $C$ is the channel dimension of the
current feature map. These modulators act as shared bias terms added to non-
overlapping windows before the self-attention module. This lightweight
addition allows for flexible feature map adjustments, improving restoration
details. The multi-scale restoration modulator is particularly effective in
tasks like image deblurring [124] and denoising [125], enhancing restoration
results with minimal computational overhead. The complete Uformer architecture
is shown in Figure 18.

###  V-B How to Train Neural Networks for Flare Removal

Wu et al. [117] introduced the first deep learning method to remove flare
using U-Net [84], they modeled the flare as a layer that can be added linearly
on top of flare-free images to generate flare-corrupted images where flare-
free images are sampled from Flickr24K used in [126]. The generation of flare
images is explained in detail in the Section VI.

Figure 21: Dai et al.’s approach in Flare7K++ where paired flare and light
source images are randomly selected from the Flare7K++ dataset. The flare
image will be added to the background image to synthesize the flare-corrupted
image. From [127].

One of the main reasons why flare removal is hard is that if the deep learning
model is trained naively, it tends to darken the image and remove the light
source as a part of the flare which is not the intended function. To avoid
that, they blended the light source with the network’s output before computing
the loss to avoid penalizing the light source region. The binary saturation
mask $M$ for the light source which is blended with the output can be obtained
using thresholding on luminance based on the observation that the flare-
causing light source is likely saturated in the input image. Then by applying
morphological operations, small saturated regions are excluded from the mask.
Any pixel inside the mask $M$ will be replaced with the ground truth $I_{0}$.
So now, the predicted flare-free image $\hat{I}_{0}$ can be computed using the
following equation:

$\hat{I}_{0}=I_{0}\odot M+f(I_{F},\theta)\odot(1-M)$ (1)

Where $f(I_{F},\theta)$ represents the output of the network which is U-Net.

To encourage the predicted flare-free $\hat{I}_{0}$ image to be close to the
ground truth $I_{0}$, they used absolute loss on the RGB values between them.
In addition to that, they used perceptual loss by feeding both images into
pre-trained VGG-19 like in [126]. the image loss can be computed using the
following equation:

$L_{I}=\|\hat{I_{0}}-I_{0}\|_{1}+\sum_{l}\lambda_{l}\|\phi_{l}(\hat{I_{0}})-\phi_{l}(I_{0})\|_{1}$
(2)

They also wanted to encourage the similarity between predicted flare $\hat{F}$
and the ground truth flare $F$. They applied the same loss as in Equation 2
but to the flare ones where $\hat{F}$ can be obtained by calculating the
difference between the network input and the masked network output:

$\hat{F}=I_{F}-f(I_{F},\theta)\odot(1-M)$ (3)

So now, the total loss is

$L=L_{I}+L_{F}$ (4)

For post-processing during inference only, the mask $M$ is feathered to create
a gradual transition at its boundaries to construct $M_{f}$, and then the
network’s input and output and be blended using said feathered mask in linear
space

$I_{B}=I_{F}\odot M_{f}+f(I_{f},\theta)\odot(1-M_{f})$ (5)

where $I_{F}$ is the flare-corrupted image entering the network. The whole
training pipeline can be shown in Figure 20.

###  V-C Flare7K++

Dai et al. [127] introduced a modification to the previous approach. They
addressed the limitations of obtaining a light source mask through
thresholding, which could fail to accurately segment light sources that are
not sufficiently bright. Moreover, in some cases, the streak region might
become overexposed and be mistakenly considered as part of the light source.
To avoid these issues, they trained the network end-to-end to directly predict
a 6-channel output as shown in Figure 21, with the first three channels as a
flare-free image $\hat{I}_{0}$ and the last three channels as a flare image
$\hat{F}$ using the light source annotation which they introduced in Flare7K
dataset [128]. Light source image $I_{L}$ is added to the background images
$I_{B}$ to synthesize the ground truth of flare-free images $I_{0}$. Then, the
difference between the flare image $I_{F}$ and light source image $I_{L}$ is
calculated to get the ground truth flare image $F$. They tried different
network architectures like U-Net [84], MPRNet [83], HINet [87], Restormer
[90], and Uformer [101].

To linearize each image before addition or subtraction, they applied an
inverse gamma correction curve with $\gamma$ sampled from [1.8,2.2]. They
encouraged similarity between predicted flare-free image and the ground truth
using absolute and perceptual losses

$L_{I}=L_{1}(\hat{I}_{0},I_{0})+L_{vgg}(\hat{I}_{0},I_{0})$ (1)

The same loss is applied to the predicted flare and the actual flare to
encourage their similarity too. They also introduced new reconstruction loss
to ensure that the predicted flare-free image and predicted flare can be added
to get the original input

$L_{rec}=|I-Clip(\hat{I}_{0},\hat{F})|$ (2)

where $\oplus$ means the addition operation in the linearized gamma-decoded
domain with the previously sampled $\gamma$. Then, the addition is clipped to
the range of [0,1]. Overall, the final loss function aims to minimize a
weighted sum of all these losses:

$L=w_{1}L_{I}+w_{2}L_{F}+w_{3}L_{rec}$ (3)

where $w_{1}$, $w_{2}$, and $w_{3}$ are respectively set to 0.5, 0.5, and 1.0
in their experiments.

###  V-D Optical Center Symmetry Prior

Figure 22: Dai et al.’s training pipeline in Bracket Flare. From [129].

Dai et al. [129] proposed an optical center symmetry prior, which suggests
that the reflective flare and light source are always symmetrical around the
lens’s optical center. This prior helps to locate the reflective flare’s
proposal region more accurately and can be applied to most smartphone cameras.

As the optical center symmetry prior is a global prior, it is difficult for
CNN-based networks to learn it, and even Transformer-based networks struggle
to learn such symmetry. Therefore, they incorporated this prior by applying a
gamma correction of $I_{g}=I^{\gamma}$ to the input flare-corrupted image with
$\gamma$ = 10. This operation extracts the nearly saturated areas of the light
source in each channel. We then rotate this nearly saturated image by 180
degrees to obtain a prior image, which is concatenated with the flare-
corrupted image to form a six-channel input. This rotated prior image provides
an initial approximation of the reflective flare’s pattern. The entire
pipeline for training the network is shown in Figure 22. Obtaining both long
and short-exposure shots is explained in Section VI.

They adopted different network structures for their experiments, they used the
default network settings of MPRNet, HINet, and Uformer and reduced the feature
channels of Restormer from 48 to 24 due to GPU memory constraint

For the loss functions, they followed the previous work using absolute and
perceptual loss for both the background and the flare image. In addition to
this, they also used reconstruction loss as in Flare7K++ but removed the
clipping, so the reconstruction loss can be expressed as:

$L_{rec}=|I-(\hat{I}_{0}\oplus\hat{F})|$ (1)

Where $\hat{I}_{B}$ and $\hat{I}_{F}$ are the predicted flare-free image and
predicted flare respectively. The main difference from the previous work is
the $L_{mask}$. Since the regions of reflective flares are always small, a
simple $L_{1}$ loss may ignore these regions, leading to the local optimum. To
encourage the network to focus on restoring the flare-corrupted regions, they
calculated the flare’s mask $I_{M}$ from the ground truth of flare image
$I_{F}$. This masked loss can be written as:

$L_{mask}=|I_{M}*(\hat{I}-I)|$ (2)

So now, the background and flare loss can be written as:

$L_{I/F}=w_{1}L_{1}+w_{2}L_{vgg}+w_{3}L_{mask}$ (3)

where $w_{1}$, $w_{2}$, and $w_{3}$ are respectively set to 0.5, 0.1, and
20.0. So now, the total loss is:

$L=L_{I}+L_{F}+L_{rec}$ (4)

##  VI Flare Datasets

Data availability has always been the major concern for most of the ground-
breaking research in the field of flare removal due to the lack of paired
flare-corrupted and flare-free images. Because constructing paired images is a
labor-intensive task and can’t be 100% reliable and accurate due to image
alignment issues, most of the researches focused on synthetic and semi-
synthetic flare generation. This survey not only covers the used datasets but
also how they were created.

(a) Aperture of dirty lens

(b) Simulated Flare

Figure 23: To simulate a scattering flare component, they sampled a number of
apertures with random dots and lines that resemble defects as in (a). Wave
optics then can be used to compute the flare image (b) of any light source
imaged by that synthetic aperture. Figure 24: Wu et al.’s setup for capturing
images of real lens flare using a strong light source in a dark room using a
motorized rotation stage that reproduces a wide range of incident angles. From
[117].

###  VI-A Wu el al

Wu et al. [117] took a major step toward making the flare removal dataset
available where they noticed that the additive nature of light implies that we
can model flare as an additive artifact on top of the flare-free image. So
they tried to synthesize flare using physics-based methods to model and
generate the flare image so that they could add it to flare-free images to
generate flare-corrupted images. For the scattering flare, they assumed that
under the thin-lens approximation, an optical imaging system can be
characterized by the complex-valued pupil function $P(u,v)$: a 2D field
describing, for each point $(u,v)$ on the aperture plane, the lens’s effect on
the amplitude and phase of an incident wave with wavelength $\lambda$:

$P_{\lambda}(u,v)=A(u,v)\cdot\exp(i\phi_{\lambda}(u,v)).$ (1)

Here, $A$ is an aperture function, a property of the optics that represents
its attenuation of the incident wave’s amplitude.222Strictly speaking, lens
optics can also introduce a phase shift, in which case A becomes a complex-
valued function. However, this has shown little difference in their simulation
results, so they assumed A is real-valued. In its simplest form, a camera with
an aperture of a finite radius $r$ has an aperture function of

$A(u,v)=\begin{cases}1&\text{if }u^{2}+v^{2}<r^{2}\\\
0&\text{otherwise}\end{cases}.$ (2)

$\phi_{\lambda}$ in Eq. 1 describes the phase shift, which depends on the
wavelength as well as the 3D location of the light source$(x,y,z)$. Omitting
the aperture coordinates$(u,v)$, $\phi_{\lambda}$ can be written as:

$\phi_{\lambda}(x,y,z)=\phi^{S}_{\lambda}(x/z,y/z)+\phi^{DF}_{\lambda}(z)$ (3)

where the linear term $\phi_{\lambda}^{S}$ is determined by the angle of
incidence, and the defocus term $\phi^{DF}_{\lambda}$ depends on the depth z
of the point light source. Once fully specified, the pupil function P in Eq. 1
can be used to calculate the point spread function (PSF) by a Fourier
transform (F) [2]:

$PSF_{\lambda}=|F\\{P_{\lambda}\\}|^{2}$ (4)

Figure 25: The pipeline for scattering flare synthesis in Flare7K [128]. To
synthesize scattering flare, they first obtained streak section curve
$\alpha$, glare descent curve $\beta$, noise patch near the light source
$\gamma$, and the vanishing corner’s angle $\theta$ around the streak from the
reference image. $\alpha$ and $\theta$ are used to synthesize the glare effect
while $\alpha$ and $\beta$ are used to simulate the streak. To simulate the
degradation around the light source, they added a blurred fractal noise
pattern on the shimmer to create a realistic flare. From [127]. Figure 26: The
pipeline of reflective flare synthesis. Since clipping effects and caustics
are not obvious in a single image, they captured video clips as references.
While synthesizing reflective flares, they first filtered most similar irises
from Optical Flares Plug-in’s iris bank. Then, they manually adjusted the
position, size, and color of these irises to fit the reference. Finally, these
irises are fused to create a reflective flare template in Adobe After Effect.
From [127].

which, by definition, is the image of a point light source at $(x,y,z)$ formed
by a camera with aperture function A. This is the flare image that we desire.

In order to recreate the effect of dust and scratches on a camera lens, they
simulated these imperfections by adding random dots and streaks of varying
sizes and transparency to a basic model of the lens opening (aperture
function), as described in Equation 2. You can see an example of this
synthetic lens opening in Figure 23 (a), they created a total of 125 different
variations of this lens opening.

When we have a specific point of light at a given location (x, y, z) with a
single color (wavelength represented by $\lambda$), we can calculate the two
aspects of how light is altered as it passes through the lens, which is
described in Equation 3 ($\phi_{\lambda}^{S}$ and $\phi_{\lambda}^{DF}$).
These calculations are deterministic, meaning we can precisely determine how
the point of light will appear in the final image. We refer to this final
appearance as the Point Spread Function (PSF) for that particular light
source, denoted as $PSF_{\lambda}$. This PSF is determined by combining the
lens opening (A) and the phase shift ($\phi_{\lambda}$) as specified in
Equation 4.

To recreate the effect of a light source spanning the entire visible spectrum,
they calculated the $PSF_{\lambda}$ for all wavelengths ($\lambda$) ranging
from 380nm (violet) to 740nm (red), with intervals of 5nm. This results in a
73-value set of data for each pixel of the PSF, considering the different
wavelengths. To get the PSF as observed by an RGB camera sensor, This full-
spectrum PSF is left-multiplied by a spectral response function (SRF) which
describes the sensitivity of color channel c to wavelength $\lambda$, where c
$\in$ ${R,G,B}$ and is represented as a 3×73 matrix. This process allows us to
derive the PSF as it would be measured by an RGB camera sensor:

$\begin{bmatrix}PSF_{R}(s,t)\\\ PSF_{G}(s,t)\\\ PSF_{B}(s,t)\\\
\end{bmatrix}=SRF\begin{bmatrix}PSF_{\lambda}=380nm(s,t)\\\ \vdots\\\
PSF_{\lambda}=740nm(s,t)\\\ \end{bmatrix}$ (5)

where $(s,t)$ are the image coordinates. This produces a flare image for a
light source located at $(x,y,z)$.

They constructed their dataset of scattering flare by randomly sampling the
aperture function A, the light source’s 3D location $(x,y,z)$, and the
spectral response function (SRF). they also applied optical distortion as
barrel and pincushion to augment the PSF RGB images. They generated a total of
3,000 such flare images. An example of the simulated aperture function and the
resultant flare image can be shown in Figure 23 (b). It is worth noting that
the exact details for sampling can be found in the Appendix of Wu et al.’s
paper [117].

For the reflective flare, Due to the difficulty of simulation of this type of
flare as it requires an accurate characterization of the optics, which is
often unavailable, they captured images of reflective flare in a laboratory
setting consisting of a bright light source, a programmable rotation stage,
and a fixed-aperture smartphone camera with a f = 13mm lens. The setup is
clearly shown in Figure 24. The setup is insulated from ambient light during
capture, The camera is also rotated programmatically so that the light source
traces (and extends beyond) the diagonal field of view, from $-75\degree$ to
$75\degree$. They captured one HDR image every $0.15\degree$, resulting in
1,000 samples. Adjacent captures are then interpolated by 2x using the frame
interpolation algorithm of Context-aware Synthesis for Video Frame
Interpolation [130], giving a total of 2,000 reflective flare images. It is
worth noting that Wu et al. [117] introduced the very first test dataset
consisting of 20 paired flare-corrupted and flare-free images. However, this
dataset is extremely homogeneous and only contains images where the light
source doesn’t exist in the scene.

TABLE I: A comparison between our Flare7K++ dataset (Flare7K+Flare-R) and Wu et al.’s dataset. The ‘type (s+r)’ indicates the number of different patterns of scattering (s) flare and reflective (r) flare. In the Flare-R dataset, the three smartphones they used possess 9 rear cameras which result in 9 types of reflective flares. They provide glare annotations that also contain the shimmer effect due to the difficulty of separation. From [127]. Dataset | Statistics | Annotations
---|---|---
Number | Synthetic | Real | Type | Light Source | Reflective Flare | Streak | Glare
Wu et al. | 5,001 | 3,000 | 2,001 | 2+1 | $\times$ | $\times$ | $\times$ | $\times$
Flare7K | 7,000 | 7,000 | 0 | 25+10 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
Flare-R | 962 | 0 | 962 | 962+9 | $\checkmark$ | $\times$ | $\times$ | $\times$

###  VI-B Flare7K

Figure 27: Visualization and distribution of Bracket Flare dataset. It
contains different types of light sources in diverse scenes. Based on light
source types, they were classified into eight categories with different flare
patterns. From [129].

So far, the only existing flare dataset is the one proposed by Wu et al.
[117], which is mainly designed for daytime flare removal. Thus, the streak
effect and glare effect that commonly exists in nighttime flares are not
considered in Wu et al.’s dataset. In terms of nighttime flares. The variety
of contamination types on the lens makes it difficult for physics-based
methods such as Wu et al.’s approach to collect real nighttime flares by
traversing all different pupil functions. This results in the lack of
diversity and the domain gap between synthetic flares in Wu et al.’s dataset
and the real-world scenes. That’s why Dai el al. [128] tried to solve this
issue by proposing a new nighttime flare dataset [128] based on hundreds of
nighttime flare images with different types of lenses (smartphone and camera)
and various light sources as reference images. They also assumed that the
flare image can be linearly added to a flare-free image to generate a flare-
corrupted image.

They aggregated the captured images and summarized the scattering flares into
25 typical types based on their patterns. Reflective flares, which are
associated with the type of the camera’s lens group, were also considered.
They captured video clips using different cameras as references for simulating
reflective flare effects. By referring to these real-world nighttime flare
videos, they designed specific flare patterns for each type of camera,
resulting in 10 typical types of reflective flares. For each flare type, they
generated 200 images with varying parameters like glare radius and streak
width, totaling 5,000 scattering flares and 2,000 reflective flares. To
utilize how mature the flare rendering has become, they used the Video Copilot
Optical Flares plugin in Adobe After Effects. The resulting flare patterns are
more diverse and closely resemble real-world nighttime scenarios. A comparison
between their dataset [128] and Wu et al.’s [117] dataset in Table 1 reveals
that their new dataset offers a wider range of patterns and annotations, which
can benefit various applications such as lens flare segmentation and light
source extraction.

In the scattering flare generation process, they followed a specific pipeline,
as shown in Figure 25. They broke down the lens flare into four components:
shimmer, streak, glare, and the light source. For each of these components,
they analyzed parameters such as the glare’s radius range and color-distance
curve using reference images. They then used Adobe After Effects to create
templates for the flare.

For glare synthesis, they examined the relationship between the RGB value of
each pixel and its distance from the light source in reference to flare-
corrupted images. This relationship was used to create a color correction
curve, which was applied to a gradient pattern with the glare’s radius to
produce the glare effect. To address variations in luminance around the
streak, they manually determined the vanishing angle and applied a feathered
mask to reduce the glare’s opacity in those areas.

When it came to streak synthesis, they departed from a symmetric approach in
[131] as it assumes that the streak is always generated with a 2-point star
PSF. However, the streak effect is not usually symmetric, and one side is
often much sharper than the other side. They instead manually created masks
for each type of streak in Adobe After Effects, allowing for different widths.
They used the RGB values of the streak and glare sections to colorize the
streak and blur the mask’s edges. The blur size was determined based on the
curve’s half-life value.

For shimmer synthesis, they utilized a shimmer template in the Optical Flares
template and adjusted its parameters to approximate the flare in the image. In
areas around the light source where images often experience significant
degradation, they employed Adobe After Effect’s default plugin fractal noise
and radial blur effects to generate a noise pattern, which was then combined
with the shimmer template to create a realistic shimmer effect.

In terms of light source synthesis, they applied thresholding [42] to the
flare-corrupted images to identify overexposed tiny shapes. To simulate the
glow effect of the light source, they used a plugin called Real Glow in Adobe
After Effects on these shapes. The light source region was deliberately made
larger than the glare’s overexposed part to ensure a realistic visual result.
The final step involved adding the light source to the flare using a screen
blend mode, preserving the overexposed region’s boundaries for a realistic
appearance.

For reflective flares, they utilized the same Optical Flares plugin, which
included a collection of 51 high-quality iris images. These iris images served
as the iris bank. They compared these irises with the reference video and
selected the most similar ones 26. Then they manually adjusted the size and
color of these irises using the Optical Flares plugin. To maintain the
proportional relationship between the irises’ distances from the light source,
they aligned different iris components along a line with proportional
distances from the light source. This process resulted in the creation of a
reflective flare template.

For specific types of reflective flares, they took into account dynamic
triggering mechanisms, including caustics and the clipping effect.

Caustics are optical patterns that occur when light rays refract and converge
at a point. In this case, caustics happened when the light source was
positioned far from the optical center of the camera lens. To replicate this
caustic effect realistically, they utilized a premade template provided by the
Optical Flares plugin. This template generated a caustic pattern, a
distinctive pattern of light and dark areas, at the center of the iris.

To mimic the dynamic triggering effect associated with caustics, they made the
intensity or opacity of the caustics pattern proportional to the distance
between the iris and the light source. This meant that as the distance between
the iris and the light source changed, the intensity of the caustic effect
would also vary accordingly, creating a more authentic representation.

The clipping effect was another phenomenon they considered. It occurred when
the path of reflected light was blocked by more than two lenses’ apertures.
Essentially, it resembled the intersection of two irises. When the distance
between the iris and the light source exceeded a certain threshold (the
clipping threshold), they initiated a process to remove parts of the iris. To
do this, they used another iris as a masking tool. This secondary iris served
solely as a mask and was not visible in the final image. This technique
allowed them to simulate the effect where portions of the iris were blocked or
clipped by the camera’s optical components, resulting in a more realistic
rendering of the flare.

In nighttime situations, matrix LED light is common which is a distinctive
form of flare known as a lattice-shaped reflective flare can manifest in
captured images. This specific type of flare occurs due to the arrangement of
the LED lights and their impact on the camera lens. To imitate this effect,
they synthesized some irises in the shape of the lattice as shown in Figure
26.

Due to limited nighttime flare removal test dataset, they created their own
synthetic dataset and gathered their own real nighttime flare dataset for
evaluation using full-reference image quality assessment metrics.

For synthetic test data collection, they followed the same pipeline discussed
earlier. In this approach, neither flare images nor flare-free images were
part of their training dataset. To create more realistic synthetic test data,
they captured images of the same scenes using both a Huawei P40 smartphone
camera (left uncleaned to introduce natural lens flare) and a Sony $\alpha$
6400 camera with a Sigma 16mm F1.4 lens [132] (which did not introduce lens
flare). Using the smartphone’s flare-corrupted images as a reference, they
synthesized flare images and merged them with images taken by the professional
camera to create pairs of flare-free and flare-corrupted images. They
generated a total of 100 such pairs for testing.

For real test data collection, they aimed to replicate the flares found in
nighttime scenes, which are typically caused by smudges or scratches on the
camera lens (or windshield for nighttime driving). To simulate these real-
world flares, they used their fingers and a cloth to introduce common smudges
on the camera’s front lens. They then used lens tissue to lightly clean the
lens, resulting in flare-free images. However, this cleaning process could
still cause slight misalignment between the paired images. To address this,
they manually aligned the paired images and obtained 100 pairs of real-world
flare-corrupted and flare-free images for their test dataset. It’s important
to note that because the ground truth images may still contain minor flares
due to lens defects, the evaluations using these real-world paired images
serve as reference points and may not fully reflect the actual performance of
flare removal methods.

###  VI-C Flare7K++

Since the synthetic dataset did not include complex degradations caused by
diffraction and dispersion in the lens system, a real-world flare dataset,
Flare-R [127], was introduced, consisting of 962 real flare patterns. They
replicated common lens contaminants encountered in daily use to diversify
Flare7K dataset. Various liquids, including water, oil, ethyl alcohol, and
carbonated drinks, were applied to the lens surface and wiped with fingers and
different types of clothing. After each wipe, a new lens flare image was
captured. Automatic white balance on phone cameras was disabled, and
different-colored lens flare images were obtained by adjusting the color
temperature of the light source.

The Huawei P40, iPhone 13 Pro, and ZTE Axon 20 5G phones, each equipped with
three rear lenses of varying focal lengths, were used. For each lens, they
altered the distance between the camera and the light source and captured
approximately 100 images. In total, they amassed 962 flare images, covering a
wide range of common scenarios. Table I shows how effective the Flare7K [128]
and Flare7K++ [127] is compared to Wu et al.’s [117] dataset.

In contrast to the synthetic Flare7K dataset, obtaining light source
annotations for real-captured flares posed challenges. To address this, they
trained a network using Flare7K data and its light source annotations to
automatically extract light sources from the real flare images. This network
was applied to process all images in the Flare-R dataset to obtain light
source annotations. In very few instances where reflective flares were not
removed, impacting the accuracy of light source extraction, erroneous bright
spots were manually removed to refine the light source annotations.

For the Flare7K test dataset, the ground truth may still be influenced by the
slight flares brought by the lens’s defects, global PSNR cannot fully reflect
the performance of flare removal methods. To address this problem, they
manually labeled masks for all streak and glare regions. Masked PSNR can be
used to evaluate the restored results in the regions of different components
of flares. they named the masked area of glare and streak regions G-PSNR and
S-PSNR respectively. All of those evaluation metrics are discussed in further
detail in Section VII.

###  VI-D Bracket Flare

To tackle the challenge of creating paired images for removing reflective
flares, Dai et al. [129] introduced a semi-synthetic dataset called Bracket
Flare. Reflective flares typically follow the same pattern as the brightest
part of a light source. To generate reflective flare examples, they adjusted
the camera’s exposure settings to capture the patterns of the light source’s
brightest region. Their approach involved using a professional camera which is
a Sony A7 III [133] mounted on a tripod. They took a series of five photos of
the same stationary scene with a 3 EV step (exposure amount) between each
shot. From each group of images, they selected the normally exposed image and
the one with the lowest exposure to form a pair.

To enhance the dataset’s diversity and realism, they carefully selected scenes
with various bright light sources that could produce reflective flares. In the
end, they curated a dataset comprising 440 pairs of 4K resolution images for
training purposes and an additional 40 images for testing. You can get the
distribution of the dataset’s contents from Figure 27, which illustrates the
dataset’s diversity, featuring various common flare patterns. However, it is
worth noting that this dataset mainly consists of only reflective flare which
is much less common than scattering flare. In addition to this, they relied on
optical symmetry prior, which means that bright spots of reflective flares and
light sources are always symmetrical around the lens’s optical center.
However, this only holds true for many smartphones and a limited number of
professional cameras, so it doesn’t generalize well for all types of flare

##  VII Performance Review

Performance review in the context of flare removal is essential for evaluating
the effectiveness of methods and algorithms designed to mitigate or eliminate
lens flare artifacts from images. It serves as a critical step in assessing
how well a flare removal technique has performed and whether it meets the
intended objectives. It is crucial because it provides valuable insights into
the quality and reliability of flare removal results.

###  VII-A Performance Metrics

####  VII-A1 Peek Signal-to-Noise Ratio

PSNR [134] for short, is a commonly used performance metric in image
processing tasks, including flare removal. It quantifies the quality of an
image by measuring the similarity between the original, unaltered image and a
processed or reconstructed image.

The PSNR is calculated using the Mean Squared Error (MSE) between the original
image $I$ and the processed image $\hat{I}$.

$\text{PSNR}=10\cdot\log_{10}\left(\frac{{\text{MAX}^{2}}}{{\text{MSE}}}\right)$
(1)

Where:

  * •

MAX is the maximum possible pixel value in the image. For typical 8-bit
grayscale images, MAX is 255. For color images, it is 255 for each channel (R,
G, B).

  * •

MSE [135] is the Mean Squared Error, which quantifies the average squared
difference between corresponding pixels in the original ${I}$ and processed
images $\hat{I}$

$\text{MSE}=\frac{1}{N}\sum_{i=1}^{N}(I_{i}-\hat{I_{i}})^{2}$ (2)

Where:

  * •

N is the total number of pixels.

  * •

${I_{i}}$ is the value of pixel at position ${i}$ in the original image.

  * •

${\hat{I_{i}}}$ is the value of pixel at position ${i}$ in the processed
image.

If the PSNR is very high (above 30 dB), it suggests that the processed image
is nearly identical to the original, and the quality is excellent. However,
it’s important to note that while PSNR provides a quantitative measure of
image quality, it may not always align perfectly with human perceptual
quality. Other metrics and visual inspection are often used in conjunction
with PSNR to comprehensively evaluate the success of flare removal methods.

####  VII-A2 Structural Similarity Index Measure

SSIM [136] for short, is a performance metric commonly used in image
processing tasks, including flare removal. It quantifies the similarity
between two images, typically an original image and a processed or
reconstructed image. SSIM assesses not only pixel-level differences but also
structural and perceptual aspects, making it a valuable metric for evaluating
the quality of flare removal. The SSIM index is computed using three main
components: luminance, contrast, and structure.

Luminance ($\mu_{x},\mu_{y}$): Represents the brightness or intensity of the
images. SSIM evaluates whether the luminance of the processed image matches
that of the original image.

Contrast ($\sigma_{x}^{2},\sigma_{y}^{2}$): Measures the variation in pixel
intensity within each image. SSIM assesses whether the contrast in the
processed image is consistent with that of the original image.

Structure ($\sigma_{xy}$): Captures the structural similarity between the
images. It considers how patterns, edges, and textures align between the
processed and original images.

$\text{SSIM}(x,y)=\frac{{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}}{{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}}$
(1)

Where:

  * •

$\mu_{x}$ represents the mean (average) of image x.

  * •

$\mu_{y}$ represents the mean (average) of image y.

  * •

$\sigma_{x}^{2}$ represents the variance of image x.

  * •

$\sigma_{y}^{2}$ represents the variance of image y.

  * •

$\sigma_{xy}$ represents the covariance between images x and y.

  * •

$C_{1}$ and $C_{2}$ are the small constants added to stabilize the formula.

The SSIM index produces a value between -1 and 1, where a value of 1 indicates
that the two images being compared are identical in terms of luminance,
contrast, and structure. If it is close to 1 then it suggests a high degree of
similarity between the two images. On the other hand, if it is equal to -1 it
indicates dissimilarity or a significant difference between the images. A
value close to 0 implies that the images are essentially uncorrelated or
dissimilar.

By considering not only pixel-level differences but also structural and
perceptual similarities, SSIM provides a more comprehensive assessment of
image quality in the context of flare removal compared to metrics like PSNR.

####  VII-A3 Learned Perceptual Image Patch Similarity

LPIPS for short, is a performance metric used in image processing tasks,
including flare removal. LPIPS assesses the perceptual similarity between two
images by considering their structural and textural characteristics. It is
particularly useful when evaluating image quality from a human perception
perspective. LPIPS is based on deep neural networks that have been trained to
measure the perceptual similarity between images. The mathematical
representation of LPIPS involves the use of deep neural networks, specifically
convolutional neural networks (CNNs). A lower LPIPS score indicates greater
similarity between the images, while a higher score suggests greater
dissimilarity. The score is obtained by comparing features extracted from
different layers of the neural network, capturing information about structural
and textural details in the images.

$\text{LPIPS}(x,y)=\sum_{i=1}^{N}w_{i}\cdot d_{i}(x,y)$ (1)

Where:

  * •

$d_{i}(x,y)$: The feature distance between images x and y at layer $i$ of the
deep neural network. Can be MSE, Euclidean distance, cosine distance, or a
combination of them

  * •

$w_{i}$: Weights associated with layer $i$ of the network.

  * •

$N$: Total number of layers in the network

Unlike traditional metrics that only consider pixel-level differences as PSNR,
LPIPS evaluates the perceptual similarity between the original and processed
images, taking into account structural and textural information. This makes it
well-suited for evaluating flare removal techniques that aim to maintain the
overall visual quality of the image while reducing flare artifacts.

TABLE II: Comparison of different methods on the Flare7K++ [127] real test
dataset.

’*’ denotes models with reduced parameters due to the limited GPU memory.
Metric/Method PSNR SSIM LPIPS G-PSNR S-PSNR Input 22.561 0.857 0.0777 19.556
13.105 Wu et al. (U-Net) [117] 24.613 0.871 0.0598 21.772 16.728 Flare7K
(U-Net) [128] 26.978 0.890 0.0466 23.507 21.563 Flare7K++ (U-Net) [127, 84]
27.189 0.894 0.0452 23.527 22.647 Flare7K++ (HINet) [127, 87] 27.548 0.892
0.0464 24.081 22.907 Flare7K++ (MPRNet*) [127, 83] 27.036 0.893 0.0481 23.490
22.267 Flare7K++ (Restormer*) [127, 90] 27.597 0.897 0.0447 23.828 22.452
Flare7K++ (Uformer) [127, 101] 27.633 0.894 0.0428 23.949 22.603

####  VII-A4 Extra Metrics

The Glare PSNR and Streak PSNR are specialized performance metrics used in
flare removal introduced in Flare7K++ [127] dataset, both of which employ the
Peak Signal-to-Noise Ratio (PSNR). These metrics focus on evaluating the
quality of specific components within an image: glare and streaks,
respectively. Glare PSNR calculates the PSNR specifically for the glare
portion of the image. It quantifies how well a flare removal method has
reduced or eliminated the glare artifacts while preserving the rest of the
image’s quality. Streak PSNR, on the other hand, computes the PSNR for streak
artifacts within an image. It measures how well an algorithm has handled
streak-like flare artifacts, such as elongated lines or streaks that may
result from intense light sources.

It’s important to note that while Glare PSNR and Streak PSNR are valuable
metrics, they have a limitation. These metrics require the availability of
glare and streak masks for the test data. In practice, obtaining accurate
glare and streak masks may not always be feasible or straightforward and
sometimes may not be accurate. This limitation can make it challenging to
apply G-PSNR and S-PSNR in all scenarios. Nevertheless, when glare and streak
masks are available, these metrics provide valuable insights into the
performance of flare removal methods, allowing for targeted evaluation of
algorithms.

###  VII-B Quantitative Analysis

Due to limited paired flare-corrupted and flare-free image datasets, the
currently available dataset contains a relatively small number of samples.
Furthermore, each test dataset has a uniform distribution and can sometimes be
limited to only one type of flare. It is worth noting that most of the
published papers conduct a user study to compare different methods in addition
to quantitative analysis. In this section, we will analyze the state-of-the-
art methods on currently used benchmarks333Due to unavailability of some
checkpoints, not all of the variations for each method is considered in all of
the conducted quantitative analysis..

####  VII-B1 Wu et al.’s Test Dataset

Method | Synthetic | Real
---|---|---
PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS
Input | 20.449 | 0.832 | 0.1570 | 18.573 | 0.782 | 0.1083
Wu et al. (U-Net) [117, 84] | 28.475 | 0.9344 | 0.0502 | 25.551 | 0.849 | 0.0616
Flare7K++ (Uformer) [127, 101] | 22.911 | 0.880 | 0.0964 | 19.570 | 0.794 | 0.1016
Bracket Flare (MPRNet) [129, 83] | 20.454 | 0.832 | 0.1567 | 18.579 | 0.782 | 0.1082

TABLE III: Comparison for different methods on Wu et al.’s [117] test dataset
(Both synthetic and real).

Method | PSNR | SSIM | LPIPS | G-PSNR | S-PSNR
---|---|---|---|---|---
Input | 22.770 | 0.921 | 0.0601 | 18.804 | 13.927
Flare7K++ (Uformer) [127, 101] | 29.498 | 0.962 | 0.0210 | 26.685 | 24.686
Bracket Flare (MPRNet) [127, 83] | 22.994 | 0.922 | 0.0578 | 18.862 | 14.309

TABLE IV: Comparison for different methods on Flare7K++ [127] synthetic test
dataset.

Wu et al. [117] introduced two test datasets, one is real containing 20
samples, and the other one is synthetic containing 38 samples. From Table III,
we can see the Wu et al.’s approach demonstrates state-of-the-art performance
on this benchmark. However, the real test dataset contain only images where
light source is not present in the image, which means that it only consist of
limited types of flare and does not include scattering flare since all of
images are captured during daytime not nighttime. Moreover, the synthetic test
dataset does not look like the actual flare in real life.

####  VII-B2 Flare7K++ Test Dataset

Dai et al. [127] introduced two extra datasets where one of them is real and
the other one is synthetic, each containing 100 samples. Table II shows
metrics for different flare removal methods on Flare7K++ real test dataset
where Uformer [101] demonstrates state-of-the-art performance using Dai et
al.’s [127] approach in Flare7K++ paper. Table IV compares the discussed
approaches in this survey on Flare7K++ synthetic test dataset. Unfortunately,
as there is no checkpoint available for Wu et al.’s work, we are not able to
provide their results. It is worth noting that this benchmark contains mainly
scattering flare in nighttime with little to none reflective flare, that is
why Bracket Flare approach [129] nearly gives the same results as the plain
input since it is only trained on reflective flare.

####  VII-B3 Bracket Flare Test Dataset

Method | PSNR | SSIM | LPIPS | Masked PSNR
---|---|---|---|---
Input | 37.30 | 0.99 | 0.025 | 21.68
Wu et al. (U-Net) [117, 84] | 26.13 | 0.895 | 0.055 | 21.89
Flare7K++ (Uformer) [127, 101] | 28.29 | 0.911 | 0.052 | 22.62
Bracket Flare (MPRNet) [129, 83] | 48.41 | 0.994 | 0.004 | 32.09
Bracket Flare (Uformer) [129, 101] | 47.47 | 0.991 | 0.003 | 31.57
Bracket Flare (HINet) [129, 87] | 48.03 | 0.994 | 0.003 | 30.88
Bracket Flare (Restormer) [129, 90] | 48.11 | 0.994 | 0.004 | 31.07

TABLE V: Comparison for different methods on Bracket Flare [129] test dataset.

Dai et al. [129] introduced another synthetic test dataset consisting of 40
samples in Bracket Flare Paper [129]. This dataset is based on optical center
symmetry prior which states that the reflective flare is in the same position
as light source but rotated 180 degrees. However, this only holds true for
many smartphones and limited number of cameras. Table V shows metrics for the
discussed methods on this dataset. We can clearly see that the input gives
even a better performance than Flare7K++, this is due to the fact that this
dataset contain only reflective flare and does not even include any scattering
ones which Flare7K++ network is trained on.444Dai et al. [129] retrained a
model with the same method as Wu et al. [117], that is how we reported Wu et
al.’s [117] metrics in Bracket Flare benchmark [129].

##  VIII Challenges and Opportunities

After highlighting the significant progress that has been made, real-world
challenges persist due to the inherent complexities and data-dependent nature
of lens flare. For this section, we will introduce some of the promising
research directions that we believe will help in further advancing flare
removal methods.

###  VIII-A Video Flare Removal

Up till now, there is not a single paper addressing flare removal for a video,
this unexplored domain has a big room for improvement. Extending lens flare
removal techniques from static images to video content introduces a set of
unique challenges. Videos are inherently dynamic, with constantly changing
scenes and lighting conditions. Temporal coherence, or ensuring that flare
removal remains consistent and smooth across frames, becomes a significant
concern. Techniques must be able to handle the dynamic nature of videos while
avoiding flickering or abrupt changes in visual quality.

Additionally, motion compensation is essential in video flare removal. Objects
in videos may move across frames, causing flares to interact with dynamic
elements. Methods must account for this motion to accurately remove lens
flares without introducing artifacts or distortions. This challenge requires
the integration of computer vision and video processing techniques to track
and analyze motion patterns within the video.

Scalability is another aspect to consider. Videos come in various resolutions,
and flare removal algorithms should be scalable to handle both standard and
high-definition videos. This scalability is crucial to ensure that lens flare
removal can cater to the needs of various video production scenarios, from
consumer videos to professional filmmaking.

###  VIII-B Real Time Flare Removal

Achieving real-time lens flare removal is a challenging endeavor, especially
for applications like live video streaming, where responsiveness is crucial.
To address this challenge, researchers need to develop methods that not only
effectively remove lens flare but also do so with minimal computational
overhead. This necessitates the creation of low-latency methods that can
efficiently process each frame of a video stream or image.

One key concern is striking the right balance between computational efficiency
and maintaining high-quality results. While faster methods may yield real-time
performance, they could compromise the quality of flare removal, potentially
leaving behind artifacts or degrading the overall image or video quality. As
such, achieving real-time flare removal requires careful optimization.
Researchers must explore innovative strategies to maintain both speed and
quality.

###  VIII-C Dataset Collection

The development and evaluation of effective flare removal algorithms heavily
rely on the availability of high-quality datasets. These datasets should
encompass diverse flare patterns, representing various scenarios and
conditions in which lens flares occur. However, creating comprehensive and
representative datasets is a challenging task in itself and can be sometimes
labor-intensive. So far, the mainly used datasets are semi-synthetic.

The used semi-synthetic pipeline has a large room for improvement. For
instance, the number of different scattering flare and reflective flare
patterns proposed in Flare7K++ are 25 and 10 respectively. However, those
numbers are relatively small and do not cover all the types of flare found in
real life due to the different luminance, chromatic, and mechanical properties
of each camera. Also, different light source properties and camera settings
can contribute to the diversity of flare patterns.

One notable challenge is the limited availability of comprehensive test
datasets that accurately represent the wide array of flare scenarios
encountered in practice. This limitation hinders the assessment of different
flare removal methods, which is why some of the researchers rely on user study
as an evaluation metric.

###  VIII-D Modular-Based Approaches

Instead of attempting to tackle the flare removal using an end-to-end
approach, researchers can explore a more modular approach, breaking down the
task into distinct components, each responsible for a specific aspect. This
strategy holds significant promise due to the limited availability of
comprehensive paired datasets for training end-to-end models. For instance,
Qiao et al. [137] made a significant contribution in this direction by
recognizing the critical role of light source conditions in generating lens
flares. To address this, they introduced a deep learning-based framework that
incorporates light source-aware guidance for single-image flare removal. They
employed multiple interconnected modules, including a light source detection
module, a flare detection module, a flare removal module, and a flare
generation module. These modules work in harmony to address various aspects of
the flare removal problem. Notably, their approach was primarily designed to
work with unpaired image datasets, since the paired datasets were limited at
that time. It’s important to highlight that, despite its significance, the
paper does not have an official implementation.

Exploring component-based or modular-based strategies for flare removal opens
up a promising avenue for researchers to delve into lens flare mitigation,
allowing for more targeted and specialized solutions. This approach not only
leverages domain-specific knowledge but also offers flexibility in adapting to
various real-world scenarios and challenges associated with lens flares.

###  VIII-E Daytime to Nighttime Translation

Flare artifacts often become more pronounced and challenging to address in
nighttime images due to increased contrast between light sources and dark
backgrounds. To enhance the performance of deep learning models for flare
removal in low-light conditions, it is advantageous to train these models
specifically on nighttime imagery. Many existing methods utilize datasets like
Flickr24K used in [126], which, while valuable, consist of a substantial
number of daytime images. As an innovative approach to address this issue,
researchers can explore transforming daytime images into nighttime
counterparts using image-to-image translation techniques, such as CycleGAN
[79].

Daytime-to-nighttime translation involves adapting existing datasets by
simulating nighttime conditions, effectively expanding the available training
data. By leveraging image translation methods, researchers can introduce the
characteristics and challenges associated with nighttime imagery into their
training process.

Exploring daytime-to-nighttime translation techniques in the context of flare
removal holds significant potential for advancing the field. Researchers are
encouraged to investigate and innovate in this area, as it offers a promising
avenue for improving the performance and adaptability of deep learning models
in addressing flare artifacts under low-light conditions.

###  VIII-F Multi-Stage Approach

One common observation in the output of existing flare removal methods is the
presence of residual artifacts that resemble noise, particularly noticeable
around streaks in scattering flares. Researchers interested in advancing flare
removal techniques can explore the concept of multi-stage methods as a
potential solution.

In a multi-stage approach, the image processing pipeline is divided into
several sequential stages, with each stage addressing a specific aspect of
flare removal. By breaking down the problem into these smaller, focused
modules, it becomes possible to tackle artifact reduction more effectively.
For instance, one stage can be dedicated to detecting and localizing streak
artifacts, while another can focus on their removal, and another one to remove
any remaining noise or artifacts. This division of labor allows for greater
precision and control over the removal process, potentially leading to
significantly improved results.

Encouraging further exploration in flare removal is vital for advancing the
field. Researchers are encouraged to investigate and innovate in this area, as
it holds the promise of substantially enhancing the quality of flare-free
images and addressing persistent artifact challenges.

##  IX Conclusion

In this work, we provide a comprehensive survey of the multifaceted domain of
lens flare removal. We delve into the complex optics behind flare formation
and categorize the diverse types of flare patterns that can manifest. The
numerous factors influencing flare appearance are analyzed in depth, spanning
light source attributes, lens features, camera settings, and scene content.
The survey extensively reviews the wide range of methods proposed for
mitigating lens flare artifacts, covering hardware optimization strategies,
traditional image processing techniques, and innovative deep learning
approaches. Flare removal datasets are examined, shedding light on how paired
images were synthesized for training and evaluation. The performance of
various methods is compared quantitatively on popular benchmarks, providing
insights into the state-of-the-art capabilities and limitations.

The survey highlights that while significant progress has been made, real-
world challenges persist due to the inherent complexities and data-dependent
nature of lens flare. Striking an effective balance between efficiency,
quality, and generalizability remains an open research problem. Promising
future directions are suggested. By comprehensively reviewing the field’s
evolution, benchmarks, and innovations, this survey offers a timely
understanding of the complexities of lens flare and the potential of existing
solutions. It provides a strong foundation to motivate and guide future
efforts aimed at robustly handling flare artifacts across varied imaging
scenarios and applications.

## References

  * [1] L. L. Holladay,  “The fundamentals of glare and visibility,”  Journal of the Optical Society of America, vol. 12, pp. 271–319, 1926.
  * [2] Joseph W Goodman,  “Introduction to fourier optics,”  Introduction to Fourier optics, 3rd ed., by JW Goodman. Englewood, CO: Roberts & Co. Publishers, 2005, vol. 1, 2005.
  * [3] J. A. Seibert, O. Nalcioglu, and W. Roeck,  “Removal of image intensifier veiling glare by mathematical deconvolution techniques,”  Medical Physics, vol. 12, no. 3, pp. 281–288, 1985.
  * [4] E.N. Malamas, Euripides Petrakis, Michalis Zervakis, Laurent Petit, and Jean-Didier Legat,  “A survey on industrial vision systems, applications and tools,”  Image and Vision Computing, vol. 21, pp. 171–188, 02 2003.
  * [5] Utsho Chakraborty, Haimanti Biswas, and Sheshang Degadwala,  Survey on Real Time Application of Computer Vision,  Ph.D. thesis, Sarvajanik College of Engineering and Technology, 10 2020.
  * [6] Xiaoxue Zhang and Zhigang Liu,  “A survey on stereo vision matching algorithms,”  in Proceeding of the 11th World Congress on Intelligent Control and Automation, 2014, pp. 2026–2031.
  * [7] Mohd Saad Hamid, NurulFajar Abd Manap, Rostam Affendi Hamzah, and Ahmad Fauzan Kadmin,  “Stereo matching algorithm based on deep learning: A survey,”  Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 5, pp. 1663–1673, 2022.
  * [8] Mingliang Zhai, Xuezhi Xiang, Ning Lv, and Xiangdong Kong,  “Optical flow and scene flow estimation: A survey,”  Pattern Recognition, vol. 114, pp. 107861, 2021.
  * [9] Tomislav Dobrički, Xiahai Zhuang, Kyoung Jae Won, and Byung-Woo Hong,  “Survey on unsupervised learning methods for optical flow estimation,”  in 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), 2022, pp. 591–594.
  * [10] Stefano Savian, Mehdi Elahi, and Tammam Tillo,  Optical Flow Estimation with Deep Learning, a Survey on Recent Advances, pp. 257–287,  Springer, 01 2020.
  * [11] Martin Thoma,  “A survey of semantic segmentation,”  CoRR, vol. abs/1602.06541, 2016.
  * [12] Irem Ulku and Erdem Akagündüz,  “A survey on deep learning-based architectures for semantic segmentation on 2d images,”  Applied Artificial Intelligence, vol. 36, no. 1, pp. 2032924, 2022.
  * [13] Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, and Damayanthi Herath,  “Semantic segmentation using vision transformers: A survey,” 2023.
  * [14] Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, Pablo Martinez-Gonzalez, and Jose Garcia-Rodriguez,  “A survey on deep learning techniques for image and video semantic segmentation,”  Applied Soft Computing, vol. 70, pp. 41–65, 2018.
  * [15] Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye,  “Object detection in 20 years: A survey,”  Proceedings of the IEEE, vol. 111, no. 3, pp. 257–276, 2023.
  * [16] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul W. Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen,  “Deep learning for generic object detection: A survey,”  CoRR, vol. abs/1809.02165, 2018.
  * [17] Ershat Arkin, Nurbiya Yadikar, Yusnur Muhtar, and Kurban Ubul,  “A survey of object detection based on cnn and transformer,”  in 2021 IEEE 2nd International Conference on Pattern Recognition and Machine Learning (PRML), 2021, pp. 99–108.
  * [18] Syed Sahil Abbas Zaidi, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Naveed Asghar, and Brian Lee,  “A survey of modern deep learning based object detection models,”  CoRR, vol. abs/2104.11892, 2021.
  * [19] Seon Jong Kang, Kyung Bong Ryu, Min Su Jeong, Seong In Jeong, and Kang Ryoung Park,  “Cam-frn: Class attention map-based flare removal network in frontal-viewing camera images of vehicles,”  Mathematics, vol. 11, no. 17, 2023.
  * [20] R.C. Gonzalez and R.E. Woods,  Digital Image Processing. 4th Edition,  Pearson Education, New Yorks, 2018.
  * [21] Aaron Deever, Mrityunjay Kumar, and Bruce Pillman,  Digital Camera Image Formation: Processing and Storage, pp. 45–77,  Springer, 11 2013.
  * [22] Jinwei Gu, Ravi Ramamoorthi, Peter Belhumeur, and Shree Nayar,  “Removing image artifacts due to dirty camera lenses and thin occluders,”  ACM Transactions on Graphics, vol. 28, no. 5, pp. 1–10, dec 2009.
  * [23] John McCann and Alessandro Rizzi,  “Camera and visual veiling glare in hdr images,”  Journal of The Society for Information Display - J SOC INF DISP, vol. 15, 09 2007.
  * [24] Eino-Ville Talvala, Andrew Adams, Mark Horowitz, and Marc Levoy,  “Veiling glare in high dynamic range imaging,”  ACM Transactions on Graphics, vol. 26, no. 3, pp. 37–es, jul 2007.
  * [25] Ramesh Raskar, Amit Agrawal, Cyrus A. Wilson, and Ashok Veeraraghavan,  “Glare aware photography: 4d ray sampling for reducing glare effects of camera lenses,”  in ACM SIGGRAPH 2008 Papers, New York, NY, USA, 2008, SIGGRAPH ’08, Association for Computing Machinery.
  * [26] Floris Chabert,  “Automated lens flare removal,”  in Stanford. Stanford, 2015.
  * [27] Matthias Hullin, Elmar Eisemann, Hans-Peter Seidel, and Sungkil Lee,  “Physically-based real-time lens flare rendering,”  ACM Transactions on Graphics, vol. 30, no. 4, jul 2011.
  * [28] Sungkil Lee and Elmar Eisemann,  “Practical real-time lens-flare rendering,”  Computer Graphics Forum, vol. 32, 07 2013.
  * [29] Patricia Vitoria and Coloma Ballester,  “Automatic flare spot artifact detection and removal in photographs,”  Journal of Mathemtaical Imaging and Vision, vol. 61, no. 4, pp. 515–533, may 2019.
  * [30] Ivana Tomić, Igor Karlovits, and Ivana Jurič,  “Practical assessment of veiling glare in camera lens system,”  Journal of Graphic Engineering and Design, 12 2014.
  * [31] Piotr Marczuk, Mario Gerlach, Rudolf Guthoff, Valentin Fuchs, Karsten Sperlich, Julia Schubert, Sebastian Bohn, Heinrich Stolz, and Oliver Stachs,  “Characterization of starburst and halo size for different virtually implanted intraocular lenses in comparison to subject’s quality of vision,”  Investigative ophthalmology and visual science, vol. 58, 06 2017.
  * [32] Find Light,  “Secrets of Antireflective Coatings: Interference as a Key Mechanism,” .
  * [33] Dr. Vladan Blahnik and Dr. Benjamin Voelker,  “About the reduction of reflections for camera lenses: How T*-coating made glass invisible,” .
  * [34] K. Faulkner, C.J. Kotre, and M. Louka,  “Veiling glare deconvolution of images produced by x-ray image intensifiers,”  in Third International Conference on Image Processing and its Applications, 1989., 1989, pp. 669–673.
  * [35] O. Nalcioglu, W. W. Roeck, A. V. Lando, and J. A. Seibert,  “On the relationship of image intensifier contrast ratios measured by two techniques,”  Physics in Medicine and Biology, vol. 29, no. 10, pp. 1231–6, 1984.
  * [36] Paul Adrien Maurice Dirac,  “The physical interpretation of the quantum dynamics,”  Proceedings of The Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 113, pp. 621–641, 1927.
  * [37] W.T. Cochran, J.W. Cooley, D.L. Favin, H.D. Helms, R.A. Kaenel, W.W. Lang, G.C. Maling, D.E. Nelson, C.M. Rader, and P.D. Welch,  “What is the fast fourier transform?,”  Proceedings of the IEEE, vol. 55, no. 10, pp. 1664–1674, 1967.
  * [38] E. O. Brigham and R. E. Morrow,  “The fast fourier transform,”  IEEE Spectrum, vol. 4, no. 12, pp. 63–70, 1967.
  * [39] Henri J. Nussbaumer,  The Fast Fourier Transform, pp. 80–111,  Springer Berlin Heidelberg, Berlin, Heidelberg, 1982.
  * [40] A. Criminisi, P. Perez, and K. Toyama,  “Region filling and object removal by exemplar-based image inpainting,”  IEEE Transactions on Image Processing, vol. 13, no. 9, pp. 1200–1212, 2004.
  * [41] OpenCV,  “Blob detection,” Last accessed on October 2023.
  * [42] Analytics Vidhya,  “A brief Study of Image Thresholding Algorithms,” .
  * [43] Prasanna Sahoo, Sasan Soltani, and Andrew Wong,  “A survey of thresholding techniques,”  Computer Vision, Graphics, and Image Processing, vol. 41, pp. 233–260, 02 1988.
  * [44] Satoshi Suzuki and KeiichiA be,  “Topological structural analysis of digitized binary images by border following,”  Computer Vision, Graphics, and Image Processing, vol. 30, no. 1, pp. 32–46, 1985.
  * [45] Vitalflux,  “Different Types of CNN Architectures Explained,” .
  * [46] Keiron O’Shea and Ryan Nash,  “An introduction to convolutional neural networks,”  CoRR, vol. abs/1511.08458, 2015.
  * [47] Deepak Ghimire, Dayoung Kil, and Seong-heum Kim,  “A survey on efficient convolutional neural networks and hardware acceleration,”  Electronics, vol. 11, no. 6, 2022.
  * [48] Moez Krichen,  “Convolutional neural networks: A survey,”  Computers, vol. 12, no. 8, 2023.
  * [49] Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi,  “A survey of the recent architectures of deep convolutional neural networks,”  CoRR, vol. abs/1901.06032, 2019.
  * [50] Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou,  “A survey of convolutional neural networks: Analysis, applications, and prospects,”  IEEE Transactions on Neural Networks and Learning Systems, vol. PP, pp. 1–21, 06 2021.
  * [51] Kunihiko Fukushima,  “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,”  Biological Cybernetics, vol. 36, pp. 193–202, 1980.
  * [52] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K.J. Lang,  “Phoneme recognition using time-delay neural networks,”  IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pp. 328–339, 1989.
  * [53] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner,  “Gradient-based learning applied to document recognition,”  Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  * [54] Li Deng,  “The mnist database of handwritten digit images for machine learning research [best of the web],”  IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
  * [55] Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Chaudhuri,  “Activation functions in deep learning: A comprehensive survey and benchmark,”  Neurocomputing, vol. 503, 07 2022.
  * [56] Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete,  “A survey on modern trainable activation functions,”  CoRR, vol. abs/2005.00817, 2020.
  * [57] Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri,  “A comprehensive survey and performance analysis of activation functions in deep learning,”  CoRR, vol. abs/2109.14545, 2021.
  * [58] Hossein Gholamalinezhad and Hossein Khosravi,  “Pooling methods in deep neural networks, a review,”  CoRR, vol. abs/2009.07485, 2020.
  * [59] Bohdan Macukow,  “Neural networks – state of art, brief history, basic models and architecture,”  in Computer Information Systems and Industrial Management: 15th IFIP TC8 International Conference, CISIM 2016, Vilnius, Lithuania, September 14-16, 2016, Proceedings, 09 2016, vol. 9842, pp. 3–14.
  * [60] Jürgen Schmidhuber,  “Deep learning in neural networks: An overview,”  Neural Networks, vol. 61, pp. 85–117, jan 2015.
  * [61] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  in Advances in Neural Information Processing Systems, F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, Eds. 2012, vol. 25, Curran Associates, Inc.
  * [62] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  in International Conference on Learning Representations, 2015.
  * [63] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  * [64] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich,  “Going deeper with convolutions,”  CoRR, vol. abs/1409.4842, 2014.
  * [65] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam,  “Mobilenets: Efficient convolutional neural networks for mobile vision applications,”  CoRR, vol. abs/1704.04861, 2017.
  * [66] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger,  “Densely connected convolutional networks,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2261–2269.
  * [67] Analytics Vidha,  “Auto-Encoders for Computer Vision: An Endless world of Possibilities,” .
  * [68] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla,  “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481–2495, 2017.
  * [69] Wenhan Yang, Robby T. Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu,  “Single image deraining: From model-based to data-driven and beyond,” 2019.
  * [70] Jie Gui, Xiaofeng Cong, Yuan Cao, Wenqi Ren, Jun Zhang, Jing Zhang, Jiuxin Cao, and Dacheng Tao,  “A comprehensive survey and taxonomy on single image dehazing based on deep learning,” 2022.
  * [71] Michael Elad, Bahjat Kawar, and Gregory Vaksman,  “Image denoising: The deep learning revolution and beyond – a survey paper –,” 2023.
  * [72] Juncheng Li, Zehua Pei, and Tieyong Zeng,  “From beginner to master: A survey for deep learning-based single-image super-resolution,” 2021.
  * [73] Dor Bank, Noam Koenigstein, and Raja Giryes,  “Autoencoders,” 2021.
  * [74] C. O. S. Sorzano, J. Vargas, and A. Pascual Montano,  “A survey of dimensionality reduction techniques,” 2014.
  * [75] Raghavendra Chalapathy and Sanjay Chawla,  “Deep learning for anomaly detection: A survey,” 2019.
  * [76] neptune.ai,  “Generative Adversarial Networks and Some of GAN Applications: Everything You Need to Know,” .
  * [77] Hands-On Artificial Intelligence for IoT,  “Cycle GAN ,” .
  * [78] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial networks,” 2014.
  * [79] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros,  “Unpaired image-to-image translation using cycle-consistent adversarial networks,”  CoRR, vol. abs/1703.10593, 2017.
  * [80] Qiqi Shang, Lingxi Hu, Quanfeng Li, Wei Long, and Linhua Jiang,  “A survey of research on image style transfer based on deep learning,”  in 2021 3rd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM), 2021, pp. 386–391.
  * [81] Jeremy Zhang,  “UNet — Line by Line Explanation,” .
  * [82] Wei Liu, Andrew Rabinovich, and Alexander C. Berg,  “Parsenet: Looking wider to see better,”  CoRR, vol. abs/1506.04579, 2015.
  * [83] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao,  “Multi-stage progressive image restoration,”  in CVPR, 2021.
  * [84] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  CoRR, vol. abs/1505.04597, 2015.
  * [85] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu,  “Image super-resolution using very deep residual channel attention networks,”  CoRR, vol. abs/1807.02758, 2018.
  * [86] Augustus Odena, Vincent Dumoulin, and Chris Olah,  “Deconvolution and checkerboard artifacts,”  Distill, vol. 1, 10 2016.
  * [87] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen,  “Hinet: Half instance normalization network for image restoration,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2021, pp. 182–192.
  * [88] Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan,  “Spatially-attentive patch-hierarchical network for adaptive motion deblurring,”  CoRR, vol. abs/2004.05343, 2020.
  * [89] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz,  “Deep stacked hierarchical multi-patch network for image deblurring,”  CoRR, vol. abs/1904.03468, 2019.
  * [90] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang,  “Restormer: Efficient transformer for high-resolution image restoration,”  in CVPR, 2022.
  * [91] Pierre Charbonnier, Laure Blanc-Féraud, Gilles Aubert, and Michel Barlaud,  “Two deterministic half-quadratic regularization algorithms for computed imaging,”  Proceedings of 1st International Conference on Image Processing, vol. 2, pp. 168–172 vol.2, 1994.
  * [92] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang,  “Multi-scale progressive fusion network for single image deraining,”  CoRR, vol. abs/2003.10985, 2020.
  * [93] Andrew L. Maas,  “Rectifier nonlinearities improve neural network acoustic models,”  in The International Conference on Machine Learning, 2013.
  * [94] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao,  “Cycleisp: Real image restoration via improved data synthesis,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2696–2705.
  * [95] Jiahui Yu, Yuchen Fan, Jianchao Yang, Ning Xu, Zhaowen Wang, Xinchao Wang, and Thomas S. Huang,  “Wide activation for efficient and accurate image super-resolution,”  CoRR, vol. abs/1808.08718, 2018.
  * [96] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  CoRR, vol. abs/1502.03167, 2015.
  * [97] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha,  “Recurrent squeeze-and-excitation context aggregation net for single image deraining,”  CoRR, vol. abs/1807.05698, 2018.
  * [98] Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Björn Stenger, Ming-Hsuan Yang, and Hongdong Li,  “Deep image deblurring: A survey,”  CoRR, vol. abs/2201.10700, 2022.
  * [99] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi,  “Photo-realistic single image super-resolution using a generative adversarial network,”  CoRR, vol. abs/1609.04802, 2016.
  * [100] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  CoRR, vol. abs/1607.08022, 2016.