# VEDA: Uneven light image enhancement via a vision-based exploratory data
analysis model

Tian Pu, Shuhang Wang, Zhenming Peng, and Qingsong Zhu Tian Pu, and Zhenming
Peng are with the School of Information and Communication Engineering,
University of Electronic Science and Technology of China, e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Shuhang Wang is with the Center for
Ultrasound Research&Translation, Massachusetts General Hospital, Harvard
Medical School, email<EMAIL_ADDRESS>Qingsong Zhu is with the
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences,
Shenzhen, email<EMAIL_ADDRESS>(corresponding author)

###### Abstract

Uneven light image enhancement is a highly demanded task in many industrial
image processing applications. Many existing enhancement methods using
physical lighting models or deep-learning techniques often lead to unnatural
results. This is mainly because: 1) the assumptions and priors made by the
physical lighting model (PLM) based approaches are often violated in most
natural scenes, and 2) the training datasets or loss functions used by deep-
learning technique based methods cannot handle the various lighting scenarios
in the real world well. In this paper, we propose a novel vision-based
exploratory data analysis model (VEDA) for uneven light image enhancement. Our
method is conceptually simple yet effective. A given image is first decomposed
into a contrast image that preserves most of the perceptually important scene
details, and a residual image that preserves the lighting variations. After
achieving this decomposition at multiple scales using a retinal model that
simulates the neuron response to light, the enhanced result at each scale can
be obtained by manipulating the two images and recombining them. Then, a
weighted averaging strategy based on the residual image is designed to obtain
the output image by combining enhanced results at multiple scales. A similar
weighting strategy can also be leveraged to reconcile noise suppression and
detail preservation. Extensive experiments on different image datasets
demonstrate that the proposed method can achieve competitive results in its
simplicity and effectiveness compared with state-of-the-art methods. It does
not require any explicit assumptions and priors about the scene imaging
process, nor iteratively solving any optimization functions or any learning
procedures.

###### Index Terms:

contrast, image enhancement, residual image, vision-based exploratory data
model.

##  I Introduction

Many image/video processing applications, such as surveillance, daily
photography and aerial imaging, require high-quality images that preserve
scene contrasts faithfully [1]. However, images taken in unevenly-lit scenes
often fail to meet this requirement due to limited dynamic ranges.

To address this problem, uneven light image enhancement methods are highly
demanded and extensive researches have been made over the past decades. In the
following, we classify and briefly review the related studies from the
viewpoint of uneven light image enhancement.

###  I-A Histogram specification based enhancement

Histogram specification (HS) is one of the most widely used techniques. Early
HS based methods attempt to achieve desired output histogram shapes under the
assumption that visually-pleasing images have ideal histogram characteristics.
However, due to the lack of universal criteria for determining the ideal
histogram for natural images, these methods often lead to detail loss and
over-stretching of contrasts. Later studies improve the performance by
applying restrictions on contrast stretch, such as hue and range preservation
[2], gamma correction [3], saliency preservation [4], and contrast limitation
[5]. Recently, swarm optimization technique is introduced to the HS, focusing
on brightness and feature preservation[6]. However, the enhanced results are
prone to unnatural artifacts.

###  I-B Physical lighting model based enhancement

Taking the imaging process of natural images into account is a viable approach
for image enhancement, and many existing methods vary in their construction of
physical lighting models. An especially worth citing category of techniques is
enhancement methods based on Retinex theory, which assumes that an image is
the element-wise product of the illumination image and the reflectance image.
Early Retinex-based methods take the reflectance image as the enhanced result
[7], but the results often suffer from unnatural appearance. To overcome this
issue, subsequent advances aim to modify the estimated illumination image
instead of removing it. Wang et al. [8] propose a naturalness preserved
enhancement method by designing a bright-pass filter to recover the
illumination image. Liang et al. recover illumination by iteratively solving a
diffusion filtering equation. This approach is effective in preserving texture
details[9]. Wang and Luo present a multi-layer model to decompose an image
into a reflectance layer and a cascaded sequence of illumination [10].
Contrast can be enhanced by adjusting the dynamic range of each illumination
layer. The variational approach for Retinex, originally proposed by Kimmel et
al. [11], provides a unified framework to formulate the illumination
estimation as an optimization problem. Since decomposing an image into the
illumination and the reflectance is a mathematically ill-posed problem, and
there are no ground truth reflectance and illumination images for natural
scenes, various assumptions on the illumination[11], or both the illumination
and the reflectance [12], [13],[14] are imposed on the Retinex framework.
Within this framework, the subsequent studies differ mainly in modeling such
assumptions and priors through different norms and regularization terms,
focusing on different aspects of illumination recovery, such as edge-aware
smoothing, illumination structure preservation, and local texture extraction
[12, 14, 15, 16, 17]. Variation-based methods require solving complex
objective functions.

Unlike Retinex-based techniques, some studies assume that the observed image
is the result of an ideal image being degraded by the air light scattering
model, which is typically used in image dehazing methods. Following this idea,
several methods employ dehazing techniques and related priors to enhance
images by treating the inverted unevenly-lit images as hazy images [18], [19].
Yu and Zhu recently propose a method to enhance images by iteratively
estimating the ambient light and the light-scattering attenuation rate [20].
In another recent work, Wang et al. propose an enhancement method by
estimating the absorption light and the transmittance [21]. Both methods
produce impressive results.

The performance of the PLM-based methods depends on the assumptions or priors
about real scenes. For example, a common assumption is that scene lighting
varies smoothly in space. However, this assumption is easily violated in
unevenly-lit scenes [22]. In addition, since image enhancement is closely
related to subjective preferences, assumptions or priors based on physical
lighting may lead to discrepancies between the enhanced results and the human
perception of scenes.

Figure 1: The flowchart of the proposed method. All computations are element-
wise.

###  I-C Deep-learning based enhancement

Recent advances in convolutional neural networks(CNNs) have shown that deep-
learning techniques can help with image enhancement. Based on paired training
images, some highly-cited studies include: Wei et al. propose RetinexNet to
enhance low-light images [23]. Wang et al. propose a convolution neural
network to enhance underexposed images by learning the illumination map from
paired low-light/normal images [24]. Xu et al. construct a hierarchical
feature mining network to enhance low-light images[25]. Zhang et al. propose
KIND and its upgraded network KIND++[26, 27]. KIND and KIND++ decompose the
input image into the reflectance and illumination images and adjust both
images to achieve the enhanced result. In contrast to the paired image based
methods, EnlightenGAN proposed by Jiang et al. is an unsupervised deep-
learning based image enhancement method trained on unpaired datasets[28]. Guo
et al. propose a deep curve estimation network named ZeroDCE and its light-
weighted version ZeroDCE++ [29, 30]. ZeroDCE and ZeroDCE++ achieve image
enhancement by learning elegantly designed mapping curves instead of learning
image-to-image mapping, so the training procedure does not require paired
data. More recently, Zhao et al. propose a zero-reference RetinexDIP network
to achieve image enhancement by learning the reflectance and illumination
images [31]. Based on a semantically contrastive learning network, Liang et
al. propose an impressive SCL-LLE method for image enhancement [32].

The performance of learning based methods heavily hinges on the training
datasets, the carefully fine-tuned loss functions, the image-specific mapping
curves, and the expensive hardware resources.

Is it possible to enhance unevenly-lit images neither requiring learning
procedures nor imposing sophisticated priors or assumptions on scenes? The
ease with which the HVS perceives real scenes across a wide range of lighting
variations suggests that visual models may help to solve this problem of
naturally enhancing the images taken in unevenly-lit conditions. In this
paper, we propose an enhancement method using a vision-based exploratory data
analysis model (VEDA). The major contributions of our work are as follows:

  * •

We propose a new perspective on image decomposition for image enhancement. The
key idea is to decompose the image into two images: the output image of a
retinal model (RM), which contains the scene contrasts, and the residual
image, which contains most of the scene lighting variations. The enhanced
image is obtained by manipulating the two images. To the best of our
knowledge, little work has been reported on decomposing an image into the
contrast and the residual for image enhancement. Our method is strikingly
simple. The processing flow consists of merely a few simple computations,
without any explicit assumptions or priors about physical lighting or surface
reflectance, without iteratively solving any complex optimization functions,
and without any learning procedures.

  * •

We propose a residual image based weighted averaging strategy to combine
enhanced results at multiple scales. A similar residual image based weighting
scheme can also be leveraged to reconcile noise suppression and detail
preservation.

  * •

Experiments on a variety of image datasets demonstrate the performance of our
method is competitive with several state-of-the-art methods.

The remainder of this paper is organized as follows: Section II illustrates
the proposed method. Section III presents the experimental results. Section IV
conducts a discussion. The conclusion is drawn in Section V.

##  II Image enhancement by the VEDA

###  II-A Motivation and Overview

The motivation for our method is twofold: firstly, it is widely accepted that
the receptive field (RF) in the retina primarily responds to local contrast
changes rather than absolute light levels. This mechanism allows the retina to
function analogously to an excellent image processing model capable of
perceiving details in natural scenes across the wide range of light levels.
Secondly, a well-known concept in exploratory data analysis is to view a
signal as the sum of the output of a particular model and the residual between
the model output and the input [33]. Accordingly, we regard an input image as
the superimposition of two images: one image is the output of a retina model,
and the other one is the residual image, simply given by image = RM +
residual. The RM preferentially extracts the scene contrasts, while the
residual image contains the lighting variations that cannot be fully processed
by the RM. Thus, the enhanced image can be achieved by adjusting these two
images and recombining them.

Fig. 1 shows the flowchart of the proposed method. The first step is to
convert input image $\bf{T}$ to the logarithmic domain by
${\bf{I}}=\log\left({{\bf{T}}}\right)$, as the logarithm of the incident light
is a rough approximation to the perceived brightness [11]. Second, the
contrast image ${{\bf{R}}_{P}}$ is extracted from $\bf{I}$ by the RM, and then
the residual image ${{\bf{L}}_{R}}$ is obtained by subtracting
${{\bf{R}}_{P}}$ from $\bf{I}$. Third, since the residual image contains most
of lighting variations, ${{\bf{L}}_{R}}$ is manipulated through a luminance
modification (LM) unit to achieve ${{\bf{L}}_{P}}$, referred to as the
perceived residual image in this paper. Finally, the enhanced intensity image
${{\bf{T}}_{E}}$ is generated by converting back the sum of ${{\bf{L}}_{P}}$
and ${{\bf{R}}_{P}}$ to the intensity domain. Note that all computations are
element-wise and are applied to the V channel in the HSV color space to avoid
color shifts.

###  II-B Extracting the contrast image by the retinal model

A variety of RMs have been proposed to explain various characteristics of the
HVS [34, 35, 36, 37, 38, 39, 40, 41]. After testing many of them, we find that
the center-surround shunting equation, which simulates the rate of change in
neuron activities [42, 43], provides the best performance within the scope of
this paper. A simplified version of this time-varying equation takes the form:

$\frac{d}{{dt}}{\bf{r}}=-m{\bf{r}}+\left({g-{\bf{r}}}\right){\bf{C}}-\left({g+{\bf{r}}}\right){{\bf{S}}_{\sigma}}$
(1)

where $\bf{r}$ denotes the neuron activity of the center-surround RF, $m$ is
the decay rate, $g$ is the positive gain, $\bf{C}$ is the center input
stimulus which generally has one pixel width for practical image processing,
namely ${\bf{C}}={\bf{I}}$, and ${{\bf{S}}_{\sigma}}$ is the surround input
stimulus given by

${{\bf{S}}_{\sigma}}={\bf{I}}*{w_{\sigma}}$

where $*$ is the convolution operator, ${w_{\sigma}}$ is a Gaussian kernel
with a standard deviation $\sigma$. We set the Gaussian kernel width ${W_{S}}$
as ${W_{S}}=6ceil\left(\sigma\right)+1$ according to the Pauta criterion,
where ${ceil\left(\bullet\right)}$ is the function that rounds the argument
value to the nearest integer greater than or equal to the value.

Eq. (1) is an initial value problem of an ordinary differential equation.
Given $t=0$ , ${\bf{r}}=0$, its solution is

${\bf{r}}=g\frac{{{\bf{I}}-{{\bf{S}}_{\sigma}}}}{{m+{\bf{I}}+{{\bf{S}}_{\sigma}}}}\left\\{{1-\exp\left[{-\left({m+{\bf{I}}+{{\bf{S}}_{\sigma}}}\right)t}\right]}\right\\}$
(2)

Eq. (2) simulates the adaptation mechanism of the RF neurons in response to
light stimuli. When the neuron activity reaches to the steady state, i.e.,
$t\to\infty$ , the time-decaying term vanishes, and the output of the RM,
denoted as ${{\bf{R}}_{P}}$, is achieved by

${{\bf{R}}_{P}}={\bf{r}}=g\frac{{{\bf{I}}-{{\bf{S}}_{\sigma}}}}{{m+{\bf{I}}+{{\bf{S}}_{\sigma}}}}$
(3)

Thus, the RM extracts the scene contrasts by the ratio of the difference-of-
Gaussian and the biased sum-of-Gaussian.

###  II-C Yielding the residual image

(a)

(b)

(c)

(d)

(e)

Figure 2: Decomposition example. (a) Input. (b) Contrast image. (c) Residual
image. (d) Perceived residual image. (e) Enhanced result.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3:  Enhanced results at multiple scales and result with WGIF. (a)
Input. (b) $\sigma=1$. (c) $\sigma=4$. (d) $\sigma=16$. (e) Multi-scale. (f)
Result using WGIF. Larger $\sigma$ values provide better overall contrast than
smaller ones, but tend to produce halos. The multi-scale form produces better
visual quality than any single scale, but halos are still visible, as
indicated by the yellow arrows. WGIF effectively removes halos. Please zoom in
to see details.

The residual image ${{\bf{L}}_{R}}$ is obtained by subtracting the contrast
image ${{\bf{R}}_{P}}$ from ${\bf{I}}$

${{\bf{L}}_{R}}={\bf{I}}-{{\bf{R}}_{P}}$ (4)

Figs. 2(a), (b), and (c) show an example of decomposing an image into the
contrast image and the residual image, respectively. It can be observed that
the contrast image retains the visual details, while the residual image mainly
retains the scene lighting variations.

The dynamic range of scene lighting is usually vast. Hence, we develop a
luminance modification (LM) function to compress the dynamic range of the
residual image. The LM function should be progressive, suppressing larger
luminance values more heavily than smaller ones[44]. In addition, the LM
function is desired to be analogous to the visual response to light intensity.
After examining a number of functions that satisfy these goals, we find that
power law [45] performs the best, which is a linear function in the
logarithmic domain:

${{\bf{L}}_{P}}=LM\left({{{\bf{L}}_{R}}}\right)=\gamma{{\bf{L}}_{R}}+k$ (5)

where $\gamma$ and $k$ are both constants, and ${{\bf{L}}_{P}}$ is the
perceived residual image.

Fig. 2(d) shows the perceived residual image. Compared with (c), the
brightness in shaded areas is improved.

###  II-D Achieving the enhanced image

The single-scale enhanced image ${{\bf{T}}_{E}}$ is obtained by recombining
the contrast image and the perceived residual image, and converting the result
back to the intensity domain, given by

${{\bf{T}}_{E}}=\exp\left({{{\bf{R}}_{P}}}+{{{\bf{L}}_{P}}}\right)$ (6)

The enhanced result of Fig. 2(a) is shown in (e). Our method succeeds in
bringing out details from the shadowed areas while maintaining good contrasts
elsewhere.

Complex images contain contrasts at multiple scales [46]. Therefore, we
develop a multi-scale strategy to produce the final result ${{\bf{T}}_{MSE}}$
by taking the weighted average of enhanced images at all scales:

${{\bf{T}}_{MSE}}=\sum\nolimits_{n=1}^{N}{{\varphi_{n}}{{\bf{T}}_{E,n}}}$ (7)

where $N$ is the number of scales, ${{\bf{T}}_{E,n}}$ is the enhanced image at
the $n$th scale, and ${\varphi_{n}}$ is the weight associated with the $n$th
scale. Since the HVS is more likely to see details in brightly-lit regions
than in weakly-lit regions, larger weights are assigned to bright regions:

${\varphi_{n}}=\frac{{{{\bf{L}}_{R,\;n}}}}{{\sum\nolimits_{n=1}^{N}{{{\bf{L}}_{R,\;n}}}}}$
(8)

where ${{\bf{L}}_{R,\;n}}$ is the residual image at the $n$th scale.

Algorithm 1 Image enhancement based on the VEDA

Input:
Image ${\bf{T}}$, parameters ${m}$, $g$, $\gamma$, and $k$.
Output:
Enhanced image ${{\bf{T}}_{MSE}}$.
Begin

1: Convert ${\bf{T}}$ to the logarithmic domain by
${\bf{I}}=\log\left({{\bf{T}}}\right)$;

2: Construct the surround images ${{\bf{S}}_{\sigma}}={{\bf{S}}_{\sigma
1}},...,{{\bf{S}}_{\sigma n}}$ using the weighted guided image filter at
different scales;

3: for each ${{\bf{S}}_{\sigma}}$ do

4: Extract the contrast image ${{\bf{R}}_{P}}$ via Eq. (3);

5: Calculate the residual image ${{\bf{L}}_{R}}$ via Eq. (4);

6: Achieve the perceived residual image ${{\bf{L}}_{P}}$ via Eq. (5);

7: Obtain the enhanced image ${{\bf{T}}_{E}}$ via Eq. (6);

8: end for

9: Calculate the weight ${\varphi_{n}}$ via Eq. (8);

10: Obtain ${{\bf{T}}_{MSE}}$ via Eq. (7);

End

The difference between ${{\bf{T}}_{E,n}}$ and ${{\bf{T}}_{E}}$ is essentially
governed by the surround images associated with different values of $\sigma$.
We construct the multi-scale surround images by the sequence
$\sigma\in\left\\{{{\sigma_{1}},{\sigma_{2}},\cdots,{\sigma_{n}},\cdots{\sigma_{N}}}\right\\}$,
where ${\sigma_{n}}={4^{n-1}}{\sigma_{1}}$. In this paper, we set
${\sigma_{1}}=1$ to detect small details and the maximum number of scales
$N=3$ due to the limited size of the RF. In addition, the values of $g$, $m$,
$\gamma$, and $k$ are invariant across scales, respectively. The influence of
the four free parameters is discussed in Section III-B.

Figs. 3(b) to (d) show the enhanced results for (a) at different scales.
Larger $\sigma$ values achieve better overall contrast than smaller $\sigma$
values, while smaller $\sigma$ values produce better local details. It is
worth noting that the difference-of-Gaussian in Eq. (3) tends to yield halo
artifacts around sharp edges, as shown in (c) and (d). The weighted multi-
scale averaging is capable of a high degree of halo removal, but cannot
eliminate them entirely. To better remove halos, we produce the surround image
${{\bf{S}}_{\sigma}}$ by the weighted guided image filter(WGIF) [47] instead
of the Gaussian filter. The result produced by the WGIF is shown in (f), where
the halos in (e) are effectively eliminated.

In summary, the entire procedure of the proposed method is outlined in
Algorithm 1.

##  III Experimental results

In this section, we first present the experimental settings. Then, we study
the effect of involved parameters on results. Finally, we make qualitative and
quantitative comparisons with several state-of-the-art methods to demonstrate
the performance of our method.

###  III-A Experimental settings

Computational environment: All non-deep-learning methods are run in the MATLAB
2019a environment on a PC with 16G RAM and 2.9GHz Intel i7-10700k CPU. All
compared deep-learning based methods are deployed on dual NVIDIA TITAN GTX
GPUs.

Compared methods: The compared methods include: DFE[9], LIME[15], MLLP[10],
PLME[20], ALSM[21], PnpRetinex(PnpRtx)[16], NRMOE[48], EnlightenGAN(EGAN)[28],
SCL-LLE[32], RetinexDIP(RtxDIP)[31], KIND++[27], and ZeroDCE[29]. For a fair
comparison, all results are produced by publicly available codes with
parameters set as exactly as given in their papers.

Datasets: We compare all methods on two test sets. The first test set, denoted
as Testset-1, contains 266 images collected from a variety of publicly
available datasets without reference images, including NPE (85 images)[8],
MLLP (76 images)[10], VV (24 images)[49], DICM (64 images) [50] and MEF (17
images)[51]. The second test set (Testset-2) is built on the Part2 subset of
SICE dataset[52], which consists of 229 multi-exposure sequences and their
corresponding reference image (Ref.Image). We select the first three under-
exposed images in each multi-exposure sequence of Part2 subset for testing
since the compared methods are not specially designed for over-exposed images.
The test images are resized to 25${\%}$ of their original size due to the
memory limitation of a previous study[16].

Objective Metrics: To the best of our knowledge, there is no widely accepted
measure to quantitatively assess the quality of enhanced images since image
quality assessment (IQA) is highly related to subjective preferences. We
evaluate all compared methods on Testset-1 by three representative image
quality assessment (IQA) metrics: NIQE[53], BIQI[54], and NFERM[55]. NIQE
estimates the deviations between the target image and a statistical model of
natural scenes. BIQI measures the human perception of the naturalness of an
image. NFERM scores the image quality based on the free-energy based brain
theory and HVS features. For these three metrics, smaller values represent
better image quality. For Testset-2, three fully referenced IQA metrics: peak
signal-to-noise ratio(PSNR), structure similarity(SSIM)[56] and lightness
order error(LOE)[8] are also adopted. Higher PSNR and SSIM values indicate
better image quality and lower LOE values indicate better degree of
naturalness preservation[8].

###  III-B Parameter study

There are four free parameters in the proposed method: two power law related
parameters $\gamma$ and $k$ in Eq. (5), the decay rate $m$, and the gain ${g}$
in Eq. (3).

Firstly, both $\gamma$ and $k$ modify the shape of the LM function, which
should be a progressively attenuating curve, thus, $\gamma<1$. Smaller
$\gamma$ values tend to bring out contrasts in shadow regions, while larger
$\gamma$ values result in over-exposure of bright regions. To determine their
reasonable values, we run experiments on DCIM dataset and compute the three
averaged IQA metrics of the results for different $\left({\gamma,k}\right)$
pairs, where ${\gamma}$ ranges from 0.1 to 0.9 in steps of 0.1, and ${k}$
ranges from ${log(5)}$ to ${log(55)}$ in steps of 5. We set ${m=g=1}$ to avoid
extensive parameter tuning. Fig. LABEL:ref_params_gamma_k(a) shows an example
with different $\left({\gamma,k}\right)$ pairs. Results demonstrates that the
pair $\left({\gamma,k}\right)=\left({0.6,log(10)}\right)$ achieves the best
average objective assessment score, as evidenced by the lowest NIQE, BIQI and
NFERM values in Fig. LABEL:ref_params_gamma_k(b).

Secondly, since the RM produces ratio-type contrast images rather than high-
pass filtered images, the value of ${m}$ should satisfy
${m\ll\left({{\bf{I}}+{{\bf{S}}_{\sigma}}}\right)}$. An example is presented
in the first row of Fig.5, where
$\left({\gamma,k}\right)=\left({0.6,log(10)}\right)$ and ${g=1}$. It can be
seen that the ${m}$ values change from 0.1 to 20 with minor effects on the
output quality. The IQA curves in Fig.5 show that larger ${m}$ values produce
slightly higher NIQE, BIQI, and NFERM values, indicating slightly lower image
quality. We achieve an appropriate balance between the quantitative scores and
visual quality by setting ${m=1}$.

Finally, the gain parameter ${g}$ controls the amplitude of the contrast
image. Fig.6 presents the visual results and IQA curves when $g$ changes from
0.5 to 2.0 in steps of 0.25 with constant ${m=1}$ and
$\left({\gamma,k}\right)=\left({0.6,log(10)}\right)$. Larger ${g}$ values lead
to stronger image enhancement, but are more likely to result in over-
enhancement. As can be seen in Fig.6, results with ${g=1}$ have the lowest
average NIQE and NFERM values, and almost the lowest BIQI value.

Based on the visual quality and the quantitative evaluation, we set
$\left({\gamma,k}\right)=\left({0.6,log(10)}\right)$, ${g=1}$, and ${m=1}$ for
our method.

(a) (a) ${m=0.1}$

(b) (b) ${m=1}$

(c) (c) ${m=5}$

(d) (d) ${m=10}$

(e) (e) ${m=15}$

(f) (f) ${m=20}$

(g) (g) NIQE-BIQI-NFERM

Figure 5: The influence of ${m}$. The ${m}$ values change from 0.1 to 20 with
minor effects on the output quality.

(a) (a) ${g=0.5}$

(b) (b) ${g=0.75}$

(c) (c) ${g=1}$

(d) (d) ${g=1.25}$

(e) (e) ${g=1.5}$

(f) (f) ${g=1.75}$

(g) (g) ${g=2.0}$

(h) (h) NIQE-BIQI-NFERM

Figure 6: The influence of ${g}$. Larger ${g}$ values lead to stronger image
enhancement, but are more likely to result in over-enhancement.

###  III-C Comparisons

(a) Input

(b) DFE

(c) ALSM

(d) LIME

(e) MLLP

(f) PnpRtx

(g) NRMOE

(h) PLME

(i) EGAN

(j) KIND++

(k) ZeroDCE

(l) SCL-LLE

(m) RtxDIP

(n) Proposed

Figure 7: Comparison 1. The images obtained by LIME and MLLP are significantly
over-enhanced. The result of RetinexDIP contains noticeable artifacts. Other
methods yield graceful results. Please zoom in to see details.

(a) Input

(b) DFE

(c) ALSM

(d) LIME

(e) MLLP

(f) PnpRtx

(g) NRMOE

(h) PLME

(i) EGAN

(j) KIND++

(k) ZeroDCE

(l) SCL-LLE

(m) RtxDIP

(n) Proposed

Figure 8: Comparison 2. ALSM and LIME achieve over-exposed results. There are
evident artifacts in the results of KIND++, and RetinexDIP. EnlightenGAN,
NRMOE, ZeroDCE, and SCL-LLE produce results with varying degrees of color
distortion. DFE, PnpRetinex, PLME, and the proposed method generate visually-
pleasing results. Please zoom in to see details.

(a) Input

(b) DFE

(c) ALSM

(d) LIME

(e) MLLP

(f) PnpRtx

(g) NRMOE

(h) PLME

(i) EGAN

(j) KIND++

(k) ZeroDCE

(l) SCL-LLE

(m) RtxDIP

(n) Proposed

Figure 9: Comparison 3. DFE, MLLP, PnpRetinex, PLME, RetinexDIP, and the
proposed method produce better results than the other methods in terms of
fidelity. Please zoom in to see details.

(a) Input

(b) DFE

(c) ALSM

(d) LIME

(e) MLLP

(f) PnpRtx

(g) NRMOE

(h) PLME

(i) EGAN

(j) KIND++

(k) ZeroDCE

(l) SCL-LLE

(m) RtxDIP

(n) Proposed

Figure 10: Comparison 4. DFE, PnpRetinex, and the proposed method produce more
visually-appealing results than the other methods. However, the result of DFE
suffers from slight halo artifacts. Please zoom in to see details.

1) Qualitative assessment

Figs. 7 to 12 are representative comparisons. Referring to these results, we
make the following comments:

  1. 1.

Both ALSM and LIME can significantly improve contrast, but tend to over-
enhance images, resulting in unnatural appearance, see Figs.8(c), and (d) to
12(c), and (d).

  2. 2.

MLLP is capable of a good degree of highlighting details, but may produce
noticeable plaque artifacts (see Figs.7(e) and 10(e).

  3. 3.

NRMOE tends to introduce color distortion in the results, as shown in
Figs.9(g), 11(g), and 12(g).

  4. 4.

DFE, PnpRetinex, and PLME are all effective in enhancing images with vivid
color. Comparatively, DFE and PnpRetinex are better at bringing out details,
as demonstrated in Figs.10(b), (f), and (h), and Figs.11(b), (f), and (g),
respectively. However, DFE may give rise to prominent halos around abrupt
bright edges (see the marked region in Fig.11(b)). PnpRetinex tends to
generate slightly over-sharpened edges, as seen in Figs.8(f) and 9(f).

  5. 5.

All deep-learning based methods can significantly enhance contrast in shadow
areas. However, the results frequently suffer from unstable image quality.
EnlightenGAN, KIND++, and RetinexDIP easily generate unwanted artifacts when
highlighting shadow areas (see Figs.7(i), (j), and (m) to 10(i), (j), and (m),
respectively). Both ZeroDCE and SCL-LLE usually produce results with reduced
color saturation. The reason may be that their limited training datasets are
incapable of covering the various natural scenes.

Compared to these enhancement methods, the proposed method offers a better
balance between improving contrast and preserving naturalness without
introducing undesired artifacts or color shifts.

(a) Input

(b) DFE

(c) ALSM

(d) LIME

(e) MLLP

(f) PnpRtx

(g) NRMOE

(h) PLME

(i) EGAN

(j) KIND++

(k) ZeroDCE

(l) SCL-LLE

(m) RtxDIP

(n) Proposed

Figure 11: Comparison 5. DFE produces noticeable halos around abrupt bright
edges, see the marked region. PnpRetinex, PLME, SCL-LLE, RetinexDIP, and the
proposed method produce more natural-looking results than other methods.

(a) Input

(b) DFE

(c) ALSM

(d) LIME

(e) MLLP

(f) PnpRtx

(g) NRMOE

(h) PLME

(i) EGAN

(j) KIND++

(k) ZeroDCE

(l) SCL-LLE

(m) RtxDIP

(n) Proposed

Figure 12: Comparison 6. The results of ALSM, LIME, and PLME suffer from
varying degrees of overexposure. ZeroDCE,SCL-LLE, and NRMOE produce rather
pale images. The results of MLLP, EnlightenGAN and KIND++ contain noticeable
artifacts. DFE, PnpRetinex, RetinexDIP, and the proposed method yield similar
results.

2) User study

We conduct a user study with 13 observers to quantify the subjective
assessment of the compared methods on Testset-1. The observers are trained
from three aspects: 1) no severe artifacts such as over-, under-exposed
regions, and halos are introduced, 2) the color rendition of the scene is
perceptually natural, and 3) enhanced details are visually-pleasing [10].

TABLE I: Comparison of MOS on Testset-1. The best score is in red and the second best one is in blue. | Testset-1
---|---
Method | NPE | MLLP | VV | DCIM | MEF | Average
DFE | 4.36 | 4.23 | 4.49 | 4.59 | 4.72 | 4.48
LIME | 4.34 | 4.07 | 4.20 | 4.12 | 4.29 | 4.20
ALSM | 4.33 | 3.89 | 4.16 | 4.03 | 4.31 | 4.14
MLLP | 4.29 | 4.01 | 3.83 | 3.94 | 4.06 | 4.03
NRMOE | 3.63 | 3.54 | 2.67 | 3.23 | 3.53 | 3.32
PLME | 4.20 | 4.13 | 4.49 | 4.48 | 4.67 | 4.39
PnpRetinex | 4.45 | 4.28 | 4.54 | 4.66 | 4.79 | 4.54
EnlightenGAN | 3.91 | 3.92 | 3.45 | 4.15 | 4.47 | 3.98
KIND++ | 3.57 | 3.62 | 3.31 | 3.68 | 4.04 | 3.64
ZeroDCE | 4.29 | 4.11 | 4.21 | 4.43 | 4.71 | 4.35
SCL-LLE | 4.13 | 4.12 | 4.32 | 4.40 | 4.58 | 4.31
RetinexDIP | 3.74 | 3.87 | 3.75 | 3.85 | 4.22 | 3.89
Proposed | 4.40 | 4.31 | 4.56 | 4.66 | 4.79 | 4.54

The user study is designed as follows: 1) an original image and one of its
randomly ordered enhanced results are simultaneously displayed on the screen.
The original is used as the reference in each trial. 2) The observers evaluate
the enhanced images based on their understanding of the scene. The observers
can switch enhanced images back and forth in each trial to make the final
rating [57]. 3) The quality score ranges from 1 to 5 (worst to best quality)
with a step-size of one. Results achieved by different methods may have the
same score, which means that the visual differences between the results are
not enough for the observer to make a preferred rating. These scores are then
averaged to yield the Mean Opinion Score (MOS) value for each method.

The MOS values for each image set are listed in Table I. The proposed method
obtains the second highest score in NPE dataset, and the highest scores in the
other four datasets. It also achieves the highest average MOS value aross all
test images, tied for first place with PnpRetinex. This small-scale user study
provides additional support that the proposed method outperforms other
compared methods in terms of visual quality.

3) Quantitative assessment

Table II shows the evaluation results on Testset-1. The proposed method
obtains the lowest BIQI values on all datasets. Therefore, our method yields
results with better naturalness perception than other compared methods. In
terms of the NIQE metric, the proposed method achieves the lowest average
values on NPE, MLLP, and DCIM, and the second lowest values on VV and MEF,
slightly higher than DFE and EnlightenGAN, respectively. This means that our
method can compete with state-of-the-art methods in producing results similar
to natural images. Our method ranks behind PnpRetinex on NPE, MLLP, DCIM, and
VV in terms of average NFERM values, and behind PLME and PnpRetinex on MEF.
This indicates that our method is comparable to the state-of-the-art methods
in terms of image quality based on the HVS-aware features of natural scenes.

TABLE II:  Comparison of NIQE(G), BIQI(B) and NFERM(F) values for Testset-1. The scores for the first, second, and third places are marked in red, blue, and green, respectively. | NPE | MLLP | DCIM | VV | MEF
---|---|---|---|---|---
Method | N | B | F | N | B | F | N | B | F | N | B | F | N | B | F
DFE | 2.84 | 30.13 | 22.99 | 2.82 | 29.88 | 15.97 | 2.81 | 27.88 | 14.84 | 2.38 | 30.82 | 35.78 | 2.70 | 28.75 | 17.14
LIME | 3.16 | 38.78 | 23.21 | 3.28 | 40.87 | 18.78 | 3.02 | 31.62 | 20.38 | 2.49 | 32.89 | 32.47 | 3.52 | 35.09 | 24.05
MLLP | 2.96 | 34.07 | 23.01 | 2.93 | 32.35 | 17.79 | 3.21 | 28.44 | 19.32 | 2.56 | 35.69 | 40.03 | 2.76 | 29.87 | 19.08
ALSM | 3.04 | 36.37 | 25.66 | 3.09 | 35.59 | 16.67 | 2.95 | 29.52 | 20.05 | 2.50 | 31.47 | 34.90 | 2.71 | 30.50 | 19.07
NRMOE | 3.51 | 36.19 | 25.71 | 3.35 | 34.45 | 20.29 | 3.08 | 29.08 | 21.81 | 2.66 | 35.2 | 39.76 | 3.17 | 29.79 | 20.22
PLME | 2.91 | 32.28 | 23.79 | 2.78 | 29.85 | 16.53 | 2.93 | 28.32 | 12.01 | 2.51 | 31.48 | 34.61 | 2.93 | 28.64 | 17.47
PnpRetinex | 2.85 | 30.86 | 21.50 | 2.97 | 34.52 | 15.01 | 2.85 | 27.85 | 15.07 | 2.51 | 30.97 | 22.47 | 2.78 | 32.07 | 18.23
RetinexDIP | 3.31 | 35.09 | 25.21 | 3.26 | 29.93 | 18.29 | 3.35 | 30.52 | 16.99 | 3.04 | 31.08 | 32.89 | 3.06 | 31.09 | 24.10
EnlightenGAN | 3.35 | 31.02 | 28.17 | 2.98 | 26.10 | 17.51 | 3.56 | 32.41 | 18.84 | 3.38 | 37.65 | 37.46 | 2.59 | 30.92 | 18.95
KIND++ | 3.11 | 33.53 | 28.46 | 3.32 | 34.01 | 17.16 | 2.83 | 31.50 | 19.08 | 3.25 | 32.66 | 44.07 | 3.32 | 34.52 | 20.65
SCL-LLE | 3.03 | 31.83 | 24.36 | 2.97 | 30.77 | 16.69 | 3.58 | 28.08 | 17.86 | 2.61 | 31.28 | 37.09 | 2.82 | 28.73 | 19.30
ZeroDCE | 3.05 | 32.54 | 24.46 | 3.14 | 32.13 | 17.25 | 3.66 | 28.25 | 14.90 | 2.68 | 32.81 | 31.39 | 3.11 | 29.39 | 19.18
Proposed | 2.79 | 30.03 | 22.04 | 2.77 | 28.68 | 15.64 | 2.66 | 27.12 | 14.49 | 2.47 | 30.54 | 27.02 | 2.65 | 27.85 | 17.93

(a) Input

(b) Ref.Image

(c) DFE

(d) ALSM

(e) LIME

(f) MLLP

(g) PnpRtx

(h) NRMOE

(i) PLME

(j) EGAN

(k) KIND++

(l) ZeroDCE

(m) SCL-LLE

(n) RtxDIP

(o) Proposed

Figure 13: Comparison 1 of results from SICE. TABLE III:  Quantitative assessment on Testset-2. The scores for the first, second, and third places are marked in red, blue, and green, respectively. Method | NIQE | BIQI | NFERM | PSNR | SSIM | LOE
---|---|---|---|---|---|---
DFE | 2.55 | 30.67 | 20.46 | 16.57 | 0.651 | 383.3
LIME | 2.72 | 35.40 | 22.55 | 14.52 | 0.623 | 417.9
MLLP | 2.45 | 30.46 | 23.50 | 16.68 | 0.647 | 528.8
ALSM | 2.57 | 33.55 | 23.86 | 14.55 | 0.635 | 389.7
NRMOE | 2.78 | 31.24 | 23.61 | 15.72 | 0.637 | 416.6
PLME | 2.63 | 30.35 | 19.33 | 16.82 | 0.657 | 425.9
PnpRetinex | 2.58 | 33.17 | 19.99 | 17.09 | 0.661 | 372.2
RetinexDIP | 2.90 | 31.90 | 26.53 | 15.62 | 0.641 | 544.9
EnlightenGAN | 2.68 | 30.76 | 21.55 | 16.47 | 0.683 | 528.9
KIND++ | 2.93 | 32.67 | 25.76 | 16.64 | 0.658 | 586.5
SCL-LLE | 2.75 | 30.71 | 26.28 | 16.93 | 0.650 | 378.2
ZeroDCE | 2.70 | 30.94 | 23.26 | 18.82 | 0.693 | 408.9
Proposed | 2.55 | 29.55 | 20.20 | 17.33 | 0.672 | 359.0

(a) Input

(b) Ref.Image

(c) DFE

(d) ALSM

(e) LIME

(f) MLLP

(g) PnpRtx

(h) NRMOE

(i) PLME

(j) EGAN

(k) KIND++

(l) ZeroDCE

(m) SCL-LLE

(n) RtxDIP

(o) Proposed

Figure 14: Comparison 2 of results from SICE.

Table III presents the quantitative assessment results on Testset-2. The
proposed method achieves the best BIQI score, the second-best NIQE score, and
the third-best NFERM score. This means that our method performs consistently
in terms of the three no-reference IQA metrics. Our method also obtains the
lowest LOE value for this dataset, indicating the best performance on
naturalness preservation. ZeroDCE obtains the highest PSNR and SSIM values,
and EnlightenGAN obtains the second highest SSIM value, while our method ranks
second and third, respectively. The reason may be that ZeroDCE and
EnlightenGAN are trained on the Part1 of SICE dataset and have learned how to
enhance the test images with parameters trained from the training set. Figs.13
and 14 present two examples from Testset-2 for visual comparison. We can
observe that ZeroDCE and EnlightenGAN suffer from color deviation. The same
drawback occurs in the results of SCL-LLE and NRMOE. KIND++ and RetinexDIP
yield results with noticeable artifacts. We consider that the results obtained
by DFE, PnpRetinex, MLLP, and our method are closer to the reference images
than those by the other methods.

4) Comparison on speed

We compare the average computation time of all compared methods over 100
images with size of 1368*912. As shown in Table IV, the proposed method is the
fastest method on CPU platform and the averaged computation time of our method
is 0.23s. The deep-learning based methods ZeroDCE and SCL-LLE are faster than
our method, but they require expensive GPU resources.

TABLE IV: Comparison of the average computation time (in seconds). The top three fastest methods are marked in red,blue, and green respectively. Platform | Method | Time | Method | Time
---|---|---|---|---
CPU | DFE | 8.65 | LIME | 0.42
| ALSM | 65.37 | MLLP | 45.08
| NRMOE | 3.56 | PLME | 0.77
| PnpRetinex | 13.24 | Proposed | 0.23
GPU | EnlightenGAN | 0.84 | KIND++ | 3.58
| SCL-LLE | 0.0069 | RetinexDIP | 42.37
| ZeroDCE | 0.0041 |  |

##  IV Discussion

###  IV-A Comparison with Retinex decomposition

Assuming that an image is the product of an illumination image and a
reflectance image, Retinex-based enhancement methods deal with the challenge
of decomposing a given image into the illumination image and the reflectance
image. Computing such a decomposition for real scenes is an ill-posed problem.
Consequently, any attempt to solve it must make some simplifying assumptions
about the scene, such as the spatial smoothness of illumination, or the piece-
wise consistency/similarity of albedo [15, 16]. These assumptions about the
scene imaging process are easily violated in real scenes. An example is shown
in Fig.15(a). The lower half of (a) is the reflection image from a highly
reflective surface (e.g., a mirror or a still pond). The nearly identical
reflectance of the pond surface suggests that the incident illumination of
this reflection region is not spatially smooth. In addition, the light source
regions in (a) go against the assumption that an image is the product of
illumination and reflectance. The images in Fig.15(b) to (d), from left to
right, are the reflectance, illumination, and enhanced images produced by
three Retinex-based methods DFE, LIME, and PnpRetinex, respectively. It can be
seen that the pond regions in the three illumination images are locally
smooth. The images in (e), from left to right, are the contrast, residual, and
enhanced images produced by our method. The contrast image looks similar to
the reflectance images, while the residual image is similar to the
illumination images. This phenomenon could be explained by two widely accepted
and closely related assumptions about the HVS[58]: 1) the HVS is mostly
sensitive to surface reflectance, and 2) the HVS primarily responds to local
scene contrasts. It might be argued that the contrast, to some extent, is a
function of perceived reflectance. Although the contrast and residual images
are similar to the reflectance and illumination images, their physical
implications are fundamentally different. Our model attempts to decompose the
light variations from an image in a functionally similar manner to the visual
perception of scenes instead of making physical assumptions about the imaging
process of the scene.

(a) (a) Input

(b)

(c) (b) DFE

(d)

(e)

(f) (c) LIME

(g)

(h)

(i) (d) PnpRetinex

(j)

(k)

(l) (e) Proposed

(m)

Figure 15: Comparison with Retinex decomposition. The input image violates the
assumptions made by Retinex theory.

(a) Input

(b) Weber

(c) Michaelson

(d) RMS

(e) Proposed

Figure 16:  Generality of the proposed model. The three ratio-type contrasts
achieve different degrees of image enhancement within the proposed model. The
Weber contrast obtains comparable performance to the proposed method, but
tends to wash out the details, see the marked region in (b).

###  IV-B Generality of the proposed method

The core perspective of our method is to decompose an image into the contrast
image and the residual image. An interesting question is whether other
contrast definitions can be applied to the proposed model. We discuss this
question through three well-known contrasts: the Weber contrast, the Michelson
contrast, and the root mean square (RMS) contrast [59].

The three ratio-type contrasts are defined as follows:

  * •

Weber contrast

${\bf{C}_{Weber}}=\frac{{{\bf{I}}-{{\bf{I}}_{S}}}}{{{\bf{I}}_{S}}}$ (9)

  * •

Michelson contrast

${{\bf{C}}_{Michelson}}=\frac{{{{\bf{I}}_{\max}}-{{\bf{I}}_{\min}}}}{{{{\bf{I}}_{\max}}+{{\bf{I}}_{\min}}}}$
(10)

  * •

RMS contrast

${{\bf{C}}_{RMS}}=\sqrt{\frac{1}{{{M_{\Omega}}}}\sum\limits_{\Omega}{{{\left({\frac{{{\bf{I}}-{\bf{\bar{I}}}}}{{{\bf{\bar{I}}}}}}\right)}^{2}}}}$
(11)

In Eqs. (9) to (11), $\bf{I}$ denotes the image intensity, ${{\bf{I}}_{S}}$ is
the local background intensity, ${{\bf{I}}_{\max}}$ and ${{\bf{I}}_{\min}}$
are the highest and lowest intensities in a local image patch, respectively,
${\bf{\bar{I}}}$ is the mean intensity of the local area $\Omega$, and
${M_{\Omega}}$ is the number of pixels in $\Omega$. As far as we know,
directly applying the three contrasts to enhance unevenly-lit images is less
reported. In this paper, we treat these three contrast functions as three
different retinal models to generate the contrast images. By testing a number
of images, we find that the Weber contrast produces better-enhanced results in
terms of fidelity than the other two contrasts and approaches comparable
performance to Eq. (3). However, the Weber contrast sometimes tends to lose
the details in bright regions. A representative example is shown in Fig. 16.
It might be argued that: 1) Eq. (9) can be regarded as a simplified version of
Eq. (3). 2) Eq. (3) and Eq. (9) are capable of estimating the contrast in
natural images more accurately than the other two contrasts.

###  IV-C Post-processing for noise suppression

Noise in dark regions is easily amplified after enhancement. Therefore,
denoising is sometimes required in some applications. Given that noise in
different regions of an unevenly-lit image is often amplified to different
degrees, we propose a fusion scheme to suppress noise in dark regions while
preventing details in bright regions from being over-smoothed, given by

${{\bf{T}}_{F}}={{\bf{L}}_{NR}}{{\bf{T}}_{E}}+\left({1-{{\bf{L}}_{NR}}}\right){{\bf{T}}_{DE}}$
(12)

where ${{\bf{T}}_{F}}$ is the final recomposed result, ${{\bf{L}}_{NR}}$ is
the residual image normalized to [0,1], and ${{\bf{T}}_{DE}}$ is the denoised
result of ${{\bf{T}}_{E}}$. This paper adopts the widely used denoising method
BM3D [60] to achieve ${{\bf{T}}_{DE}}$. Fig. 17 exhibits an example of noise
suppression. Compared to the images in (b) and (c), the result in (d) shows
that fine details in the bright regions are nicely preserved, while noise in
the dark regions is effectively smoothed out. We would like to mention that
the denoising in Eq.(12) can be appended as a post-processing step to any
enhancement method for unevenly-lit images.

We must emphasize that noise suppression usually results in a loss of detail
in shadows. Suppressing noise while preserving details remains an ongoing
research topic in image enhancement. In this paper, we focus on improving
contrasts rather than suppressing noise to avoid distraction. Introducing
noise estimation methods to determine whether to suppress noise could be one
of our following works.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i) (a)

(j) (b)

(k) (c)

(l) (d)

Figure 17: Example of noise suppression. (a) Input. (b) Enhanced result
without denoising. (c) Denoised version of (b). (d) Result of Eq. (12).

###  IV-D Limitations

It will be of interest to improve the proposed method in our future work in
the following aspects: 1) Our method is not good at enhancing over-exposed
images because the residual image is adjusted by a fixed gamma curve. The
gamma function cannot bring out details in both dark and bright regions. A
possible way to alleviate this issue is to train a CNN to fit light
enhancement curves under various lighting conditions. 2) The proposed method
does not take into account more characteristics of the HVS, such as frequency
selectivity and directional selectivity[59]. Incorporating more visual
mechanisms related to natural image analysis into our model may help to
improve the fidelity of the enhanced results.

##  V Conclusion

In this paper, we have proposed a biological vision based exploratory data
model to enhance images taken under unevenly-lit conditions. The proposed
model decomposes the input image into its contrast image and residual image.
The perceptually important details in the scene are preserved in the contrast
image, while the lighting variations are retained in the residual image. The
enhanced result can be achieved by manipulating the two images and recombining
them. The major advantage of the proposed method is its simplicity and
effectiveness. Unlike existing enhancement methods based on physical lighting
models or deep-learning techniques, the proposed method does not require any
explicit assumptions and prior knowledge of the natural scenes, nor any
learning procedures. Despite its simplicity, experimental results demonstrate
that the proposed method is comparable to several state-of-the-art methods.

## References

  * [1] S. Chen and A. Beghdadi, “Natural enhancement of color image,” _EURASIP Journal on Image and Video Processing_ , vol. 2010, no. 1, pp. 1–19, 2010.
  * [2] M. Nikolova and G. Steidl, “Fast hue and range preserving histogram specification: Theory and new algorithms for color image enhancement,” _IEEE Transactions on Image Processing_ , vol. 23, no. 9, pp. 4087–4100, 2014\.
  * [3] S.-C. Huang, F.-C. Cheng, and Y.-S. Chiu, “Efficient contrast enhancement using adaptive gamma correction with weighting distribution,” _IEEE Transactions on Image Processing_ , vol. 22, no. 3, pp. 1032–1041, 2013.
  * [4] K. Gu, G. Zhai, X. Yang, W. Zhang, and C. W. Chen, “Automatic contrast enhancement technology with saliency preservation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 25, no. 9, pp. 1480–1494, 2015.
  * [5] H. Xu, G. Zhai, X. Wu, and X. Yang, “Generalized equalization model for image enhancement,” _IEEE Transactions on Multimedia_ , vol. 16, no. 1, pp. 68–82, 2014.
  * [6] A. K. Bhandari, P. Kandhway, and S. Maurya, “Salp swarm algorithm-based optimally weighted histogram framework for image enhancement,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 9, pp. 6807–6815, 2020.
  * [7] D. Jobson, Z. Rahman, and G. Woodell, “A multiscale retinex for bridging the gap between color images and the human observation of scenes,” _IEEE Transactions on Image Processing_ , vol. 6, no. 7, pp. 965–976, 1997.
  * [8] S. Wang, J. Zheng, H.-M. Hu, and B. Li, “Naturalness preserved enhancement algorithm for non-uniform illumination images,” _IEEE Transactions on Image Processing_ , vol. 22, no. 9, pp. 3538–3548, 2013.
  * [9] Z. Liang, W. Liu, and R. Yao, “Contrast enhancement by nonlinear diffusion filtering,” _IEEE Transactions on Image Processing_ , vol. 25, no. 2, pp. 673–686, 2016.
  * [10] S. Wang and G. Luo, “Naturalness preserved image enhancement using a priori multi-layer lightness statistics,” _IEEE transactions on image processing_ , vol. 27, no. 2, pp. 938–948, 2018.
  * [11] R. Kimmel, M. Elad, D. Shaked, R. Keshet, and I. Sobel, “A variational framework for retinex,” _International Journal of Computer Vision_ , vol. 52, no. 1, pp. 7–23, 2003.
  * [12] X. Fu, Y. Liao, D. Zeng, Y. Huang, X. Zhang, and X. Ding, “A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation,” _IEEE Transactions on Image Processing_ , vol. 24, no. 12, pp. 4965–4977, 2015.
  * [13] L. Wang, L. Xiao, H. Liu, and Z. Wei, “Variational bayesian method for retinex,” _IEEE Transactions on Image Processing_ , vol. 23, no. 8, pp. 3381–3396, 2014.
  * [14] J. Xu, M. Yu, L. Liu, F. Zhu, D. Ren, Y. Hou, H. Wang, and L. Shao, “STAR: A structure and texture aware retinex model,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 5022–5037, 2020.
  * [15] X. Guo, Y. Li, and H. Ling, “LIME: Low-light image enhancement via illumination map estimation,” _IEEE Trans Image Process_ , vol. 26, no. 2, pp. 982–993, 2017.
  * [16] Y.-H. Lin and Y.-C. Lu, “Low-light enhancement using a plug-and-play retinex model with shrinkage mapping for illumination estimation,” _IEEE Transactions on Image Processing_ , vol. 31, pp. 4897–4908, 2022.
  * [17] M. Zhou, X. Wu, X. Wei, T. Xiang, B. Fang, and S. Kwong, “Low-light enhancement method based on a retinex model for structure preservation,” _IEEE Transactions on Multimedia_ , pp. 1–13, 2023.
  * [18] X. Dong, G. Wang, Y. Pang, W. Li, J. Wen, W. Meng, and Y. Lu, “Fast efficient algorithm for enhancement of low lighting video,” in _2011 IEEE International Conference on Multimedia and Expo_ , 2011, pp. 1–6.
  * [19] X. Zhang, P. Shen, L. Luo, L. Zhang, and J. Song, “Enhancement and noise reduction of very low light level images,” in _Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012)_ , 2012, pp. 2034–2037.
  * [20] S.-Y. Yu and H. Zhu, “Low-illumination image enhancement algorithm based on a physical lighting model,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 1, pp. 28–37, 2019.
  * [21] Y.-F. Wang, H.-M. Liu, and Z.-W. Fu, “Low-light image enhancement via the absorption light scattering model,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5679–5690, 2019.
  * [22] M. Li, J. Liu, W. Yang, X. Sun, and Z. Guo, “Structure-revealing low-light image enhancement via robust retinex model,” _IEEE Transactions on Image Processing_ , vol. 27, no. 6, pp. 2828–2841, 2018.
  * [23] C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” _British Machine Vision Conference,155,2018_ , 2018\.
  * [24] R. Wang, Q. Zhang, C.-W. Fu, X. Shen, W.-S. Zheng, and J. Jia, “Underexposed photo enhancement using deep illumination estimation,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 6849–6857.
  * [25] K. Xu, H. Chen, X. Tan, Y. Chen, Y. Jin, Y. Kan, and C. Zhu, “Hfmnet: Hierarchical feature mining network for low-light image enhancement,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 71, pp. 1–14, 2022.
  * [26] Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” in _Proceedings of the 27th ACM international conference on multimedia_ , 2019, pp. 1632–1640.
  * [27] Y. Zhang, X. Guo, J. Ma, W. Liu, and J. Zhang, “Beyond brightening low-light images,” _International Journal of Computer Vision_ , vol. 129, pp. 1013–1037, 2021.
  * [28] Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang, “Enlightengan: Deep light enhancement without paired supervision,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2340–2349, 2021.
  * [29] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zero-reference deep curve estimation for low-light image enhancement,” in _CVPR 2020: Computer Vision and Pattern Recognition_ , 2020, pp. 1780–1789.
  * [30] C. Li, C. Guo, and C. L. Chen, “Learning to enhance low-light image via zero-reference deep curve estimation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 8, pp. 4225–4238, 2021.
  * [31] Z. Zhao, B. Xiong, L. Wang, Q. Ou, L. Yu, and F. Kuang, “Retinexdip: A unified deep framework for low-light image enhancement,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 3, pp. 1076–1088, 2022\.
  * [32] D. Liang, L. Li, M. Wei, S. Yang, L. Zhang, W. Yang, Y. Du, and H. Zhou, “Semantically contrastive learning for low-light image enhancement,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 2, 2022, pp. 1555–1563.
  * [33] G. Deng, “A generalized unsharp masking algorithm,” _IEEE Transactions on Image Processing_ , vol. 20, no. 5, pp. 1249–1261, 2011.
  * [34] J. K. Eshraghian, K. Cho, S. Baek, J.-H. Kim, and K. Eshraghian, “Biological modeling of vertebrate retina: Rod cell to bipolar cell,” in _2017 40th International Conference on Telecommunications and Signal Processing (TSP)_ , 2017, pp. 391–394.
  * [35] T. Hansen and H. Neumann, “A simple cell model with dominating opponent inhibition for robust image processing,” _Neural Networks_ , vol. 17, no. 5-6, pp. 647–662, 2004.
  * [36] T. Lindeberg, “A computational theory of visual receptive fields,” _Biological cybernetics_ , vol. 107, no. 6, pp. 589–635, 2013.
  * [37] S. Wienbar and G. W. Schwartz, “The dynamic receptive fields of retinal ganglion cells.” _Progress in Retinal and Eye Research_ , vol. 67, pp. 102–117, 2018.
  * [38] T.-R. Huang and S. Grossberg, “Cortical dynamics of contextually cued attentive visual learning and search: spatial and object evidence accumulation.” _Psychological Review_ , vol. 117, no. 4, p. 1080, 2010.
  * [39] M. W. Cho and M. Y. Choi, “A model for the receptive field of retinal ganglion cells,” _Neural Networks_ , vol. 49, pp. 51–58, 2014.
  * [40] M. R. Silver, S. Grossberg, D. Bullock, M. H. Histed, and E. K. Miller, “A neural model of sequential movement planning and control of eye movements: Item-order-rank working memory and saccade selection by the supplementary eye fields,” _Neural Networks_ , vol. 26, pp. 29–58, 2012.
  * [41] X.-S. Zhang, S.-B. Gao, R.-X. Li, X.-Y. Du, C.-Y. Li, and Y.-J. Li, “A retinal mechanism inspired color constancy model,” _IEEE Transactions on Image Processing_ , vol. 25, no. 3, pp. 1219–1232, 2016.
  * [42] S. Grossberg, “Toward autonomous adaptive intelligence: building upon neural models of how brains make minds,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 51, no. 1, pp. 51–75, 2020.
  * [43] N. A. Browning, S. Grossberg, and E. Mingolla, “A neural model of how the brain computes heading from optic flow in realistic scenes,” _Cognitive Psychology_ , vol. 59, no. 4, pp. 320–356, 2009.
  * [44] L. D. Stroebel, _Basic photographic materials and processes_ , 2000.
  * [45] M. L. Bolton, “Modeling human perception could stevens’ power law be an emergent feature?” in _2008 IEEE International Conference on Systems, Man and Cybernetics_. IEEE, 2008, pp. 1073–1078.
  * [46] E. Peli, “Contrast in complex images,” _JOSA A_ , vol. 7, no. 10, pp. 2032–2040, 1990.
  * [47] Z. Li, J. Zheng, Z. Zhu, W. Yao, and S. Wu, “Weighted guided image filtering,” _IEEE Transactions on Image Processing_ , vol. 24, no. 1, pp. 120–129, 2015.
  * [48] M. Kumar and A. K. Bhandari, “No-reference metric optimization-based perceptually invisible image enhancement,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 71, pp. 1–10, 2022.
  * [49] https://sites.google.com/site/vonikakis/datasets.
  * [50] C. Lee, C. Lee, and C.-S. Kim, “Contrast enhancement based on layered difference representation,” in _IEEE International Conference on Image Processing (ICIP)_ , 2012.
  * [51] K. Ma, K. Zeng, and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” _IEEE Transactions on Image Processing_ , vol. 24, no. 11, pp. 3345–3356, 2015.
  * [52] J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” _IEEE Transactions on Image Processing_ , vol. 27, no. 4, pp. 2049–2062, 2018.
  * [53] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a completely blind image quality analyzer,” _IEEE Signal processing letters_ , vol. 20, no. 3, pp. 209–212, 2012.
  * [54] A. Moorthy and A. Bovik, “A two-step framework for constructing blind image quality indices,” _IEEE Signal Processing Letters_ , vol. 17, no. 5, p. 516, 2010.
  * [55] K. Gu, G. Zhai, X. Yang, and W. Zhang, “Using free energy principle for blind image quality assessment,” _IEEE Transactions on Multimedia_ , vol. 17, no. 1, pp. 50–63, 2014.
  * [56] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [57] P. Cao, Z. Wang, and K. Ma, “Debiased subjective assessment of real-world image enhancement,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 711–721.
  * [58] A. Soranzo and A. Gilchrist, “Layer and framework theories of lightness,” _Attention, Perception, & Psychophysics_, vol. 81, pp. 1179–1188, 2019\.
  * [59] A. Beghdadi, M. A. Qureshi, S. A. Amirshahi, A. Chetouani, and M. Pedersen, “A critical analysis on perceptual contrast and its use in visual information analysis and processing,” _IEEE Access_ , vol. 8, pp. 156 929–156 953, 2020.
  * [60] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” _IEEE Transactions on image processing_ , vol. 16, no. 8, pp. 2080–2095, 2007.


11institutetext: Princeton University22institutetext: Institute for Research
and Innovation in Software for High Energy Physics (IRIS-HEP)

# An Object Condensation Pipeline for Charged Particle Tracking at the High
Luminosity LHC

Kilian Lieret 1122<EMAIL_ADDRESS>Gage DeZoort 11<EMAIL_ADDRESS>
###### Abstract

Recent work has demonstrated that graph neural networks (GNNs) trained for
charged particle tracking can match the performance of traditional algorithms
while improving scalability to prepare for the High Luminosity LHC experiment.
Most approaches are based on the edge classification (EC) paradigm, wherein
tracker hits are connected by edges, and a GNN is trained to prune edges,
resulting in a collection of connected components representing tracks. These
connected components are usually collected by a clustering algorithm and the
resulting hit clusters are passed to downstream modules that may assess track
quality or fit track parameters. In this work, we consider an alternative
approach based on object condensation (OC), a multi-objective learning
framework designed to cluster points belonging to an arbitrary number of
objects, in this context tracks, and regress the properties of each object. We
demonstrate that OC shows very promising results when applied to the pixel
detector of the trackML dataset and can, in some cases, recover tracks that
are not reconstructable when relying on the output of an EC alone. The results
have been obtained with a modular and extensible open-source implementation
that allows us to efficiently train and evaluate the performance of various OC
architectures and related approaches.

##  1 Introduction

The exploration of tracking algorithms based on graph neural networks (GNNs)
is motivated by the poor computational scaling of combinatorial Kalman filter
algorithms with pileup Cerati_2016 . In recent years, many approaches have
been developed around the edge classification (EC) paradigm (see e.g., Ref.
ju_exatrkx_2021 ), in which GNNs are designed to predict whether or not edges
drawn between tracker hits represent physical trajectories. These
architectures have been shown to demonstrate excellent physics performance
and, importantly, scalability with respect to pileup lazar_accelerating_2023 .
In the majority of these approaches, tracks are rendered from edge-weighted
graphs directly, either by a graph walk algorithm, spatial clustering, or
simply collecting connected components.

This work instead explores a learned track rendering stage based on object
condensation (OC), a multi-loss training scheme we use to cluster hits
belonging to the same track in a learned clustering space. Employing only a
very lightweight EC network (without any message passing), we show that the OC
approach is able to deliver excellent performance when applied to simulated
high-pileup TrackML pixel detector events. We also show that the algorithm is
able to reconstruct tracks with missing edges, which is not possible when
relying on the output of an EC alone. Therefore, this algorithm could also be
used at the end of an EC pipeline, leveraging both node and edge embeddings to
resolve ambiguities that may exist at the output of the EC.

##  2 Dataset and input features

This study is performed using the TrackML dataset amrouche_tracking_2020 ;
amrouche_tracking_2021  that simulates the worst-case HL-LHC pileup conditions
($\langle\mu\rangle=200$) in a generic tracking detector geometry111We use the
version corresponding to throughput phase of the TrackML challenge, which is
hosted at Codalab amrouche_tracking_2021 .. Our studies are limited to the
innermost pixel detector layers, including 4 barrel layers and 7 layers in
each endcap. In each pixel tracker event, we embed hits as graph nodes with 14
features, including:

  * •

The cylindrical coordinates $r$, $\phi$, $z$ of the hits, where $z$
corresponds to the line of the colliding proton-proton pairs; the
corresponding pseudorapidity $\eta$ and the conformal tracking coordinates $u$
and $v$ Hansroul_conformal_1988  are also included.

  * •

The hit’s charge fraction (sum of charge in the cluster divided by the number
of activated channels), as well as variables describing the shape and
orientation of the cluster introduced in Ref. fox4DTrackingUsing2021 . For the
latter, we closely follow the implementation in Ref. ju_exatrkx_2021 .

##  3 Tracking metrics

We study several tracking definitions to evaluate the performance of our
pipeline. They are defined with respect to target populations of tracks; for
example, we commonly report the tracking efficiency on reconstructable
particles that produce at least three hits and have $|\eta|<4.0$. The
efficiencies are defined with respect to various matching criteria between
reconstructed tracks and truth tracks. For a given reconstructed track $c$, we
define the _majority particle_ $\pi_{c}$ as the particle with the largest
number of hits within $c$ (and choose a random one if this condition applies
to multiple particles). We write $\\#c$ for the number of hits in $c$ and
$\\#\pi_{c}$ for the number of hits of $\pi_{c}$ anywhere. We define the
_majority fraction_ $f^{\text{in}}_{c}$ as the number of hits of $\pi_{c}$
within $c$ divided by $\\#c$ and the _majority outside fraction_
$f^{\text{out}}_{c}$ as the number of hits of $\pi_{c}$ outside of $c$ divided
by $\\#\pi_{c}$.

  * •

Perfect match efficiency ($\epsilon^{\text{perfect}}$): the number of
reconstructed tracks with $\\#c>3$, $\pi_{c}$ reconstructable,
$f^{\text{in}}_{c}=f^{\text{out}}_{c}=1$ normalized to the number of
reconstructable particles.

  * •

LHC-style match efficiency ($\epsilon^{\text{LHC}}$): the number of
reconstructed tracks with $\\#c>3$, $\pi_{c}$ reconstructable, $f_{c}>75\%$
normalized to the number of clusters with reconstructable $\pi_{c}$. Note that
duplicates, wherein multiple reconstructed tracks match to one particle, are
possible with this definition.

  * •

Double majority match efficiency ($\epsilon^{\text{DM}}$): the number of
reconstructed tracks with $\\#c>3$, $\pi_{c}$ reconstructable,
$f^{\text{in}}_{c}>50\%$, and $f^{\text{out}}_{c}>50\%$ normalized to the
number of reconstructable particles. This definition produces unique cluster-
track assignments.

We also define the fake rate $f^{\text{DM}}$ based on the double majority
metric as the number of reconstructed tracks with $\\#c\geq 3$ and $\pi_{c}$
reconstructable that do not satisfy the double majority criterion normalized
to the number of clusters with reconstructable $\pi_{c}$.

As we are mostly interested in high-$p_{\text{T}}$ tracks, we also consider
these metrics with an additional $p_{\text{T}}>0.9\,\mathrm{GeV}$ threshold
applied to particles and majority particles in the definition of the metrics.
The corresponding metrics are denoted
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$,
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}$,
$\epsilon^{\text{LHC}}_{p_{\text{T}}>0.9}$ and
$f^{\text{DM}}_{p_{\text{T}}>0.9}$.

##  4 Graph construction

The edges of the input graph are constructed by connecting hits on different
detector layers that satisfy a series of geometric constraints and pass a
classifier threshold.

###  4.1 Edges based on geometric constraints

This procedure is nearly identical to the geometric graph construction
procedure described in Ref. dezoort_interaction_2021 , but without applying
any cuts based on truth information. Edges between nodes $i$ and $j$ are
selected based on the following geometric variables:

$\displaystyle z_{0}\coloneqq
z_{i}-r_{i}\frac{z_{j}-z_{i}}{r_{j}-r_{i}},\quad\phi_{\text{slope}}\coloneqq\frac{\phi_{j}-\phi_{i}}{r_{j}-r_{i}},\quad\text{and}\quad\Delta
R\coloneqq\sqrt{(\eta_{j}-\eta_{i})^{2}+(\phi_{j}-\phi_{i})^{2}}.$ (1)

Here, we require candidate edges to satisfy $z_{0}<197.4\,\mathrm{mm}$,
$\phi_{\text{slope}}<0.001825/\mathrm{mm}$, and $\Delta R<1.797$. The cutoff
points were optimized to maximize the fraction of reconstructable track edges
appearing in the graph, while simultaneously minimizing the number of un-
physical edges constructed. In contrast to Ref. dezoort_interaction_2021 , no
barrel intersection cut is applied. Note that the performance of the graph
construction can be translated to an approximate upper bound for the
performance of the pipeline downstream. Assuming that the pipeline can build
tracks exactly out of those hits that are connected, that is, assuming a
pipeline with perfect EC followed by identifying tracks as connected
components of the resulting edge subgraph, we obtain
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}\leq_{\text{EC}}97.4\%$ and
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}\leq_{\text{EC}}84.0\%$. However,
we will also show that OC can, in some cases, surpass this limit.

We provide four initial edge features based on the coordinates of the two hits
involved: $\Delta r$, $\Delta\phi$, $\Delta z$, and $\Delta R$. The resulting
graphs are denoted $\mathcal{G}\coloneqq(X,R_{a},I)$, where
$X\coloneqq(x_{i})_{i=1,\dots,N}\in\mathbb{R}^{N\times 14}$ are the node
features, $I\in\mathbb{N}^{2\times N_{\text{edges}}}$ is the list of edges in
coordinate format, and $R_{a}\coloneqq(e_{ij})_{(i,j)\in I}$ with
$e_{ij}\in\mathbb{R}^{4}$ are the edge features222While edge attributes and
features are vectors of length $N_{\text{edges}}$, they are indexed by double
indices $(i,j)\in I$ (abbreviated to $ij$ in subscript) for notational
convenience.. We also define truth labels $l_{i}\in\\{0,1,...,N_{\text{t}}\\}$
(where $N_{\text{t}}$ is the number of particles in the graph) indicating
whether the hit is noise ($l_{i}=0$) or belongs to track $t$ ($l_{i}=t$,
$1\leq t\leq N_{\text{t}}$); in this work, we do not consider shared hits
between tracks. The truth label $y_{ij}$ indicates whether an edge connects
two non-noise hits of the same particle ($l_{i}=l_{j}>0$, $(i,j)\in I$). The
geometric cuts alone achieve a purity of
$N_{\text{true}}^{\text{built}}/N_{\text{total}}^{\text{built}}=4.5\%$ at
$2.8\times 10^{6}$ edges per graph.

###  4.2 Edge filtering

We then apply a lightweight EC to reduce the number of false edges. For this,
we train a fully connected neural network (FCNN) that takes node and edge
features as inputs. The node and edge features described in section 2 and
subsection 4.1 are concatenated, $z^{(0)}_{ij}\coloneqq[x_{i},x_{j},e_{ij}]$,
$(i,j)\in I$, and embedded into a 256-dimensional space by a fully connected
layer: $z^{(1)}_{ij}\coloneqq W^{(1)}z^{(0)}_{ij}$, with learnable weights
$W^{(1)}\in\mathbb{R}^{256\times(14+14+4)}$. We then apply a fully connected
network of five hidden layers of width 256 with $\operatorname{ReLU}$
activations and residual connections of the form
$z^{(\ell+1)}_{ij}\coloneqq\sqrt{\beta}\,W^{(\ell+1)}\operatorname{ReLU}\bigl{(}z^{(\ell)}_{ij}\bigr{)}+\sqrt{1-\beta}\,z^{(\ell)}_{ij}$,
where $l=1,\dots,5$, $(i,j)\in I$, and $\beta=0.4$.

To obtain an edge weight, we apply the logistic activation function $\sigma$:
$w_{ij}\coloneqq\sigma(W^{(7)}\operatorname{ReLU}(z^{(6)}_{ij}))\in(0,1)^{\mathrm{N_{\mathrm{edges}}}}$,
where $W^{(7)}\in\mathbb{R}^{1\times 256}$. This output is trained with binary
cross entropy loss to classify whether an edge connects two hits of the same
particle. As we are more interested in tracks with a high value of
$p_{\text{T}}$, we exclude true edges connecting hits of particles with
$p_{\text{T}}<0.9\,\mathrm{GeV}$ from the loss, i.e.,

$\displaystyle\begin{split}&\ell_{\text{EF}}(y,w)\coloneqq-\frac{1}{N_{\text{edges}}}\sum_{(i,j)\in
I}\biggl{(}\delta_{(p_{\text{T}}>0.9)}\,y_{ij}\log
w_{ij}+(1-y_{ij})\log(1-w_{ij})\biggr{)},\\\
&\qquad\text{where}\quad\delta_{(p_{\text{T}}>0.9)}\coloneqq\begin{cases}0&l_{i}=0\lor
p_{T}^{l_{i}}<0.9\,\mathrm{GeV},\\\ 1&\text{else}\end{cases},\end{split}$ (2)

and $p_{\text{T}}^{l_{i}}$ denotes the $p_{\text{T}}$ of the particle
belonging to particle $i$.

The classifier achieves a ROC AUC of $93.3\%$ when evaluated on all tracks and
$99.8\%$ when evaluated on all tracks of interest. Here, _tracks of interest_
refers to tracks with $p_{\text{T}}>0.9\,\mathrm{GeV}$ and the
_reconstructable_ constraints described in section 3. To find an appropriate
threshold, we calculate the perfect EC upper bounds to
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ for the subgraphs satisfying
$w_{ij}<w_{\text{thld}}$ for all edges. Based on 1(a), we set
$w_{\text{thld}}=0.03$, resulting in $\mathrm{TPR}=48.3\%$,
$\mathrm{TPR}\text{ (tracks of interest)}=98.5\%$, $\mathrm{FPR}=1.1\%$. The
perfect EC upper bounds for this threshold are
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}\leq_{\text{EC}}97.7\%$,
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}\leq_{\text{EC}}92.1\%$. The
purity of the graphs is $68\%$ at $89\times 10^{3}$ edges per graph.

We can also establish a “lower bound” on the performance of the pipeline by
reconstructing tracks directly based on this stage. For this, we identify
tracks with connected components of the aforementioned subgraphs (though with
a stricter value of $w_{\text{thld}}$) and calculate the efficiencies. A scan
over $w_{\text{thld}}$ is shown in 1(b) and shows a maximum of
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ at $w_{\text{thld}}=0.31$ with
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}=78.9\%$,
$\epsilon^{\text{LHC}}_{p_{\text{T}}>0.9}=77.1\%$, and
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}=41.1\%$. However, it should be
noted that more elaborate probabilistic schemes to build tracks based on edge
scores might surpass these numbers.

(a) Upper efficiency bounds

(b) Lower efficiency bounds

Figure 1: Performance of the EC applied during edge construction. The left
plot shows the true positive and false positive rates together with
approximate upper bounds on the recoverable performance based on a perfect EC.
The right plot shows the achievable tracking performance by applying a cut on
the EC and identifying tracks with connected components of the resulting
subgraph. Note that $\epsilon^{\text{LHC}}_{p_{\text{T}}>0.9}$ is not strictly
monotonous because it is normalized to the number of reconstructed tracks
rather than particles.

##  5 Object condensation

Our architecture extends traditional EC pipelines with an additional step
based on object condensation (OC) (Kieseler_condensation_2020, ), a set of
truth definitions and loss functions designed to cluster hits belonging to the
same object and regress the properties of the reconstructed objects. OC has
been extensively validated in applications to calorimetry
(Kieseler_condensation_2020, ; qasim_multi-particle_2021, ; qasim2022endtoend,
), but its applications to tracking have to-date been relatively unexplored.

###  5.1 Loss functions

For each hit $i$, the OC network predicts condensation strengths
$\beta_{i}\in\mathbb{R}$ and clustering coordinates
$c_{i}\in\mathbb{R}^{d_{c}}$. During training, the highest-$\beta_{i}$ hit in
each track is dubbed the track’s condensation point (CP); the goal of OC is to
cluster hits around their track’s CP in the learned clustering coordinate
space. The condensation strength of a track $t$ is that of its CP, i.e.,
$\beta^{(t)}=\max_{\\{i|l_{i}=t\\}}\ \beta_{i}$. The condensation strength
predicted for each hit is used to calculate an un-physical “charge” defined by
$q_{i}=\text{arctanh}^{2}\beta_{i}+q_{\mathrm{min}}$ (here, $q_{\mathrm{min}}$
is treated as a hyperparameter). The charge corresponding to a track’s CP is
denoted $q^{(t)}$, located at the position $c^{(t)}$. During training, the CPs
for each track are used to define attractive and repulsive losses designed to
produce well-separated clusters of hits belonging to the same track in the
$c_{i}$ coordinates:

$\displaystyle L_{\mathrm{V}}$
$\displaystyle\coloneqq\frac{1}{N}\sum_{i=1}^{N}q_{i}\sum_{t=1}^{N_{t}}\bigg{(}\delta_{(l_{i}=t)}V^{\mathrm{att}}_{t}(c_{i})+s_{\text{rep}}\big{(}1-\delta_{(l_{i}=t)}\big{)}V^{\mathrm{rep}}_{t}(c_{i})\bigg{)}.$
(3)

Here, $\delta_{(l_{i}=t)}$ is $1$ when the node’s track label is $t$ and $0$
otherwise, and $s_{\text{rep}}$ is a hyperparameter. The potential functions
are a quadratic attractive loss and a repulsive hinge loss:

$\displaystyle
V^{\mathrm{att}}_{t}(c)\coloneqq\delta_{(p_{\text{T}}>0.9)}\,q^{(t)}\,\|c^{(t)}-c\|^{2},\quad
V^{\mathrm{rep}}_{t}(c)\coloneqq q^{(t)}\max(0,1-\|c^{(t)}-c\|),$ (4)

where $\delta_{(p_{\text{T}}>0.9)}$ (defined as in Equation 2) excludes hits
from noise or low-$p_{\text{T}}$ particles from the attraction. An additional
loss term $L_{\beta}$ is designed to encourage a unique CP for each track and
suppress the condensation strengths of noise hits:

$L_{\beta}\coloneqq\frac{1}{N_{t}}\sum_{t=1}^{N_{t}}(1-\beta^{(t)})+s_{\text{B}}\frac{\sum_{i=1}^{N}\beta_{i}\,\delta_{(l_{i}=0)}}{\sum_{i=1}^{N}\delta_{(l_{i}=0)}}.$
(5)

Here, $s_{B}$ is a hyperparameter that controls the strength of noise
suppression. All loss terms are finally combined as $L\coloneqq
L_{V}+s_{\beta}L_{\beta}$. For the results in this paper, we choose
$s_{\text{rep}}=0.6$, $s_{\beta}=0.004$, $q_{\text{min}}=0.34$, and
$s_{\text{B}}=0.09$. To reduce the memory footprint of the loss functions, the
graphs are split in 32 sectors during training.

###  5.2 Model

The GNN that is doing the heavy lifting of this tracking pipeline is built
from _interaction network_ layers battaglia_interaction_2016  with residual
connections in the node updates. Node and edge features are first encoded,
$x_{i}^{(1)}\coloneqq W^{\text{enc}}_{\text{node}}x_{i}$,
$e_{ij}^{(1)}\coloneqq W^{\text{enc}}_{\text{edge}}e_{ij}$ , where $(i,j)\in
I$, $W^{\text{enc}}_{\text{node}}\in\mathbb{R}^{192\times 14}$,
$W^{\text{enc}}_{\text{edge}}\in\mathbb{R}^{192\times 4}$. Then, five
iterations of message passing are performed with

$\displaystyle\begin{split}e_{ij}^{(k+1)}&\coloneqq\bigl{(}\Phi^{(k+1)}\circ\operatorname{ReLU}\bigr{)}\Big{(}\bigl{[}x_{i}^{(k)},x_{j}^{(k)},e_{ij}^{(k)}\bigr{]}\Big{)},\\\
x_{i}^{(k+1)}&\coloneqq\sqrt{\beta}\,\Psi^{(k+1)}\Bigl{(}\bigl{[}\operatorname{ReLU}\bigl{(}x_{i}^{(k)}\bigr{)},{\textstyle\sum\nolimits_{j\in\mathcal{N}_{i}}}e_{ij}^{(k+1)}\bigr{]}\Bigr{)}+\sqrt{1-\beta}\,x_{i}^{(k)}.\end{split}$
(6)

Here, $\Phi$ and $\Psi$ are FCNNs with $\operatorname{ReLU}$ activations, a
layer width of 192, and one hidden layer. $\beta$ has been chosen to be $0.2$.
Finally, the outputs are decoded as $c_{i}\coloneqq
W^{\text{dec}}_{\text{c}}\operatorname{ReLU}(x^{(6)})$ (clustering
coordinates),
$\beta_{i}\coloneqq\sigma\bigl{(}W^{\text{dec}}_{\beta}\operatorname{ReLU}(x^{(6)})\bigr{)}$
(condensation likelihoods), where $\sigma$ is the logistic function, and
$W_{\text{c}}^{\text{dec}}\in\mathbb{R}^{192\times 24}$,
$W^{\text{dec}}_{\beta}\in\mathbb{R}^{192\times 1}$. The total number of
parameters of this model is $1.9\times 10^{6}$.

###  5.3 Postprocessing and results

Hit clusters produced in the OC clustering space must be rendered by a
downstream algorithm. For this, we use the Density-Based Spatial Clustering of
Applications with Noise (DBSCAN), an iterative clustering algorithm that has
two parameters, $\epsilon$ (defining the size of the neighborhood of a point
that is considered when merging clusters), and $k$ (minimum number of points
within a neighborhood for the points to be considered a _core point_)
ester1996density . For this application, $k=1$ is optimal; maximizing
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ vs $\epsilon$ yields
$\epsilon=0.279$ (see Figure 3). With this, we obtain
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}=95\%$,
$\epsilon^{\text{LHC}}_{p_{\text{T}}>0.9}=97\%$,
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}=80\%$ and
$f_{p_{\text{T}}>0.9}=1.7\%$. All metrics are presented vs $p_{\text{T}}$ and
vs $\eta$ in Figure 4.

In a side study, we have also tested the ability of the OC network to
reconstruct tracks with missing edges after graph construction or edge
filtering. For this, all edges between the barrel and the right endcap have
been removed after graph construction, limiting the upper bound for
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}$ for a perfect EC to almost zero
for tracks with $2<\eta<3$. However, as OC is using edges only as a means to
exchange information, it is not subject to this upper bound. Indeed, the OC
pipeline achieves $\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}=60\%$ in this
region. This is shown in Figure 3.

Figure 2: Optimizing the DBSCAN hyperparameters to obtain the maximum
performance. The dashed lines are the upper and lower bounds after the EF step
for reference (see Figure 1).

Figure 3: An OC pipeline outperforms a perfect EC when edges between the
barrel and the right endcap have been removed.

Figure 4: Tracking performance in bins of $p_{\text{T}}$ and $\eta$.

Figure 5: Comparing the performance of the OC pipeline with the upper and
lower bounds introduced in this paper.

##  6 Summary

This paper presents the first GNN-based charged particle tracking pipeline
that uses the OC approach to reconstruct tracks in the worst-case pileup
conditions expected at the HL-LHC. Our pipeline shows excellent performance
with respect to several metrics when applied to the pixel detector of the
TrackML dataset. We also demonstrate that OC approach can join partial tracks
that are not connected by any of the edges used for message passing, allowing
it to outperform algorithms that solely rely on the output of an EC in certain
scenarios. This suggests that the use of OC at the output stage of EC-based
pipelines may lead to a boost in performance. Future applications of OC may
also allow for the regression of track parameters, for example transverse
momentum, as part of an architecture capable of rendering tracks and
performing preliminary fits in one shot. This incorporation of track physics
may well also lead to a more robust model.

All results were produced with the open-source project
Lieret_gnn_tracking_An_open-source  that implements various OC tracking
architectures in a modular and extensible Python package.

## References

  * (1) G. Cerati, P. Elmer, S. Krutelyov, S. Lantz, M. Lefebvre, K. McDermott, D. Riley, M. Tadel, P. Wittich, F. WÃrthwein et al., EPJ Web of Conferences 127, 00010 (2016)
  * (2) X. Ju, D. Murnane, P. Calafiura, N. Choma, S. Conlon, S. Farrell, Y. Xu, M. Spiropulu, J.R. Vlimant, A. Aurisano et al., The European Physical Journal C 81, 876 (2021), htttps://doi.org/10.1140/epjc/s10052-021-09675-8
  * (3) A. Lazar, X. Ju, D. Murnane, P. Calafiura, S. Farrell, Y. Xu, M. Spiropulu, J.R. Vlimant, G. Cerati, L. Gray et al., Journal of Physics: Conference Series 2438, 012008 (2023)
  * (4) S. Amrouche, L. Basara, P. Calafiura, V. Estrade, S. Farrell, D.R. Ferreira, L. Finnie, N. Finnie, C. Germain, V.V. Gligorov et al., _The Tracking Machine Learning challenge : Accuracy phase_ , in _The NeurIPS ’18 Competition_ , edited by S. Escalera, R. Herbrich (Springer International Publishing, Cham, 2020), pp. 231–264, https://doi.org/10.1007/978-3-030-29135-8_9
  * (5) S. Amrouche, L. Basara, P. Calafiura, D. Emeliyanov, V. Estrade, S. Farrell, C. Germain, V.V. Gligorov, T. Golling, S. Gorbunov et al., arXiv:2105.01160 [cs.LG] (2021), submitted to Computing and Software for Big Science, https://doi.org/10.48550/arXiv.2105.01160
  * (6) M. Hansroul, H. Jeremie, D. Savard, Nucl. Instrum. Methods Phys. Res., A 270, 498 (1988)
  * (7) P.J. Fox, S. Huang, J. Isaacson, X. Ju, B. Nachman, Journal of Instrumentation 16, P05001 (2021), 2012.04533
  * (8) G. DeZoort, S. Thais, J. Duarte, V. Razavimaleki, M. Atkinson, I. Ojalvo, M. Neubauer, P. Elmer, Computing and Software for Big Science 5 (2021)
  * (9) J. Kieseler, The European Physical Journal C 80 (2020)
  * (10) S.R. Qasim, K. Long, J. Kieseler, M. Pierini, R. Nawaz, arXiv:2106.01832 [physics.ins-det] (2021), https://doi.org/10.48550/arXiv.2106.01832
  * (11) S.R. Qasim, N. Chernyavskaya, J. Kieseler, K. Long, O. Viazlo, M. Pierini, R. Nawaz, Eur. Phys. J. C 82, 753 (2022), https://doi.org/10.1140/epjc/s10052-022-10665-7
  * (12) P.W. Battaglia, R. Pascanu, M. Lai, D. Rezende, K. Kavukcuoglu (2016), preprint at https://arxiv.org/abs/1612.00222, 1612.00222
  * (13) M. Ester, H.P. Kriegel, J. Sander, X. Xu et al., _A density-based algorithm for discovering clusters in large spatial databases with noise._ , in _kdd_ (1996), Vol. 96, pp. 226–231
  * (14) K. Lieret, G. DeZoort, _gnn_tracking: An open-source GNN tracking project_ , https://github.com/gnn-tracking/gnn_tracking