# Self-Gated Memory Recurrent Network for Efficient Scalable HDR Deghosting

K Ram Prabhakar*, Susmit Agrawal*, and R Venkatesh Babu * equal contribution.
K. R. Prabhakar, S. Agrawal and R. V. Babu are with the Department of
Computational and Data Sciences, Indian Institute of Science, Bengaluru, KA,
560012, INDIA. e-mail<EMAIL_ADDRESS>This paper has supplementary
downloadable material available at http://ieeexplore.ieee.org, provided by the
author. The material includes additional results. This material is 15MB in
size.

###### Abstract

We propose a novel recurrent network-based HDR deghosting method for fusing
arbitrary length dynamic sequences. The proposed method uses convolutional and
recurrent architectures to generate visually pleasing, ghosting-free HDR
images. We introduce a new recurrent cell architecture, namely Self-Gated
Memory (SGM) cell, that outperforms the standard LSTM cell while containing
fewer parameters and having faster running times. In the SGM cell, the
information flow through a gate is controlled by multiplying the gate’s output
by a function of itself. Additionally, we use two SGM cells in a bidirectional
setting to improve output quality. The proposed approach achieves state-of-
the-art performance compared to existing HDR deghosting methods quantitatively
across three publicly available datasets while simultaneously achieving
scalability to fuse variable length input sequence without necessitating re-
training. Through extensive ablations, we demonstrate the importance of
individual components in our proposed approach. The code is available at
https://val.cds.iisc.ac.in/HDR/HDRRNN/index.html.

###### Index Terms:

High Dynamic Range image fusion, Exposure Fusion, Deghosting, Computational
Photography, Convolutional Neural Networks.

This article has been accepted for publication in IEEE Transactions on
Computational Imaging. This is the author’s version which has not been fully
edited and content may change prior to final publication.Citation information:
DOI 10.1109/TCI.2021.3112920© 2021 IEEE. Personal use is permitted, but
republication/redistribution requires IEEE permission. See
https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on
Computational Imaging. This is the author’s version which has not been fully
edited and content may change prior to final publication.Citation information:
DOI 10.1109/TCI.2021.3112920© 2021 IEEE. Personal use is permitted, but
republication/redistribution requires IEEE permission. See
https://www.ieee.org/publications/rights/index.html for more information.

##  I Introduction

Standard digital cameras have limited sensor capability that can capture only
a part of the natural scene illumination. This results in the captured images
being too bright or too dark in certain regions, causing loss of textural and
structural details in those regions. Specialized camera sensors capable of
capturing natural high dynamic ranges are prohibitively expensive for day-to-
day use cases [1, 2]. A cheaper and more practical substitute is to provide a
software alternative to advanced hardware. One commonly used software solution
is to capture multiple Low Dynamic Range (LDR) images, each with a different
exposure value, and combine them to generate an image containing a High
Dynamic Range (HDR) of illumination. The generated HDR image has enriched
details and textures in both highlight and shadow regions.

In the absence of camera or object motion across the captured LDR images, HDR
generation is a simple process [3]. However, if the images contain relative
object or camera motion, fusing them with a standard HDR merging algorithm
will result in ghost-like artifacts in the final result. The process of
obtaining ghost-free HDR images even in the presence of movement among
captured images is known as HDR deghosting. The HDR deghosting problem has a
rich history spanning two decades, and several methods have been proposed in
the literature to address it.

Rejection based methods attempt to remove pixels affected by motion (or
dynamic pixels) from largely static exposure stacks. These dynamic pixels are
replaced by LDR content from one of the captured images [4, 5, 6, 7, 8, 9].
While these methods offer good quality results for mostly static scenes, they
suffer from low HDR content in moving regions. Alignment based methods
transform the input stack to resemble a chosen reference image, enforcing
similar structure across all inputs [10, 11, 12, 13, 14]. The aligned images
are then fused using HDR merging methods such as [3]. The drawback is that
most alignment methods are time and compute-intensive and are often unable to
remove all artifacts. Patch-based optimization methods synthesize an HDR
output that resembles a chosen reference image in well-exposed regions of
reference and borrow details from other LDR images in regions where reference
is poorly exposed [15, 16, 17]. Despite their better performance, they are
complex algorithms that suffer from high computational costs.

Data-driven methods use learning algorithms, primarily Convolutional Neural
Networks (CNN), to learn fusion as well as deghosting [18, 19, 20, 21, 22].
Once trained, they can generate high-quality HDR images faster than most other
non-deep methods. A potential drawback, however, is that most existing
learning-based methods today are not scalable [18, 19, 21, 22]. We refer to
scalability as a trait of a method to fuse arbitrary length LDR image
sequences during inference. Most existing CNN-based methods are trained to
operate on three images, with the middle image being the default reference.
Hence, they can only fuse three LDR images during inference, not more, and not
less than three.

In practical use cases, some challenging natural scenes may contain a very
high illumination range that requires more than three LDR images to cover
completely. In such cases, most existing CNN based HDR deghosting methods
cannot be used to merge arbitrary length LDR images. These methods need to be
re-trained with a new number of input images. However, such a solution poses
two challenges: 1) A new and separate model must be trained for each possible
sequence length. Hence, it increases the memory and storage footprint on the
device. 2) New datasets have to be captured for different sequence lengths.
Additionally, it is tedious to capture ground truth HDR, as it involves
capturing static and dynamic sequences of the same scene with controlled
motion. This shows the importance and need of having a method than can handle
arbitrary exposure brackets and varying input lengths without necessitating
re-training.

Among state-of-the-art non-deep methods, Sen et al. [15] and Hu et al. [16]
methods are scalable. However, these methods have significant computation
overhead and generate images with artifacts (see Fig. 1). The CNN-based method
proposed by Prabhakar et al. [20] is capable of fusing a variable number of
images, but at a considerable cost in terms of quality due to their internal
aggregation strategy of features (see Fig. 7c and 10). This indicates that
there is still room for developing a better and efficient scalable deep HDR
deghosting method.

We address the problem as mentioned above by exploiting the internal memory
capabilities of recurrent architectures. In our proposed approach, we
introduce a novel recurrent cell called Self-Gated Memory (SGM) cell for
efficient HDR deghosting. The SGM cell’s internal vector representations make
an ideal way to encode pixel color and intensity data while simultaneously
filtering out structural information of non-reference images. Since the
misaligned structures in non-reference images are the major cause of ghosting
artifacts, removing them results in higher quality images. Also, we improve
the accuracy of our model by using two SGM cells in a bidirectional setting.

Figure 1: A qualitative example to highlight results by various scalable HDR
deghosting methods on a sequence with seven dynamic LDR images (top row).
Existing scalable HDR deghosting methods are either too slow or introduce
visible artifacts (highlighted by black arrow). Our proposed method generates
high-quality artifact-free HDR images with less computational cost.

In summary, the main contributions of our work are as follows:

  * •

We propose a novel network that uses recurrent architectures for efficient and
scalable HDR deghosting. Our network can fuse an arbitrary number of LDR
images into a ghost-free HDR image without re-training. To the best of our
knowledge, our model is the first RNN-based approach proposed for multi-shot
HDR deghosting.

  * •

We present a novel Self-Gated Memory recurrent cell that uses self-gating to
control data flow. The proposed SGM cell outperforms existing state-of-the-art
HDR deghosting methods and other standard recurrent architectures.

  * •

We perform extensive experiments to demonstrate the superiority of the
proposed approach. Additionally, through rigorous ablation experiments, we
justify the importance of each module in our proposed method.

The rest of the paper is organized as follows. Section II presents a brief
review of existing HDR deghosting methods in the literature. In Section III,
we present our novel recurrent architecture for HDR deghosting and describe
the methodology in detail. In Section IV, we compare the proposed approach
with existing state-of-the-art deghosting methods. In Section 9, we present
the discussion on several ablations and running time comparison. Finally, we
conclude the paper in Section V.

##  II Related Works

Naively fusing images with camera and object motion will result in visible
ghosting artifacts. Several methods have been proposed in the past to avoid
such ghosting artifacts111Please refer to [23, 24, 25] for more detailed
literature review..

Alignment methods: This class of algorithms aligns all input images with
camera motion to a chosen reference image. Thus, the generated image sequence
is mostly static and can be fused using standard HDR fusion algorithms with
minimal ghosting artifacts. These methods are not primarily deghosting
approaches suitable to handle object motion, hence they are used as a pre-
processing step for such algorithms to eliminate global camera or scene
motion. The methods within this category mostly differ by the choice of
alignment strategy such as frequency domain matching and cross correlation
[26, 27, 28], elastic registration [29, 30], SIFT [10], CIFT [31],
neighbourhood similarity [32], etc. The early work by Mann et al. [33]
proposed to jointly estimate CRF and register images using global homography.
Candocia et al. [34] register inputs by optimizing to reduce pixel intensity
variance. An alignment method proposed by Tomaszewska and Mantiuk [10] use
SIFT to perform a key-point search in consecutive images of the input stack,
and then use RANSAC to improve the matching. The images are warped using a
homography computed from the matched key-points.

Ward [13] proposed an algorithm that runs in linear time and uses grayscale
images to create image pyramids. Bogoni [11] perform both global and local
alignment in two steps using affine transforms and optical flow correction.
Zimmer et al. [12] propose a method involving the optimization of a two-term
energy function, containing a data term to promote alignment to reference and
a regularizer term to ensure a smooth flow in poorly exposed regions. This
optimization is applied after aligning images with optical flow. A fast
alignment method introduced by Gallo et al. [14] computes sparse
correspondences between an input and the reference image and propagates the
sparse flow in an edge-aware fashion to approximate the dense flow map. Kang
et al. [35] globally align frames in a video sequence and then use gradient-
based optical flow to correct local regions to generate HDR video.

Rejection based methods: This class of algorithms works well with mostly
static image stacks [36, 8, 37]. The static pixels that are void of motion are
fused using traditional static HDR merging techniques such as [3]. The
algorithms in this category can be further divided into two subgroups,
depending on how they identify and handle dynamic pixels. The first subclass
of these algorithms replaces moving regions with content from the chosen
reference image. Grosch [4] generate an error map based on thresholded color
differences between pixels. The generated error map represents moving regions,
which can then be filled from the reference image. An et al. [38] use zero-
mean normalized cross-correlation to obtain dynamic pixels between input
images. Wu et al. [5] find pixels that violate brightness consistency to
detect dynamic regions. Gallo et al. [6] threshold the difference in the
logarithmic domain instead of directly using pixel values to identify regions
affected by motion.

Lee et al. [39] detect moving pixels by thresholding the difference of rank
normalized input images. Li et al. [40] compute bi-directional pixel
similarity metric between input and reference to identify motion region. Min
et al. [7] identify moving regions by performing multi-level thresholding of
intensity histograms and use these regions to construct a radiance map. Raman
et al. [9] group superpixels to identify dynamic regions. The result is used
to create a largely static version of the input stack. Heo et al. [8] locate
the ghosted pixels by thresholding the joint probability computed between the
reference and the other source images. Lin et al. [41] threshold the
difference between normalized input images to find motion regions.

The second subclass of rejection based methods contain algorithms that work
without selecting a reference image from the inputs [42, 43, 44]. They ignore
images affected by the motion for the dynamic pixels and only use the rest of
the stack images. The final result contains static regions of the whole scene.
Khan et al. [45] use a kernel density estimator function to compute pixel
weights and iteratively optimize the function. Their approach does not need
explicit object detection or motion estimation. Pedone et al. [46] improve
deghosting performance of [45] with morphological operations on the estimated
motion bitmaps. Pece et al. [47] detect clusters of pixels affected by motion
using binary operations and pick the least saturated clusters. Eden et al.
[48] use a graph-cut method to address object motion within the input stack.

Granados et al. [49] iteratively reconstruct the HDR image by estimating a
mean radiance map with minimum variance. An earlier work by Reinhard et al.
[50] utilizes normalized variance to identify motion affected pixels. Building
on top of [50]’s approach, Jacobs et al. [51] proposed to use an entropy
measure instead of variance. Sidibe et al. [52] increase the exposure and
locate dynamic pixels that does not proportionally increase its intensity
value. Oh et al. [53] proposed a rank minimization approach to generate final
HDR output. In their approach, authors calculate rank-1 matrix of stacked
input images. The residual noise of the estimate is used to locate the moving
dynamic pixels.

Non-rigid registration methods: The following set of methods use optical flow
to handle both global camera motion as well as local object motion. These
algorithms warp moving regions in images of the input stack, such that these
regions align with a chosen reference image. The final HDR result is obtained
by merging the aligned static sequence. Ferradans et al. [54] use GMM to model
the difference between optical flow warped images. The pixels that do not fit
to estimated modes within pre-defined variance receives less weight in the
fusion process. Jinno et al. [55] jointly estimate flow, occlusion and
saturation maps by minimizing an energy function, which are later used to fuse
the images. Hafner et al. [56] jointly estimate fused HDR and displacement
fields in a optimization framework enforced with spatial smoothness
constraints. This category methods suffer from erroneous dense correspondence
or optical flow due to complex non-rigid motions and occlusion.

Patch based optimization methods: This class of methods generates a final HDR
result similar to the reference image in regions where the reference image is
properly exposed [57, 58]. Sen et al. [15] propose a method to optimize both
structure and content of the HDR image. They achieve it by parameterizing
these quantities in an image synthesis equation and optimize for the same. Hu
et al. [16] generate latent images from the input stack that are similar in
structure but vary in an exposure. While this class of methods addresses the
shortcomings of rejection based methods and registration methods, they suffer
from high computational complexity.

Data driven techniques: These approaches make use of Convolutional Neural
Networks trained on data that captures the essence of the HDR deghosting
problem. Fusion and Deghosting rules are approximated by learning from
examples. Kalantari et al. [18] use an input stack that has been aligned using
optical flow and train the model to correct warping artifacts. Wu et al. [19]
directly use neural networks to learn both alignment and fusion, thus
attempting to remove the overhead of optical flow. Yan et al. [59] propose
attention mechanisms to focus only on the relevant information from the input
stack.

Yan et al. [60] use a network with a non-local module to identify matching
neighbor features to fill in ill-exposed regions of the reference image.
Prabhakar et al. [20] aggregate features derived by shared CNN modules to
create a scalable architecture that can fuse an arbitrary number of images
without retraining. Prabhakar et al. [22] propose a method to minimize the
consumption of memory and enable the fusion of high-resolution images. They
perform alignment on low-resolution inputs and upsample intermediate features
while generating the final HDR using a Bilateral Grid Upsampler [61]. In spite
of their better performance, most of the existing deep-learning methods [18,
19, 59, 21, 60, 22] are not scalable to fuse arbitrary number of images
without re-training.

Recurrent Neural Networks have been extensively explored for processing
temporal data such as speech [62, 63, 64], natural language processing [65,
66, 67], video captioning [68, 69], video segmentation [70, 71], low-level
video processing [72, 73] and in so many other applications. The closest
matching related application to HDR deghosting that utilizes recurrent
architecture is burst denoising approach by Godard et al. [72] and the video
denoising method by Chen et al. [73]. To the best of our knowledge, there has
not been any previous attempt to use recurrent networks for HDR deghosting.

##  III Proposed method

###  III-A Method overview

Figure 2: (a) The architecture of the proposed approach. The current input
data $X_{n}$ concatenated with the reference data $X_{ref}$ is passed as input
to the encoder ($\mathcal{E}$). The encoded features are fed to the proposed
bi-directional SGM recurrent cells ($\overrightarrow{\text{C}}$ and
$\overleftarrow{\text{C}}$). The final output of the recurrent cells in both
directions are concatenated and passed to the decoder ($\mathcal{D}$) to
generate fused HDR ($\mathbf{Y}$). (b) Architecture of encoder ($\mathcal{E}$)
and decoder ($\mathcal{D}$). The number below each block denotes the number of
output channels of the corresponding block. Architecture of (c) standard LSTM
cell, and (d) proposed Self-Gated Memory (SGM) recurrent cell.

Given a list of $N$ Low Dynamic Range (LDR) images
$S=\\{I_{1},I_{2},\cdots,I_{N}\\}$, the goal of our approach is to merge them
into a single HDR image $\mathbf{Y}$ without any ghosting artifacts. Assuming
that the input sequence images are camera motion aligned, the challenge is to
combine these images while accounting for object motion. Our proposed method
is a reference-based method, where the final result will be structurally
similar to one of the input images chosen as reference ($I_{ref}$). In
particular, for static regions (regions without any object motion), the result
contains HDR content fused from all input images. For dynamic regions (regions
affected by motion), the result will have the same structure as the chosen
reference image but with HDR content added from other images. Typically, the
image with the least saturated pixels is chosen as a reference.

Motivation: Most deep learning-based approaches ([18, 19, 59, 22]) are trained
to fuse fixed number of LDR images, usually 3 images. These methods require
re-training with the corresponding number of images for fusing a sequence with
the different number of input images. Another shortcoming is the lack of a
training dataset for different sequence lengths. It is very tedious to collect
large labeled datasets for different sequence lengths, as it requires
capturing both static and dynamic sequences of same scene. Prabhakar et
al.[20] addressed scalability in their work by concatenating the mean and max
of input features. However, their method still suffers from artifacts in
challenging scenarios (shown in Fig. 1).

Our proposed approach consists of a single neural network that uses
convolutional and recurrent architectures. As shown in Fig. 2a, the overall
network can be logically divided into three sub-modules: an encoder, two
recurrent cells in a bidirectional setting, and a decoder.

Encoder ($\mathcal{E}$): As shown in Fig. 2b, the encoder is a block of
3$\times$3 convolutional layers. It acts as a basic feature extractor and
processes the input images for fusion by the recurrent module. Since the
number of input images can vary, we use a single encoder to process all $N$
images in the input sequence. For the rest of the paper, let $I_{n}$ represent
the $n^{th}$ image in the input sequence. The gamma-corrected HDR version
($H_{n}$) of $I_{n}$ is obtained by $H_{n}=I_{n}^{2.2}/t_{n}$, where $t_{n}$
is the exposure time of $I_{n}$. Let $X_{n}$ be the concatenation of $I_{n}$
and $H_{n}$. $\mathcal{E}$ then takes concatenated $X_{n}$ and $X_{ref}$ as
input,

$\displaystyle X_{n}$ $\displaystyle=I_{n}\oplus H_{n}$ (1) $\displaystyle
X_{ref}$ $\displaystyle=I_{ref}\oplus H_{ref}$ (2) $\displaystyle E_{n}$
$\displaystyle=\mathcal{E}\big{(}\\{X_{n},X_{ref}\\})$ (3)

where, $\oplus$ denotes the feature concatenation operation along the channel
axis.

Recurrent Module: This is the core module in the proposed architecture. The
recurrent module takes as input the list of encoded images
${E_{1},E_{2},...,E_{n}}$ and iteratively fuses them into output vectors
${h_{1},h_{2},...,h_{N}}$, such that an output vector $h_{n}$ at the $n^{th}$
timestep contains information from all input vectors $E_{1}$ to $E_{n}$. The
final output $h_{N}$ thus contains fused information from the entire input
sequence.

The recurrent module may contain a single recurrent cell in a unidirectional
setting, or it may use two cells in a bidirectional arrangement. In the
bidirectional setting, one of the cells processes the input sequence starting
from $E_{1}$ to $E_{N}$, while the second cell processes inputs starting from
$E_{N}$ to $E_{1}$. The outputs for the cells are $\overrightarrow{h_{N}}$ and
$\overleftarrow{h_{1}}$ respectively, with the arrows indicating the direction
in which the input sequence is processed. The module’s output is then given by
concatenating the two vectors $\overrightarrow{h_{N}}$ and
$\overleftarrow{h_{1}}$.

Figure 3: Qualitative comparison between proposed SGM cell against baseline
recurrent cell types: LSTM and GRU.

In the forward direction, $h_{0}$ and $c_{0}$ are zero vectors of the same
height and width as each input image, named $\overrightarrow{h_{init}}$ and
$\overrightarrow{c_{init}}$. Then, in the reverse direction, $h_{N+1}$ and
$c_{N+1}$ are $\overleftarrow{h_{init}}$ and $\overleftarrow{c_{init}}$
respectively. We use cell $\overrightarrow{\text{C}}$ in forward and a
separate cell $\overleftarrow{\text{C}}$ in reverse direction for images
$i=(1,\cdots,N)$,

$\displaystyle(\overrightarrow{h_{i}},\overrightarrow{c_{i}})$
$\displaystyle=\overrightarrow{\text{C}}\big{(}E_{i},\overrightarrow{h_{i-1}},\overrightarrow{c_{i-1}}\big{)},$
(4) $\displaystyle(\overleftarrow{h_{N-i+1}},\overleftarrow{c_{N-i+1}})$
$\displaystyle=\overleftarrow{\text{C}}\big{(}E_{N-i+1},\overleftarrow{h_{N-i+2}},\overleftarrow{c_{N-i+2}}\big{)}$
(5)

For $N$=3, the forward cell unrolls as,

$\displaystyle(\overrightarrow{h_{1}},\overrightarrow{c_{1}})$
$\displaystyle=\overrightarrow{\text{C}}\big{(}E_{1},\overrightarrow{h_{init}},\overrightarrow{c_{init}}\big{)},$
(6) $\displaystyle(\overrightarrow{h_{2}},\overrightarrow{c_{2}})$
$\displaystyle=\overrightarrow{\text{C}}\big{(}E_{2},\overrightarrow{h_{1}},\overrightarrow{c_{1}}\big{)},$
(7) $\displaystyle(\overrightarrow{h_{3}},\overrightarrow{c_{3}})$
$\displaystyle=\overrightarrow{\text{C}}\big{(}E_{3},\overrightarrow{h_{2}},\overrightarrow{c_{2}}\big{)}$
(8)

Whereas the reverse cell unrolls as,

$\displaystyle(\overleftarrow{h_{3}},\overleftarrow{c_{3}})$
$\displaystyle=\overleftarrow{\text{C}}\big{(}E_{3},\overleftarrow{h_{init}},\overleftarrow{c_{init}}\big{)},$
(9) $\displaystyle(\overleftarrow{h_{2}},\overleftarrow{c_{2}})$
$\displaystyle=\overleftarrow{\text{C}}\big{(}E_{2},\overleftarrow{h_{3}},\overleftarrow{c_{3}}\big{)},$
(10) $\displaystyle(\overleftarrow{h_{1}},\overleftarrow{c_{1}})$
$\displaystyle=\overleftarrow{\text{C}}\big{(}E_{1},\overleftarrow{h_{2}},\overleftarrow{c_{2}}\big{)}$
(11)

Feature decoder ($\mathcal{D}$): After processing the complete input sequence,
the output features produced by the recurrent module are passed to a decoder
block that generates the final HDR image. In bidirectional architecture, the
concatenation of the final outputs from both recurrent cells serves as the
input to this module (Fig. 2b),

$\displaystyle\mathbf{Y}=\mathcal{D}(\overrightarrow{h_{N}}\oplus\overleftarrow{h_{1}})$
(12)

In our implementation, we use two stacked SDC blocks [74] to capture details
across different receptive fields, followed by a single convolutional layer to
generate the final image.

Loss: The model produces an HDR image $\mathbf{Y}$ as its output. This image
is tonemapped using the $\mu$-law tonemapping function defined as,

$T(\mathbf{Y})=\dfrac{log(1+\mu\mathbf{Y})}{log(1+\mu)}$ (13)

where $\mu=5000$. The loss $\mathcal{L}$ between the ground truth HDR
$\mathbf{\hat{Y}}$ and the predicted image $\mathbf{Y}$ is obtained by
computing the $\ell_{2}$ loss between their tonemapped representations:

$\mathcal{L}=\ell_{2}(T(\mathbf{Y}),T(\mathbf{\hat{Y}}))$ (14)

###  III-B Self-Gated Memory (SGM) recurrent cell

Before we present the details of the proposed SGM cell, we first discuss
existing standard recurrent cells, their shortcomings and finally on how they
are addressed in the SGM cell. We start with the three standard recurrent cell
types, the standard RNN cell, the Long Short-Term Memory (LSTM), and the Gated
Recurrent Unit (GRU) cells. Out of all three, LSTM performed better than other
two cell types in terms of quantitative metrics (Table III). Qualitatively,
images generated by LSTM has less ghosting artifacts compared to GRU (Fig. 3).
However, we identify three challenges in LSTM cell design that can be
addressed to improve the performance specific to HDR deghosting task.

The problems encountered with LSTM cell are discussed below, along with the
corresponding change in the proposed SGM cell architecture to address the
issue:

  1. (i)

The forgetting of information from the internal state in both LSTM and SGM can
be represented in the form: $c_{n}=\alpha\cdot c_{n-1}+\beta$. It is evident
that $\alpha$ plays a large role in deciding what information is forgotten. In
case of LSTM, $\alpha$ is obtained from the forget gate as shown below:

LSTM | SGM
---|---
$\alpha=\sigma((h_{n-1}\oplus E_{n})\odot k+b)$ |  | $\alpha=\sigma((t_{n}\oplus i_{n})\odot k+b)$
---
$t_{n}=\psi(c_{n-1}\odot k+b)$

It is obvious through the equations that $c_{n-1}$ is not directly involved in
deciding what information is to be “forgotten” (Fig. 2c). Thus, the important
exposure information may be forgotten without much importance given to
$c_{n-1}$. The forgotten information is then overwritten with information from
the current image, resulting in a large contribution of final images to the
output. In some cases, this manifests as saturated regions in the output, even
though intermediate images may have the required detail to generate
saturation-free results.

To prevent this, we use $c_{n-1}$ in computation of the weight maps used for
forgetting in the SGM cell. In the SGM cell, as shown in above table and in
Fig. 2d, $c_{n-1}$ contributes significantly in computation of the new state
$c_{n}$. Visually, we see that SGM cell has retained structural information in
locations where LSTM fails to, as shown in Fig. 4A. Quantitatively, cell
architecture ablations types 6 and 7 in Table I show that not using $c_{n-1}$
actively for computing the new state results in degraded performance.

  2. (ii)

LSTM performs minimal computation before deleting information from the long-
term memory. The bulk of computation is performed in the input gate, and the
result is unconditionally integrated with $c$. If any structural artifacts are
not removed by the input gate, they will propagate through the rest of the
sequence with high probability. These show up as warping artifacts in the
final output of the LSTM cell, as shown in Fig. 4B.

To address this issue in the SGM, we propose a set of architectural changes.
First, we have individual layers for both $h$ and $c$, resulting in
preliminary filtering from both long-term and short-term memory before state
update. Next, we use both states to compute the weight map that ultimately
updates $c$. It can be seen in Fig. 4B that this design helps to eliminate
warping artifacts. The cell architecture ablations for these changes are Type
4, Type 6 and Type 7.

  3. (iii)

Finally, LSTM maps high values generated internally as well as in its output
to $[-1,1]$ range, due to the Tanh activations. This means that if two pixel
values from two different images are high, they will be mapped close together,
losing relative intensity information between them. This is especially
problematic if the number of images during evaluation is higher than the
number of training images since they will probably have more over-saturated
regions than what the LSTM has seen during training. In such cases, having
additional images will not result in higher quality output, as shown in Fig.
4C. Both models were trained with only 3 images and tested on 3 and 7 length
sequences. As highlighted by the arrows, an LSTM model trained with 3 images,
generates over-saturation when tested with 7 images. In contrast, SGM cell can
generate output without such artifacts.

SGM implicitly addresses this due to the Swish activation, which has an
unbounded upper limit. Fig. 4C shows that SGM is able to extract information
from long sequences, where LSTM saturates in quality.

With these major considerations, we perform a series of cell architecture
ablations to empirically get the optimal architecture for our application. The
final cell contains the following structures:

Input gate: This gate is used to extract relevant features from the input
$E_{n}$ to the cell, and its previous output $h_{n-1}$. It uses a convolution
block followed by self-gating:

$\displaystyle\overrightarrow{i_{n}}=\psi((\overrightarrow{h_{n-1}}\oplus
E_{n})\odot\overrightarrow{k_{i}}+\overrightarrow{b_{i}})$ (15)

where $k_{i}$ is the kernel used to convolve the input, $b_{i}$ is the bias,
$\oplus$ is the concatenation operation, and $\odot$ is the convolution
operation.

Transform gate: This gate is used to extract relevant features from the
previous internal state $c_{n-1}$ of the cell. It uses a convolution block
followed by self-gating:

$\displaystyle\overrightarrow{t_{n}}=\psi(\overrightarrow{c_{n-1}}\odot\overrightarrow{k_{t}}+\overrightarrow{b_{t}})$
(16)

where $k_{t}$ is the kernel used to convolve the previous state, and $b_{t}$
is the bias.

Update gate: The update gate is used to update the internal state $c_{n}$ of
the cell, using the features extracted from the previous state $c_{n-1}$,
previous output $h_{n-1}$ and the input $E_{n}$. It performs a weighted sum of
the two features to derive the new state $c_{n}$. The weights are determined
using a convolutional block having sigmoid activation:

$\displaystyle\overrightarrow{w_{n}}$
$\displaystyle=\sigma((\overrightarrow{i_{n}}\oplus\overrightarrow{t_{n}})\odot\overrightarrow{k_{w}}+\overrightarrow{b_{w}})$
(17) $\displaystyle\overrightarrow{c_{n}}$
$\displaystyle=\overrightarrow{w_{n}}\\!\cdot\\!\overrightarrow{t_{n}}\\!+\\![1\\!-\\!\overrightarrow{w_{n}}]\\!\cdot\\!\overrightarrow{i_{n}}$
(18)

where $k_{w}$ is the kernel used to convolve the input, and $b_{w}$ is the
bias.

Figure 4: Qualitative comparison between LSTM and SGM cells.

Output Gate: This gate produces the final output, $h_{n}$. It uses a
convolution block followed by self-gating. The inputs to this gate are the old
cell state ($c_{n-1}$), the new state ($c_{n}$) and the current input
($E_{n}$):

$\displaystyle\overrightarrow{h_{n}}=\psi((\overrightarrow{c_{n-1}}\oplus(\overrightarrow{c_{n}}+E_{n}))\odot\overrightarrow{k_{o}}+\overrightarrow{b_{o}})$
(19)

where $k_{o}$ is the kernel used to convolve the previous state, and $b_{o}$
is the bias.

We use two cells in a bidirectional arrangement in our implementation, each
having 64 filters with 3$\times$3 kernel size for all its convolutional
layers. The equations for the cell operating on the sequence $\mathcal{S}$ in
the reverse direction are identical to the equations mentioned above, applied
from $I_{N}$ to $I_{1}$:

$\displaystyle\overleftarrow{i_{n}}$
$\displaystyle=\psi((\overleftarrow{h_{n+1}}\oplus
E_{n})\odot\overleftarrow{k_{i}}+\overleftarrow{b_{i}})$ (20)
$\displaystyle\overleftarrow{t_{n}}$
$\displaystyle=\psi(\overleftarrow{c_{n+1}}\odot\overleftarrow{k_{t}}+\overleftarrow{b_{t}})$
(21) $\displaystyle\overleftarrow{w_{n}}$
$\displaystyle=\sigma((\overleftarrow{i_{n}}\oplus\overleftarrow{t_{n}})\odot\overleftarrow{k_{w}}+\overleftarrow{b_{w}})$
(22) $\displaystyle\overleftarrow{c_{n}}$
$\displaystyle=(\overleftarrow{w_{n}}*\overleftarrow{t_{n}})+((1-\overleftarrow{w_{n}})*\overleftarrow{i_{n}})$
(23) $\displaystyle\overleftarrow{h_{n}}$
$\displaystyle=\psi((\overleftarrow{c_{n}}\oplus(\overleftarrow{c_{n+1}}+E_{n}))\odot\overleftarrow{k_{o}}+\overleftarrow{b_{o}})$
(24)

Table I: Comparison of various SGM cell architectures. Metrics ($\rightarrow$) | PSNR-L | PSNR-T
---|---|---
| Images ($\rightarrow$)
---
/ Cell types ($\downarrow$)
3 | 5 | 7 | 3 | 5 | 7
| LSTM | 43.10 | 43.90 | 50.30 | 47.02 | 47.63 | 48.28
| Type1 | 43.21 | 43.69 | 49.25 | 46.61 | 47.37 | 47.06
| Type2 | 42.68 | 42.99 | 49.02 | 46.35 | 46.44 | 46.82
Proposed | Type3 | 42.13 | 43.02 | 49.49 | 45.76 | 46.38 | 46.35
Type4 | 43.01 | 43.25 | 48.12 | 46.05 | 45.89 | 45.05
Type5 | 42.29 | 43.34 | 49.19 | 46.09 | 46.82 | 47.21
Type6 | 43.08 | 43.73 | 49.98 | 46.95 | 47.20 | 47.43
| Type7 | 43.16 | 43.55 | 49.28 | 46.19 | 46.70 | 46.50
| Final SGM | 43.88 | 44.37 | 51.04 | 47.68 | 48.05 | 48.69
Figure 5: Qualitative comparison between state-of-the-art approaches and the
proposed approach on three validation sequences from Kalantari17 dataset [18].

To show the contribution of each structure and connection within the cell, we
perform a series of architecture experiments with the particular component
removed. The following ablations are performed:

  * •

Type 1: We remove the connection feeding $c_{n-1}$ to the remaining inputs of
the output gate. The output of the cell now becomes a function of the input of
the current internal state and input, with the information of previous inputs
present as a nonlinear transformation within the new state. We observe that
removing this connection results in a minor dip in overall performance of the
network.

  * •

Type 2: We do not concatenate $E_{n}$ to the other inputs of the output gate.
The output is thus derived only from the new state that integrates information
from the current input, and previous state. Important information about the
specific frame may be lost during this computation. As such, the drop in
performance after removing this connection is significant.

  * •

Type 3: The output gate only takes $c_{n}$ as its input; $c_{n-1}$ and $E_{n}$
are not used. As expected, drop in overall performance is greater than
removing any of the individual connections.

  * •

Type 4: We remove the transform gate completely from the cell. The remaining
parts are same as the final version. The drop in performance indicates that it
is advantageous to have some of the local information in $c_{n-1}$ forgotten
before the state update, without interference from the new input.

  * •

Type 5: We replace self-gating with two separate layers having sigmoid and
Tanh as their activations, and multiply them to get the output of the gate. We
follow the LSTM-style connections in the output gate. This shows that it is
harder to optimize gates having two separate components.

  * •

Type 6: We perform LSTM-style connections to “forget” parts of $c_{n-1}$. That
is, $c_{n-1}$ is not used to compute the weightmaps to perform state update.

  * •

Type 7: We evaluate whether the order of processing $\\{E_{n},h_{n-1}\\}$ and
forgetting from $c_{n-1}$ impacts performance. We modify the cell structure
from Type 6 ablation to perform this experiment.

##  IV Evaluation and Results

Table II: Quantitative comparison between the proposed method against nine
state-of-the-art methods. The best score is highlighted by blue cell color,
and the second-best score is highlighted by gray cell color. The abbrevations
used are as follows: T1 - linear domain, T2 - log tonemapper, T3 - Krawczyk et
al. [75], T4 - Reinhard et al. [50], and T5 - Durand et al. [76] tonemappers.

| PSNR | SSIM |
---|---|---|---
| T1 | T2 | T3 | T4 | T5 | T1 | T2 | T3 | T4 | T5 |  HDR- VDP-2
Hu13 | 31.25 | 35.75 | 30.18 | 28.87 | 31.69 | 0.941 | 0.963 | 0.940 | 0.942 | 0.944 | 62.07
Sen12 | 38.57 | 40.94 | 27.81 | 30.33 | 31.35 | 0.971 | 0.978 | 0.955 | 0.966 | 0.969 | 64.74
Endo17 | 8.846 | 21.33 | 12.26 | 12.10 | 20.65 | 0.107 | 0.622 | 0.715 | 0.787 | 0.738 | 54.00
Eilertsen17 | 14.21 | 14.13 | 25.23 | 20.26 | 26.82 | 0.350 | 0.882 | 0.925 | 0.923 | 0.935 | 57.95
Kalantari17 | 41.27 | 42.74 | 34.12 | 33.70 | 32.99 | 0.981 | 0.987 | 0.980 | 0.979 | 0.974 | 66.10
Wu18 | 40.91 | 41.65 | 34.98 | 34.54 | 33.69 | 0.986 | 0.986 | 0.982 | 0.982 | 0.977 | 67.44
Prabhakar19 | 39.68 | 40.47 | 33.56 | 34.08 | 32.60 | 0.980 | 0.975 | 0.966 | 0.975 | 0.961 | 66.50
Prabhakar20 | 41.33 | 42.82 | 35.18 | 36.94 | 36.29 | 0.986 | 0.989 | 0.984 | 0.985 | 0.982 | 67.15
Yan19 | 41.08 | 41.21 | 29.83 | 33.28 | 29.51 | 0.989 | 0.989 | 0.962 | 0.978 | 0.974 | 67.53
Proposed | 41.68 | 42.07 | 36.09 | 37.85 | 36.29 | 0.990 | 0.990 | 0.986 | 0.988 | 0.983 | 67.59
(a)

PSNR | SSIM |
---|---|---
T1 | T2 | T3 | T4 | T5 | T1 | T2 | T3 | T4 | T5 |  HDR- VDP-2
29.47 | 32.58 | 27.61 | 26.84 | 26.20 | 0.954 | 0.949 | 0.912 | 0.919 | 0.917 | 63.50
32.93 | 33.43 | 30.22 | 29.15 | 30.91 | 0.972 | 0.964 | 0.950 | 0.948 | 0.950 | 65.47
9.760 | 8.980 | 11.18 | 13.07 | 20.60 | 0.132 | 0.641 | 0.675 | 0.763 | 0.711 | 55.76
14.19 | 15.66 | 22.47 | 22.04 | 24.97 | 0.442 | 0.869 | 0.879 | 0.897 | 0.904 | 58.74
32.50 | 35.63 | 30.08 | 28.81 | 30.45 | 0.969 | 0.961 | 0.938 | 0.943 | 0.940 | 65.40
34.40 | 38.03 | 33.35 | 31.82 | 32.66 | 0.977 | 0.971 | 0.962 | 0.957 | 0.956 | 66.59
32.74 | 36.08 | 30.66 | 29.83 | 30.54 | 0.967 | 0.959 | 0.942 | 0.935 | 0.939 | 66.10
34.98 | 38.30 | 32.99 | 31.76 | 32.53 | 0.978 | 0.970 | 0.960 | 0.952 | 0.953 | 66.25
35.28 | 38.65 | 33.82 | 32.08 | 33.31 | 0.980 | 0.973 | 0.963 | 0.961 | 0.957 | 66.88
36.38 | 39.03 | 34.45 | 32.43 | 32.85 | 0.983 | 0.975 | 0.967 | 0.963 | 0.960 | 67.95
(b)

###  IV-A Implementation

Our models and pipelines were implemented using Tensorflow. The network was
trained on a machine with Intel core i7-8700 CPU and a NVIDIA RTX 2080ti 11GB
GPU. We use the Adam optimizer [77] with $2\times 10^{-4}$ learning rate and a
batch size of four to train the model for 200 epochs. The learning rate is
halved after every 25 epochs. We train the model using Kalantari et al. [18]
and Prabhakar et al. [20] datasets for 3-image fusion. [18] dataset consists
of 74 training and 15 validation sequences, while [20] dataset consists of 466
training and 116 validation sequences. In both [18] and [20] datasets, each
sequence consists of three varying exposure images with exposure bias
$\\{-2,0,+2\\}$ or $\\{-3,0,+3\\}$. Additionally, we trained on Prabhakar et
al. [78] dataset for validating scalability performance. Dataset given by [78]
consists of 70 training sequences with 7 LDR images, and the sequence length
(3, 5 or 7) is selected randomly during training. It also contains 14
sequences with 7 LDR images for testing. The sequence
$\\{I_{2},I_{4},I_{6}\\}$ is used for validating 3-image performance and
sequence $\\{I_{2},I_{3},I_{4},I_{5},I_{6}\\}$ is used for performing 5-image
validation. Images ${I_{1}}$ through ${I_{7}}$ are all used for 7-image
validation. Apart from [18], [20] and [78], we have validated our model on
publicly available datasets like Sen et al. [15], Cambridge [79] and Tursun et
al. [80].

###  IV-B Quantitative results

In Table IIb, we compare the proposed method trained for 3 images on [18] and
[20] datasets against nine other HDR deghosting approaches. The approaches
against which we compare the proposed approach are:  (i) Hu13 - Hu et al.
[16],  (ii) Sen12 - Sen et al. [15],  (iii) Endo17 - Endo et al. [81],  (iv)
Eilertsen17 - Eilertsen et al. [82],  (v) Kalantari17 - Kalantari et al. [18],
(vi) Wu18 - Wu et al. [19],  (vii) Prabhakar19 - Prabhakar et al. [20],
(viii) Prabhakar20 - Prabhakar et al. [22] (ix) Yan19 - Yan et al. [59].  We
evaluate the performance of different methods using three popular metrics:
HDR-VDP-2 [83], SSIM [84] and PSNR. HDR-VDP-2 is a full reference HDR image
quality assessment metric for evaluating prediction quality in all luminance
ranges.

To get an overall view of quality, we compute SSIM and PSNR using four
different tonemapping functions, in addition to comparison in the linear
domain. The tonemapping functions used are: Log tonemapper (Eqn. 13), Krawczyk
et al. [75], Reinhard et al. [85] and Durand et al. [76]. In total, we compare
against nine existing methods on eleven different metrics. Table IIba shows
quantitative comparison on Kalantari17 dataset with 15 test sequences. Our
proposed method outperforms all other existing methods on ten out of eleven
metrics. Similarly, in Table IIbb, we present a quantitative comparison on
Prabhakar19 dataset with 116 test images. We have trained all existing methods
and our method on [20] for a fair comparison. The proposed approach
outperforms all approaches in every metric except for PSNR on the [76]
tonemapping function.

Table III: Comparison of the proposed method against existing methods to fuse variable number of images from [78] dataset. Metrics ($\rightarrow$) | PSNR-L | PSNR-T
---|---|---
| Images ($\rightarrow$)
---
/ Methods ($\downarrow$)
3 | 5 | 7 | Mean | 3 | 5 | 7 | Mean
Sen12 | 40.61 | 41.71 | 47.08 | 43.13 | 46.26 | 47.51 | 47.27 | 47.01
Hu13 | 35.71 | 36.38 | 39.31 | 37.13 | 41.85 | 42.65 | 40.81 | 41.77
Prabhakar19 | 39.53 | 39.47 | 45.43 | 41.47 | 39.79 | 40.02 | 40.10 | 39.97
Proposed | Bi-Vanilla | 43.26 | 43.88 | 50.07 | 45.74 | 46.15 | 46.52 | 47.16 | 46.61
Bi-LSTM | 43.10 | 43.90 | 50.30 | 45.77 | 47.02 | 47.63 | 48.28 | 47.64
Bi-GRU | 42.46 | 43.02 | 49.23 | 44.90 | 45.69 | 46.01 | 46.45 | 46.05
Bi-SGM | 43.88 | 44.37 | 51.04 | 46.43 | 47.68 | 48.05 | 48.69 | 48.14

In Table III, we present quantitative comparison against existing scalable HDR
deghosting methods on Prabhakar et al. dataset [78] with sequences of length
3,5 and 7. We also compare against different recurrent cells to demonstrate
the proposed SGM cell’s effectiveness.

###  IV-C Qualitative results

Among the classical non-deep HDR deghosting methods, Hu13 [16] and Sen12 [15]
are popular patch-based optimization approaches222Please refer to
Supplementary material for more results.. As both Hu13 and Sen12 methods
synthesize the result with reference images, these methods’ success depends on
carefully choosing a reference image with the least amount of saturation. In
the presence of heavily saturated regions in the reference image, they tend to
introduce artifacts. As shown in Fig. 1, the result by Hu13 method has
structural artifacts in heavily saturated regions of the reference image. In
the same region, the Sen12 method generates results without any texture
information. Similarly, in 5c, we present a qualitative comparison between
different state-of-the-art methods and proposed method on a challenging
sequence from the Kalantari17 dataset. The results in Fig. 5 and 10 show the
artifacts introduced by both Hu13 and Sen12 methods on various sequences with
arbitrary length sequences.

Fig. 5c shows a qualitative comparison between Kalantari17 and the proposed
method on a difficult example from the Kalantari17 dataset. The Kalantari17
method has residual optical flow warping artifacts present in the result shown
in the zoomed red box. Similarly, in Fig. 5b, their result has warping
artifact along the boundaries. In another example from the Prabhakar19 dataset
shown in Fig. 7c, Kalantari17 method fails to correct warping artifacts in the
dynamic regions. In Fig. 7a, Kalantari17 method’s result has artifacts even in
static regions.

As shown in Fig. 5c, 5 and 7, Wu18 and Yan19 methods suffer from artifacts in
reference saturated dynamic regions. In such regions, Wu18 and Yan19 methods
fail to reconstruct complete texture from other LDR input images. In 5c, both
Wu18 and Yan19 generate results with missing branches. Also, in Fig. 5, both
methods introduce artifacts in regions affection by saturation and motion.
Similarly, in Fig. 7a and Fig. 7b, Wu18 and Yan19 results fail to faithfully
reconstruct window bars; instead, they smoothen out the details.

Figure 6: A qualitative example from Kalantari17 dataset [18]. The red zoomed
box highlight the artifact introduced by the Prabhakar20 method on saturated
dynamic regions. Comparatively, the proposed method can generate plausible
textures, even in saturated areas. Image best viewed electronically.

Prabhakar20 method generates output with artifacts as shown in Fig. 6. The
zoomed highlighted region is affected by motion and saturation. As the
Prabhakar20 method’s input is optical flow corrected, their approach
hallucinates incorrect textures in the saturated areas. In Fig. 5b, a similar
artifact is noticed in the saturated and moving region. In Fig. 7c, both
Prabhakar20 and Kalantari17 methods result in artifacts due to optical flow
alignment. In Fig. 7a and Fig. 7b, Prabhakar20 method has residual ghosting
artifact left in the result.

Figure 7: Qualitative comparison on sequences from (a) Cambridge dataset [79],
(b,c) Prabhakar19 dataset [20]. Image best viewed electronically.

Prabhakar19 is the only scalable deep learning-based method. In Fig. 1, we
show an example with seven input images. Prabhakar19 method’s output has color
distortion in the region marked with a blue box. In Fig. 7c, the Prabhakar19
method’s output has an over-smoothing artifact in the saturated reference
region. Similarly, in Fig. 10a the output of Prabhakar19 method contains
inaccurate texture details. In Fig. 7b, Prabhakar19 method over-smoothens the
cloud region, thus losing the necessary details in them.

In comparison, the proposed method generates results with accurate texture and
vivid color details for variable-length sequences without re-training. The
proposed method’s result is void of ghosting artifacts and can generate
plausible textures in regions affected by motion and saturation (Fig. 6 and
5). It should be noted that, as the existing state-of-the-art deep learning
methods except Prabhakar19, were trained on three input images, we tested
those methods only on sequences with three input images.

In Fig. 9, we present an example for adding more LDR images in a dynamic
scene. As seen from the results, adding more images leads to increase in the
dynamic range.

Figure 8: Qualitative comparison between (a) unidirectional and (b)
bidirectional SGM cells. As highlighted by the black arrow, the output by
unidirectional cell has artifacts. The ground truth is shown in (c). Figure 9:
An qualitative example to highlight effect of adding more LDRs to the network.
Left column: seven input varying exposure images with foreground motion.
Second column image: output by proposed method by fusing middle three images
($I_{3},I_{4},I_{5}$). Third column image: by fusing middle five images
$I_{2}$ to $I_{6}$. Last image: by fusing all seven images.

###  IV-D Ablations

Uni vs Bi-directional SGM: We run various experiments to show the
effectiveness of our approach in a variety of baseline settings. To validate
the importance of bi-directional setting, we train a model using only one SGM
cell in one direction. The input to decoder consists of final time step
feature maps. The results are provided in Table IV. Also, we observe that
using optical flow provides slight boost across all metrics. We observe that
bi-directional SGM cell offers better results compared to uni-directional cell
in all three comparison metrics. The qualitative comparison in Fig. 8 shows
that the output by uni-directional model has visible artifacts in saturated
regions, which is corrected in bi-directional model.

Scalability: To verify the effectiveness of our model to fuse arbitrary number
of images without retraining, we have tested a model trained with UCSD dataset
[18] on [78] dataset without retraining for different input image lengths. We
achieved a average PSNR-L of 43.64dB, while the only other learnable scalable
architecture [20] achieves 38.02dB. Also, we show qualitative comparisons for
HDR Deghosting on the METU dataset consisting of 9-image sequences without re-
training (Fig. 10). Additionally, we also show qualitative results for the
Multi-Exposure fusion problem using our method, where the models have been
trained on sequence lengths ranging from 3 images to 9 images and tested on
examples having 30 images in a sequence (Fig. 9 in supplementary file).

Table IV: Quantitative comparison between several baseline ablation experiments on Kalantari17 dataset. | Uni/Bi
---
directional
| Optical
---
flow
PSNR-L | PSNR-T |  | HDR-
---
VDP-2
$\rightarrow$ | ✗ | 41.16 | 41.71 | 67.29
✓ | 41.64 | 41.84 | 67.35
$\rightleftarrows$ | ✗ | 41.26 | 41.93 | 67.33
✓ | 41.68 | 42.07 | 67.59

Image order: In order to understand the importance of image order, we train a
model after randomly shuffling the low, medium and high exposure images in the
[18] dataset. In this experiment, we have considered all permutations of the
three sequences, including those where the reference image was displaced from
the central spot. The medium exposure image is used as reference, however its
location in the sequence is random. For validation, the same strategy is used.
The setup with optical flow corrected images gives a PSNR-L of 41.53 dB, which
is comparable to existing state-of-the-art methods (Table V). This shows that
our model can provide high quality HDR images even if the input LDR images are
not sorted by exposure time. We have also trained a model with 5 images in
random order which achieved a PSNR-L of 43.62dB versus 43.75dB without
shuffling.

SGM cell: We also conduct experiments regarding the structure of the proposed
SGM Cell (Table V). They are not included as separate Type Ablations, since
they are minimal changes on top of the final version of the cell. In the first
experiment, we replace the swish activation at the output gate with the Tanh
activation (change in equations 19 and 24), and observe a drop of 0.94 dB in
PSNR-L. In another ablation experiment, we remove the Swish activations and
use the outputs of the gates in equations (15), (16) and (19) (and
corresponding reverse equations) directly without applying any nonlinearity.
This experiment resulted in a drop of 0.63 dB in PSNR-L.

Figure 10: A qualitative example from (a) Prabhakar19 dataset [20], and (b) METU dataset [25]. Table V: An ablation study on the importance of SGM cell components. Ablations | PSNR-L (dB)
---|---
Shuffled input sequence | 41.53
Tanh at the output gate | 40.74
Without Sigmoid in Eqn. (16), (18) and (19) | 41.05
Proposed final architecture | 41.68

###  IV-E Running times

In Table VI, we present the running time comparison between existing methods
and the proposed method for variable length sequences. Our approach can fuse a
sequence with three images of resolution 1000$\times$1500 in 0.03 seconds on a
NVIDIA Quadro RTX 6000 GPU and a Intel Core i7 3.00 GHz CPU. Also, our method
is atleast 17$\times$ faster than other scalable HDR deghosting approaches.

Table VI: Running time (in seconds) and number of parameters comparison between existing methods and proposed method. | Number of Images | Parameters (Millions)
---|---|---
| Architecture
---
3 | 5 | 7 | Network | Cell
Sen12 | 146.0 | 496 | 1187 | - | -
Hu13 | 264.2 | 550 | 798 | - | -
Kalantari17 | 50.02 | - | - | 0.38 | -
Wu18 | 6.800 | - | - | 16.61 | -
Yan19 | 0.096 | - | - | 1.44 | -
Prabhakar19 | 0.663 | 1.274 | 1.873 | 12.21 | -
Proposed | Bi-LSTM | 0.042 | 0.058 | 0.075 | 1.30 | 0.33
Bi-GRU | 0.032 | 0.049 | 0.059 | 1.12 | 0.22
Bi-SGM | 0.037 | 0.052 | 0.072 | 1.19 | 0.26

##  V Conclusion

We propose a efficient scalable HDR deghosting method based on recurrent
neural networks. In our approach, we introduce a novel Self-Gated Memory
recurrent cell that can control the information flow by gating the output with
a function of itself. The SGM cell offers high accuracy than standard GRU and
has less parameters than LSTM cell. We utilize SGM cell in bi-directional
setting to achieve better performance and visually pleasing results. The major
advantage of our proposed method compared to many existing deep learning based
HDR deghosting methods, is that our approach can fuse arbitrary length
sequence without a need for re-training. Further, we demonstrate the
superiority of our method over existing state-of-the-art methods on three
publicly available datasets.

## Acknowledgment

This work was supported by a project grant from MeitY (No.4(16)/2019-ITEA) and
Uchhatar Avishkar Yojana (UAY) project (IISC_10), MHRD, Govt. of India.

## References

  * [1] M. D. Tocci, C. Kiser, N. Tocci, and P. Sen, “A versatile HDR video production system,” in _ACM Transactions on Graphics (TOG)_ , vol. 30, no. 4. ACM, 2011, p. 41.
  * [2] H. Zhao, B. Shi, C. Fernandez-Cull, S.-K. Yeung, and R. Raskar, “Unbounded high dynamic range photography using a modulo camera,” in _IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2015, pp. 1–10.
  * [3] P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in _ACM SIGGRAPH 2008 classes_. ACM, 2008, p. 31.
  * [4] T. Grosch, “Fast and robust high dynamic range image generation with camera and object movement,” _Vision, Modeling and Visualization, RWTH Aachen_ , pp. 277–284, 2006.
  * [5] S. Wu, S. Xie, S. Rahardja, and Z. Li, “A robust and fast anti-ghosting algorithm for high dynamic range imaging,” in _2010 IEEE International Conference on Image Processing_. IEEE, 2010, pp. 397–400.
  * [6] O. Gallo, N. Gelfandz, W.-C. Chen, M. Tico, and K. Pulli, “Artifact-free high dynamic range imaging,” in _ICCP_. IEEE, 2009, pp. 1–7.
  * [7] T.-H. Min, R.-H. Park, and S. Chang, “Histogram based ghost removal in high dynamic range images,” in _2009 IEEE International Conference on Multimedia and Expo_. IEEE, 2009, pp. 530–533.
  * [8] Y. S. Heo, K. M. Lee, S. U. Lee, Y. Moon, and J. Cha, “Ghost-free high dynamic range imaging,” in _Asian Conference on Computer Vision_. Springer, 2010, pp. 486–500.
  * [9] S. Raman and S. Chaudhuri, “Reconstruction of high contrast images for dynamic scenes,” _The Visual Computer_ , vol. 27, no. 12, pp. 1099–1114, 2011.
  * [10] A. Tomaszewska and R. Mantiuk, “Image registration for multiexposure high dynamic range image acquisition,” in _Proceedings of the International Conference on Computer Graphics, Visualization and Computer Vision_ , 2007.
  * [11] L. Bogoni, “Extending dynamic range of monochrome and color images through fusion,” in _Proceedings of the 15th International Conference on Pattern Recognition_ , 2000.
  * [12] H. Zimmer, A. Bruhn, and J. Weickert, “Freehand HDR imaging of moving scenes with simultaneous resolution enhancement,” in _Computer Graphics Forum_ , vol. 30, no. 2. Wiley Online Library, 2011, pp. 405–414.
  * [13] G. Ward, “Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures,” _Journal of Graphics Tools_ , vol. 8, no. 2, pp. 17–30, 2003.
  * [14] O. Gallo, A. Troccoli, J. Hu, K. Pulli, and J. Kautz, “Locally non-rigid registration for mobile HDR photography,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2015, pp. 49–56.
  * [15] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, “Robust patch-based HDR reconstruction of dynamic scenes.” _ACM Trans. Graph._ , vol. 31, no. 6, p. 203, 2012.
  * [16] J. Hu, O. Gallo, K. Pulli, and X. Sun, “HDR deghosting: How to deal with saturation?” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2013.
  * [17] K. Ma, H. Li, H. Yong, Z. Wang, D. Meng, and L. Zhang, “Robust multi-exposure image fusion: A structural patch decomposition approach,” _IEEE Transactions on Image Processing_ , vol. 26, no. 5, pp. 2519–2532, 2017.
  * [18] N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes,” _ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017)_ , vol. 36, no. 4, 2017.
  * [19] S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” in _European Conference on Computer Vision_ , 2018, pp. 120–135.
  * [20] K. R. Prabhakar, R. Arora, A. Swaminathan, K. P. Singh, and R. V. Babu, “A fast, scalable, and reliable deghosting method for extreme exposure fusion,” in _2019 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2019, pp. 1–8.
  * [21] Q. Yan, D. Gong, P. Zhang, Q. Shi, J. Sun, I. Reid, and Y. Zhang, “Multi-scale dense networks for deep high dynamic range imaging,” in _IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 41–50.
  * [22] K. R. Prabhakar, S. Agrawal, D. Singh, B. Ashwath, and R. V. Babu, “Towards practical and efficient high-resolution HDR deghosting with CNN,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [23] P. Sen and C. Aguerrebere, “Practical high dynamic range imaging of everyday scenes: photographing the world as we see it with our own eyes,” _IEEE Signal Processing Magazine_ , vol. 33, no. 5, pp. 36–44, 2016.
  * [24] P. Sen, “Overview of state-of-the-art algorithms for stack-based high-dynamic range (HDR) imaging,” _Electronic Imaging_ , vol. 2018, no. 5, pp. 1–8, 2018.
  * [25] O. T. Tursun, A. O. Akyüz, A. Erdem, and E. Erdem, “The state of the art in HDR deghosting: a survey and evaluation,” in _Computer Graphics Forum_ , vol. 34, no. 2. Wiley Online Library, 2015, pp. 683–707.
  * [26] L. Cerman and V. Hlavac, “Exposure time estimation for high dynamic range imaging with hand held camera,” in _Proc. of Computer Vision Winter Workshop, Czech Republic_. Citeseer, 2006\.
  * [27] A. A. Rad, L. Meylan, P. Vandewalle, and S. Süsstrunk, “Multidimensional image enhancement from a set of unregistered differently exposed images,” in _Computational Imaging V_ , vol. 6498. International Society for Optics and Photonics, 2007, p. 649808.
  * [28] S. Yao, “Robust image registration for multiple exposure high dynamic range image synthesis,” in _Image Processing: Algorithms and Systems IX_ , vol. 7870. International Society for Optics and Photonics, 2011, p. 78700Q.
  * [29] J. Im, S. Jang, S. Lee, and J. Paik, “Geometrical transformation-based ghost artifacts removing for high dynamic range image,” in _2011 18th IEEE International Conference on Image Processing_. IEEE, 2011, pp. 357–360.
  * [30] J. Im, S. Lee, and J. Paik, “Improved elastic registration for removing ghost artifacts in high dynamic imaging,” _IEEE Transactions on Consumer Electronics_ , vol. 57, no. 2, pp. 932–935, 2011.
  * [31] M. Gevrekci and B. K. Gunturk, “On geometric and photometric registration of images,” in _2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07_ , vol. 1. IEEE, 2007, pp. I–1261.
  * [32] A. O. Akyüz, “Photographically guided alignment for hdr images.” in _Eurographics (Areas Papers)_ , 2011, pp. 73–74.
  * [33] S. Mann, C. Manders, and J. Fung, “Painting with looks: Photographic images from video using quantimetric processing,” in _Proceedings of the tenth ACM international conference on Multimedia_ , 2002, pp. 117–126.
  * [34] F. M. Candocia, “Simultaneous homographic and comparametric alignment of multiple exposure-adjusted pictures of the same scene,” _IEEE Transactions on Image Processing_ , vol. 12, no. 12, pp. 1485–1494, 2003.
  * [35] S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High dynamic range video,” in _ACM Transactions on Graphics (TOG)_ , vol. 22, no. 3. ACM, 2003, pp. 319–325.
  * [36] A. Srikantha, D. Sidibé, and F. Mériaudeau, “An svd-based approach for ghost detection and removal in high dynamic range images,” in _International Conference on Pattern Recognition_ , 2012.
  * [37] H.-S. Sung, R.-H. Park, D.-K. Lee, and S. Chang, “Feature based ghost removal in high dynamic range imaging,” _International Journal of Computer Graphics & Animation_, vol. 3, no. 4, p. 23, 2013.
  * [38] J. An, S. H. Lee, J. G. Kuk, and N. I. Cho, “A multi-exposure image fusion algorithm without ghost effect,” in _2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2011, pp. 1565–1568.
  * [39] D.-K. Lee, R.-H. Park, and S. Chang, “Improved histogram based ghost removal in exposure fusion for high dynamic range images,” in _2011 IEEE 15th International Symposium on Consumer Electronics (ISCE)_. IEEE, 2011, pp. 586–591.
  * [40] Z. Li, S. Rahardja, Z. Zhu, S. Xie, and S. Wu, “Movement detection for the synthesis of high dynamic range images,” in _2010 IEEE International Conference on Image Processing_. IEEE, 2010, pp. 3133–3136.
  * [41] H.-Y. Lin and W.-Z. Chang, “High dynamic range imaging for stereoscopic scene representation,” in _2009 16th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2009, pp. 4305–4308.
  * [42] W. Zhang and W.-K. Cham, “Gradient-directed multiexposure composition,” _IEEE Transactions on Image Processing_ , vol. 21, no. 4, pp. 2318–2323, 2011\.
  * [43] S. Silk and J. Lang, “Fast high dynamic range image deghosting for arbitrary scene motion,” in _Proceedings of Graphics Interface 2012_ , 2012, pp. 85–92.
  * [44] W. Zhang and W.-K. Cham, “Gradient-directed composition of multi-exposure images,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2010.
  * [45] E. A. Khan, A. Akyiiz, and E. Reinhard, “Ghost removal in high dynamic range images,” in _IEEE International Conference on Image Processing_ , 2006.
  * [46] M. Pedone and J. Heikkilä, “Constrain propagation for ghost removal in high dynamic range images.” in _VISAPP (1)_ , 2008, pp. 36–41.
  * [47] F. Pece and J. Kautz, “Bitmap movement detection: HDR for dynamic scenes,” in _2010 Conference on Visual Media Production_. IEEE, 2010, pp. 1–8.
  * [48] A. Eden, M. Uyttendaele, and R. Szeliski, “Seamless image stitching of scenes with large motions and exposure differences,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2006.
  * [49] M. Granados, B. Ajdin, M. Wand, C. Theobalt, H.-P. Seidel, and H. P. Lensch, “Optimal HDR reconstruction with linear digital cameras,” in _Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2010, pp. 215–222.
  * [50] E. Reinhard and K. Devlin, “Dynamic range reduction inspired by photoreceptor physiology,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 11, no. 1, pp. 13–24, 2005.
  * [51] K. Jacobs, C. Loscos, and G. Ward, “Automatic high-dynamic range image generation for dynamic scenes,” _IEEE Computer Graphics and Applications_ , no. 2, pp. 84–93, 2008.
  * [52] D. Sidibé, W. Puech, and O. Strauss, “Ghost detection and removal in high dynamic range images,” in _17th European Signal Processing Conference_ , 2009\.
  * [53] T.-H. Oh, J.-Y. Lee, Y.-W. Tai, and I. S. Kweon, “Robust high dynamic range imaging by rank minimization,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 6, pp. 1219–1232, 2014.
  * [54] S. Ferradans, M. Bertalmío, E. Provenzi, and V. Caselles, “Generation of hdr images in non-static conditions based on gradient fusion.” in _VISAPP (1)_ , 2012, pp. 31–37.
  * [55] T. Jinno and M. Okuda, “Multiple exposure fusion for high dynamic range image acquisition,” _IEEE Transactions on image processing_ , vol. 21, no. 1, pp. 358–365, 2011.
  * [56] D. Hafner, O. Demetz, and J. Weickert, “Simultaneous HDR and optic flow computation,” in _22nd International Conference on Pattern Recognition (ICPR)_. IEEE, 2014, pp. 2065–2070.
  * [57] N. Menzel and M. Guthe, “Freehand hdr photography with motion compensation.” in _VMV_ , 2007, pp. 127–134.
  * [58] S.-C. Park, H.-H. Oh, J.-H. Kwon, W. Choe, and S.-D. Lee, “Motion artifact-free hdr imaging under dynamic environments,” in _2011 18th IEEE International Conference on Image Processing_. IEEE, 2011, pp. 353–356.
  * [59] Q. Yan, D. Gong, Q. Shi, A. van den Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1751–1760.
  * [60] Q. Yan, L. Zhang, Y. Liu, Y. Zhu, J. Sun, Q. Shi, and Y. Zhang, “Deep hdr imaging via a non-local network,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4308–4322, 2020.
  * [61] M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff, and F. Durand, “Deep bilateral learning for real-time image enhancement,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, pp. 1–12, 2017.
  * [62] Y. Zhao, X. Jin, and X. Hu, “Recurrent convolutional neural network for speech processing,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 5300–5304.
  * [63] S. Pascual and A. Bonafonte, “Multi-output RNN-LSTM for multiple speaker speech synthesis and adaptation,” in _2016 24th European Signal Processing Conference (EUSIPCO)_. IEEE, 2016, pp. 2325–2329.
  * [64] G. Gelly and J.-L. Gauvain, “Optimization of rnn-based speech activity detection,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 3, pp. 646–656, 2017.
  * [65] T. Mikolov and G. Zweig, “Context dependent recurrent neural network language model,” in _2012 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2012, pp. 234–239.
  * [66] M. Sundermeyer, T. Alkhouli, J. Wuebker, and H. Ney, “Translation modeling with bidirectional recurrent neural networks,” in _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014, pp. 14–25.
  * [67] S. Ghosh, O. Vinyals, B. Strope, S. Roy, T. Dean, and L. Heck, “Contextual lstm (clstm) models for large scale nlp tasks,” _arXiv preprint arXiv:1602.06291_ , 2016.
  * [68] W. Pei, J. Zhang, X. Wang, L. Ke, X. Shen, and Y.-W. Tai, “Memory-attended recurrent network for video captioning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8347–8356.
  * [69] H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu, “Video paragraph captioning using hierarchical recurrent neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4584–4593.
  * [70] Y.-T. Hu, J.-B. Huang, and A. G. Schwing, “Maskrnn: Instance level video object segmentation,” _arXiv preprint arXiv:1803.11187_ , 2018.
  * [71] S. Valipour, M. Siam, M. Jagersand, and N. Ray, “Recurrent fully convolutional networks for video segmentation,” in _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2017, pp. 29–36.
  * [72] C. Godard, K. Matzen, and M. Uyttendaele, “Deep burst denoising,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 538–554.
  * [73] X. Chen, L. Song, and X. Yang, “Deep RNNs for video denoising,” in _Applications of Digital Image Processing XXXIX_ , vol. 9971. international Society for optics and Photonics, 2016, p. 99711T.
  * [74] R. Schuster, O. Wasenmuller, C. Unger, and D. Stricker, “Sdc-stacked dilated convolution: A unified descriptor network for dense matching tasks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2556–2565.
  * [75] G. Krawczyk, K. Myszkowski, and H.-P. Seidel, “Lightness perception in tone reproduction for high dynamic range images,” in _Computer Graphics Forum_ , vol. 24, no. 3. Citeseer, 2005, pp. 635–646.
  * [76] F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” in _ACM transactions on graphics (TOG)_ , vol. 21, no. 3. ACM, 2002, pp. 257–266.
  * [77] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [78] K. R. Prabhakar and R. V. Babu, “High dynamic range deghosting dataset,” 2020, data retrieved from IISc VAL website. [Online]. Available: https://val.cds.iisc.ac.in/HDR/HDRD/
  * [79] K. Karaduzovic-Hadziabdic, T. J. Hasic, and R. K. Mantiuk, “Multi-exposure image stacks for testing HDR deghosting methods,” 2017.
  * [80] O. T. Tursun, A. O. Akyüz, A. Erdem, and E. Erdem, “An objective deghosting quality metric for HDR images,” in _Computer Graphics Forum_ , vol. 35, no. 2. Wiley Online Library, 2016, pp. 139–152.
  * [81] Y. Endo, Y. Kanamori, and J. Mitani, “Deep reverse tone mapping.” _ACM Trans. Graph._ , vol. 36, no. 6, pp. 177–1, 2017.
  * [82] G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, “HDR image reconstruction from a single exposure using deep CNNs,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 6, p. 178, 2017.
  * [83] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions,” _ACM Transactions on graphics (TOG)_ , vol. 30, no. 4, p. 40, 2011.
  * [84] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [85] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda, “Photographic tone reproduction for digital images,” in _ACM Transactions on Graphics (TOG)_ , vol. 21, no. 3. ACM, 2002, pp. 267–276.