CEN-HDR: Computationally Efficient neural Network for real-time High
  Dynamic Range imaging
High dynamic range (HDR) imaging is still a challenging task in modern
digital photography. Recent research proposes solutions that provide
high-quality acquisition but at the cost of a very large number of operations
and a slow inference time that prevent the implementation of these solutions on
lightweight real-time systems. In this paper, we propose CEN-HDR, a new
computationally efficient neural network by providing a novel architecture
based on a light attention mechanism and sub-pixel convolution operations for
real-time HDR imaging. We also provide an efficient training scheme by applying
network compression using knowledge distillation. We performed extensive
qualitative and quantitative comparisons to show that our approach produces
competitive results in image quality while being faster than state-of-the-art
solutions, allowing it to be practically deployed under real-time constraints.
Experimental results show our method obtains a score of 43.04 mu-PSNR on the
Kalantari2017 dataset with a framerate of 33 FPS using a Macbook M1 NPU.
Steven Tel, Barthélémy Heyrman, Dominique Ginhac
# CEN-HDR: Computationally Efficient neural Network for real-time High Dynamic Range imaging

###### Abstract

High dynamic range (HDR) imaging is still a challenging task in modern digital photography. Recent research proposes solutions that provide high-quality acquisition but at the cost of a very large number of operations and a slow inference time that prevent the implementation of these solutions on lightweight real-time systems. In this paper, we propose CEN-HDR, a new computationally efficient neural network by providing a novel architecture based on a light attention mechanism and sub-pixel convolution operations for real-time HDR imaging. We also provide an efficient training scheme by applying network compression using knowledge distillation. We performed extensive qualitative and quantitative comparisons to show that our approach produces competitive results in image quality while being faster than state-of-the-art solutions, allowing it to be practically deployed under real-time constraints. Experimental results show our method obtains a score of 43.04 \(\mu\)-PSNR on the _Kalantari2017_ dataset with a framerate of 33 FPS using a Macbook M1 NPU. The proposed network will be available at _[https://github.com/steven-tel/CEN-HDR_](https://github.com/steven-tel/CEN-HDR_)

Keywords:High Dynamic Range Imaging, Efficient computational photography

## 1 Introduction

In the last decades, applications based on computer vision have become increasingly important in everyday life. Currently, many research works are conducted to propose more reliable algorithms in areas such as object detection, action recognition, or scene understanding. However, the accuracy of these algorithms depends largely on the quality of the acquired images. Most standard cameras are unable to faithfully reproduce the illuminations range of a natural scene, as the limitations of their sensors generate a loss of structural or textural information in under-exposed and over-exposed regions of the acquired scene. To tackle this challenge, sensors with a higher dynamic range (HDR) have been proposed [17, 26] to capture more intensity levels of the scene illumination, but
these solutions are expensive, preventing high dynamic range acquisition from being readily available.

Towards making HDR imaging practical and accessible, software solutions were proposed, based on the emergence of deep learning in computer vision applications. They acquire one Low Dynamic Range (LDR) image and try to expends its dynamic range thanks to a generative adversarial network [12, 37, 8]. Although these methods produce images with a higher illumination range, they have limitations in extending the dynamic of the input image to the dynamic of the acquired scene. A more effective approach is to acquire multiple LDR images with different exposure times and merge them into one final HDR image. Traditional computer vision algorithms [3] allow the acquisition of good-quality static scenes when there is no camera or object motion between images with different exposure times. However, in a lot of use cases, images are captured in a rapid sequence from a hand-held device resulting in inevitable misalignments between low dynamic range shots. Therefore scenes with motions introduce new challenges as ghost-like artifacts for large motion regions or loss of details in occluded regions. Following recent advances in the field of deep learning, several methods based on Convolutional Neural Network (CNN) were proposed to spatially align input frames to a reference one before merging them into a final HDR image.

State-of-the-art deep learning solutions for multi-frame merging HDR [34, 13] tend to be based on a previously proposed method[30] and add additional processing to increase the accuracy of the HDR merging system. As a consequence, the computational cost and execution time are significantly increased, preventing these solutions to be used in lightweight systems and/or in real-time applications. Then, the primary goal of HDR imaging software solutions, which was to make HDR imaging more widely available compared to hardware solutions, is therefore not being achieved at all. Therefore, in this paper, we propose a Computationally Efficient neural Network for High Dynamic Range imaging (CEN-HDR). CEN-HDR is based on an encoder-decoder neural network architecture for generating ghost-free HDR images from scenes with large foreground and camera movements. Unlike previously published solutions, we decided to develop a new approach keeping in mind the constraint of inference time and computational cost.

1. We propose CEN-HDR a novel efficient convolutional neural network based on a new attention mechanism and sub-pixel convolution that overcomes ghost-like artifacts and occluded regions while keeping a low computational cost, allowing our solution to be implemented in real-time on a lightweight system.
2. We demonstrate the efficiency of network compression for the realization of CEN-HDR by applying a knowledge distillation scheme.
3. We perform extensive experiments to determine the best trade-off between accuracy and inference cost with the main objective to demonstrate the relevance of CEN-HDR.
## 2 Related Works

We briefly summarize existing HDR merging approaches into two categories: deep learning-based architectures and efficient learning-based architectures. In the first category, the proposed methods aim to achieve better quality in HDR imaging without taking inference cost into account. Approaches belonging to the second category seek to optimize the compromise between the quality of the generated images and the computation cost. This leads to the use of new operators in the proposed deep learning architectures.

### Deep learning based HDR merging

Using multiple input images for HDR generation leads to the need to align the features of the LDR images to the reference image. The first common method for feature registration is by computing the motion between inputs features using an optical flow algorithm. Multiple studies [31, 7, 22] used Liu[11] optical flow algorithm. In _Kalantari et al.[7]_, the input images are aligned by selecting the image with better pixels as a reference and computing the optical flow between this reference and other input LDR images. Then, the warped images are fed to a supervised convolutional neural network (CNN) to merge them into an HDR image. However, since the optical flow algorithm initially assumes that the input images have the same exposure time, trying to warp the different exposures with occluded regions can result in artifacts in the final HDR image. To address this issue, _DeepHDR[28]_ proposes an image translation network able to hallucinate information in the occluded regions without the need for optical flow. Moreover, many solutions have been developed to correct the ghost effect introduced by the misalignment of the input images. In _AHDRNet[30]_ an attention module is

Figure 1: Architecture of the proposed CEN-HDR solution. The spatial size of input features is divided by 2 at the encoding step. The attention module allows registering non-reference features to the reference ones. The full spatial size is recovered thanks to the pixel shuffle operation.
proposed to emphasize the alignment of the input images to the reference image. Input images are then merged using several dilated residual dense blocks. The high performance of the AHDRNet network led it to be used as a base network for other methods. For example, _ADNet_[13] follows the same main architecture that AHDRNet but adds a Pyramidal alignment module based on deformable convolution allowing a better representation of the extracted features but in the counterpart of a larger number of operations.

### Efficient learning-based HDR merging architectures

To our best knowledge, the first architecture which aims to be efficient was proposed by _Prabhakar et al._[21] by processing low-resolution images and upscaling the result to the original full resolution thanks to a bilateral guided upsampling module. Recently, the HDR community tends to focus more on efficient HDR image generation[20] and no longer only aims at improving the image quality but also at significantly limiting the number of processing operations. This results in efficient solutions such as _GSANet_[9] that propose efficient ways to process gamma projections of input images with spatial and channel attention blocks to increase image quality while limiting the number of parameters. _Yu et al._[36] introduce a multi-frequency lightweight encoding module to extract features and a progressive dilated u-shape block for features merging. Moreover, the different standard convolution operations are replaced by depth-wise separable convolution operations firstly proposed in [2], they are composed of a depth-wise convolution followed by a pointwise convolution which allows more efficient use of model parameters. Another efficient method, proposed by _Yan et al._[33] is a lightweight network based on an u-net[23] like encoder-decoder architecture, allowing for spatially reduced processed features. While these solutions focus on the number of performed operations, their inference time still remains too long to be considered as real-time solutions.

## 3 Proposed Method

We consider three LDR images \(I_{i}\in\mathbb{R}^{3\times H\times W}\) with their respective exposure times \(t_{i}\) as inputs. The generated HDR image is spatially aligned with the central LDR frame \(I_{2}\) selected as the reference image. To make our solution more robust to exposure difference between inputs, the respective projection of each LDR input frame into the HDR domain is calculated using the gamma encoding function described in Eq. 1, following previous works [13, 18, 30]:

\[H_{i}=\frac{I_{i}^{\gamma}}{t_{i}},\quad\gamma=2,2 \tag{1}\]

Where \(H_{i}\in\mathbb{R}^{3\times H\times W}\) is the gamma-projected input. Then, each input is concatenated with their respective gamma-projection to obtain \(L_{i}\in\mathbb{R}^{6\times H\times W}:\)

\[L_{i}=I_{i}\oplus H_{i} \tag{2}\]
where \(\oplus\) represents the concatenation operation. \(L_{i}\) will then be fed to our proposed merging network whose architecture is detailed in Fig. 1.

### Feature encoding

Using high-resolution images as inputs presents an additional challenge in the design of a real-time HDR merging network. To solve such a problem, previous works [29, 33] propose to use a U-net[23] like architecture to reduce the spatial size of the features processed by the merging network. However, a too large reduction of the spatial dimensions causes the extraction of coarse features that degrade the final result. So, we decide to limit the spatial reduction to 2 by using an encoder block composed of 2 sequential convolutions as described in Eq. 3.

\[F_{i}=conv_{E_{1}}(conv_{E_{2}}(L_{i})) \tag{3}\]

where \(F_{i}\in\mathbb{R}^{32\times\frac{H}{2}\times\frac{W}{2}}\) is the features map extracted from the encoder for each LDR input. \(conv_{E_{1}}\) and \(conv_{E_{2}}\) are 3x3-convolution layers extracting respectively 16 and 32 features map. The spatial size is divided by 2 setting a stride of 2 for \(conv_{2}\).

### Attention module

The final generated HDR image must be aligned with the reference image. To address this requirement [30] demonstrates the effectiveness of using a spatial

Figure 2: Illustration of the attention module composed of 2 branches, respectively responsible for spatial attention and channel attention. A sigmoid activation function is used to keep the value between 0 and 1.
attention module after the encoding step. Since then, many spatial and channel attention modules have been proposed in the literature which can be integrated into networks to improve their performance. According to the inference cost study done in Table 5, we propose the _Spatial-Channel Reference Attention Module_ (SCRAM) a slightly modified version of the Bottleneck Attention Module (BAM) proposed in [19]. Indeed while BAM aims to generate a mask of its input feature maps, in our case we want to generate attention maps from the concatenation of reference and non-reference features, these attention maps are then applied to non-reference features only resulting in a reduction of the number of feature maps in the proposed attention module. Moreover, batch normalization is not applied in SCRAM. The detailed structure of SCRAM is illustrated in Fig. 2.

The non-reference features \(F_{i\neq 2}\) are concatenated with the features of the reference image:

\[X_{i}=F_{i}\oplus F_{2=ref},\quad i\neq 2 \tag{4}\]

where \(X_{i}\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}\) is the input of SCRAM and \(\oplus\) is the concatenation operator.

Following [19], SCRAM is composed of two branches respectively responsible for the spatial and channel features alignment of the non-reference images to the reference ones:

\[A_{i}=\sigma(s(X_{i})+c(X_{i})),\quad i\neq 2 \tag{5}\]

where \(s\) is the spatial attention branch and \(c\) is the channel attention branch. The sum of produced features by each branch passes through \(\sigma\), a sigmoid activation function to keep the output values between 0 and 1.

_Spatial attention:_ The objective of the spatial branch is to produce an attention map allowing to keep the most relevant information for the spatial alignment of the non-reference images to the reference one. To limit the computation, we first reduce the number of features map by 3 using a pointwise convolution. With the objective to extract more global features while keeping the same computation, we then make the receptive field larger by employing 3 dilated convolutions [35] layers with a factor of dilatation set to 2. The final attention map of size \((1,H,W)\) is produced by using a pointwise convolution and then expanded across the channel dimension to obtain \(A_{S}\in\mathbb{R}^{32\times H\times W}\).

_Channel attention:_ This branch aims to perform a channel-wise feature recalibration. We first squeeze the spatial dimension by applying a global average pooling which sums out the spatial information to obtain the features vector of size \((64,1,1)\). A multilayer perceptron with three hidden layers is then used in purpose to estimate cross-channel attention. The last activation size is set to 32 to fit the number of channels of the non-reference features \(F_{i}\). Finally, the resulting vector map is spatially expanded to obtain the final feature map \(A_{C}\in\mathbb{R}^{32\times H\times W}\).
The Attention features \(A_{i}\) are then used to weight the non-reference features \(F_{i}\):

\[F_{i}^{\prime}=F_{i}\otimes A_{i},\quad i\neq 2 \tag{6}\]

where \(\otimes\) is the element-wise product and \(F_{i}^{\prime}\) is the aligned non-reference features. For the reference features we set \(F_{2}^{\prime}=F_{2}\).

### Features merging

While most of the computation is usually done in the merging block [13, 18, 30], we propose a novel efficient feature merging block that first focuses on merging the non-references features. Each feature map \(F_{i}^{\prime}\) produced by the encoder goes through a convolution layer:

\[M_{i}=conv_{M_{1}}(F_{i}^{\prime}) \tag{7}\]

Where \(conv_{M_{1}}\) is a 3x3-convolution producing \(M_{i}\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}\). Then we focus on merging the non-reference features maps by concatenating them and feeding the result features in a convolution layer:

\[M_{\text{non-ref}}=conv_{M_{2}}(M_{1}\oplus M_{3}) \tag{8}\]

where \(conv_{M_{2}}\) is a 3x3-convolution and \(M_{\text{non-ref}}\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}\) is the non-reference merged features.

As we emphasize the reference features throughout all our network, here we merge our reference features with the non-reference \(M_{non-ref}\) only by adding them together to limit the number of features map processed later:

\[M=conv_{M_{4}}(conv_{M_{3}}(M_{2}+M_{non-ref})) \tag{9}\]

where \(conv_{M_{3}}\) and \(conv_{M_{4}}\) are 3x3-convolutions producing each 64 features map and \(M\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}\) contains features from all LDR input images.

### Features decoding

The role of the decoder is to produce the final HDR image from the features produced by the merger block. At the encoding stage, we divided the spatial dimensions by 2. While the original spatial size is usually recovered using bilinear upsampling or transposed convolution[14] operation, we propose to use the pixel shuffle operation first proposed in [25], it is presented as an efficient sub-pixel convolution with a stride of \(1/r\) where \(r\) is the upscale factor. In our case, we set \(r=2\). As illustrated in Fig. 3, the pixel shuffle layer rearranges elements in a tensor of shape (\(C\times r^{2},H,W\)) to a tensor of shape (\(C,H\times r,W\times r\)):

\[D=PixelShuffle(M) \tag{10}\]

where \(D\in\mathbb{R}^{16\times H\times W}\) is the resulting upscaled features.
The final HDR image is obtained by following the Eq 11.

\[HDR=\sigma(conv_{D}(D+S_{2})) \tag{11}\]

where \(S_{2}\) is the reference features extracted by the first convolution layer of our network \(conv_{E_{1}}\) to stabilize the training of our network. Finally, we generate the final HDR image using a 3x3-convolution layer, followed by a Sigmoid activation function.

## 4 Experimental Settings

### Datasets

The CEN-HDR network has been trained using the dataset provided by [7] composed of 74 training samples and 15 test samples. Each sample represents the acquisition of a dynamic scene caused by large foreground or camera motions and is composed of three input LDR images (with EV of -2.00, 0.00, +2.00 or -3.00, 0.00, +3.00) and a reference HDR image aligned with the medium exposure image. The network has also been separately trained and tested using the dataset from the NTIRE[20] dataset, where the 3 LDR images are synthetically generated from the HDR images provided by [4]. The dataset is composed of 1500 training samples, 60 validation samples, and 201 testing samples. The ground-truth images for the testing sample are not provided.

### Loss function

Following previous works [7, 30], the images have been mapped from the linear HDR domain to the LDR domain before evaluating the loss function. In order to train the network, the tone-mapping function has to be differentiable around zero, so, the \(\mu\)-law function is defined as follows:

\[T(H)=\frac{log(1+\mu H)}{log(1+\mu)},\quad\mu=5000 \tag{12}\]

Figure 3: Illustration of the pixel rearrangement by the pixel shuffle layer for an upscale factor set to \(r=2\) and an input shape of \((4,4,4)\). In the proposed solution, the input shape is \((64,\frac{H}{2},\frac{W}{2})\), the produced output size is \((16,H,W)\).
where H is the linear HDR image and \(\mu\) the amount of compression.

To make an efficient network, a network compression method defined as knowledge distillation proposed in [5] has been used. By using knowledge distillation, we assume that the capacity of a large network is not fully exploited, so the objective is to transfer the knowledge of this large teacher network to our lighter network as described in the Eq. 13.

\[\mathcal{L}=\alpha\times\mathcal{L}(T,T_{GT})+(1-\alpha)\times\mathcal{L}(T,T_{ Teacher}) \tag{13}\]

where \(\mathcal{L}\) is the \(L_{1}\) loss function. \(T\) is the tone mapped prediction of our network, \(T_{GT}\) the tone mapped ground truth provided in the dataset and \(T_{Teacher}\) the tone mapped prediction of the large teacher network. we use the HDR-GAN [18] model as the teacher. \(\alpha\) is a trade-off parameter set to 0.2. Moreover, the method proposed by [7] to produce the training dataset focuses mainly on foreground motion. It does not allow the generation of reliable ground-truth images for chaotic motions in the background, such as the movement of tree leaves due to wind, which results in a ground truth image with blurred features that do not reflect reality. This has the effect of producing a greater error when the predicted image contains sharper features than in the ground truth image. Using also a predicted image from a teacher model allows for dealing with this data misalignment. The comparison of the performance obtained between training done with knowledge distillation and without is made in Table. 1.

Figure 4: Qualitative comparison of the proposed CEN-HDR solution with other HDR merging methods. The cropped patch demonstrates that the proposed efficient network has equivalent capabilities to the state-of-the-art methods to correct the ghost effect due to the large movement in the scene.
### Implementation details

The CEN-HDR network has been trained using cropped patches of size \(256\times 256\) pixels with a stride of 128 and evaluated on the full-resolution test images. During training, random augmentations are applied on the cropped patch such as horizontal symmetry and rotation of 90, 180, or 270 degrees. Training has been done using the Adam optimizer with a batch size of 8. The learning rate is initially set to \(10^{-4}\), keep fixed for 80 epochs, and decreased by 0.8 every 20 epochs after. The training lasts for 500 epochs.

\begin{table}
\begin{tabular}{l|l l l l l} \hline \hline Training method & \(\mu\)-PSNR & PSNR & \(\mu\)-SSIM & SSIM & HDR-VDP2 \\ \hline w/o knowledge distillation & 40.8983 & 40.0298 & 0.9772 & 0.9926 & 62.17 \\ with knowledge distillation & 43.0470 & 40.5335 & 0.9908 & 0.9956 & 64.34 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of the proposed CEN-HDR architecture performances with and without knowledge distillation. In the first case, the network is trained with the HDR ground-truth proposed in [7]. In the second case, we also use the prediction of HDR-GAN [18] as label (Eq. 13). In both cases, the network is trained for 500 epochs using the \(l_{1}\) criterion.

Figure 5: Comparison of the proposed CEN-HDR solution with other HDR merging methods. The X-axis represents the mean runtime using the M1 NPU with an input size of 1280x720 pixels. The Y-axis is the fidelity score on the test images from [7] dataset. The best solutions tend to be in the upper left corner. The radius of circles represents the number of operations, the smaller the better.
## 5 Experimental Results

### Fidelity performance

In Table 2 and Fig. 4, the proposed CEN-HDR solution is compared against seven lightweight state-of-the-art methods: [24] and [6] are based on input patch registration methods. [7] is based on a sequential CNN, the inputs need first to be aligned thanks to an optical flow algorithm. For [28], the background of each LDR input is aligned by homography before being fed to an encoder-decoder-based CNN. [32] proposes an encoder-decoder architecture with a non-local attention module. [30] is a CNN based on an attention block for features registration and on multiple dilated residual dense blocks for merging. [18] is the first GAN-based approach for HDR merging with a deep supervised HDR method. Quantitative evaluation is done using objective metrics. The standard peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are computed both directly in the linear domain and after tone mapping by applying the \(\mu\)-law function (Eq. 12). The HDR-VDP2[15] metric predicts the quality degradation with the respect to the reference image. We set the diagonal display size to 24 inches and the viewing distance to 0.5 meter. In addition, PU-PSNR and PU-SSIM are calculated by applying the encoding function proposed in [16] with a peak luminance set to 4000.

In Fig. 6, we present the results for 3 test scenes of the NTIRE[20] dataset. While the network can produce images with a high dynamic range, we notice that the high sensor noise present in the input images is not fully corrected in the dark areas of the output HDR images. Moreover, the motion blur introduced in the dataset produces less sharp characteris

\begin{table}
\begin{tabular}{l|r r r r r r r} \hline \hline Method & \(\mu\)-PSNR & PU-PSNR & PSNR & \(\mu\)-SSIM & PU-SSIM & SSIM & HDR-VDP2 \\ \hline Sen et al.[24] & 40.80 & 32.47 & 38.11 & 0.9808 & 0.9775 & 0.9721 & 59.38 \\ Hu et al.[6] & 35.79 & \(-\) & 30.76 & 0.9717 & \(-\) & 0.9503 & 57.05 \\ Kalantari et al.[7] & 42.67 & 33.82 & 41.23 & 0.9888 & 0.9832 & 0.9846 & 65.05 \\ DeepHDR[28] & 41.65 & 31.36 & 40.88 & 0.9860 & 0.9815 & 0.9858 & 64.90 \\ NHDRRNet[32] & 42.41 & \(-\) & \(-\) & 0.9887 & \(-\) & \(-\) & 61.21 \\ AHDRNet[30] & 43.61 & 33.94 & 41.03 & 0.9900 & 0.9855 & 0.9702 & 64.61 \\ HDRGAN[18] & 43.92 & 34.04 & 41.57 & 0.9905 & 0.9851 & 0.9865 & 65.45 \\ CEN-HDR(our) & 43.05 & 33.23 & 40.53 & 0.9908 & 0.9821 & 0.9856 & 64.34 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Quantitative comparison with lightweight state-of-the-art methods on the Kalantari2017[7] test samples. PSNR and SSIM are calculated in the linear domain while \(\mu\)-PSNR and \(\mu\)-SSIM are calculated after \(\mu\)-law tone mapping (Eq. 12). For compared methods, the results are from [18]. PU-PSNR and PU-SSIM are calculated applying the encoding function proposed in [16].
In Table 6 we study the effect of the attention module on the performance of proposed HDR deghosting architecture. SCRAM-C and SCRAM-S respectively correspond to the SCRAM module composed only of the channel attention branch and the spatial attention branch. The proposed SCRAM allows to achieve a similar quality as the spatial attention module proposed in AHDRNet[28], while Table 5 shows that the inference cost of SCRAM is lower.

### Efficiency comparison

As we want to propose an efficient HDR generation method, in Table 3 we compare the computation cost and the inference time of our network with state-of-the-art HDR networks that achieve similar performance on quality metrics.

The number of operations and parameters are measured using the script provided by the NTIRE[20] challenge. To evaluate runtimes, all the compared networks are executed on the Neural Processing Unit (NPU) of a MacBook Pro (2021) powered with an M1 chip. The time shown is the average for 500 inference runs after a warm-up of 50 runs. The input size is set to \(1280\times 720\) pixels. The gamma-projection of LDR inputs (Eq. 1) and the tone mapping of the HDR output are included in the inference time measurement. Note that the background alignment of inputs frame using homography for [28] is not included.

Table 4 compares the number of parameters and operations of the proposed CEN-HDR solution with recent efficient methods [9, 33, 36]. The input size is set to \(1900\times 1060\) pixels corresponding to the size of the inputs from the dataset proposed by [20]. For compared methods [9, 33, 36] the measurements are provided by the NTIRE[20] challenge. We could not compare the inference time of the CEN-HDR solution with these three architectures as they were recently proposed and their implementation is not yet available.

Fig. 5 compares the trade-off between fidelity to the ground truth label and runtime of the proposed CEN-HDR solution with other HDR merging methods.

\begin{table}
\begin{tabular}{l|r r r r} \hline \hline Method & Num. of params. & Num. of op. (GMAccs) & Runtime(s) & FPS \\ \hline DeepHDR[28] & 14618755 & 843.16 & 0.1075 & 9.30 \\ AHDRNet[30] & 1441283 & 1334.95 & 0.4571 & 2.18 \\ NHDRNet[32] & 7672649 & 166.11 & 0.0431 & 23.20 \\ HDR-GAN[18] & 2631011 & 479.78 & 0.2414 & 4.14 \\ CEN-HDR(our) & 282883 & 78.36 & 0.0277 & 36.38 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Inference cost comparison of the proposed CEN-HDR solution against state-of-the-art lightweight deep learning-based methods. The number of operations and parameters are measured using the script provided by the NTIRE[20] challenge. To measure the inference time, all the compared networks are executed on an M1 NPU. The input size is set to \(1280\times 720\) pixels.
The X-axis represents the mean runtime using an M1 NPU with an input size of 1280x720 pixels. The Y-axis is the fidelity score on the test images from [7] dataset. The best solutions tend to be in the upper left corner. The radius of circles represents the number of operations. Our solution is shown as the best solution for real-time HDR merging with a high-fidelity score.

## 6 Conclusions

In this paper, we propose CEN-HDR, a novel computationally efficient HDR merging network able to correct the ghost effect caused by large object motions in the scene and camera motion. The proposed lightweight network architecture effectively succeeds in generating real-time HDR images with a dynamic range close to that of the original scene. By integrating the knowledge distillation methods in our training scheme, we demonstrate that the majority of the representation capabilities of a large HDR merging network can be transferred into a lighter network, opening the door to real-time HDR embedded systems.

\begin{table}
\begin{tabular}{l|r r} \hline \hline Method & Num. of params. & Num. of op. (GMAccs) \\ \hline GSANet[9] & 80650 & 199.39 \\ Yan et al.[33] & 18899000 & 156.12 \\ Yu et al.[36] & 1013250 & 199.88 \\ CEN-HDR(our) & 282883 & 128.78 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Inference cost comparison of the proposed CEN-HDR solution versus recent efficient merging networks. The number of operations and parameters for [9, 33, 36] and our solution are computed following the method described in [20].The input size is set to \(1900\times 1060\) pixels.

\begin{table}
\begin{tabular}{l|r r|r r r} \hline \hline Method & \multicolumn{2}{c|}{Attention type} & params. & GMAccs & Runtime(s) \\ \cline{2-5}  & Spatial & Channel & & & \\ \hline AHDRNet attention [30] & ✓ & & 55392 & 20.772 & 0.0085 \\ EPSANet[38] & ✓ & & 42560 & 15.768 & 0.0111 \\ SK attention [10] & & ✓ & 125984 & 43.104 & 0.0155 \\ Double attention[1] & ✓ & ✓ & 33216 & 12.456 & 0.0101 \\ CBAM[27] & ✓ & ✓ & 22689 & 7.525 & 0.0734 \\ BAM[19] & ✓ & ✓ & 17348 & 5.008 & 0.0060 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Inference cost comparison of attention modules. Spatial and Channel attention modules are studied by feeding a tensor of size \((1,\frac{H}{4},\frac{W}{4})\) corresponding to the concatenation of the reference and non-reference tensors after the encoding step.
\begin{table}
\begin{tabular}{l|c c c c} \hline \hline Method & \(\mu\)-PSNR & PSNR & \(\mu\)-SSIM & SSIM \\ \hline Without attention module & 42.12 & 39.95 & 0.9850 & 0.9823 \\ AHDRNet[30] attention module & 42.94 & 40.49 & 0.9903 & 0.9852 \\ SCRAM-C & 42.32 & 40.14 & 0.9854 & 0.9829 \\ SCRAM-S & 42.89 & 40.41 & 0.9884 & 0.9835 \\ SCRAM & 43.05 & 40.53 & 0.9908 & 0.9856 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Effect of the attention module on the performance of proposed HDR deghosting network. SCRAM-C and SCRAM-S respectively correspond to the SCRAM module composed only of the channel attention branch and the spatial attention branch.

Figure 6: Qualitative results of the proposed CEN-HDR solution on samples from the NTIRE[20] challenge dataset. The ground truth images are not provided. We notice that the high sensor noise present in the input images is not fully corrected in the dark areas of the output HDR images. Moreover, the motion blur introduced in the dataset produces less sharp characteristics in the output image.