# End-to-end Rain Streak Removal with RAW Images

First Author GuoDong DU
Institution1: National University of Singapore
Institution1 address: 21 Lower Kent Ridge Rd, 119077
<EMAIL_ADDRESS>Second Author HaoJian Deng
Institution2 Harbin Institute of Technology, Shenzhen
First line of institution2 address: Harbin Institute of Technology, Shenzhen
Third author JiaHao Su
Institution3: Harbin Institute of Technology, Shenzhen
First line of institution3 address: Harbin Institute of Technology, Shenzhen
Fourth author Yuan Huang
Institution4: Waseda University
First line of institution4: address West Waseda 1-6-1, Shinjuku ku, Tokyo

###### Abstract

In this work we address the problem of rain streak removal with RAW images.
The general approach is firstly processing RAW data into RGB images and
removing rain streak with RGB images. Actually the original information of
rain in RAW images is affected by image signal processing (ISP) pipelines
including none-linear algorithms, unexpected noise, artifacts and so on. It
gains more benefit to directly remove rain in RAW data before being processed
into RGB format. To solve this problem, we propose a joint solution for rain
removal and RAW processing to obtain clean color images from rainy RAW image.

[Method] To be specific, we generate rainy RAW data by converting color rain
streak into RAW space and design simple but efficient RAW processing
algorithms to synthesize both rainy and clean color images. The rainy color
images are used as reference to help color corrections.

[Result] Different backbones show that our method conduct a better result
compared with several other state-of-the-art deraining methods focused on
color image. In addition, the proposed network generalizes well to other
cameras beyond our selected RAW dataset. Finally, we give the result tested on
images processed by different ISP pipelines to show the generalization
performance of our model is better compared with methods on color images.

##  1 Introduction

As a fundamental problem in computer vision, rain removal aims to reconstruct
the visibility of objects and scenes in photos captured under rainy days.
Since the atmospheric process is complicated, the visibility degradation
caused by rain have different types in terms of raindrop size, rain density,
and wind velocity [18]. For example, nearby rain streak can distort background
scene content; adherent raindrops that fall and flow on camera lenses can
obstruct or blur the scenes; distant rain accumulated throughout the scene,
creating a mist-like phenomenon in a manner more similarly to fog. Rain
removal has thus become a necessary preprocessing step for subsequent outdoor
tasks, like visual tracking [24], person reidentification [17], or event
detection [22].

In the last decade, we have witnessed a continuous progress on rain removal
research with many methods for both video and images. While conventional
methods utilize the prior knowledge of background scenes and encoding physical
properties, most recent methods employ the data-driven manner and design
specific convolution neural network architectures. Some works focus on better
visual quality using the generative adversarial network[30], others focus on
rain removal on the real world rain/rain-free image pairs besides on
synthesized dataset[26]. All existing methods remove rain for images in RGB
format which are processed from original RAW images. However there is
significant information loss when RAW data is processed into RGB format
through image signal processing (ISP) pipeline. No existing methods explore
whether the original information in RAW image is helpful for rain removal
tasks. Not only the rainy image suffer from degradation of scene visibility
outside the camera, but also it is affected by ISP pipeline within cameras.
The main affection of ISP in summarized as below. Firstly, the non-linear
operation give fluctuated digit gains to the RAW pixels within a single image.
For example, lens correction gives a bigger gain to the pixels away from the
image center[2]; Gamma correction makes the dark regions lighter[20]; Tone
mapping gives more contrast to the moderate pixels[21]. These non-linear
operations process the RAW data to match human vision but are dishonest to the
original rain streak distribution in linear RAW data space. In addition,
camera’s exposure algorithm usually gives the output color images a biased
digit gain without considering image intensities. Secondly, the demosaicing
process within ISP pipeline may produce construction noise, blurry and ringing
artifacts[19]. These unexpected image quality degradation caused by digital
image processing may also deteriorate the feature learning ability for
deraining problems. Finally, the pixels in RAW data are typically 12 or 14
bits, but are reduced to 8 bits color pixels. The information loss brings more
difficulties especially for high frequency learning tasks. Hence it is
conspicuous that the ISP pipelines surely affect rain removal tasks and no
existing methods remove rain directly in RAW images. Furthermore, because
different camera factories have different RAW processing pipelines, the final
output RGB images exhibit different ISP characteristics. Thus the learned
deraining model from a kind of ISP images may generalize poorly on color
images processed by another factory. A deep learning model always needs a new
dataset for different camera factories. While the domain distance of different
RAW images is shorter than color images, the rain removal architectures
proposed for color images are not general enough to model the difference
between ISP pipelines.

The RAW data from cameras is accessible easily, and significant progress has
been made for image restoration and enhancement by exploiting the RAW data in
some low-level vision tasks. These tasks are devoted to solve the degradation
reconstruction problems such as denoising [1], super resolution [27] and
illumination enhancement [4]. In this work, we attempt to exploit RAW data
information to further boost rain removal performance based on deep network
architectures which are prominent for strong representation capability.
Specially, we propose a joint solution for rain removal and RAW processing by
designing a network architecture. What is more, to obtain paired RAW rainy
images and clean RGB images, we design algorithms to model rain streak in RAW
domain and synthesize RAW rainy images.

Figure 1: RAW rainy image pipeline.

To synthesize rainy images in RAW data and reserve realistic details, we
firstly select a high quality RAW image dataset as our clean or background RAW
images. Since we mainly focus on the rain removal task in RAW images, we model
the rain image considering only the type of rain streak. We render RAW rain
streaks which are then added to the clean RAW images to synthesize a rainy RAW
image. RAW rain streaks are modeled by converting rain streaks in color space,
using the inverse ISP technology, which is proposed by [1]. Then we design a
simple but efficient ISP pipeline processing both the RAW clean and rainy data
to obtain corresponding images in RGB format separately. Our pipeline for
rainy RAW image synthesis is shown in figure 1. Besides, we choose another two
RAW image datasets and implement several different ISP pipelines to establish
a benchmark for generalization ability test. For the Joint Rain Removal and
Raw Processing network (JoRRRP) architecture, we first pack RAW rainy data
according to its bayer pattern and then extract a RAW detail layer obtained by
using low-pass filtering[25, 9]. The layer is then fed into deep residual
network (ResNet) [13] followed by an upsampling layer using pixel shuffle
method[23]. Hence we can obtain a demosaiced color rain mask. Then we apply a
spatially-variant color transformation estimated from color rainy image to
help recover the fidelity of color appearances. Thus we obtain the rain mask
in RGB format which is the negative residual mapping as proposed in [9]. After
combining the rain mask in RGB format with processed rainy RGB image, we can
get the final clean RGB image. Our overall framework for end-to-end rain
streak removal in RAW images is shown is Figure 2.

We conduct experiments on our RAW synthetic rain data to demonstrate the
importance of each component of our model. In addition, we replace ResNet with
other efficient backbones[32, 31, 14] for further ablation study. The result
shows that the performance of rain streak removal with RAW images is always
better than RGB methods, with average 0.7 dB and 0.05 improvement, according
to the evaluation metric of peak signal-to-noise ratio (PSNR) and structural
similarity (SSIM) index separately. Finally, we give the result tested on
images processed by different ISP pipelines to show the generalization
performance of our model compared with methods on color images. Our
contributions are summarized as follows:

  1. 1.

We present a technique to synthesize rainy image on RAW data, especially a
pipeline of how to generate rain streak on RAW data. To the best of our
knowledge, this is the first dataset that is aimed to use RAW data for
deraining problems.

  2. 2.

We propose a joint rain removal and RAW processing deep learning model.
Experiments show that using RAW data achieves better accuracy and visual
improvements against state-of-the-art methods on only RGB space images.

  3. 3.

Extensive experiments are conducted on images from different ISP pipelines and
results show that our me-thod gives a better generalization performance.

Figure 2: Joint Rain Removal and RAW Processing.

##  2 Related Work

Our work is related to RAW image processing and rain removal methods, which
are reviewed briefly as follows.

A. RAW image processing

Many joint solutions for image restoration tasks using deep neural network
have been proposed by exploiting RAW data. [12] proposed an efficient CNN for
effective joint denoising and demosaicking. [19] address demosaicing and
super-resolution simultaneously and claim that super-resolution with RAW data
helps recover fine details and clear structure. [1] present procedures for
unprocessing color images into synthetic RAW data and the purpose is to obtain
more realistic noisy and clean pairs. The problem of lack of realistic
training data also exists in super resolution. [27] develop a dual
convolutional neural network to exploit the originally captured radiance
information in RAW images and they simulate the imaging process of digital
cameras to remedy the information loss of the input. Their proposed method is
declared to enable super-resolution for real captured images. [4] introduce a
dataset of RAW short-exposure low-light images, with corresponding long-
exposure reference images and develop a network for processing low-light
images.
There exists a few datasets with different imaging scenarios that can provide
metadata of camera sensors for rendering RAW rain streak. can be used as clean
background RAW data. The HDR+ dataset is featured with burst denoising and
sophisticated style retouching [21]; [7] proposed an unique database of HDR
photographs accompanied by detailed luminance measurements and visual
appearance scaling from the original scene; and the FiveK dataset contains
high quality images that are retouched by five photographers to have different
color styles[3].

B. Rain removal methods

When a rainy image is taken, the visual effects of rain on that digital image
further hinges on many camera parameters, such as exposure time, depth of
field, and resolution[11]. Garg and Nayar proposed a camera based rain gauge
and improve the performance of deraining by selecting camera parameters
without appreciably altering the scene appearance. Besides, the author claimed
that the visibility of rain is affected by both the brightness of scene and
camera parameters such as exposure time and depth of field.
While multi-frame based deraining method [10, 15] take the advantages of
sequential information, the prior based algorithms for single image deraining
are more available. Chen and Hsu [5] decomposed the background and rain streak
layers based on low-rank priors. Li et al. [16] use patch-based priors for
both the clean background and rain layers in the form of Gaussian mixture
models. These approaches tend to have unsatisfactory performances on real
images with complicated scenes and rain forms. Numerous deep learning
techniques for image deraining algorithms have been recently proposed. Yang et
al. attempt to jointly detect and remove rains taking a contextualized dilated
network[28]. Zhang et al. apply a residual-aware classifier to efficiently
determine the density-level of a given rainy image for joint rain density
estimation and deraining[29]. Wang et al. design a novel SPANet in a local-to-
global attentive manner[26]. None of the existing methods exploit the RAW data
for rain removal tasks.

##  3 Method

###  3.1 RAW Image Pipeline

Most existing deraining datasets are synthesized by assuming a specific rain
model, such as rain streak, raindrop, as well as rain and mist. The most
common model assumed by the majority of deraining algorithms is a rain streak
image $R_{s}$, which is modeled as a linear super- imposition of the clean
background scene $B$ and the sparse, line-shape rain streak component $S$:
$R_{s}=B+S$ Since our research mainly concern how the RAW data information
could help for deraining problems, we only use this certain type of synthetic
images. Then we explore the technique to ”unprocess” rain streak to its
corresponding RAW format by inverting each component in RAW processing
pipeline. The original RAW data will be used as background of our rainy RAW
image, and we generate rain streak according to the clean RAW data.

Datasets such as MiT-Adobe Fivek [3] record the camera metadata of their
images which is very import when converting RAW data to RGB space by image
signal processing (ISP). The metadata usually comprises important information
such as black level, saturation , Bayer pattern, color matrix, flip and so on.
Given the metadata of each RAW data image, we can process our rain streak to
its corresponding RAW rain streak. See Figure 1 for an overview of our steps
of rain streak generation on RAW data.

A traditional ISP pipeline usually contains denoising, lens correction,
exposure white balance, color correction, tone mapping, gamma correction and
so on. We design a simple but efficient ISP pipeline to process RAW image,
then we inverse each step of this pipeline for rendering RAW rain streak.

RGB clean.

exposure

RAW rainy

demosaic

tone mapping

Figure 3: Different ISP pipelines

What is more, we propose several different RAW processing algorithms
simulating different styles of ISP to obtain various color images for testing
of generalization performance. See Figure 3 for comparison of different ISP
processing results.

Noise and Lens Correction:

Our selected RAW sensor data actually have its real noise, primarily coming
from two sources: photo arrival statistics (“shot” noise) and imprecision in
the readout circuitry (“read” noise). Hence we just assume the generated rainy
RAW data keep the same noise as its original clean background, it is trivial
to consider how to imitate RAW noise in rain streak. The lens profile
correction adjustments is to remove typical optical issues that camera lenses
create, such as vignetting or distortion[2]. We assume a simple radial
distortion lens correction function:

$RAW_{corrected}=RAW\cdot\left(\alpha r^{2}+\beta\right)$

where $r$ is the distance from image center to each pixel. $\alpha,\beta$ are
fluctuated constant for different color channels. For RAW rain streak
generation, we inverse this step through division by gain map.

Demosaicing:

In the beginning of digital image processing pipeline we need to reconstruct a
full color image from the incomplete color samples output from an image sensor
overlaid with a color filter array (CFA), such as R-G-G-B. This can be
achieved by interpolation or more complicated demosaicing algorithms [15]. To
invert this step we omit two of its three color values according to the Bayer
filter pattern for each pixel in the our rain streak images. In our forward
image signal processing (ISP) of rendering RGB images from both clean and
rainy RAW data, we adopt a demosaic algorithm with linear interpolation for
each color pixel to perform color construction. For test dataset, we adopt
another advance demosaicking method using finite impulse response filter in
frequency domain to extract necessary components.

Exposure:

As a extremely simple brightening measure, we find the mean luminance of the
image and then scale it so that the mean luminance is some more reasonable
value. We fairly arbitrarily scale the image so that the mean luminance is
about 1/4 the maximum, with a normal distribution centered at 0.25 with
standard deviation of 0.05, and then clamped into range
$\left[0.045,0.75\right]$ for a safe content reserving. For the ISP pipelines
designed for test benchmark, we choose a larger range of mean luminance
values.

White Balance:

It is very important to estimate illumination chromaticity correctly to avoid
invalid overall color cast in the final image. The goal of white balancing is
to estimate accurately the color of the overall scene illumination and to make
the image look as if is taken under canonical light. When synthesizing rain
streak on RAW data, we take the product of the inverse digital and white
balance gains to get an inverse gain for per-channel, then we apply this gain
to our rain streak. This inverse gain is almost always less than unity, which
means that naively gaining down our synthetic imagery will result in a dataset
that systematically lacks highlights and contains almost no clipped pixels.
This is problematic, as correctly handling saturated image intensities is
critical when generating rain streak on RAW data. This problem is the same in
[1] when the author investigates the pipeline of generation of realistic
synthetic RAW data for denoising. The author then apply a highlight-preserving
transformation $f(x,g)$ to perform safe white balance inverse.

$\displaystyle a(x)$
$\displaystyle=\left(\frac{\max\left(x-t,0\right)}{1-t}\right)^{2}$ (1)
$\displaystyle f(x,g)$
$\displaystyle=\max\left(\frac{x}{g},\left(1-\alpha\left(x\right)\right)\left(\frac{x}{g}\right)+\alpha\left(x\right)x\right)$
(2)

We apply this function for our rain streak generation problem.

Color Correction:

In general, the color space in a camera space is different from expected sRGB
color space because the color filters of a camera sensor vary from different
factories. Then the camera RAW data applies a 3 x 3 color correction matrix
(CCM) to address this problem. Thus we apply the inverse of a RAW data CCM to
transmit a rain streak from sRGB space to the original “camera space”.

Gamma Compression:

Gamma Compression is is a nonlinear operation used to make dark regions
lighter by allocating more bits of dynamic range to low intensity pixels,
because humans are more sensitive to relative differences between darker tones
than between lighter tones.

$\displaystyle\Gamma\left(x\right)$
$\displaystyle=\max(x,\epsilon)^{\frac{1}{2.4}}$ (3)
$\displaystyle\Gamma^{-1}\left(y\right)$
$\displaystyle=\max(y,\epsilon)^{2.4}$ (4)

Tone Mapping:

Tone mapping addresses the problem of strong contrast reduction from the scene
radiance to the displayable range while preserving the image details and color
appearance important to appreciate the original scene content. It is
conspicuous that RAW data are always high dynamic range images and processing
it to color images prefer a well designed tone mapping algorithm[21]. The
reverse-engineering for a complex edge-aware local tone mapping is difficult
and trivial. We choose a simple tone mapping curve the same as equation 5 and
inverse the curve function for rain streak unprocessing.

$\displaystyle simple(x)$ $\displaystyle=3x^{2}-2x^{3}$ (5) $\displaystyle
simple^{-1}(y)$
$\displaystyle=\frac{1}{2}-\sin\left(\frac{\sin^{-1}\left(1-2y\right)}{3}\right)$
(6)

both are only defined on inputs in [0, 1].
Figure 4: tone mapping $\displaystyle Reinhard\left(x\right)$
$\displaystyle=\frac{x}{1+x}$ (7) $\displaystyle CE\left(x\right)$
$\displaystyle=1-e^{-x}$ (8) $\displaystyle ACES\left(x\right)$
$\displaystyle=\frac{x(2.51x+0.03))}{x(2.43x+0.59)+0.14}$ (9)

For the ISP pipelines designed for test benchmark, we choose another 3 common
tone mapping method: Reinhard tone mapping, Cry Engine 2 tone mapping(CE), the
Academy Color Encoding System (ACES) tone mapping, [21, 8, 6]. See Figure 3.1
for the comparison of different tone mapping curves.

###  3.2 Joint Rain Removal and RAW Processing

Figure 3.1 illustrates the proposed architecture and comprises two modules:
Joint Rain Removal and RAW Processing (JoRRRP) module to learn a mapping
function from RAW details to color details; Color estimation for predicting a
local transformation matrix.

Figure 5: Jodd Table 1: Ablation study for our joint RAW processing and deraining network Methods | MIT-Adobe | HDR+ | Fairchild
---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
only rgb input |  28.95 |  0.9174 |  28.41 |  0.9320 |  28.60 |  0.9373
w/o color branch |  27.28 |  0.8858 |  26.42 |  0.8798 |  26.96 |  0.9056
w/o RAW input, w/o pack |  28.73 |  0.9182 |  28.67 |  0.9333 |  28.70 |  0.9366
w/o RAW input, pack+DC |  29.07 |  0.9225 |  29.03 |  0.9377 |  29.08 |  0.9413
w/o RAW input, DC+pack |  29.04 |  0.9221 |  28.96 |  0.9374 |  28.94 |  0.9408
w/o feature fusion |  29.51 |  0.9272 |  29.28 |  0.9422 |  29.20 |  0.9441
Ours full model |  29.53 |  0.9278 |  29.33 |  0.9431 |  29.25 |  0.9482
Table 2: comparison of backbones for joint rain removal and RAW processing. Backbones | MIT-Adobe | HDR+ | Fairchild
---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
ResN |  only rgb input |  28.95 |  0.9174 |  28.41 |  0.9322 |  28.59 |  0.9373
|  w/o RAW input |  29.07 |  0.9225 |  29.02 |  0.9377 |  29.07 |  0.9413
|  Ours full model |  29.52 |  0.9278 |  29.32 |  0.9431 |  29.25 |  0.9482
RDN |  only rgb input |  29.00 |  0.9229 |  28.50 |  0.9355 |  28.67 |  0.9407
|  w/o RAW input |  29.36 |  0.9261 |  29.00 |  0.9396 |  28.98 |  0.9424
|  Ours full model |  29.94 |  0.9332 |  29.41 |  0.9474 |  29.49 |  0.9520
RCAN |  only rgb input |  29.34 |  0.9263 |  29.24 |  0.9422 |  29.28 |  0.9466
|  w/o RAW input |  29.46 |  0.9270 |  29.34 |  0.9433 |  29.21 |  0.9460
|  Ours full model |  30.80 |  0.9339 |  29.62 |  0.9492 |  29.54 |  0.9532
IMDN |  only rgb input |  28.48 |  0.9205 |  28.91 |  0.9377 |  28.79 |  0.9390
|  w/o RAW input |  29.16 |  0.9239 |  29.03 |  0.9402 |  28.91 |  0.9420
|  Ours full model |  29.74 |  0.9303 |  29.48 |  0.9464 |  29.48 |  0.9508
Table 3: Experiments on different ISP pipelines. ISP pipelines | MIT-Adobe | HDR+ | Fairchild
---|---|---|---
RGB | RAW | RGB | RAW | RGB | RAW
ResN |  simple |  28.602 |  +0.511 |  28.415 |  +0.787 |  28.595 |  +0.607
|  demosaic |  26.243 |  +0.642 |  25.939 |  +0.912 |  26.224 |  +0.819
|  exposure |  27.864 |  +0.635 |  27.511 |  +0.993 |  27.854 |  +0.944
|  tone mapping |  30.221 |  +0.793 |  29.920 |  +1.341 |  30.533 |  +1.171
RDN |  simple |  28.784 |  +0.542 |  28.500 |  +0.919 |  28.675 |  +0.819
|  demosaic |  26.081 |  +0.708 |  25.831 |  +1.194 |  26.140 |  +1.075
|  exposure |  28.042 |  +0.718 |  27.593 |  +1.229 |  27.956 |  +1.203
|  tone mapping |  30.374 |  +0.923 |  30.008 |  +1.546 |  30.597 |  +1.416
Table 4: comparison with other rain removal methods. Methods | MIT-Adobe | HDR+ | Fairchild
---|---|---|---
|  PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
DDN |  28.95 |  0.9174 |  28.42 |  0.9322 |  28.60 |  0.9373
JORDER |  28.33 |  0.9088 |  29.24 |  0.9327 |  28.85 |  0.9340
Ours model |  29.52 |  0.9278 |  29.33 |  0.9431 |  29.25 |  0.9482
Ours model(RCAN) |  30.80 |  0.9339 |  29.62 |  0.9492 |  29.55 |  0.9532

While there are methods directly learning a mapping function from RAW images
to color images with neural networks, there is ground truth color images
corresponding to RAW input since different ISP pipelines provide various RAW
image processing algorithms. Hence, we need the color rainy image as
complementary of RAW data limitation. The original rainy color image input can
provide a reference to help recover the fidelity of color appearances.

Firstly, we pack the RAW image into 4 color channel and then adopt
decomposition method[9] to obtain RAW details. The RAW details then are fed
into a deep CNN module for the purpose of joint negative rain streak feature
learning and converting RAW data to RGB space. The module is followed by an
upsampling layer with the pixel shuffle method[23], as the demosaicking
algorithm is approached via super resolution. We also application
decomposition method to extract color details.

It is difficult to recover the relevant color corrections conducted within the
ISP system, and thereby the networks trained with it could only be used for
one specific camera. Similar with [27], we adopt a shallow U-Net structure to
predict color correction matrix with 12 channels output. This local color
estimation contain a $3\times 3$ per pixel channel color matrix and $1\times
3$ offset color information. It is beneficial to combine the color details
with the feature output from JoRRRP module, this feature fusion make the color
estimation module aware of the information from RAW details.

###  3.3 Backbone

A straight forward backbone for the JoRRRP module is ResNet, but we need more
backbones for ablation study of whether the RAW data information really help
the rain removal task. We deploy three more compact and efficient backbones:
residual dense network (RDN)[32], residual channel attention network
(RCAN)[31], light weight information multi-distillation network (IMDN)[14].

The RDN backbone exploit the hierarchical features by local feature fusion in
RDB [32]. After fully obtaining dense local features, the author then use
global feature fusion to jointly and adaptively learn global hierarchical
features. The RCAN backbone allows abundant low-frequency information to be
bypassed through multiple skip connections by using residual in residual (RIR)
structure. Furthermore, the network adaptively rescale features and
considering inter dependencies among feature channels by channel attention
mechanism. This IMDN network construct the cascaded information multi-
distillation blocks (IMDB), which contains distillation and selective fusion
parts. In addition, the network propose contrast-aware channel attention
mechanism according to the importance of candidate features.

###  3.4 Implementation Details

We obtain 2000 clean RAW images from MIT-Adobe 5K dataset [3] to generate RAW
and RGB training image pairs. The original dataset is composed of high
resolution RAW photos photographed from different kinds of light source. We
carefully remove photos that is indoor, night or abstract patterns, which is
not likely to be a rainy scene. We randomly selected 1800 images as training,
100 images as validation and 100 images as test. For each RAW image , we
generate 4 RAW rain streak mask with different angles. The RAW data is resized
to quarter of its original resolution to save computation. In addition, we
separately choose 100 RAW images from HDR plus dataset and Fairchild dataset
for testing the generalization ability. RAW images for testing is processed
with different ISP algorithms containing different demosaicing methods,
exposure scale and tone mapping methods.

##  4 Experiments and Results

###  4.1 Results on Two RAW Synthetic Rain Dataset

We first conduct experiments on our RAW synthetic rain data in order to
clearly demonstrate the importance of each component of our model. The result
show that the performance is worsen without color branch, RAW input or feature
fusion.

###  4.2 Ablation study of backbones

To evaluate the stability of our JoRRRP, we replace the ResN backbone with
different backbones: RDN, RCAN and IMDN. The result shows that the performance
of rain streak removal with RAW images is always better than RGB methods, with
average 0.7 PSNR and 0.05 SSIM improvement.

###  4.3 ISP settings

We also tested the performance of our solution with other RGB de-raining
piplines and the results show that the model using RAW data have a better
performance than models trained on RGB space images especially when the
testing data is processed with different ISP pipelines.

###  4.4 Comparison With The SOTA Methods

We show comparisons about model size and performance in Figure 8. Although our
RCAN is the deepest network, it has less parameter number than that of EDSR
and RDN. Our RCAN and RCAN+ achieve higher performance, having a better
tradeoff between model size and performance. It also indicates that deeper
networks may be easier to achieve better performance than wider networks.

##  5 Conclusions

In this work, we propose a joint solution of rain streak removal and RAW
processing. The results show that using RAW data achieves better accuracy and
visual improvements against state-of-the-art methods on only RGB space images.
In addition, we design We present a technique to synthesize rainy image on RAW
data, especially pipeline how to generate rain streak on RAW data. What is
more, our method achieve better generalization performance than methods on
color image when testing on different ISP pipelines.

## References

  * [1] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron.  Unprocessing images for learned raw denoising.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11036–11045, 2019.
  * [2] Faisal Bukhari and Matthew N Dailey.  Automatic radial distortion estimation from a single image.  Journal of mathematical imaging and vision, 45(1):31–45, 2013.
  * [3] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand.  Learning photographic global tonal adjustment with a database of input / output image pairs.  In The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition, 2011.
  * [4] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to see in the dark.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3291–3300, 2018.
  * [5] Yi-Lei Chen and Chiou-Ting Hsu.  A generalized low-rank appearance model for spatio-temporally correlated rain streaks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1968–1975, 2013.
  * [6] S Cooper, T Mansencal, and K Wheatley.  Aces retrospective and enhancements, 2017.
  * [7] Mark D Fairchild.  The hdr photographic survey.  In Color and imaging conference, volume 2007, pages 233–238. Society for Imaging Science and Technology, 2007.
  * [8] Karl-Ingo Friese, Marc Herrlich, and Franz-Erich Wolter.  Using game engines for visualization in scientific applications.  In Entertainment Computing Symposium, pages 11–22. Springer, 2008.
  * [9] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley.  Removing rain from single images via a deep detail network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3855–3863, 2017.
  * [10] Kshitiz Garg and Shree K Nayar.  Detection and removal of rain from videos.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pages I–I. IEEE, 2004.
  * [11] Kshitiz Garg and Shree K Nayar.  When does a camera see rain?  In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, volume 2, pages 1067–1074. IEEE, 2005.
  * [12] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand.  Deep joint demosaicking and denoising.  ACM Transactions on Graphics (TOG), 35(6):1–12, 2016.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang.  Lightweight image super-resolution with information multi-distillation network.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 2024–2032, 2019.
  * [15] Jin-Hwan Kim, Jae-Young Sim, and Chang-Su Kim.  Video deraining and desnowing using temporal correlation and low-rank matrix completion.  IEEE Transactions on Image Processing, 24(9):2658–2670, 2015.
  * [16] Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown.  Rain streak removal using layer priors.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2736–2744, 2016.
  * [17] Jiayuan Mao, Tete Xiao, Yuning Jiang, and Zhimin Cao.  What can help pedestrian detection?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3127–3136, 2017.
  * [18] Sudipta Mukhopadhyay and Abhishek Kumar Tripathi.  Combating bad weather part i: Rain removal from video.  Synthesis Lectures on Image, Video, and Multimedia Processing, 7(2):1–93, 2014.
  * [19] Guocheng Qian, Jinjin Gu, Jimmy S Ren, Chao Dong, Furong Zhao, and Juan Lin.  Trinity of pixel enhancement: a joint solution for demosaicking, denoising and super-resolution.  arXiv preprint arXiv:1905.02538, 2019.
  * [20] Shanto Rahman, Md Mostafijur Rahman, Mohammad Abdullah-Al-Wadud, Golam Dastegir Al-Quaderi, and Mohammad Shoyaib.  An adaptive gamma correction for image enhancement.  EURASIP Journal on Image and Video Processing, 2016(1):1–13, 2016.
  * [21] Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda.  Photographic tone reproduction for digital images.  In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 267–276, 2002.
  * [22] Mohamed S Shehata, Jun Cai, Wael Maged Badawy, Tyson W Burr, Muzamil S Pervez, Robert J Johannesson, and Ahmad Radmanesh.  Video-based automatic incident detection for smart roads: The outdoor environmental challenges regarding false alarms.  IEEE Transactions on Intelligent Transportation Systems, 9(2):349–360, 2008.
  * [23] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874–1883, 2016.
  * [24] Yibing Song, Chao Ma, Xiaohe Wu, Lijun Gong, Linchao Bao, Wangmeng Zuo, Chunhua Shen, Rynson WH Lau, and Ming-Hsuan Yang.  Vital: Visual tracking via adversarial learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8990–8999, 2018.
  * [25] Carlo Tomasi and Roberto Manduchi.  Bilateral filtering for gray and color images.  In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271), pages 839–846. IEEE, 1998.
  * [26] Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau.  Spatial attentive single-image deraining with a high quality real rain dataset.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12270–12279, 2019.
  * [27] Xiangyu Xu, Yongrui Ma, and Wenxiu Sun.  Towards real scene super-resolution with raw images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1723–1731, 2019.
  * [28] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan.  Deep joint rain detection and removal from a single image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1357–1366, 2017.
  * [29] He Zhang and Vishal M Patel.  Density-aware single image de-raining using a multi-stream dense network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 695–704, 2018.
  * [30] He Zhang, Vishwanath Sindagi, and Vishal M Patel.  Image de-raining using a conditional generative adversarial network.  IEEE transactions on circuits and systems for video technology, 2019.
  * [31] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 286–301, 2018.
  * [32] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472–2481, 2018.