A Learnable Color Correction Matrix for RAW Reconstruction
Anqi Liu Shiyi Mu Shugong Xu


Geometry of temporal chiral structures 
    Andres F. Ordonez^1,2, Aycke Roos ^3, Pablo M. Maier ^3, David Ayuso^1,2,3 and Olga Smirnova^3,4
    September 9, 2024
====================================================================================================



§ ABSTRACT 
Autonomous driving algorithms usually employ sRGB images as model input due to their compatibility with the human visual system. However, visually pleasing sRGB images are possibly sub-optimal for downstream tasks when compared to RAW images. The availability of RAW images is constrained by the difficulties in collecting real-world driving data and the associated challenges of annotation. To address this limitation and support research in RAW-domain driving perception, we design a novel and ultra-lightweight RAW reconstruction method. The proposed model introduces a learnable color correction matrix (CCM), which uses only a single convolutional layer to approximate the complex inverse image signal processor (ISP). Experimental results demonstrate that simulated RAW (simRAW) images generated by our method provide performance improvements equivalent to those produced by more complex inverse ISP methods when pretraining RAW-domain object detectors, which highlights the effectiveness and practicality of our approach.






§ INTRODUCTION


The construction of datasets is crucial for autonomous driving algorithms, as a large volume of data is needed to train models for recognizing and understanding road environments. Existing widely used real-world autonomous driving datasets, such as <cit.>, only provide sRGB images that are derived from RAW images through a camera image signal processor (ISP). Unlike sRGB images, RAW images offer physically meaningful and interpretable information due to their linear relationship between image intensity and the radiant energy incident on the camera sensor. This characteristic of RAW images has been effectively utilized in low-level tasks such as denoising <cit.>, deblurring <cit.>, HDR <cit.>, and super-resolution <cit.>. Additionally, RAW images have demonstrated advantages in high-level tasks such as object detection <cit.> and segmentation <cit.>.

Due to the suitability with the human visual system and the abundance and diversity of sRGB images, most autonomous driving algorithms use sRGB images as input. However, sRGB images may not always be the most suitable data type for computer learning compared to RAW images. As demonstrated by Tesla <cit.>, using 12-bit RAW photon count images as input to the network bypasses the complex process of manually tuning the ISP and improves algorithm processing speed. Existing research on RAW-domain object detection <cit.> shows that using RAW images as model input can lead to better performance.

However, the availability of RAW images is limited due to the constraints of data collection and the high cost of RAW image transmission, storage, and processing. Additionally, annotating RAW images is expensive and burdensome. As a result, only a few RAW datasets <cit.> of real-world driving scenarios are available. One solution to expand the availability of RAW images is to reconstruct them from existing sRGB images using inverse ISP algorithms. Traditional methods <cit.> gradually convert sRGB images to RAW images by simulating ISP system modules. Metadata-based methods <cit.> encode necessary metadata from RAW images into the inverse ISP model to achieve high-accuracy reconstruction. Learning-based methods <cit.> aim to approximate the mapping from sRGB images to RAW images using neural networks in an end-to-end pipeline. Most of these aforementioned methods <cit.> require paired sRGB and RAW images for training, which limits their application to converting existing sRGB datasets captured by unknown cameras into RAW format. To address this limitation, <cit.> propose an unpaired RAW reconstruction pipeline based on CycleGAN <cit.>, which can convert sRGB images to RAW images for any camera style, making it ideal for performing RAW reconstruction in the absence of sRGB and RAW image pairs.

In practical industrial applications, enterprises have accumulated extensive sRGB images and high-quality labels through long-term real-world road tests and simulation trials, which are used for iterative optimization of autonomous driving algorithms. These sRGB images are invaluable for RAW-domain driving perception tasks, as they can be converted into RAW format using inverse ISP algorithms. Additionally, the labels associated with these images can be directly reused for subsequent tasks, eliminating the need for costly relabeling. However, converting large volumes of sRGB images into RAW format is time-consuming and computationally expensive. To advance RAW-domain research, there is a pressing need for a time-efficient RAW reconstruction method with high reconstruction quality. To address this challenge, we propose a novel learning-based method for RAW reconstruction. Our proposed model is exceptionally simple and lightweight. Specifically, the complex inverse ISP system is replaced with a simple and learnable color correction matrix, implemented using just a single convolutional layer. Our contributions are summarized as follows:

  
  * We propose a novel inverse ISP method for RAW reconstruction. Our proposed model consists of only one convolution layer to learn the mapping from sRGB images to RAW images.
  
  * In the absence of real-world driving scene RAW datasets, our proposed model can rapidly expand the quantity of available RAW images, making it highly practical for advancing research in RAW-domain autonomous driving algorithms.
  
  * Experimental results show that when used to pretrain a RAW-domain object detector, the simulated RAW (simRAW) images generated by our learnable color correction matrix achieve equivalent performance gains comparable to those generated by more complex inverse ISP methods. This demonstrates the feasibility and effectiveness of our proposed approach.




§ RELATED WORKS




RAW datasets of driving scenarios. To our knowledge, only a few RAW datasets are available as shown in Table <ref>. The PASCALRAW dataset <cit.> is collected using a Nikon D3200 DSLR camera and contains 4259 high-resolution 12-bit RAW images of daytime scenes. The ROD dataset <cit.> is collected using the Sony IMX490 sensor and contains 25k high-resolution 24-bit RAW images of both daytime and nighttime scenarios. The multiRAW dataset <cit.> is collected using five different cameras and contains 7469 high-bit RAW images from various times and locations (rural, tunnel, and urban areas), with corresponding sRGB images converted using the respective camera ISP are provided.

RAW Reconstruction. Traditional inverse ISP methods <cit.> typically involve non-learnable operations based on camera parameters, such as inverse tone mapping, inverse gamma transformation, inverse color correction, inverse white balance and mosaic. This paradigm requires manually setting the true internal camera parameters which are often proprietary and not publicly available. Additionally, they do not account for complex nonlinear ISP operations, leading to inaccurate reconstruction of sRGB and RAW images. Metadata-based methods <cit.> incorporate necessary metadata of RAW images in the network to guide RAW reconstruction process. Studies such as <cit.> focus on learning-based methods, where ISP operations are learned end-to-end by neural networks. For instance, CycleISP <cit.> uses cycle consistency to learn both the forward RAW-to-sRGB and reverse sRGB-to-RAW translations. InvISP <cit.> proposes an flow-based invertible ISP architecture using a single invertible neural network for both forward and inverse translations. ParamISP <cit.> presents a hybrid model that combines model-based and data-driven approaches, leveraging standard ISP operations in a learnable and interpretable manner. In this approach, camera parameters from EXIF data are converted into feature vectors to control the ISP network. Additionally, <cit.> introduces an unpaired RAW-to-sRGB ISP and sRGB-to-RAW inverse ISP based on CycleGAN <cit.>, which eliminates the need for paired data and is more accessible. While existing methods achieve high-quality reconstruction, there remains a need for a more concise and equally accurate inverse ISP method to enable high-speed RAW reconstruction for practical applications, as discussed in Sec <ref>.

RAW-domain Object Detection. Existing research on RAW-domain object detection <cit.> indicates that using RAW images as model input can yield better performance compared to sRGB images. For instance, <cit.> proposes a RAW adapter that is image-adaptive and jointly optimized with downstream detectors. <cit.> argues that ISPs should be optimized for specific tasks, suggesting that Bayer pattern RAW images can be fed directly into detectors after a simple module such as Yeo-Johnson transformation. <cit.> indicates that detector performance can be improved by using RAW images with gamma correction. In these approaches, an adjustment module is usually employed to modify the data distribution of RAW images and enhance feature visibility.



§ METHOD




To achieve high-speed RAW reconstruction, we propose an innovative and ultra-lightweight model called the Learnable Color Conversion Matrix (LCCM). Unlike complex inverse ISP models <cit.>, LCCM utilizes only a single convolutional layer to approximate the mapping from sRGB images to RAW images. The parameters of the color conversion matrix are learnable during the training process. Inspired by knowledge distillation <cit.>, we use the unpaired RAW reconstruction method Unpaired-CycleR2R <cit.> as the teacher model, while the LCCM serves as the student model. The main framework of our RAW reconstruction method is illustrated in Figure <ref>.



 §.§ Unpaired RAW Reconstruction Teacher Model


Modern ISP systems in cameras typically consist of five major steps to convert RAW input into the corresponding sRGB output, as illustrated in Figure <ref> (a): demosaic f_dm, auto white balance f_wb, brightness adjustment f_ba, color correction f_cc, and gamma correction f_gc. The overall conversion performed by the camera ISP system can be defined as a composite function f_ISP composed of multiple invertible and tractable functions as follows <cit.>:

    f_ISP=f_gc∘ f_cc∘ f_ba∘ f_wb∘ f_dm


    y=f_ISP∘x

where x denotes a RAW image, y denotes an sRGB image, and ∘ denotes the function composition operation. The inverse ISP function g_InvISP that mirrors the ISP function f_ISP, can be expressed as <cit.>:

    g_InvISP=g_d m∘ g_w b∘ g_b a∘ g_c c∘ g_g c


    x=g_InvISP∘y

where g = f^-1 assumed for simplicity.
As shown in Figure <ref> (a), the teacher model Unpaired-CycleR2R <cit.> is designed based on a reversible, modular paradigm. It employs unpaired sRGB and RAW images to train both an sRGB-to-RAW inverse ISP for RAW reconstruction and a RAW-to-sRGB ISP. This unsupervised learning approach based on unpaired samples is highly applicable and practical for converting existing sRGB images into RAW format.



 §.§ Learnable Color Correction Matrix


In a camera ISP system, a 3x3 color correction matrix is utilized for color correction and adjustment, ensuring that the colors are restored to match the human visual system. Our proposed method replaces the complex ISP system with a single learnable color correction matrix to learn the mapping from sRGB images to RAW images. The color correction matrix C can be defined as below:

    X=Y ∙[[ C_11 C_12 C_13; C_21 C_22 C_31; C_31 C_32 C_33 ]]


where X denotes a RAW image matrix, Y denotes an sRGB image matrix, the • denotes the dot product of matrix. The calculation process of the convolutional layer can be represented by the following formula:

    x_i j=∑_u=1^U∑_v=1^V w_u v y_i+u-1, j+v-1

where x denotes a RAW image, y denotes an sRGB image, w denotes the convolutional kernel weight parameters, i and j denote the pixel positions, U and V denote the size of the convolutional kernel. A convolutional layer with a kernel size of 1, stride of 1, input and output channels of 3, and padding of 0, contains 9 weight parameters and 3 bias parameters. We innovatively use the 9 weight parameters of the convolutional layer to simulate the 9 parameters of the color correction matrix. Thus, a color correction matrix with trainable parameters is created, allowing it to learn and approximate the mapping from sRGB images to RAW images. Additionally, the bias parameters are used to fine-tune the color conversion matrix. We calculate the mean squared error between the simRAW images generated by LCCM and those generated by the teacher model Unpaired-CycleR2R <cit.>. The loss function is formulated as follows:

    L_M S E=1/n∑_i=1^n(x_simRAW-S -x_simRAW-T )^2

where x_simRAW-S denotes the simRAW images generated by our LCCM, x_simRAW-T denotes the simRAW images generated by the teacher model Unpaired-CycleR2R <cit.>, and i denotes each pixel.





§ EXPERIMENTS



 §.§ Experiment Settings

Datasets. We evaluate our proposed method on the multiRAW dataset <cit.> and the BDD100K dataset <cit.>. The BDD100K dataset <cit.> is a large-scale dataset contains 100k sRGB images collected in diverse scenes (city streets, residential areas, and highways). The multiRAW dataset <cit.> contains driving scene images covering various conditions (daytime, nighttime, tunnels and rainy weather).

Training Details for RAW reconstruction. Since the teacher model Unpaired-CycleR2R <cit.> is capable of converting sRGB images to RAW images for any camera style, we utilized this open-source model to reconstruct sRGB images from the BDD100K dataset <cit.> into simRAW-T images. This process provides us with sRGB and RAW image pairs, which are then used to train the LCCM. Subsequently, simRAW-S images are generated using the student model LCCM. Since the mapping from sRGB images to RAW images is a pixel-level task, we train the model using the original size of input sRGB images with a batch size of 1. An Adam optimizer with a learning rate of 0.001 is used for 100 training epochs. The quality of the generated simRAW images is evaluated using PSNR and SSIM.

Training Details for RAW-domain object detection. The generated simRAW images can be used to train RAW-domain models for various tasks. We focus on high-level object detection task and select YOLOv3 <cit.> due to its widespread use. Our method can be easily extended to other object detectors. The RAW images are resized to a fixed resolution of 1920×1920. Demosaic and gamma correction are applied in the RAW image processing pipeline as outlined in <cit.>. Data augmentation techniques, including random scaling, cropping, and color jitter, are employed. We use an SGD optimizer with a batch size of 4, a learning rate of 0.02, momentum of 0.9, and weight decay of 10^-4. A linear learning rate adjustment strategy is implemented during training. We train the detector for 30 epochs following <cit.>. Our experiments consist of two stages: pretraining and fine-tuning. In the pretraining stage, the RAW-domain YOLOv3 detector is trained on the simRAW-T and simRAW-S datasets, respectively. In the fine-tuning stage, the pretrained model is used to fine-tune the RAW-domain YOLOv3 detector on the multiRAW dataset <cit.>. 



 §.§ Main Results of RAW Reconstruction



We randomly select 1000 sRGB images from the BDD100K <cit.> training set and 1000 RAW counterparts from simRAW-T images as paired training samples, with 100 of each type as testing samples. As shown in Table <ref>, the proposed LCCM demonstrates significantly lower parameter and computational complexity, allowing it to efficiently complete both training and inference and contribute to the rapid expansion of RAW image datasets. In terms of PSNR, LCCM surpasses the traditional non-learning-based method UP <cit.> by 4.03 points, achieves nearly the same point as the learning-based method InvISP <cit.>, and is only 2.08 points behind CycleISP <cit.>. 
Figure <ref> illustrates that our proposed LCCM, composed of just a single convolutional layer, can effectively achieve RAW reconstruction under varying weather and lighting conditions. 

Figure <ref> presents the histograms for each color channel of the sRGB images from BDD100K <cit.>, simRAW-S sRGB and simRAW-T sRGB images. It is noteworthy that the intensity distributions of simRAW-T sRGB and simRAW-S sRGB images are generally similar, demonstrating that our LCCM can successfully simulate the mapping from sRGB images to RAW images. However, it is observed that the peak distributions of the three color channels in simRAW-S sRGB images differ from those in simRAW-T sRGB images. We attribute this to the limitation of using a single linear convolutional layer, which cannot fully approximate the nonlinear complex inverse ISP. 
While it is expected that a model using only one convolutional layer for RAW reconstruction may not outperform more complex networks, it is important to note that the RAW images generated by this simple network have comparable effects when used to pretrain RAW-domain detectors, as compared to complex inverse ISP methods. This will be further discussed in Sec <ref>.







 §.§ Main Results of RAW-domain Object Detection



The purpose of this study is to explore an ultra-simple and highly efficient method for RAW reconstruction, aimed at expanding RAW image datasets to support research in RAW-domain perception tasks for autonomous driving. We focus on the performance of a RAW-domain detector pretrained using simRAW images to verify the feasibility of our proposed LCCM method. The quantitative experimental results are presented in Table <ref>. 
First, We randomly initialize the RAW-domain YOLOv3 model and train it from scratch. Then, we pretrain the RAW-domain YOLOv3 detector using simRAW-T and simRAW-S images, respectively, to obtain a pretrained model. This pretrained model is then used to initialize the model parameters and fine-tune the RAW-domain YOLOv3 detector on the multiRAW dataset <cit.>. 

As shown in Table <ref>, detection accuracy improves consistently across different camera styles. The results indicate that fine-tuning a simRAW-pretrained YOLOv3 detector outperforms training the model from scratch. Compared to the baseline without pretraining, using LCCM-generated simRAW-S images for pretraining results in an improvement of 5.8 points (iPhone XSmax), 4.1 points (huawei P30Pro) and 3.2 points (asi 294mcpro) in metric AP.
We emphasize that the performance gains obtained by pretraining with simRAW-T images and simRAW-S images are nearly identical. The experimental results reveal the feasibility and significant potential of our proposed LCCM, which is highly efficient, substantially reducing time and computational resources for RAW reconstruction.





 §.§ Ablation Studies

The mapping from sRGB images to RAW images is a pixel-level task, so each high-resolution image provides a large number of pixel samples for training the learnable color conversion matrix. In this section, we conduct experiments to investigate the impact of training sample quantity on the quality of RAW reconstruction images. Our proposed LCCM is particularly notable for its ability to train the color correction matrix using only a few dozen pairs of sRGB and RAW images. As shown in Figure <ref>, performance saturation occurs with approximately 100 training samples, This indicates that with just a small set of sRGB and RAW paired images from a specific camera, LCCM can effectively learn the sRGB-to-RAW mapping relationship. These results underscore the efficiency of our proposed LCCM.



§ CONCLUSION

In this paper, we present an innovative inverse ISP method for RAW reconstruction. Our approach leverages a learnable color correction matrix to approximate the mapping from sRGB images to RAW images. The proposed model is exceptionally simple and lightweight, consisting of only a single convolutional layer. Experimental results show that when used to pretrain a RAW-domain object detector, the simRAW images generated by our proposed model achieve performance gains equivalent to those generated by more complex inverse ISP models. It demonstrates the effectiveness and feasibility of our method. We hope that our work will contribute to the rapid expansion of RAW datasets to support research in autonomous driving tasks and inspire new insights into the design of efficient inverse ISP methods.



§ ACKNOWLEDGEMENTS

This work was supported in part by the National High Quality Program under Grant TC220H07D, in part by the National Key R&D Program of China under Grant 2022YFB2902002, in part by the Innovation Program of Shanghai Municipal Science and Technology Commission under Grant 20511106603, and in part by Foshan Science and Technology Innovation Team Project under Grant FS0AAKJ919-4402-0060. 


