# Image Colorization: A Survey and Dataset

Saeed Anwar∗, Muhammad Tahir∗, Chongyi Li, Ajmal Mian, Fahad Shahbaz Khan,
Abdul Wahab Muzaffar S. Anwar is a research scientist with Data61-CSIRO,
Australia, and Lecturer at Australian National University, Australia.
<EMAIL_ADDRESS>
M. Tahir is an Assistant Professor at College of Computing and Informatics,
Saudi Electronic University.
<EMAIL_ADDRESS>
Chongyi Li is with the School of Computer Science and Engineering, Nanyang
Technological University (NTU), Singapore.
<EMAIL_ADDRESS>
Ajmal Mian is a Professor of Computer Science with The University of Western
Australia.
<EMAIL_ADDRESS>
Fahad S. Khan is a Lead Scientist at the Inception Institute of Artificial
Intelligence (IIAI), Abu Dhabi, UAE and an Associate Professor at Computer
Vision Laboratory, Linkoping University, Sweden.
<EMAIL_ADDRESS>
Abdul W. Muzaffar is an Assistant Professor at College of Computing and
Informatics, Saudi Electronic University.
<EMAIL_ADDRESS>denotes equal contribution

###### Abstract

Image colorization is an essential image processing and computer vision branch
to colorize images and videos. Recently, deep learning techniques progressed
notably for image colorization. This article presents a comprehensive survey
of recent state-of-the-art colorization using deep learning algorithms,
describing their fundamental block architectures in terms of skip connections,
input etc. as well as optimizers, loss functions, training protocols, and
training data etc. Generally, we can roughly categorize the existing
colorization techniques into seven classes. Besides, we also provide some
additional essential issues, such as benchmark datasets and evaluation
metrics. We also introduce a new dataset specific to colorization and perform
an experimental evaluation of the publicly available methods. In the last
section, we discuss the limitations, possible solutions, and future research
directions of the rapidly evolving topic of deep image colorization that the
community should further address. Dataset and Codes for evaluation will be
publicly available at https://github.com/saeed-anwar/ColorSurvey

###### Index Terms:

Image colorization, deep learning, experimental survey, new colorization
dataset, colorization review, CNN model classification.

##  1 Introduction

“Don'’t ask what love can make or do! Look at the colors of the world.” \-
Rumi

Colorization of images is a challenging problem due to the varying image
conditions that need to be dealt via a single algorithm. The problem is also
severely ill-posed as two out of the three image dimensions are lost; although
the semantics of the scene may be helpful in many cases, for example, grass is
usually green, clouds are usually white, and the sky is blue. However, such
semantic priors are highly uncommon for objects such as t-shirts, desks, and
many other observable items. Further, the colorization problem also inherits
the typical challenges image enhancement, such as changes in illumination,
variations in viewpoints, and occlusions.

In recent years, with the rapid development of deep learning techniques, a
variety of image colorization models have been introduced and state-of-the-art
performance on current datasets has been reported. Diverse deep-learning
models ranging from the early brute-force networks (e.g. [1]) to the recently
carefully designed Generative Adversarial Networks (GAN) (e.g. [2]) have been
successfully used to tackle the colorization problem. These colorization
networks differ in many major aspects, including but not limited to network
architecture, loss types, learning strategies, network depth, etc.

In this paper, we provide a comprehensive overview of single-image
colorization and focus on the recent advances in deep neural networks for this
task. To the best of our knowledge, no survey of either traditional or deep
learning colorization has been presented in current literature. Our study
concentrates on many important aspects, both in a systematic and comprehensive
way, to benchmark the recent advances in deep-learning based image
colorization.

Contributions: Our contributions are as follows

  1. 1.

We provide a thorough review of image colorization techniques including
problem settings, performance metrics and datasets.

  2. 2.

We introduce a new benchmark dataset, named Natural Color Dataset (NCD),
collected specifically for the colorization task.

  3. 3.

We provide a systematic evaluation of state-of-the-art models on NCD dataset.

  4. 4.

We propose a new taxonomy of colorization networks based on the differences in
domain type, network structure, auxiliary input, and final output.

  5. 5.

We also summarize the vital components of networks, provide new insights,
discuss the challenges, and possible future directions of image colorization.

---
Figure 1: Taxonomy: Classification of the colorization networks based on
structure, input, domain, and type of network.

Related Surveys: Image colorization has been the focus of significant research
efforts over the last two decades while most early image colorization methods
were primarily influenced by conventional machine learning approaches [3, 4,
5, 6, 7, 8] \- the past few years have seen a huge shift in focus to deep
learning-based models due to their success in a number of different
application domains [1, 9, 10, 11, 12].

Automatic image colorization systems built upon deep-learning methodologies
have recently shown impressive performance. However, despite this success, no
literature survey or review covering the latest developments in this field
currently exists. Therefore, inspired by surveys in deep image super-
resolution [13], VQA [14], etc. we aim to provide a survey for deep image
colorization.

Problem Formulation: The aim of colorization is to convert a grayscale image
to color image, typically captured in previous decades, when technological
advancements were limited. Hence this process is more a form of image
enhancement than image restoration. It is also possible that the color images
such as RGB are converted to grayscale or Y-Channel of YUV color-space,
possibly to save storage space or communication bandwidth. The grayscale
images may also exist due to old technology or low-cost sensors. Therefore, in
this case, a trivial formulation can be written as:

$I_{g}=\Phi(I_{rgb}),$ (1)

where $\Phi(\cdot)$ is a function that converts the RGB image $I_{rgb}$ to a
grayscale image $I_{g}$, for example as follows:

$I_{g}=0.2989*I_{r}+0.5870*I_{g}+0.1140*I_{b}.$ (2)

Typically, colorization methods aim to restore the color in YUV space, where
the model needs to predict only two channels, i.e. U and V - instead of the
three channels in RGB.

##  2 Single-Image Deep Colorization

This section introduces the various deep learning techniques for image
colorization. As shown in Figure 1, these colorization networks have been
classified into various categories, in terms of different factors, such as
structural differences, input type, user-assistance etc. Some networks may be
eligible for multiple categories; however, we have simply placed these in the
most appropriate category.

###  2.1 Plain Networks

Like in other CNN tasks, early colorization architectures were plain networks.
We classify a network as plain if it possesses a simple, straightforward
architecture with stacked convolutional layers, i.e., no, or naive skip
connections. Networks that fall into this category are shown in Figure 2 and
are discussed below.

####  2.1.1 Deep Colorization

Deep colorization111Code available at https://shorturl.at/cestD [1] can be
regarded as the first work to have attempted to incorporate convolutional
neural networks (CNNs) for the colorization of images. However, this method
does not fully exploit the CNNs; instead, it also includes joint bilateral
filtering [15] as a post-processing step to remove artifacts introduced by the
CNN network.

In the training step, five fully connected linear layers are followed by non-
linear activations (ReLU). The loss function utilized is the least-squares
error. In the proposed model, the number of neurons in the first layer depends
on the dimensions of the feature descriptor extracted from the grayscale
patch, while the output layer has only two neurons, i.e., the U and V channel
corresponding color pixel value. In the testing step, features are extracted
from the input grayscale image at three levels i.e. low-level, mid-level, and
high-level. The features at low-level are the sequential gray values, at mid-
level they are DAISY features [16], and at high-level semantic labeling is
performed. The patch and DAISY features are concatenated and then passed
through the network. As a final step for removing artifacts, joint bilateral
filtering [15] is performed.

|
---|---
Deep Colorization | Depth Colorization (DE2CO)
Colorful Colorization
Figure 2: Plain networks are the earlier models with convolutional layer
stacking with no skip or naive skip connections.

The network takes as input a 256 $\times$ 256 grayscale image and is composed
of five layers: one input layer, three hidden layers, and one output layer.
The model is trained on 2688 images from the Sun dataset [17]. The images are
segmented into 47 objects categories including cars, buildings, sea etc. for
47 high-level semantics. Furthermore, 32-dimensional mid-level DAISY features
[16] and 49-dimensional low-level features are utilized for colorization.

####  2.1.2 Colorful Colorization

Colorful image Colorization CNN222Code at
https://github.com/richzhang/colorization [18] was one of the first attempts
to colorize grayscale images. The network takes as input a grayscale image and
predict 313 “ab” pairs of the gamut showing the empirical probability
distribution, which are then transformed to “a” and “b” channels of the “Lab”
color space. The network stacks convolutional layers in a linear fashion,
forming eight blocks. Each block is made up of either two or three
convolutional layers followed by a ReLU layer and Batch Normalization
(BatchNorm [19]) layer. Striding instead of pooling is used to decrease the
size of the image. The input to the network is 256$\times$256, while the
output is 224$\times$224; however, it is resized later to the original image
size.

The colorful image colorization network removes the influence (due to
background, e.g. sky, clouds, walls etc.) of the low “ab” values by
reweighting the loss based on the pixel rarity in training—the authors’ term
this technique as class rebalancing.

The framework utilized to build the network is Caffe [20], and ADAM [21] is
used as the solver. The network is trained for 450k iterations with a learning
rate of $3\times 10^{-5}$, which is reduced to $10^{-5}$ at 200k and $3\times
10^{-6}$ at 375k iterations. The kernel size is 3$\times$3, and the feature
channels vary from 64 to 512.

####  2.1.3 Deep Depth Colorization

Deep depth colorization333Available at https://github.com/engharat/SBADAGAN
(DE)2CO [22] employs a deep neural network architecture for colorizing depth
images utilizing pre-trained ImageNet [23]. The system is primarily designed
for object recognition by learning the mapping from the depths to RGB
channels. The weights of the pre-trained network are kept frozen, and only the
last fully connected layer of (DE)2CO is trained that classifies the objects
with a softmax classifier. The pre-trained networks are merely used as feature
extractors.

The input to the network is a 228$\times$228 depth map reduced via convolution
followed by pooling to $64\times 57\times 57$. Subsequently, the features are
passed through a series of residual blocks, composed of two convolutional
layers, each followed by batch normalization [19] and a non-linear activation
function i.e. leaky-ReLU [24]. After the last residual block, the output is
passed through a final convolutional layer to produce three channels, i.e., an
RGB image as an output. To obtain the original resolution, the output is
deconvolved as a final step. When an unseen dataset is encountered, only the
last convolutional layer is retrained while keeping the weights across all
other layers frozen. (DE)2CO outperforms CaffeNet (a variant of Alexnet [25]),
VGG16 [26], GoogleNet [27], and ResNet50 [28] under the same settings on three
benchmark datasets including Washington-RGBD [29], JHUIT50 [30], and BigBIRD
[31].

###  2.2 User-guided networks

User-guided networks require input from the user either in the form of in
points, strikes, or scribbles, as presented in Figure 3. The user input can be
provided in real-time or offline. The following are examples of user-guided
networks.

####  2.2.1 Scribbler

Sangkloy et al. [32] used an end-to-end feed-forward deep generative
adversarial
architecture444https://github.com/Pingxia/ConvolutionalSketchInversion to
colorize images. To guide structural information and color patterns, user
input in the form of sketches and color strokes is employed. The adversarial
loss function enables the network to colorize the images more realistically.

The generator part of the proposed network adopts an encoder-decoder structure
with residual blocks. Following the architecture of Sketch Inversion [33], the
augmented architecture consists of three downsampling layers, seven residual
blocks preceded by three upsampling layers. The downsampling layers apply
convolutions of stride two, whereas the upsampling layers utilize bilinear
upsampling to substitute the deconvolutional layers contrary to Sketch
Inversion. All layers are followed by batch normalization [19] and the ReLU
activation function except the last layer, where the $TanH$ function is used.
The discriminator part of the proposed network is composed of a fully
convolutional network with five convolutional layers and two residual blocks.
Leaky-ReLU [24] is used after each convolutional layer except the last one,
where Sigmoid is applied.

The proposed model’s performance is assessed qualitatively on datasets from
three different domains, including faces, cars, and bedrooms.

|
---|---
Scribbler architecture | Real-Time User guided Colorization
|
Interactive Deep Colorization | Hint-Guided Anime Colorization
Figure 3: User-Guided networks are the ones that require a user to input the
color at some stage of the network during colorization.

####  2.2.2 Real-Time User-Guided Colorization

Zhang et al. [34] developed user interaction based on two variants, namely
local hint and global hint networks, both of which utilize a common main
branch for image colorization555https://github.com/junyanz/interactive-deep-
colorization. The local hint network is responsible for processing the user
input and yielding color distribution, whereas the global hint network accepts
global statistics in the form of a global histogram and average saturation of
the image. The loss functions include Huber and regression.

The main network is composed of ten blocks, made up of two or three
convolutional layers followed by the ReLU. Similarly, each block is succeeded
by batch normalization as well. In the first four blocks, feature tensors are
continuously halved, while doubling the feature dimensions. The same process
is performed in reverse order for the last four convolutional blocks.
Moreover, a dilated convolution (with a factor of two) is applied for the
fifth and sixth blocks. Symmetric shortcut connections are also established
between different blocks for the recovery of spatial information. The kernel
size for all convolutions is set to $3\times 3$. However, a $1\times 1$ kernel
is used at the last layer, mapping block ten and the final output.

####  2.2.3 Interactive Deep Colorization

To colorize grayscale images, Xiao et al. [35] developed an interactive
colorization model based on the U-Net [36] architecture, which can
simultaneously utilize global and local inputs. The network includes a feature
extraction module, a dilated module, a global input module, and a
reconstruction module.

The feature extraction module (layers 2 to 14) receives the inputs i.e. a
grayscale image, local input, and gradient map that are merged via element-
wise summation. The global input is processed by four convolution layers
independently and then merged with the output of the feature extraction module
using element-wise summation. The dilated module takes the input from the
extraction module (corresponding to convolutional layers ranging from 15 to
20). The dilated module’s output is further processed by a reconstruction
module composed of many deconvolution and convolution layers. As a final step,
the output of the network is combined with the input grayscale image to
generate the colorized version. All the layers use ReLU except the final one,
which employs a $TanH$ activation.

The Huber loss is modified to meet the requirements of the proposed model. The
testing set is composed of randomly chosen 1k images from ImageNet [23]. The
proposed model is trained for 300k iterations utilizing the remaining images
from ImageNet [23], combined with 150k images from Places dataset [37].

####  2.2.4 Anime Line Art Colorization

Ci et al. [38] proposed an end-to-end interactive deep conditional GAN (CGAN)
[39] for the colorization of synthetic anime line arts. The system operates on
the user hints and the grayscale line art.

The discriminator is conditioned on the local features computed from a pre-
trained network called Illustration2Vec [40]. The generator adopts the
architecture of U-Net [36] that has two convolution blocks and the local
feature network at the start. Afterward, four sub-networks with similar
structures are employed, each of which is composed of convolutional layers at
the front followed by ResNeXt blocks [41] with dilation and sub-pixel
convolutional (PixelShuffle) layers. Each convolutional layer utilizes
LeakyReLU as activation function, except the final one where $TanH$ activation
is employed. The discriminator, on the other hand, is inspired by the
architecture of SRGAN [42]; however, the basic blocks are replaced from the
earlier mentioned generator to remove dilation and an increased number of
layers is used.

The generator employs the perceptual and adversarial losses, and the
discriminator combines Wasserstein critic and penalty losses. The ADAM [21]
optimizer is employed with $\beta\textsubscript{1}=0.5$,
$\beta\textsubscript{1}=0.9$ and a batch size of four.

###  2.3 Domain-Specific Colorization

The aim of these networks is to colorize images from different modalities such
as infrared, or different domains such as radar. We provide the details of
such networks in the following sub-sections and are shown in Figure 4.

####  2.3.1 Infrared Colorization

An automatic Near Infrared (NIR)666Code is available at https://bit.ly/2YZQhQ4
image colorization technique [43] was developed using a multi-branch deep CNN.
NIR [43] is capable of learning luminance and chrominance channels. Initially,
the input image is preprocessed and is converted into a pyramid. Where each
pyramid-level is normalized i.e. has zero mean and unit variance. Next, each
structurally similar branch of NIR [43] is trained using a single input
pyramid-level without sharing weights between layers. All the branches are
merged into a fully connected layer to produce the final output. Additionally,
the mean input image is also fed directly to the fully connected layer.

The fundamental structure of NIR is inspired by [18]. Each branch is composed
of stacked convolutional layers with pooling layers placed in the middle after
regular intervals. The activation function after each convolutional layer is
ReLU. In order to produce the colorized image, the raw output of the network
is joint-bilaterally filtered, and the output is then further enhanced by
incorporating high-frequency information directly from the original input
image.

Each block has the same number of convolutional layers. The kernel size is
3$\times$3, while a downsampling of 2$\times$2 is utilized in the pooling
layers. Similarly, the same number of feature maps, i.e. 16, are employed in
the first block, but after each pooling layer, the number is increased by 2.
Moreover, The authors developed a real-world dataset by capturing road scenes
in summer with a multi-CCD NIR/RGB camera. The proposed model is trained on
about 32 image pairs and tested on 800 images from the mentioned dataset.

|
---|---
NIR architecture | SIC
|
RICNet architecture | SAR-GAN architecture
Figure 4: Domain-Specific Colorization networks colorize images from different
modalities such as infra-red, radar images.

####  2.3.2 SAR-GAN

To colorize Synthetic Aperture Radar (SAR) images, Wang et al. [44] proposed
SAR-GAN. The network uses a cascaded generative adversarial network as its
underlying architecture. The input SAR images are first denoised from speckles
and then colorized to produce high-quality visible images. The generator of
SAR-GAN [44] consists of a despeckling subnet and colorization subnet. The
output of the despeckling subnet is a noise-free SAR image that is further
processed by the colorization subnet to produce a colorized image.

The despeckling sub-network consists of eight convolutional layers with batch
normalization [19], an activation function, and an element-wise division
residual layer. First, the speckle component in the SAR image is estimated and
forwarded to the residual layer. To generate a noise-free image, a residual
layer equipped with skip connections performs component-wise division of the
input SAR image by the estimated speckle. The colorization sub-network
utilizes a symmetric encoder-decoder architecture with eight convolutional
layers and three skip connections that enable the network’s input and output
to share low-level characteristics.

The ADAM [21] optimization technique is adopted for training the entire
network. The discriminator component of SAR-GAN [44] utilizes a hybrid loss
function, developed by combining the pixel-level $\ell_{1}$ loss with the
adversarial loss. SAR-GAN [44] is tested on 85 out of 3292 synthetic images as
well as real SAR images. .

####  2.3.3 Radar Image Colorization

Song et al. [45] developed a feature-extractor network and a feature-
translator network for the colorization of single-polarization radar grayscale
images777Code at https://github.com/fudanxu/SAR-Colorization. To construct the
feature-extractor network, the first seven layers of the VGG16 [26] pre-
trained on ImageNet [23], are used. The output of the feature extractor
network is in the form of a hyper-column descriptor that is obtained by
concatenating the corresponding pixels from all layers along with the input
image.

The final hyper-column descriptor, thus obtained, is fed into the feature-
translator network that is composed of five fully connected layers. As a final
step, the feature-translator network uses softmax function to obtain the
output similar to classification, which is achieved by constructing nine
groups of neurons representing nine polarimetric. The ReLU activation function
follows each convolutional layer in both networks. Furthermore, to train both
the feature-extractor and feature-translator, full polarimetric radar image
patches are employed.

####  2.3.4 Sketch Image Colorization

Lee et al. [46] proposed a reference-based sketch image colorization, where a
sketch image is colorized according to an already-colored reference image. In
contrast to grayscale images, which contain the pixel intensity, sketch images
are more information-scarce, which makes sketch image colorization more
challenging.

In the training phase, the authors proposed an augmented self-reference
generation method, generated from the original image by both color
perturbation and geometric distortion. Specifically, a color image is first
converted into its sketch image using an outline extractor. Then, the
augmented self-reference image is generated by applying the thin plate splines
transformation to the reference image. The generated reference image as the
ground truth contains most of the content from the original image, thus,
providing a full information of correspondence for the sketch. Taking the
sketch image and reference image as inputs, the network first encodes these
two inputs by two independent encoders. Furthermore, the authors proposed a
spatially corresponding feature transfer module to transfer the contextual
representations obtained from the reference into the spatially corresponding
positions of the sketch (i.e., integrating the feature representations of
sketch and its reference image). The integrated features are passed through
residual blocks and a decoder to produce the colored output. During training,
a similarity-based triplet loss, L1 reconstruction loss, adversarial loss,
perceptual loss, and style loss are used to drive the learning of the proposed
network.

The proposed network is a U-net-like structure coupled with several residual
blocks and a spatially corresponding feature transfer module in between the
encoder and decoder. The sketch image and the color reference image with a
size of 256 $\times$ 256 are fed to the network and the network outputs the
colorization result of the sketch image.

###  2.4 Text-based Colorization

In these types of networks, the images are colorized based on the input, which
is usually text. We classify the following models as text-based colorization
networks. Figure 5 presents the network architectures for this category.

####  2.4.1 Learning to Color from Language

To exploit additional text input for colorization, a language-conditioned
colorization architectures888Available at
https://github.com/superhans/colorfromlanguage [47] was proposed to colorize
grayscale images with additional learning from image captions. The authors
employed an existing language-agnostic architecture called FCNN [18],
providing image captions as an additional input. FCNN [18] is composed of
eight blocks each of which consists of a sequence of convolutional layers
followed by batch normalization. The authors experimented with the _CONCAT_
network [48] to generate captions from images, but settled on the _FILM_
network [49] due to its small number of parameters. To obtain the final
output, the language-conditioned weights of the FILM [49] are utilized for
affine transformation of the convolutional blocks’ outputs.

Finally, FCNN [18] and the two language-conditioned architectures i.e. CONCAT
and FILM were trained on images from MS-COCO dataset [50]. Automatic
evaluations indicated that the FILM architecture achieves the highest
accuracy. Moreover, the models were also assessed by performing crowdsourced
assessments that authenticated the effectiveness of the architectures. The
output of the network is a 56$\times$56 color image.

|
---|---
LCL-Net architecture | Text2Color
Figure 5: Text-based colorization networks are based on the text input with
the grayscale image.

####  2.4.2 Text2Colors

The Text2Colors999Code is https://github.com/awesome-davian/Text2Colors/ [51]
model is comprised of two conditional generative adversarial networks: Text-
to-Pallette Generation Network (TPN) and Palette-based Colorization Network
(PCN). TPN is responsible for constructing color palettes learned from the
Palette-and-Text (PAT) dataset, which contains five-color palettes for each of
10,183 textual phrases. Meanwhile, PCN is responsible for colorizing the input
grayscale image given the generated palette based on the input text.

The TPN generator learns the pairing between text and color palette, while its
discriminator, simultaneously learns the characteristics of the color palette
and text, to identify the real palettes from the fake ones. The loss function
in the TPN network is the Huber loss.

On the other hand, the generator of the PCN network has two subnetworks: the
colorization network based on U-Net [36] to colorize the images and the
conditioning network to apply the palette colors to the generated image. The
PCN discriminator is based on the DCGAN architecture [52]. In the PCN’s
discriminator, first, the features from the input image and generated palette
(by the TPN network) are jointly learned by a series of Conv-LeakyReLU layers.
Then, a fully connected layer classifies the image as either real or fake.

The discriminator and generator of TPN are first trained on the PAT dataset
for 500 epochs, and then trained on the ground-truth image for 100 epochs. The
trained generators of TPN and PCN are then used to colorize the input
grayscale image at the testing stage using the input text’s color palette.
Adam optimizer is used with a learning rate of 0.0002 for all the
networks.nput text’s color palette. Adam optimizer with a learning rate of
0.0002 for all the networks.

###  2.5 Diverse Colorization

The aim of diverse colorization is to generate different colorized images,
rather than restore the original color shown in Figure 6. Diverse colorization
is usually achieved via GANs or variational autoencoders. GANs attempt to
generate the colors in a competitive manner where the Generator tries to fool
the Discriminator while the Discriminator tries to differentiate between the
ground-truth and the generated colors. We present the vanilla GANs employed
for colorization.

####  2.5.1 Unsupervised Diverse Colorization

Cao et al. [53] proposed the utilization of conditional GANs for the diverse
colorization of real-world objects101010Code is available at
https://github.com/ccyyatnet/COLORGAN. In the generator of the proposed GAN
network, the authors employed five fully convolutional layers with batch
normalization and ReLU. To provide continuous conditional supervision for
realistic results, the grayscale image is concatenated with every layer of the
generator. Furthermore, to diversify the colorization outputs, noise channels
are added to the first three convolutional layers of the generator network.
The discriminator is composed of four convolutional layers and a fully-
connected layer to differentiate between the fake and the real values of the
image.

During the convolution operations, the stride is set to $1$ to keep the
spatial information the same across all layers. The performance of the
proposed method was assessed using the Turing test methodology. The authors
provided questionnaire surveys to 80 subjects asking them 20 different
questions regarding the produced results for the publicly available LSUN
bedroom dataset [54]. The proposed model obtained a convincible rate of
$62.6\%$ compared to $70\%$ for ground-truth images. Additionally, a
significance t-test generated a p-value of 0.1359, indicating that the
generated colorized images are not significantly different from the real
images.

For implementation, the authors opted for the Tensorflow framework. The batch
size was selected as 64 with a learning rate of 0.0002 and 0.0001 for the
discriminator and generator, respectively. The model was trained for 100
epochs using RMSProp as an optimizer. The output of the network is
64$\times$64 in size.

####  2.5.2 Tandem Adversarial Networks

For the colorization of raw line-art, Frans [55] proposed two adversarial
networks in tandem. The first network, namely the color-prediction network,
predicts the color scheme from the outline, while the second network, called
the shading network, produces the final image from the color scheme generated
by the color-prediction network in conjunction with the outline. Both the
color-prediction and shading networks have same structure as U-Net [36] while
the discriminator has the same structure as [53]. The adversarial training is
performed using the discriminator formed by stacking the four convolutional
layers and a fully connected layer at the end.

The convolutional layers are transposed once the density of the feature matrix
reaches a certain level. The author also added skip connections between the
corresponding layers to directly allow the gradient to flow through the
network. Further, $\ell_{2}$ and adversarial losses are incorporated in the
color-prediction and shading-networks, respectively. The convolutional filter
size is set to $5\times 5$ with a stride of two for every convolutional layer
in the network. The number of feature maps is increased by two times after
every layer, where the initial number of feature maps is 64.

|
---|---
UDCGAN/CGAN | Diverse Image Colorization
|
TANDEM architecture | Memopainter
ChromaGAN
Figure 6: Diverse colorization networks generate different colorized images
instead of aiming to restore the original color only.

####  2.5.3 ICGAN

Image Colorization Using Generative Adversarial Networks, abbreviated as ICGAN
[56] proposed by Nazeri et al., has demonstrated better performance in image
colorization tasks than traditional convolutional neural networks. Fully
convolutional networks for semantic segmentation [57] inspire the baseline
model and is constructed by replacing the fully connected layers of the
network by fully convolutional layers. The basic idea is to downsample the
input image gradually via multiple contractive encoding layers and then apply
a reverse operation to reconstruct the output with many expansive decoding
layers, similar to U-Net [36].

The ICGAN [56] generator takes grayscale input image as opposed to random
noise, like traditional GANs. The authors also proposed the generator’s
modified cost function to maximize the discriminator’s probability of being
incorrect instead of minimizing the likelihood of being correct. Moreover, the
baseline model’s generator is used without any modifications, while the
discriminator is composed of a series of convolutional layers with batch
normalization [19].

The filter size in each convolutional layer is 4$\times$4 as opposed to the
traditional 3$\times$3, and the activation function is Leaky-ReLU [24] with a
slope value of 0.2. The final one-dimensional output is obtained by applying a
convolution operation after the last layer, predicting the input’s nature with
a certain probability.

To train the network, the authors employed ADAM optimization [21] with weight
initialization using the guidelines from [58]. The performance of the system
is assessed using CIFAR10 [59] and Places356 datasets [60]. Overall, the
visual performance of the ICGAN [56] is favorable compared to that of
traditional CNN.

####  2.5.4 Learning Diverse Image Colorization

Deshpande et al. [61] employed Variational Auto-Encoder (VAE) and Mixture
Density Network (MDN) to yield multiple diverse yet realistic colorizations
for a single grayscale image111111Code is available at
https://github.com/aditya12agd5/divcolor. The authors first utilized VAE to
learn a low-dimensional embedding for a color field of size $64\times 64\times
2$, and then the MDN was employed to generate multiple colorizations.

The VAE architecture encoder consists of four convolutional layers with a
kernel size of $5\times 5$ and stride of two. The feature channels start at
128 and double in the successive encoder layer. Each convolutional layer is
followed by batch normalization, and ReLU is used as an activation function.
The last layer of the encoder is fully connected. On the other hand, the
decoder of the VAE architecture has five convolutional layers, each one
preceded by linear upsampling and followed by batch normalization, and ReLU.
The input to the decoder is a d-dimensional embedding, and the output is a
$64\times 64\times 2$ color field. The convolutional kernel size is $4\times
4$ in the first layer, and in the remaining, it is of size $5\times 5$. As
mentioned earlier, activation function in all layers is ReLU, except in the
last layer, where the activation function is _TanH_. The output channels at
each layer are halved in the decoder.

MDN consists of twelve convolutional layers and two fully connected layers and
is activated by the ReLU function, which is followed by batch normalization.
During the testing embeddings sampled from the MDN output can be used by the
Decoder of the VAE to produce multiple colorizations. To enhance the overall
performance of the proposed architecture, the authors designed three loss
functions: specificity, colorfulness, and gradient.

####  2.5.5 ChromaGAN

ChromaGAN121212Code available at https://github.com/pvitoria/ChromaGAN [62]
exploits geometric, perceptual, and semantic features via an end-to-end self-
supervised generative adversarial network. By utilizing the semantic
understanding of the real scenes, the proposed model can colorize the image
realistically (actual colors) rather than merely pleasing the human eyes.

The Generator of ChromaGAN [62] is composed of two branches, which receive a
grayscale image of size 224$\times$224 as input. One of the branches yields
chrominance information, whereas the other generates a class distribution
vector. The network composition is as follows: the first stage is shared
between both the branches that implements VGG16 [26] while removing the last
three fully connected layers. The pre-trained VGG16 [26] weights are used to
train this stage without freezing them. In the second stage, each branch
follows its dedicated path. The first branch has two modules, each one
designed by combining a convolutional layer followed by Batch normalization
[19] and ReLU. The second path has four modules with the same composite
structure (Conv-BN-ReLU) as of the first branch but is further followed by
three fully connected layers and provides the class distribution. In the third
stage, the outputs of these two distinct paths fused. The features then pass
through six modules. Each module has a convolutional layer and ReLU with two
upsampling layers within.

The discriminator is based on PatchGAN [63] architecture that holds high-
frequency structural information of the generator’s output by focusing on the
local patches rather than the entire image. ChromaGAN [62] is trained on 1.3M
images of ImageNet [23] for a total of five epochs. The optimizer used is ADAM
with an initial learning rate of 2$e^{-5}$.

####  2.5.6 MemoPainter: Coloring with Limited Data

MemoPainter131313https://github.com/dongheehand/MemoPainter-PyTorch [2] is
capable of learning from limited data, which is made possible by the
integration of external memory networks [64] with the colorization networks.
This technique effectively avoids the dominant color effect and preserves the
color identity of different objects.

Memory networks keep the history of rare examples, enabling them perform well
even with insufficient data. To train memory networks in an unsupervised
manner, a novel threshold triplet loss was introduced by the authors [2]. In
memory networks, the “key-memory” holds spatial information and computes
cosine similarity against the input. Likewise, “value-memory” keep records
about color information utilized by the colorization networks as a condition.
Similarly, “age” keeps the time-stamp for each entry in the memory. The “key”
and “value” memories are generated from the training data. The memory is
updated by averaging the “key-memory” and a new query image owing to the
distance between the color information of the new query image and the
available top-1 “value-memory” element lies below the threshold. Otherwise, a
new record is stored for the input image color information.

The colorization networks are constructed using conditional GANs [39]
consisting of a generator and discriminator, and color information is learned
from RGB images during training. The generator inspired by U-Net [36], is
composed of ten convolutional layers, while the discriminator is fully
convolutional, consisting of four layers. The color feature is extracted and
provided to the generator after being passed forward from the MLP.

During testing, the color information is retrieved from the memory networks
and fed to the generator as a condition. The colorization networks employ
adaptive instance normalization for enhanced colorization as it is considered
as a style transfer. The performance of MemoPainter [2] is evaluated on
different datasets, including Oxford102 Flower [65], Monster [66], Yumi
141414https://comic.naver.com/webtoon/list.nhn?titleId=651673, Superheroes
[2], and Pokemon 151515https://www.kaggle.com/kvpratama/pokemon-images-
dataset.

###  2.6 Multi-path networks

The multi-path networks follow different parts to learn features at different
levels or different paths. The following are examples of multi-path networks,
and Figure 7 provides their architectures.

####  2.6.1 Let there be Color

Iizuka et al. [67] designed a neural
network161616https://github.com/satoshiiizuka/siggraph2016_colorization to
primarily colorize the grayscale images and provide scene classification as a
secondary task. The proposed system has two branches to learn features at
multiple scales. Both branches are further divided into four subnets, i.e. the
low-level features subnet, mid-level features subnet, global features subnet,
and colorization network. The low-level subnet decreases the resolution and
extract edges and corners. To learn the textures, mid-level features come in
handy.

Similarly, a 256D vector is computed by the global subnet to represent the
image. The mid-level and global features are combined and fed into the
colorization network to predict the chrominance channel. The final output is
restored to the original input resolution. The global features help the
classification network to classify the scene in the image. Both the
colorization network and scene classification network are jointly trained.

The low-level features subnet is composed of six convolution layers, the mid-
level features subnet has two convolutional layers, and the global features
network is comprised of four convolutional layers as well as three fully-
connected layers. The colorization network is made up of a fusion layer, four
convolutional, and two upsampling layers. To feed input to the network, the
image is resized to 256$\times$256, and then output is upsampled to its
original resolution. The kernel is of size 3$\times$3; feature channels vary
between 64 to 512, and striding is used to reduce the feature map size.
Moreover, the learning rate is determined automatically using ADADELTA [68],
the network is optimized using Stochastic Gradient Descent (SGD) [69], and the
system is designed using Torch [70].

####  2.6.2 Learning Representations for Automatic Colorization

Learning Representations for Automatic Colorization171717Code is available at
https://github.com/gustavla/autocolorize [71] aims to learn a mapping function
taking per-pixel features as hypercolumns of localized slices from existing
CNN networks to predict hue and chroma distributions for each pixel. The
authors use VGG16 [26] as a feature extractor while discarding the
classification layer and using a grayscale image as input rather than RGB. For
each pixel, a hypercolumn is extracted from all the VGG16 layers except the
last classification layer resulting in a 12k channel feature descriptor that
is fed into a fully connected layer of 1k channels, which predicts the final
hue and chroma outputs.

The KL divergence is employed as a loss to predict distributions over a set of
color bins. The input size is 256$\times$256, and the framework used is Caffe
[20], where the network is fine-tuned for ten epochs, each of which takes
about 17 hours. Other parameters and optimizations are similar to VGG16 [26].

####  2.6.3 PixColor

PixColor, proposed by [72], employs a conditional PixelCNN to produce a low-
resolution color image from a given grayscale image. Then, a refinement CNN is
then trained to process the original grayscale and the low-resolution color
image to produce a high-resolution colorized image. The colorization of an
individual pixel is determined by the color of the previous pixels.

PixelCNN is composed of two convolutional layers and three ResNet blocks [28].
The first convolutional layer uses kernels of size $7\times 7$ while the last
one employs $3\times 3$. Similarly, ResNet blocks contain 3, 4, and 23
convolutional layers, respectively. The PixelCNN colorization network is
composed of three masked convolutional layers: one in the beginning and second
at the end of the network, whereas a Gated convolutional Block with ten layers
is surrounded by the Gated convolutional layers.

The PixelCNN model is trained by applying a maximum likelihood function with
cross-entropy loss. Then, the refinement network - composed of 16
convolutional layers followed by two bilinear upsampling layers, each with two
internal convolutional layers - is trained on the ground-truth chroma images
downsampled to $28\times 28$. The network ends with three convolutional
layers, where the final layer outputs the colorized image.

|
---|---
Let there be color | MHC
|
Learning Automatic Colorization | Pixelated
|
Pixel Level Segmentation | ColorCapsNet
PixColor
Figure 7: Examples of multi-path networks. These colorization networks learn
different features via several network paths.

####  2.6.4 ColorCapsNet

Colorize Capsule Network (ColorCapsNet) [73] is built upon CapsNet [74] with
three modifications. Firstly, the single convolutional layer is replaced by
the first two convolutional layers of VGG19 [26] and initialized via its
weights. Secondly, Batch normalization [19] is inserted in between the first
two convolutional layers. Thirdly, the number of capsules is reduced from ten
to six in the capsule layer. The architecture of ColorCapsNet is similar to an
autoencoder. The colorization specific hidden variables are in between the
encoder and decoder, are processed in the latent space.

To train the model, the input RGB image is first converted into the CIE Lab
colorspace, and extracted patches are fed to the network in order to learn the
color distribution. The output is in the form of colorized patches which are
combined to obtain the complete image in Lab colorspace and later converted to
RGB colorspace.

The difference between real and generated images is minimized using the Mean
Squared Error loss. ColorCapsNet is trained on ILSVRC 2012 to learn the
general color distribution of objects and then be finetuned using the DIV2K
dataset [75]. ColorCapsNet shows comparable performance to other models,
despite its shallow architecture. Adam is used as the optimizer with a
learning rate of 0.001 with different kernel sizes.

####  2.6.5 Pixelated

Pixelated, introduced by Zhao et al. [76] is an image colorization model
guided by pixelated semantics to keep the colorization consistent across
multiple object categories. The network architecture is composed of a color
embedding branch and a semantic segmentation branch. The network is built from
gated residual blocks, each of which contains two convolutional layers, a skip
connection, and a gating mechanism.

The learning mechanism is composed of three components. _Firstly_ , an
autoregressive model, is adopted to utilize pixelated semantics for image
colorization. A shared CNN is used for modeling to achieve colorization
diversity by generating per-pixel distributions using a conditional Pixel-CNN.
_Secondly_ , semantic segmentation is incorporated into color embedding by
introducing atrous spatial pyramid pooling at the top layer capable of
extracting multi-scale features using several parallel filters. The output is
obtained by fusing multi-scale features. The loss function for semantic
segmentation is the cross-entropy loss with softmax function. _Thirdly_ , to
produce better and more accurate color distributions, pixelated color
embedding is concatenated with semantic segmentation to create the semantic
generator.

During training, color embedding loss, semantic loss, and color generation
loss are combined. The performance of the network is evaluated on the Pascal
VOC2012 [77] and COCO-stuff [50] datasets. The images are rescaled to
128$\times$128 for use in the network.

####  2.6.6 Multiple Hypothesis Colorization

Mohammad & Lorenzo [78] developed a multiple hypothesis colorization
architecture, producing multiple color values for each pixel of the grayscale
image. The low-cost colorization is achieved by storing the best pixel-level
hypothesis. The shared features are computed by a common trunk consisting of
convolutional layers. The trunk is then split into multiple branches where
each branch predicts color for each pixel. The layers in the main trunk and
its subsequent branches are all fully convolutional.

The authors developed two architectures, one for CIFAR100 and the other for
ImageNet [23]. The architecture for the CIFAR100 dataset is composed of 31
blocks preceded by a stand-alone convolutional layer. The number of channels
in different layers across multiple blocks varies from 64 to 256. In contrast,
the architecture for the ImageNet [23] is composed of 21 blocks in total. The
number of channels across different layers inside blocks varies from 32 to
1024. Although the two models have a different number of blocks, the block
structure is the same. Residual connections are also used in both networks.
Moreover, batch normalization is performed in all blocks, and ReLU is applied
to all layers, except the last of each block.

Softmax is employed as a loss function. The CIFAR100 model is trained for 40k
iterations with a learning rate of 0.001, which is dropped after 10k and 20k
iterations by a factor of 10. Similarly, the ImageNet [23] model is trained
for 120k iterations with an initial learning rate of 0.0005 that is dropped
after 40k and 80k iterations by a factor of 5.

####  2.6.7 Pixel-Level Semantics Guided Image Colorization

A hierarchical network comprised of a semantic segmentation branch and a
colorization branch was developed by Zhao et al. [79]. The first four
convolution layers are shared between the two branches to learn low-level
features. Four more convolutional layers further extend the colorization
branch, while three deconvolution layers from the segmentation branch. The
output of the deconvolutional layers is concatenated and passed through the
final convolutional layer to produce class probabilities.

Semantic segmentation is achieved through weighted cross-entropy loss and
softmax function, whereas colorization is performed via multinomial cross-
entropy loss with class rebalance. During the training, the network jointly
optimizes the two tasks. During the testing, a joint bilateral upsampling
layer is introduced to generate the final output.

For semantic segmentation, the model is trained on 10,582 and tested on 1449
images from the PASCAL VOC2012 dataset for 40 epochs. Similarly, 9k images are
used for training and 1k images for testing on the COCO-stuff dataset [50] for
20 epochs.

###  2.7 Exemplar-based Colorization

Exemplar-based colorization utilizes the colors of example images provided
along with input grayscale images. Figure 8 presents the example networks for
exemplar-based colorization. We provide more details about the networks in the
subsequent sub-sections.

####  2.7.1 Deep Exemplar-based Colorization

Deep exemplar-based colorization181818https://github.com/msracver/Deep-
Exemplar-based-Colorization [80] transfers the colors from a reference image
to the grayscale one. The aim here is not to colorize images naturally but to
provide diverse colors to the same image. The system is composed of two
subnetworks: Similarity subnetwork and Colorization sub-network.

The similarity subnet takes the target and reference luminance channels
aligned before via Deep Image Analogy [81]. The authors use the standard
features of VGG19 [26] after each of its blocks resulting in coarse to fine,
five levels of feature maps. The features are upsampled to the same size. The
similarity subnet computes a bidirectional similarity map using discrete
cosine distance. Furthermore, the colorization subnet concatenates the
grayscale image, chrominance channels, and the calculated similarity maps.
U-Net [36] inspires the structure of the colorization subnet.

The loss function is $\ell_{2}$, which is a combination of chrominance
channels and perceptual loss. The network is trained by employing ADAM
optimizer [21] in the Caffe framework [20] with a learning rate of 10-3 for
ten epochs, reduced by 0.1 after 33% of training.

|
---|---
Deep Exemplar Colorization | Fast Deep Exemplar Colorization
Instance Colorization
Figure 8: Exemplar-based Colorization networks. These networks imitate the
colors of the input example image provided along with the grayscale image.

####  2.7.2 Fast Deep Exemplar Colorization

Current exemplar-based colorization methods suffer from two challenges: 1)
they are sensitive to the selection of reference images, and 2) they have high
time and resource consumption. Inspired by stylization characteristics in
feature extracting and blending, Xu et al. [82] proposed a stylization-based
architecture for fast deep exemplar colorization. The proposed architecture
consists of two parts: a transfer sub-net that learns a coarse chrominance map
(ab map in CIELab color space) and a colorization sub-net that refines the map
to generate the final colored result. The proposed method aims to generate
plausible colorization results in real-time, whether the input and exemplar
image are semantically related or not. More specifically, an encoder-decoder
structure is used for the transfer sub-net, which takes the target-reference
image pairs as the input and output initial ab map. The pre-trained VGG19
module (from $conv1_{1}$ layer to $conv4_{1}$ layer) is treated as the encoder
and a symmetrical decoder for image reconstruction. In addition, the fast
Adaptive Instance Normalization (AdaIN) [83] is utilized after convolutional
layers to accelerate feature matching and blending. The ab map generated by
the transfer sub-net is inaccurate and has some artifacts. To refine it, a
colorization sub-net that adopts an analogous U-Net structure is designed,
which takes a known luminance map along with initial chrominance ab map as
input.

In the original implementations, the transfer sub-net is trained on the
Microsoft COCO dataset [50] by minimizing the sum of the L2 loss while the
colorization sub-net is trained on the ImageNet dataset [23] by minimizing the
Huber loss [84].

####  2.7.3 Instance-Aware Image Colorization

The Existing colorization models usually fail to colorize the images with
multiple objects. To solve this issue, Su et al. [85] proposed an instance-
aware image colorization method191919Code is available at
https://cgv.cs.nthu.edu.tw/projects/instaColorization. The network includes
three parts: 1) an off-the-shelf pre-trained model to detect object instances
and produce cropped object images, 2) two backbone networks trained end-to-end
for instance, and full-image colorization and 3) a fusion module to
selectively blend features extracted from different layers of the two
colorization networks. Specifically, a grayscale image as input is fed to the
network. The network first detects the object bounding boxes using an off-the-
shelf object detector. Then, each detected instance is cropped out and
forwarded to the instance colorization network, for instance, colorization. At
the same time, the input grayscale image is also sent to another instance
colorization network (with same structure, but different weights) for full-
image colorization. Finally, a fusion module is used to fuse all the instance
features with the full-image feature at each layer until obtaining the output
colored image. In the training phase, a sequential approach strategy that
trains the full-image network followed by the instance network, and finally
trains the feature fusion module by freezing the above two networks is
adopted. The smooth-L1 loss is used to train the network.

In the original implementation, the channel numbers of the full-image feature,
instance feature, and fused feature in each of the 13 layers are 64, 128, 256,
512, 512, 512, 512, 256, 256, 128, 128, 128, and 128. The authors use three
datasets for training and evaluation, including ImageNet [23], COCO-Stuff
[86], and Places205 [87]. Moreover, the images are resized to a size of 256
$\times$ 256.

##  3 Experiments

###  3.1 Datasets

The datasets available for evaluation are the most commonly used ones in the
literature for other tasks such as detection, classification, segmentation
etc. Where the images are first converted to grayscale, and then apply
colorization models to analyze its performance. Hence, we provide a new
dataset, explicitly designed for the colorization task in the next section,
while We list the currently used datasets below.

|  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |
Figure 9: Sample images from our new proposed Natural-Color dataset (NCD).

  * •

COCO-stuff dataset: The Common Objects in COntext-stuff (COCO-stuff) [86] is
constructed by annotating the original COCO dataset [50], which originally
annotated things while neglecting stuff annotations. There are 164k images in
COCO-stuff dataset that span over 172 categories including 80 things, 91
stuff, and 1 unlabeled class.

PASCAL VOC dataset: PASCAL Visual Object Classes (PASCAL VOC) dataset [88] has
more than 11000 images that are divided into 20 object categories.

  * •

CIFAR datasets: CIFAR-10 and CIFAR-100 [59] are two subsets created and
reliably labelled from 80 million tiny image dataset [89]. CIFAR-10 is
comprised of 60k images equally distributed over mutually exclusive 10
categories with 6k images in each category. On the other hand, CIFAR-100 has
the same images distributed over 100 categories with 600 images assigned to
each category. Each image in both the subsets is of size $32\times 32$ pixels.
In CIFAR-100, two level labelling is used. At the higher level there are 20
superclasses each of which is further divided into five subclasses. Overall,
50k and 1k images comprise training and testing sets, respectively.

  * •

ImageNet ILSVRC2012 [23]: This dataset contains 1.2 million high resolution
training images spanning over 1k categories where 50k images comprise the
hold-out validation set. Images are rescaled to $128\times 128$ pixels.

  * •

Palette-and-Text dataset [51]: is constructed by making modifications to the
data collected from color-hex.com where users upload user-defined color
palettes with label names of their choice. The authors first collected 47,665
palette-text pairs and removed non-alphanumeric and non-English words from the
collection. After removing text-palette pairs that lack semantic
relationships, the final curated dataset contains 10,183 textual phrases with
their corresponding five-color palettes.

###  3.2 Evaluation Metrics

The metrics typically used to assess colorization quality are either
subjective or commonly used mathematical metrics such as PSNR and SSIM.
Subjective evaluation is the gold standard for many applications where humans
determine the accuracy of the output of the algorithms.

In colorization, although subjective evaluation is used to some extent, it has
two limitations: 1) scaling is exceptionally challenging, 2) accurately
determining the color is also very difficult. On the other hand, the
mathematical metrics are general and may not provide an accurate performance
of the algorithms. We provide more insights into the metrics for colorization
in the “Future Directions” section.

##  4 Limitations and Future Directions

Lack of Appropriate Evaluation Metrics: As mentioned earlier, colorization
papers typically employ metrics from image restoration tasks, which may not be
appropriate for the task at hand. We also propose that instead of comparing
YUV, only the predicted color channels i.e. U-channel and V-Channel should be
compared. Similarly, it will be highly sought after to use metrics
specifically designed to take color into account, such as PCQI [90] and UIQM
[91].

PCQI stands for Patch-based contrast quality index. It is a patch-based
evaluation metric. It considers three statistics of a patch i.e. mean
intensity ($p_{m}$), signal strength or contrast change ($p_{c}$), and
structure distortion ($p_{s}$) for comparison with ground-truth. It can be
expressed as

$PCQI=p_{m}(x,y)\textperiodcentered p_{c}(x,y)\textperiodcentered p_{s}(x,y).$
(3)

Similarly, UIQM is the abbreviation of underwater image quality measure. It is
different from earlier defined evaluation metrics as it does not require a
reference image. Like PCQI, it also depends on three measures i.e. image
colorfulness, image sharpness, and image contrast. It can be formulated as:

$IQM=w_{1}(ICM)+w_{2}(ISM)+w_{3}(IConM),$ (4)

where ICM, ISM, and IConM stand for image colorfulness, image sharpness and
image contrast, respectively while “$w$” controls the weight of each quantity.

Lack of Benchmark Dataset: Image colorization techniques are typically
evaluated on grayscale images from various datasets available in the
literature due to the absence of purposefully built colorization datasets. The
available datasets were originally collected for tasks such as detection,
classification, etc. contrary to the image colorization. The quality of the
images may not be sufficient for image colorization. Moreover, the datasets
contain objects, such as buses, shirts, doors, etc., that can take any color.
Hence such an evaluation environment is not appropriate for fair comparison in
terms of PSNR or SSIM. We aim to remove this unrealistic setting for image
colorization by collecting images that are true to their colors. For example,
a carrot will have an orange color in most images. Bananas will be either
greenish or yellowish. We have collected _723_ images from the internet
distributed in _20_ categories. Each image has an object and a white
background. Our benchmark outlines a realistic evaluation scenario that
differs sharply from those generally employed by the image colorization
techniques. Figure 9 shows the images from each category from our Natural
Color Dataset (NCData).

Lacking of Competitions: Currently, most vision tasks have competitions held
across different top-tier conferences e.g. (CVPR, ECCV workshops such as
NTIRE, PBVS, etc.) and online submission platforms, such as Kaggle. These
competitions help push state-of-the-art. Unfortunately, there is no such
arrangement for image colorization. Making competitions a regular feature in
top-tier conferences and online venues would thus be a drastic step forward
for image colorization.

Limited Availability of Open-Source Codes: Open-source code plays a vital role
in the advancement of the research fields, as can be seen in classification
[28], image super-resolution [92], image denoising [93] etc. In image
colorization, open-source codes are rare or codes are obsolete now as they
were built in earlier CNN frameworks. In other fields of research, the codes
are either recycled or re-implemented by volunteers in the corresponding field
to the new frameworks and environments. However, this is not currently being
done for image colorization, hindering progress.

##  5 Conclusion

Single-image colorization is a research problem with critical real-life
applications. Deep learning approaches’ exceptional success has resulted in
rapid growth in deep convolutional techniques for image colorization. Based on
exciting innovations, various methods are proposed exploiting network
structures, training methods, and learning paradigms etc. This article
presents a thorough review of deep learning methods for image colorization.

We observe image colorization performance has improved in recent years at the
cost of increasing the network complexity. However, state-of-the-art methods
application to critical real-world scenarios is restricted due to inadequate
metrics, network complexity, and failure to handle real-life degradations.

We observe the following trends in image colorization. 1) GAN-based methods
deliver diverse colorization visually compared to CNN-based methods, 2) the
existing models generally deliver a sub-optimal result for complex scenes
having a large number of objects with small sizes, 3) deep models with higher
complexity have little improvement in terms of numbers, 4) the diversity of
networks in image colorization as compared to other image restoration is
significant, 5) a future direction for image colorization is unsupervised
learning, 6) many recent advancements and techniques such as attention
mechanism, and loss functions can be incorporated for performance. We believe
this article and novel dataset will attract further attempts to resolve the
mentioned crucial problems.

## References

  * [1] Z. Cheng, Q. Yang, and B. Sheng, “Deep colorization,” in _IEEE International Conference on Computer Vision_ , 2015, pp. 415–423.
  * [2] S. Yoo, H. Bahng, S. Chung, J. Lee, J. Chang, and J. Choo, “Coloring with limited data: Few-shot colorization via memory augmented networks,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 283–11 292.
  * [3] T. Welsh, M. Ashikhmin, and K. Mueller, “Transferring color to greyscale images,” in _29th annual conference on Computer graphics and interactive techniques_ , 2002, pp. 277–280.
  * [4] A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” in _Proceedings of International Conference on Computer Graphics and Interactive Techniques’ ACM_ , 2004, pp. 689–694.
  * [5] Y.-C. Huang, Y.-S. Tung, J.-C. Chen, S.-W. Wang, and J.-L. Wu, “An adaptive edge detection based colorization algorithm and its applications,” in _13th annual ACM international conference on Multimedia_ , 2005, pp. 351–354.
  * [6] Y. Qu, T.-T. Wong, and P.-A. Heng, “Manga colorization,” _ACM Transactions on Graphics (TOG)_ , vol. 25, no. 3, pp. 1214–1220, 2006.
  * [7] L. Yatziv and G. Sapiro, “Fast image and video colorization using chrominance blending,” _IEEE transactions on image processing_ , vol. 15, no. 5, pp. 1120–1129, 2006.
  * [8] Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, and H.-Y. Shum, “Natural image colorization,” in _18th Eurographics conference on Rendering Techniques_ , 2007, pp. 309–320.
  * [9] T. Pärnamaa and L. Parts, “Accurate classification of protein subcellular localization from high-throughput microscopy images using deep learning,” _G3: Genes, Genomes, Genetics_ , vol. 7, no. 5, pp. 1385–1392, 2017.
  * [10] M. Xiao, X. Shen, and W. Pan, “Application of deep convolutional neural networks in classification of protein subcellular localization with microscopy images,” _Genetic epidemiology_ , 2019.
  * [11] T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent trends in deep learning based natural language processing [review article],” _IEEE Computational Intelligence Magazine_ , vol. 13, no. 3, pp. 55–75, 2018\.
  * [12] Z. Zhao, P. Zheng, S. Xu, and X. Wu, “Object detection with deep learning: A review,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 30, no. 11, pp. 3212–3232, 2019.
  * [13] S. Anwar, S. Khan, and N. Barnes, “A deep journey into super-resolution: A survey,” _ACM Comput. Surv._ , vol. 53, no. 3, May 2020. [Online]. Available: https://doi.org/10.1145/3390462
  * [14] N. Aafaq, A. Mian, W. Liu, S. Z. Gilani, and M. Shah, “Video description: A survey of methods, datasets, and evaluation metrics,” _ACM Comput. Surv._ , vol. 52, no. 6, Oct. 2019. [Online]. Available: https://doi.org/10.1145/3355390
  * [15] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama, “Digital photography with flash and no-flash image pairs,” _ACM transactions on graphics (TOG)_ , vol. 23, no. 3, pp. 664–672, 2004.
  * [16] E. Tola, V. Lepetit, and P. Fua, “A fast local descriptor for dense matching,” in _2008 IEEE conference on computer vision and pattern recognition_. IEEE, 2008, pp. 1–8.
  * [17] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 2010, pp. 3485–3492.
  * [18] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in _European conference on computer vision_. Springer, 2016, pp. 649–666.
  * [19] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International Conference on Machine Learning_ , 2015, pp. 448–456.
  * [20] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in _22nd ACM international conference on Multimedia_ , 2014, pp. 675–678.
  * [21] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [22] F. M. Carlucci, P. Russo, and B. Caputo, “(DE)2 CO: Deep depth colorization,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 2386–2393, 2018.
  * [23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [24] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. icml_ , vol. 30, no. 1, 2013, p. 3.
  * [25] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [26] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _International Conference on Learning Representations_ , 2015.
  * [27] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1–9.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [29] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” in _2011 IEEE international conference on robotics and automation_. IEEE, 2011, pp. 1817–1824.
  * [30] C. Li, A. Reiter, and G. D. Hager, “Beyond spatial pooling: fine-grained representation learning in multiple domains,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 4913–4922.
  * [31] A. Singh, J. Sha, K. S. Narayan, T. Achim, and P. Abbeel, “Bigbird: A large-scale 3d database of object instances,” in _2014 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2014, pp. 509–516.
  * [32] P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays, “Scribbler: Controlling deep image synthesis with sketch and color,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5400–5409.
  * [33] Y. Güçlütürk, U. Güçlü, R. van Lier, and M. A. van Gerven, “Convolutional sketch inversion,” in _European Conference on Computer Vision_. Springer, 2016, pp. 810–824.
  * [34] R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, pp. 1–11, 2017.
  * [35] Y. Xiao, P. Zhou, Y. Zheng, and C.-S. Leung, “Interactive deep colorization using simultaneous global and local inputs,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 1887–1891.
  * [36] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [37] B. Zhou, A. Lapedriza, A. Torralba, and A. Oliva, “Places: An image database for deep scene understanding,” _Journal of Vision_ , vol. 17, no. 10, pp. 296–296, 2017.
  * [38] Y. Ci, X. Ma, Z. Wang, H. Li, and Z. Luo, “User-guided deep anime line art colorization with conditional adversarial networks,” in _26th ACM international conference on Multimedia_ , 2018, pp. 1536–1544.
  * [39] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _In Advances in Neural Information Processing Systems_ , p. s 2672–2680, 2014.
  * [40] M. Saito and Y. Matsui, “Illustration2vec: a semantic vector representation of illustrations,” in _SIGGRAPH Asia 2015 Technical Briefs_ , 2015, pp. 1–4.
  * [41] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1492–1500.
  * [42] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.
  * [43] M. Limmer and H. P. Lensch, “Infrared colorization using deep convolutional neural networks,” in _2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2016, pp. 61–68.
  * [44] P. Wang and V. M. Patel, “Generating high quality visible images from sar images using cnns,” in _2018 IEEE Radar Conference (RadarConf18)_. IEEE, 2018, pp. 0570–0575.
  * [45] Q. Song, F. Xu, and Y.-Q. Jin, “Radar image colorization: Converting single-polarization to fully polarimetric using deep neural networks,” _IEEE Access_ , vol. 6, pp. 1647–1661, 2017.
  * [46] L. Junsoo, K. Eungyeup, L. Yunsung, K. Dongjun, C. Jaehyuk, and C. Jaegul, “Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence,” in _IEEE conference on computer vision and pattern recognition_ , 2020, pp. 5801–5810.
  * [47] V. Manjunatha, M. Iyyer, J. Boyd-Graber, and L. Davis, “Learning to color from language,” in _2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , 2018, pp. 764–769.
  * [48] S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, “Learning what and where to draw,” in _Advances in neural information processing systems_ , 2016, pp. 217–225.
  * [49] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [50] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [51] H. Bahng, S. Yoo, W. Cho, D. Keetae Park, Z. Wu, X. Ma, and J. Choo, “Coloring with words: Guiding image colorization through text-based palette generation,” in _IEEE European Conference on Computer Vision (ECCV)_ , 2018, pp. 431–447.
  * [52] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _In: Proc. the InternationalConference on Learning Representations (ICLR)_ , 2015.
  * [53] Y. Cao, Z. Zhou, W. Zhang, and Y. Yu, “Unsupervised diverse colorization via generative adversarial networks,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2017, pp. 151–166.
  * [54] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” _arXiv preprint arXiv:1506.03365_ , 2015.
  * [55] K. Frans, “Outline colorization through tandem adversarial networks,” _arXiv preprint arXiv:1704.08834_ , 2017.
  * [56] K. Nazeri, E. Ng, and M. Ebrahimi, “Image colorization using generative adversarial networks,” in _International Conference on Articulated Motion and Deformable Objects_. Springer, 2018, pp. 85–94.
  * [57] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [58] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _IEEE international conference on computer vision_ , 2015, pp. 1026–1034.
  * [59] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [60] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva, “Places: An image database for deep scene understanding,” _arXiv preprint arXiv:1610.02055_ , 2016.
  * [61] A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth, “Learning diverse image colorization,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6837–6845.
  * [62] P. Vitoria, L. Raad, and C. Ballester, “Chromagan: Adversarial picture colorization with semantic class distribution,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 2445–2454.
  * [63] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [64] Ł. Kaiser, O. Nachum, A. Roy, and S. Bengio, “Learning to remember rare events,” _International Conference on Learning Representations_ , 2017.
  * [65] M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_. IEEE, 2008, pp. 722–729.
  * [66] P. Docter, J. Culton, J. Pidgeon, and R. Eggleston, “Monsters, inc.” Walt Disney Pictures, 2001.
  * [67] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” _ACM Transactions on Graphics (ToG)_ , vol. 35, no. 4, pp. 1–11, 2016.
  * [68] M. D. Zeiler, “Adadelta: an adaptive learning rate method,” _arXiv preprint arXiv:1212.5701_ , 2012.
  * [69] L. Bottou, “Stochastic gradient learning in neural networks,” _Proceedings of Neuro-Nımes_ , vol. 91, no. 8, p. 12, 1991.
  * [70] R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A matlab-like environment for machine learning,” in _BigLearn, NIPS workshop_ , no. CONF, 2011.
  * [71] G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in _European Conference on Computer Vision_. Springer, 2016, pp. 577–593.
  * [72] S. Guadarrama, R. Dahl, D. Bieber, M. Norouzi, J. Shlens, and K. Murphy, “Pixcolor: Pixel recursive colorization,” _28th British Machine Vision Conference (BMVC)_ , 2017. [Online]. Available: https://arxiv.org/abs/1705.07208
  * [73] G. Ozbulak, “Image colorization by capsule networks,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [74] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” in _Advances in neural information processing systems_ , 2017, pp. 3856–3866.
  * [75] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 126–135.
  * [76] J. Zhao, J. Han, L. Shao, and C. G. Snoek, “Pixelated semantic colorization,” _International Journal of Computer Vision_ , pp. 1–17, 2019.
  * [77] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [78] M. H. Baig and L. Torresani, “Multiple hypothesis colorization and its application to image compression,” _Computer Vision and Image Understanding_ , vol. 164, pp. 111–123, 2017.
  * [79] J. Zhao, L. Liu, C. Snoek, J. Han, and L. Shao, “Pixel-level semantics guided image colorization,” in _British Machine Vision Conference 2018, BMVC 2018, Northumbria University, Newcastle, UK, September 3-6, 2018_. BMVA Press, 2018, p. 156. [Online]. Available: http://bmvc2018.org/contents/papers/0236.pdf
  * [80] M. He, D. Chen, J. Liao, P. V. Sander, and L. Yuan, “Deep exemplar-based colorization,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 4, p. 47, 2018.
  * [81] J. Liao, Y. Yao, L. Yuan, G. Hua, and S. B. Kang, “Visual attribute transfer through deep image analogy,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, pp. 1–15, 2017.
  * [82] X. Zhongyou, W. Tingting, F. Faming, S. Yun, and Z. Guixu, “Stylization-based architecture for fast deep exemplar colorization,” in _IEEE conference on computer vision and pattern recognition_ , 2020, pp. 9363–9372.
  * [83] H. Xun and B. Serge, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _ICCV_ , 2017, pp. 1501–1510.
  * [84] H. Peter, “Robust estimation of a location parameter,” in _Breakthroughs in statistics_ , 1992, pp. 492–518.
  * [85] S. Jheng-Wei, C. Hung-Kuo, and H. Jia-Bin, “Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence,” in _IEEE conference on computer vision and pattern recognition_ , 2020, pp. 7968–7977.
  * [86] H. Caesar, J. Uijlings, and V. Ferrari, “Coco-stuff: Thing and stuff classes in context,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1209–1218.
  * [87] Z. Bolei, L. Agata, X. Jianxiong, T. Antonio, and O. Aude, “Learning deep features for scene recognition using places dataset,” in _Advances in neural information processing systems_ , 2014, pp. 487–495.
  * [88] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , vol. 111, no. 1, pp. 98–136, 2015.
  * [89] A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 30, no. 11, pp. 1958–1970, 2008.
  * [90] S. Wang, K. Ma, H. Yeganeh, Z. Wang, and W. Lin, “A patch-structure representation method for quality assessment of contrast changed images,” _IEEE Signal Processing Letters_ , vol. 22, no. 12, pp. 2387–2390, 2015.
  * [91] K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired underwater image quality measures,” _IEEE Journal of Oceanic Engineering_ , vol. 41, no. 3, pp. 541–551, 2015.
  * [92] S. Anwar and N. Barnes, “Densely residual laplacian super-resolution,” _arXiv preprint arXiv:1906.12021_ , 2019.
  * [93] ——, “Real image denoising with feature attention,” in _IEEE International Conference on Computer Vision_ , 2019, pp. 3155–3164.
  * [94] G. Larsson, M. Maire, and G. Shakhnarovich, “Colorization as a proxy task for visual understanding,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6874–6883.
  * [95] W. Jang, “Sampling using neural networks for colorizing the grayscale images,” _arXiv preprint arXiv:1812.10650_ , 2018.
  * [96] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [97] H. Kim, H. Y. Jhoo, E. Park, and S. Yoo, “Tag2pix: Line art colorization using text tag with secat and changing loss,” in _IEEE International Conference on Computer Vision_ , 2019, pp. 9056–9065.