# A Further Study of Linux Kernel Hugepages on A64FX with FLASH, an
Astrophysical Simulation Code

Catherine Feldman<EMAIL_ADDRESS>,  Smeet Chheda
<EMAIL_ADDRESS>,  Alan C. Calder
<EMAIL_ADDRESS>,  Eva Siegmann<EMAIL_ADDRESS>,
John Dey<EMAIL_ADDRESS>,  Tony Curtis
<EMAIL_ADDRESS>and  Robert J. Harrison
<EMAIL_ADDRESS>Institute for Advanced Computational
ScienceStony Brook UniversityStony BrookNew YorkUSA11794-5250

(2023)

###### Abstract.

We present an expanded study of the performance of FLASH when using Linux
Kernel Hugepages on Ookami, an HPE Apollo 80 A64FX platform. FLASH is a multi-
scale, multi-physics simulation code written principally in modern Fortran and
makes use of the PARAMESH library to manage a block-structured adaptive mesh.
Our initial study used only the Fujitsu compiler to utilize standard hugepages
(hp), but further investigation allowed us to utilize hp for multiple
compilers by linking to the Fujitsu library libmpg and transparent hugepages
(thp) by enabling it at the node level. By comparing the results of hardware
counters and in-code timers, we found that hp and thp do not significantly
impact the runtime performance of FLASH. Interestingly, there is a significant
reduction in the TLB misses, differences in cache and memory access counters,
and strange behavior is observed when using thp.

high performance computing, A64FX architecture, astrophysics

††journalyear: 2023††copyright: rightsretained††doi:
10.1145/3569951.3597583††conference: Practice and Experience in Advanced
Research Computing; July 23–27, 2023; Portland, OR, USA††booktitle: Practice
and Experience in Advanced Research Computing (PEARC ’23), July 23–27, 2023,
Portland, OR, USA††isbn: 978-1-4503-9985-2/23/07††ccs: Computer systems
organization Parallel architectures††ccs: Computing methodologies Modeling and
simulation††ccs: Computing methodologies Massively parallel and high-
performance simulations††ccs: Applied computing Physics††ccs: Applied
computing Astronomy

##  1\. Introduction

###  1.1. Ookami and A64FX

The A64FX processor expects to provide high performance and reliability for
memory-intensive applications while maintaining a good performance to power
ratio. The appeal of A64FX, currently the backbone of the Fugaku
supercomputer, is that it eliminates the need to port to accelerators such as
GPUs to improve performance. Ookami is an open-access resource featuring
Fujitsu A64FX processors provided under the US NSF’s ACCESS program and
managed jointly by Stony Brook University and the University at Buffalo.
Ookami is an HPE/Cray Apollo80 system with 176 A64FX Fujitsu compute nodes,
each with 32GB high-bandwidth memory (HBM) and a 512GB SSD. Ookami’s FX700
series A64FX processors consist of four core memory groups each with 12 cores,
resulting in a total of 48 cores, 64KB L1 cache per core, and 8MB L2 cache
shared between the cores and runs at 1.8 GHz. The nodes have 32 GB of high-
bandwidth memory, where 5 GB are reserved for the OS, leaving 27 GB for the
user. These processors use the ARMv8.2–A Scalable Vector Extension (SVE) SIMD
instruction set with a 512 bit vector implementation, allowing for vector
lengths anywhere from 128–2048 bits and enabling vector length agnostic
programming (Burford et al., 2021).

###  1.2. Thermonuclear Supernovae with FLASH

Our application is a bright stellar explosion known as a thermonuclear (Type
Ia) supernova (SN Ia), which we model using FLASH, a software instrument for
addressing multi-scale, multi-physics applications (Fryxell et al., 2000).
FLASH is written in modern Fortran, parallelized through MPI, and implements
AMR (Adaptive Mesh Refinement) using the PARAMESH library. Full-star
hydrodynamics simulations such as these are memory and computationally
intensive, making our application a good candidate to try on A64FX. Early
study of the performance of FLASH on Ookami may be found in (Feldman et al.,
2022), and similar experiences are reported in (Bari et al., 2021; Domke,
2021). The unoptimized performance on A64FX did not compare well to that found
on traditional X86 architectures (Burford et al., 2021).

Profiling indicated that FLASH spent about half of its time in the
hydrodynamics routines, and within those 20% of the time was spent in routine
for the material equation of state (EOS) (Feldman et al., 2022). We therefore
settled on two test problems for further exploration: a 2-d SN Ia problem
(that exercises the material EOS) and, looking ahead to our science goal of
3-d SN Ia simulations, a 3-d hydrodynamics simulation, the Sedov explosion
problem. We dubbed these two tests “EOS” and “3-d Hydro”, and details of both
the EOS and hydrodynamics modules may be found in the original FLASH paper
(Fryxell et al., 2000).

Our motivation for investigating huge memory pages was both the observed
bountiful DTLB misses, and FLASH’s memory stride. PARAMESH manages a block-
structured adaptive mesh, where each block is separated into smaller cells
that each store requisite variables, such as density and temperature,
consecutively in an array. Thus there is a stride in memory when gathering the
same variable (i.e. density) from different cells, and a larger stride between
blocks.

###  1.3. Previous Work with Hugepages

Here, we explore both standard and transparent hugepages. Modern processors
manage memory in blocks known as pages. Hugepage support was integrated into
the Linux kernel in version 2.6. These pages are larger in size than regular
pages, which in theory means there are fewer pages for the OS to manage as
there is a finite amount of memory. Depending on the OS, hugepages come in
different sizes. Managing these pages can be challenging and at times require
changes to application code. To that extent, Transparent HugePages were
implemented in the Linux kernel where the the ”transparent” hugepages are an
abstraction layer managed by the kernel, where the kernel is responsible for
their creation, management and use (THP, 2022). Transparent hugepages are by
default disabled on Ookami.

Other studies that have tested the performance effects of using hugepages on
A64FX include (Domke, 2021), (Langarita et al., 2023), and (Alappat et al.,
2022), and suggest certain environment variable settings for best results.
(Langarita et al., 2023) explicitly shows that the greatest speedup gain from
enabling hugepages is seen for a latency-bound section of their simulation,
but is only a 1.11 $\times$ speedup. (Alappat et al., 2022) found that an
increase in L2 TLB misses caused performance degradation when using normal 64
KiB pages, but didn’t affect the performance when using 2 MiB hugepages.

This work extends our initial study of using hugepages with just the Fujitsu
compiler, which demonstrated that hugepages did not provide a significant
speedup (Calder et al., 2022). Our speculation was that TLB misses might not
make much of a difference because the A64FX has hardware to ameliorate the
cost of TLB misses by avoiding OS calls, or because the FLASH data access
patterns do not trigger a performance penalty.

##  2\. Testing Use of Hugepages

We ran the “EOS” and “3-d Hydro” test problems, as described above. The EOS
test ran a $\sim$ 1 GB 2-d SN Ia simulation for 50 time steps and the 3-d
Hydro test ran a $\sim$ 9 GB Sedov explosion simulation for 2 time steps. Both
tests were run on 1 and 12 cores. We used the round robin distribution of
processors for the runs on 12 cores because FLASH Morton orders the blocks to
be spatially located together. Filling one core memory group first will put
blocks together but round robin spreads them as much as possible. We ran each
test 7 times, removed the highest and lowest run times, and averaged the
results from the remaining 5. To investigate the effects of hugepages, we used
the Fujitsu hardware counters (Fujitsu, 2023) of the Performance Application
Programming Interface (PAPI) (pap, 2022) to monitor cycles, TLB misses, and
memory access, and used FLASH’s internal timers to obtain runtimes. Tests
consisted of running the PAPI-instrumented code without hugepages (no hp),
with 2MB standard hugepages (hp), and with 2MB transparent hugepages (thp). To
use (t)hp, we linked the GCC and ARM compilers to Fujitsu’s libmpg library,
and used compiler flags for the Fujitsu compiler. A detailed description of
the runtime environment, including library versions, compiler flags, linking
to PAPI and Fujitsu’s libmpg library, and how to enable/disable (t)hp can be
found in Appendix A.

##  3\. Results

First, we saw how the runtime, main memory bandwidth (MMB), and DTLB miss rate
changed with huge page use. To do this, we used the following PAPI counters by
setting  PAPI_EVENTS to
PERF_COUNT_HW_CPU_CYCLES,PERF_COUNT_HW_CACHE_MISSES,DTLB-LOAD-MISSES. The
results from the 1 processor runs are shown for the EOS test in Figure 1(a),
and for the 3-d hydro in Figure 1(b) – the 12 core runs exhibited similar
patterns and are therefore not shown.

(a) EOS test

(b) 3-d hydro test

Figure 1. Ratios of runs with and without hugepages for each compiler for the (a) EOS test and (b) 3-d hydro test on 1 core Table 1. Counters and derived rates for single core runs, for each test problem and two compilers. Values shown are ratios with : without standard hugepages enabled. Counter descriptions and rate calculations can be found in (Fujitsu, 2023). | EOS |  | 3-d Hydro |
---|---|---|---|---
Description | GCC | Fujitsu | GCC | Fujitsu
DTLB-LOAD-MISSES | 0.03 | 0.06 | 0.11 | 0.31
L1D_TLB_REFILL | 0.03 | 0.05 | 0.11 | 0.31
L2D_TLB_REFILL | 0.0002 | 0.01 | 0.03 | 0.03
L1I_TLB_REFILL | 0.71 | 1.01 | 0.04 | 0.65
L2I_TLB_REFILL | 1.00 | 0.99 | 0.59 | 0.16
L1D_CACHE_REFILL | 0.96 | 0.99 | 1.00 | 1.00
L2D_CACHE_REFILL | 1.08 | 1.06 | 0.96 | 1.03
LD_COMP_WAIT | 0.71 | 0.78 | 1.17 | 0.99
LD_COMP_WAIT_L1_MISS | 0.82 | 0.78 | 0.94 | 1.00
LD_COMP_WAIT_L2_MISS | 0.90 | 0.96 | 0.97 | 0.98
Average latency of L1D cache miss processing | 1.03 | 1.03 | 1.00 | 1.00
Average latency of L2 cache miss processing | 2.53 | 1.00 | 1.03 | 0.96
Bidirectional effective bandwidth between L1D cache and L2 cache | 1.01 | 1.07 | 0.91 | 1.00
Bidirectional effective bandwidth between L2 cache and memory | 1.10 | 1.11 | 0.87 | 1.04

The figures show the ratios of runs with and without (t)hp, e.g. values around
1 indicate no changes, values ¡ 1 indicate a reduction by using (t)hp, and
values ¿ 1 an increase. It is important to note that only a portion of our
code is instrumented with PAPI, namely the EOS calls for the EOS test, and the
hydrodynamics calls for the 3-d hydro test. Therefore, these counters
represent the behavior in that specific module, rather than the software as a
whole, while the timers show the full runtime. As expected and seen in our
last study (Calder et al., 2022), in both cases the hardware cycles, MMB, and
overall runtime are about the same when using hp, thp, or no hp. However,
using hp drastically decreases the DTLB miss rate, while using thp does not
have as much of an effect.

Using thp proved to be an interesting struggle. Thp would not enable in our 1
core runs with the Fujitsu compiler for the EOS test, and is therefore not
shown in Figure 1(a). We finally saw thp usage by mapping the process to NUMA
node 1 instead of NUMA node 0. When running the 3-d hydro application compiled
with GCC on 12 cores, the node would reset in the middle of execution when thp
was enabled. These difficulties using thp will be investigated in the future.

We also observed the change in selected hardware counters and their derived
rates when enabling hp. We found that most of these counters varied by only
around 1%, so we report ratios of counters from a single run rather than an
average as before. A64FX has 6 hardware counters, so these results were
collected across multiple runs. For ease of interpretation, we ran these
exploratory tests on 1 core only. The ratios of hp : no hp for the most
relevant values are shown in Table 1, and full tables showing all measured
counters and rates can be found in Appendix B. As before, values ¡ 1 indicate
a reduction by using hp, and values ¿ 1 an increase.

As expected, the TLB-related counters showed the biggest change. Although the
L2-DTLB showed the greatest improvement when hp was enabled, 99% of the total
DTLB misses resulted in an L1-DTLB miss, and only ¡ 1% resulted in a L2-DTLB
miss. The instruction TLBs were less affected. GCC typically exhibited a
greater decrease in TLB refills than Fujitsu. The runtime, number of L1D and
L2D cache misses, and the bandwidth were relatively unaffected by hp use. For
the EOS test, the number of cycles spent waiting for memory access completion
(LD_COMP_WAIT) is smaller when hp is enabled, but for the GCC compiler, the
latency of L2 cache miss processing is higher. For the 3-d Hydro test with
GCC, enabling hp slightly increased the total number of cpu cycles as well as
(LD_COMP_WAIT). Overall, enabling hp has the overwhelming effect of reducing
TLB misses, but not much else. The Fujitsu compiler seems to have less
prominent changes in its counters than GCC.

Figure 2. Ratios between the Fujitsu and other compilers (GCC and ARM), for each application and type of huge page. Table 2. Counters and derived rates for single core runs, for each test problem with either standard hp or no hp enabled. Values shown are ratios for Fujitsu : GCC compiler. Counter descriptions and rate calculations can be found in (Fujitsu, 2023). | EOS |  | 3-d Hydro |
---|---|---|---|---
Description | Hp | No hp | Hp | No hp
DTLB-LOAD-MISSES | 0.66 | 0.39 | 2.20 | 0.82
L1D_TLB_REFILL | 0.55 | 0.39 | 2.52 | 0.86
L2D_TLB_REFILL | 0.77 | 0.02 | 0.93 | 1.02
L1I_TLB_REFILL | 0.70 | 0.49 | 0.63 | 0.04
L2I_TLB_REFILL | 1.00 | 1.01 | 0.64 | 2.33
L1D_CACHE_REFILL | 0.94 | 0.90 | 0.91 | 0.92
L2D_CACHE_REFILL | 1.04 | 1.06 | 1.08 | 1.00
LD_COMP_WAIT | 0.50 | 0.46 | 0.66 | 0.78
LD_COMP_WAIT_L1_MISS | 0.56 | 0.58 | 2.58 | 2.43
LD_COMP_WAIT_L2_MISS | 0.82 | 0.77 | 2.11 | 2.09
Average latency of L1D cache miss processing | 0.90 | 0.90 | 1.04 | 1.03
Average latency of L2 cache miss processing | 0.25 | 0.64 | 0.89 | 0.94
Bidirectional effective bandwidth between L1D cache and L2 cache | 2.76 | 2.59 | 1.63 | 1.49
Bidirectional effective bandwidth between L2 cache and memory | 2.88 | 2.85 | 1.91 | 1.61

We also compared the single core results between compilers, namely to the
Fujitsu compiler, which by far produced the fastest runtime. Figure 2 shows
the ratio between the Fujitsu and other compilers (purple for GCC, pink for
ARM) for each test problem (darker colors for EOS) and type of hugepage (solid
for no hp, dotted for hp, and striped for thp), using the same dataset as that
from Figures 1(a) and 1(b). Here, values ¡ 1 indicate a reduction due to use
of the Fujitsu compiler, and values ¿ 1 indicate an increase. Regardless of
hugepage use, the Fujitsu compiler was nearly twice as fast as the others, and
nearly four times as fast as ARM for the EOS test. The Fujitsu compiler also
executes about half of the hardware cycles. For the EOS test, the Fujitsu
compiler has a 2.5-3$\times$ greater MMB than the others; this is about
1.5-2$\times$ for 3-d Hydro. This is true even though the Fujitsu compiler
exhibits a higher DTLB miss rate, which interestingly increases with huge page
use. This rate increase says nothing about the relative TLB misses between the
compilers, however, so for a better comparison we look at the ratios between
the raw counter values and derived rates.

Table 2 shows the ratio between the Fujitsu and GCC compilers of a subset of
counters, for each test problem with hp and no hp enabled. We chose to compare
only these two compilers since the ARM compiler is too slow to be a viable
choice for production runs, and we only look at no hp and hp runs because thp
did not even achieve the goal of reducing TLB misses. Again, values ¡ 1
indicate a reduction due to use of the Fujitsu compiler, and values ¿ 1
indicate an increase. The data used is the same as that used to create Table
1, and full tables showing all measured counters and rates can be found in
Appendix B.

Although the Fujitsu compiler has a much higher TLB miss rate than the GCC
compiler in most cases, it has lower total TLB misses. The Fujitsu compiler
also has a higher (1.6-2.9 $\times$) memory bandwidth and lower latency. It
has the same number of cache misses, but spends less total cycles waiting for
memory access than the GCC compiler.

##  4\. Summary and Conclusions

We found that for all compilers and both test problems, the use of both
standard and transparent huge pages did not significantly affect the
performance of FLASH, despite a drastic decrease in TLB misses. This suggests
that TLB misses indeed do not have an impact on the performance. This may be
due to the A64FX’s translation table cache (TTC), which decreases the latency
of virtual to physical address translation (Fujitsu, 2023). Higher cache miss
rates when using the Fujitsu compiler are offset by higher memory bandwidth
and lower latency, which results in a shorter runtime.

The Fujitsu compiler demonstrates 2-4 times better performance than the GCC
and ARM compilers. Although the Fujitsu compiler uses only half the total
cycles of the GCC compiler, both compilers have the same number of cache
misses. Since the bandwidth is $\sim$ 2 $\times$ larger for Fujitsu, this
means that less time is spent waiting for memory access completion (ie in
LD_COMP_WAIT), thereby shortening the runtime. However, only $\sim$ 20 % - 40
% of the total cycles are spent in LD_COMP_WAIT, so a higher bandwidth can’t
completely account for the faster runtime. A contributing factor could be that
Fujitsu may have better optimizations that take advantage of the A64FX
hardware. This includes the use of SVE – the Fujitsu executable uses the SVE
registers 21 $\times$ more than GCC. The reason why Fujitsu produces the
fastest executable, and what the performance bottlenecks are, will be explored
in detail in future work.

###### Acknowledgements.

Ookami is supported by the US NSF grant #1927880, and this research was
supported in part by the US DOE under grant DE-FG02-87ER40317. FLASH was
developed in part by the US DOE NSA-ASC and OSC-ASCR-supported Flash Center
for Computational Science at the University of Chicago. The authors gratefully
acknowledge the generous support of the Ookami community. The authors also
thank Jens Domke at RIKEN for very helpful suggestions.

## References

  * (1)
  * THP (2022) 2022\.  _5.2. Huge Pages and Transparent Huge Pages_.  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-transhuge
  * pap (2022) 2022\.  _Performance Application Programming Interface_.  http://icl.cs.utk.edu/papi/
  * Alappat et al. (2022) Christie Alappat, Nils Meyer, Jan Laukemann, Thomas Gruber, Georg Hager, Gerhard Wellein, and Tilo Wettig. 2022.  Execution-Cache-Memory modeling and performance tuning of sparse matrix-vector multiplication and Lattice quantum chromodynamics on A64FX.  _Concurrency and Computation: Practice and Experience_ 34, 20 (2022), e6512.  https://doi.org/10.1002/cpe.6512 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpe.6512
  * Bari et al. (2021) Md Abdullah Shahneous Bari, Barbara Chapman, Anthony Curtis, Robert J. Harrison, Eva Siegmann, Nikolay A. Simakov, and Matthew D. Jones. 2021.  A64FX performance: experience on Ookami. In _2021 IEEE International Conference on Cluster Computing (CLUSTER)_. 711–718.  https://doi.org/10.1109/Cluster48925.2021.00106
  * Burford et al. (2021) A. Burford, A. Calder, D. Carlson, B. Chapman, F. Coskun, T. Curtis, C. Feldman, R. Harrison, Y. Kang, B. Michalowicz, E. Raut, E. Siegmann, D. Wood, R. DeLeon, M. Jones, N. Simakov, J. White, and D. Oryspayev. 2021\.  Ookami: Deployment and Initial Experiences. In _Practice and Experience in Advanced Research Computing_ (Boston, MA, USA) _(PEARC ’21)_. Association for Computing Machinery, New York, NY, USA, Article 9, 8 pages.  https://doi.org/10.1145/3437359.3465578
  * Calder et al. (2022) A. C. Calder, C. Feldman, E. Siegmann, J. Dey, A. Curtis, S. Chheda, and R. J. Harrison. 2022.  On Using Linux Kernel Huge Pages with FLASH, an Astrophysical Simulation Code. In _2022 IEEE International Conference on Cluster Computing (CLUSTER)_. IEEE Computer Society, Los Alamitos, CA, USA, 539–544.  https://doi.org/10.1109/CLUSTER51413.2022.00070
  * Domke (2021) Jens Domke. 2021\.  A64FX – Your Compiler You Must Decide!. In _2021 IEEE International Conference on Cluster Computing (CLUSTER)_. 736–740.  https://doi.org/10.1109/Cluster48925.2021.00109
  * Feldman et al. (2022) C. Feldman, B. Michalowicz, E. Siegmann, T. Curtis, A. Calder, and R. Harrison. 2022\.  Experiences with Porting the FLASH Code to Ookami, an HPE Apollo 80 A64FX Platform. In _International Conference on High Performance Computing in Asia-Pacific Region Workshops_ (Virtual Event, Japan) _(HPCAsia 2022 Workshop)_. Association for Computing Machinery, New York, NY, USA, 72–77.  https://doi.org/10.1145/3503470.3503478
  * Fryxell et al. (2000) B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Rosner, J. W. Truran, and H. Tufo. 2000\.  FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes.  _The Astrophysical Journal Supplement Series_ 131 (2000), 273–334.
  * Fujitsu (2023) Fujitsu. 2023.  _A64FX Microarchitecture Manual_.  https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Microarchitecture_Manual_en_1.3.pdf
  * Langarita et al. (2023) R. Langarita, A. Armejach, P. Ibáñez, J. Alastruey-Benedé, and M. Moretó. 2023\.  Porting and Optimizing BWA-MEM2 Using the Fujitsu A64FX Processor.  _IEEE/ACM Transactions on Computational Biology and Bioinformatics_ (2023), 1–14.  https://doi.org/10.1109/TCBB.2023.3264514
  * RIKEN (2020) RIKEN. 2020.  _Fugaku Codesign Report_.  https://www.r-ccs.riken.jp/wp/wp-content/uploads/2022/03/fs2020-report-e.pdf

##  Appendix A Environment

###  A.1. Libraries and compiler flags

We performed this study using FLASH version 4.6.2, including additional
modules for our SN Ia application. To enable thp, we were conferred a
dedicated node of Ookami running Rocky Linux 8.4 with kernel
4.18.0-305.25.1.el8_4.aarch64. To provide a more equal comparison, all runs
used serial hdf5/1.10.1 and the same PAPI library. For this study, we were
unable to get the Cray compiler to use the same HDF5 library (necessary) as
the other compilers, and therefore it is not used here, although our earlier
study demonstrates that the Cray and GCC compilers give similar performance
for FLASH. The performance of three compilers - GCC, Fujitsu, and ARM - were
compared, and the compiler options for each are listed in Table 3.

Table 3. Compiler flags and MPI implementations used for this study Compiler | Compiler Flags | Linker Flags | MPI Implementation
---|---|---|---
GCC 12.2.0 | -O3 -mcpu=a64fx -mtune=a64fx |  | MVAPICH 2.3.7
| -fdefault-real-8 -fdefault-double-8 |  |
| -Wuninitialized -fallow-argument-mismatch |  |
ARM 21.0 | -c -O3 -armpl -mcpu=a64fx -mtune=a64fx -r8 | -lamath | MVAPICH 2.3.7
Fujitsu 4.5 | -KSVE,A64FX,ARMV8_3_A -Az -Kfast |  | Fujitsu built-in 1.0.21.01
| -CcdII4 -CcdRR8 |  | (based on OpenMPI)

We linked to PAPI at compile-time with -L /opt/cray/pe/papi/6.0.0.4/lib -lpapi
and then at run-time linked the executable to the proper library like so:
export LD_LIBRARY_PATH=/opt/cray/pe/papi/6.0.0.4/lib:${LD_LIBRARY_PATH} .

###  A.2. Enabling huge pages

Paging policy for the static data area, stack/thread stack area, and reserved
dynamic memory areas is defined by XOS_MMM_L_PAGING_POLICY. We set the paging
policy to demand for all three areas (export
XOS_MMM_L_PAGING_POLICY=demand:demand:demand) to ensure that memory used is
within the NUMA memory region as much as possible.

To use (t)hp, we linked the GCC and ARM compilers to Fujitsu’s libmpg library
by adding -Wl,-T/opt/FJSVxos/mmm/util/bss-2mb.lds -L/opt/FJSVxos/mmm/lib64
-lmpg to the compile and link flags. For the Fujitsu compiler, we added
-Klargepage and -Knolargepage to turn (t)hp on and off, respectively.

Switching between different pages is controlled by the XOS_MMM_L_HPAGE_TYPE
environment variable, when using Fujitsu’s libmpg. While the documentation
mentions that acceptable values are none or hugetlbfs, (RIKEN, 2020) mentions
another possible value, thp for the variable on Fugaku (A64FX FX1000). This is
viable on the FX700 system as well. Therefore there are three values for this
environment variable – none (No hp), hugetlbfs (default, enables hp), and thp
(enables thp).

The kernel should invoke thp on its own when it processes a file greater than
2 Gb. Thp can be enabled or disabled by selecting [always] or [never],
respectively, in /sys/kernel/mm/tranparent_hugepage/enabled. We monitored the
use of hugepages by the machine by looking at specific system variables in
/proc/meminfo: HugePages_Total should be nonzero when hp is in use;
AnonHugePages should be nonzero when thp is in use; and both variables should
be zero when using no hp.

##  Appendix B Counter Data

For completeness, the tables below report the raw counter values and derived
rates for both the Fujitsu and GCC compilers, with standard hp and without hp,
for both test problems (“EOS” and “3-d Hydro”) on 1p. This data was used to
create Tables 1 and 2. Table 4 shows the raw counter values for the EOS test;
Table 5 shows the derived rates for the EOS test; Table 6 shows the raw
counter values for the 3-d Hydro test; and Table 7 shows the derived rates for
the 3-d Hydro test. Counter descriptions and rate calculations can be found in
the Fujitsu Microarchitecture Manual (Fujitsu, 2023).

Table 4. Raw counter values and their ratios for the EOS test on 1 core. The first 3 columns show the raw counter values for the GCC compiler with and without hp, followed by the counter ratio. The next three columns show the same, but for the Fujitsu compiler. The last 2 columns show the ratio of counter values between Fujitsu : GCC compiler. Counter | GCC | GCC | GCC | Fujitsu | Fujitsu | Fujitsu | Fuj : GCC | Fuj : GCC
---|---|---|---|---|---|---|---|---
| hp | no hp | hp : no hp | hp | no hp | hp : no hp | hp | no hp
CPU_CYCLES | 341210316099 | 354894128279 | 0.96 | 118393395910 | 126126802860 | 0.94 | 0.35 | 0.36
DTLB-LOAD-MISSES | 136916841 | 4200311763 | 0.03 | 89784864 | 1624074206 | 0.06 | 0.66 | 0.39
L1D_TLB_REFILL | 135140728 | 4186687491 | 0.03 | 74337421 | 1628705951 | 0.05 | 0.55 | 0.39
L2D_TLB_REFILL | 212 | 966342 | 0.0002 | 163 | 21269 | 0.01 | 0.77 | 0.02
L1I_TLB_REFILL | 74981652 | 105364002 | 0.71 | 52578326 | 52073237 | 1.01 | 0.70 | 0.49
L2I_TLB_REFILL | 6206064 | 6207292 | 1.00 | 6203536 | 6259401 | 0.99 | 1.00 | 1.01
L1_MISS_WAIT | 41995961928 | 42462092793 | 0.99 | 35374847991 | 34746226643 | 1.02 | 0.84 | 0.82
L1D_CACHE_REFILL | 1211001903 | 1261895471 | 0.96 | 1132822066 | 1141540325 | 0.99 | 0.94 | 0.90
L1D_CACHE_REFILL_HWPRF | 19441488 | 16435557 | 1.18 | 107749070 | 103581864 | 1.04 | 5.54 | 6.30
L1D_CACHE_REFILL_PRF | 21247269 | 16573641 | 1.28 | 107955640 | 103723552 | 1.04 | 5.08 | 6.26
L1D_CACHE_REFILL_DM | 1173058042 | 1194338101 | 0.98 | 1012021771 | 1008685191 | 1.00 | 0.86 | 0.84
L1D_CACHE_WB | 568115212 | 570331006 | 1.00 | 570460305 | 548102563 | 1.04 | 1.00 | 0.96
L2_MISS_WAIT | 14156941 | 5198312 | 2.72 | 3749562 | 3539840 | 1.06 | 0.26 | 0.68
L2D_CACHE_REFILL | 1649116 | 1533521 | 1.08 | 1721394 | 1629720 | 1.06 | 1.04 | 1.06
L2D_CACHE_REFILL_HWPRF | 702309 | 646787 | 1.09 | 897424 | 805954 | 1.11 | 1.28 | 1.25
L2D_CACHE_REFILL_PRF | 702309 | 646787 | 1.09 | 900301 | 808715 | 1.11 | 1.28 | 1.25
L2D_CACHE_REFILL_DM | 946807 | 886734 | 1.07 | 821093 | 821005 | 1.00 | 0.87 | 0.93
L2D_CACHE_WB | 910868 | 883074 | 1.03 | 836069 | 814766 | 1.03 | 0.92 | 0.92
LD_COMP_WAIT | 43456361267 | 61463918085 | 0.71 | 21856142950 | 28021892897 | 0.78 | 0.50 | 0.46
LD_COMP_WAIT_EX | 14295343427 | 12725601925 | 1.12 | 4395919156 | 4817739974 | 0.91 | 0.31 | 0.38
LD_COMP_WAIT_PFP_BUSY | 0 | 0 | N/A | 49 | 0 | N/A | N/A | N/A
LD_COMP_WAIT_L1_MISS | 14764291144 | 18042715909 | 0.82 | 8197400785 | 10457746706 | 0.78 | 0.56 | 0.58
LD_COMP_WAIT_L2_MISS | 85681191 | 94715479 | 0.90 | 70471471 | 73306995 | 0.96 | 0.82 | 0.77
EU_COMP_WAIT | 192299195248 | 189702560873 | 1.01 | 62110089168 | 63986117005 | 0.97 | 0.32 | 0.34
BR_COMP_WAIT | 912951041 | 795494616 | 1.15 | 306967691 | 298776080 | 1.03 | 0.34 | 0.38
BR_MIS_PRED | 163423745 | 167518949 | 0.98 | 40681971 | 42617939 | 0.95 | 0.25 | 0.25
EFFECTIVE_INST_SPEC | 247030572236 | 241626826143 | 1.02 | 81247006439 | 81247004885 | 1.00 | 0.33 | 0.34
LD_SPEC | 55921382684 | 54938703722 | 1.02 | 20359738305 | 20359737979 | 1.00 | 0.36 | 0.37
BASE_LD_REG_SPEC | 15673422673 | 14690743711 | 1.07 | 2004533696 | 2004533370 | 1.00 | 0.13 | 0.14

Table 5. Derived rates and their ratios for the EOS test on 1 core. The first 3 columns show the rates for the GCC compiler with and without hp, followed by the rate ratio. The next three columns show the same, but for the Fujitsu compiler. The last 2 columns show the ratio of rates between Fujitsu : GCC compiler. Rate | GCC | GCC | GCC | Fujitsu | Fujitsu | Fujitsu | Fuj : GCC | Fuj : GCC
---|---|---|---|---|---|---|---|---
| hp | no hp | hp : no hp | hp | no hp | hp : no hp | hp | no hp
Cycles per instruction (CPI) | 1.38 | 1.47 | 0.94 | 1.46 | 1.55 | 0.94 | 1.05 | 1.06
Branch misprediction rate | 0.0007 | 0.0007 | 0.95 | 0.001 | 0.001 | 0.95 | 0.76 | 0.76
L1-ITLB miss rate | 0.0003 | 0.0004 | 0.70 | 0.0006 | 0.0006 | 1.01 | 2.13 | 1.47
L1-DTLB miss rate | 0.0005 | 0.02 | 0.03 | 0.0009 | 0.02 | 0.05 | 1.67 | 1.16
L2-ITLB miss rate | 2.51E-05 | 2.57E-05 | 0.98 | 7.64E-05 | 7.70E-05 | 0.99 | 3.04 | 3.00
L2-DTLB miss rate | 8.58E-10 | 4.00E-06 | 0.0002 | 2.01E-09 | 2.62E-07 | 0.01 | 2.34 | 0.07
L1D cache miss rate | 0.005 | 0.005 | 0.94 | 0.01 | 0.01 | 0.99 | 2.84 | 2.69
… attributable to demand access | 0.005 | 0.005 | 0.96 | 0.01 | 0.01 | 1.00 | 2.62 | 2.51
… attributable to prefetch access | 8.60E-05 | 6.86E-05 | 1.25 | 0.001 | 0.001 | 1.04 | 15.45 | 18.61
… attributable to software prefetch access | 7.31E-06 | 5.71E-07 | 12.79 | 2.54E-06 | 1.74E-06 | 1.46 | 0.35 | 3.05
L2 cache miss rate | 6.68E-06 | 6.35E-06 | 1.05 | 2.12E-05 | 2.01E-05 | 1.06 | 3.17 | 3.16
… attributable to demand access | 3.83E-06 | 3.67E-06 | 1.04 | 1.01E-05 | 1.01E-05 | 1.00 | 2.64 | 2.75
… attributable to prefetch access | 2.84E-06 | 2.68E-06 | 1.06 | 1.11E-05 | 9.95E-06 | 1.11 | 3.90 | 3.72
… attributable to software prefetch access | 0.00 | 0.00 | N/A | 3.54E-08 | 3.40E-08 | 1.04 | N/A | N/A
Average latency of |  |  |  |  |  |  |  |
… L1D cache miss processing | 34.68 | 33.65 | 1.03 | 31.23 | 30.44 | 1.03 | 0.90 | 0.90
… L2 cache miss processing | 8.58 | 3.39 | 2.53 | 2.18 | 2.17 | 1.00 | 0.25 | 0.64
Average number of outstanding misses |  |  |  |  |  |  |  |
… in L1D cache miss processing | 0.12 | 0.12 | 1.03 | 0.30 | 0.28 | 1.08 | 2.43 | 2.30
… in L2 cache miss processing | 4.15E-05 | 1.46E-05 | 2.83 | 3.17E-05 | 2.81E-05 | 1.13 | 0.76 | 1.92
Bidirectional effective BW (GB/s) |  |  |  |  |  |  |  |
… between L1D cache and L2 cache | 2.40 | 2.38 | 1.01 | 6.63 | 6.17 | 1.07 | 2.76 | 2.59
… between L2 cache and memory | 0.003 | 0.003 | 1.10 | 0.010 | 0.009 | 1.11 | 2.88 | 2.85
LD_COMP_WAIT/CPU_CYCLES | 0.13 | 0.17 | 0.74 | 0.18 | 0.22 | 0.83 | 1.45 | 1.28
LD_COMP_WAIT_L1_MISS/CPU_CYCLES | 0.04 | 0.05 | 0.85 | 0.07 | 0.08 | 0.84 | 1.60 | 1.63
LD_COMP_WAIT_L2_MISS/CPU_CYCLES | 0.0003 | 0.0003 | 0.94 | 0.0006 | 0.0006 | 1.02 | 2.37 | 2.18
LD_COMP_WAIT_EX/CPU_CYCLES | 0.04 | 0.04 | 1.17 | 0.04 | 0.04 | 0.97 | 0.89 | 1.07
LD_COMP_WAIT_PFP_BUSY/CPU_CYCLES | 0.0 | 0.0 | N/A | 4.14E-10 | 0.00 | N/A | N/A | N/A

Table 6. Raw counter values and their ratios for the 3-d Hydro test on 1 core. The first 3 columns show the raw counter values for the GCC compiler with and without hp, followed by the counter ratio. The next three columns show the same, but for the Fujitsu compiler. The last 2 columns show the ratio of counter values between Fujitsu : GCC compiler. Counter | GCC | GCC | GCC | Fujitsu | Fujitsu | Fujitsu | Fuj : GCC | Fuj : GCC
---|---|---|---|---|---|---|---|---
| hp | no hp | hp : no hp | hp | no hp | hp : no hp | hp | no hp
CPU_CYCLES | 2143932684764 | 1954934065987 | 1.10 | 1196335386888 | 1200402908968 | 1.00 | 0.56 | 0.61
DTLB-LOAD-MISSES | 226322429 | 1979316404 | 0.11 | 498078094 | 1615854112 | 0.31 | 2.20 | 0.82
L1D_TLB_REFILL | 200617382 | 1889017976 | 0.11 | 505910868 | 1623370615 | 0.31 | 2.52 | 0.86
L2D_TLB_REFILL | 222571 | 6424782 | 0.03 | 207882 | 6556994 | 0.03 | 0.93 | 1.02
L1I_TLB_REFILL | 5949138 | 141791086 | 0.04 | 3751271 | 5787082 | 0.65 | 0.63 | 0.04
L2I_TLB_REFILL | 84205 | 141619 | 0.59 | 53741 | 330456 | 0.16 | 0.64 | 2.33
L1_MISS_WAIT | 1174125092814 | 1175696041980 | 1.00 | 1112295369914 | 1114808777704 | 1.00 | 0.95 | 0.95
L1D_CACHE_REFILL | 28765078635 | 28695255519 | 1.00 | 26312621913 | 26442346625 | 1.00 | 0.91 | 0.92
L1D_CACHE_REFILL_HWPRF | 7220868699 | 7128347854 | 1.01 | 4454501262 | 4526188824 | 0.98 | 0.62 | 0.63
L1D_CACHE_REFILL_PRF | 7746287019 | 7653780229 | 1.01 | 4590540067 | 4673132966 | 0.98 | 0.59 | 0.61
L1D_CACHE_REFILL_DM | 19022344796 | 19035542257 | 1.00 | 20327097179 | 20468323515 | 0.99 | 1.07 | 1.08
L1D_CACHE_WB | 20130483048 | 20092819898 | 1.00 | 18212432757 | 18287841025 | 1.00 | 0.90 | 0.91
L2_MISS_WAIT | 2310792458 | 2343447319 | 0.99 | 2211054362 | 2224081481 | 0.99 | 0.96 | 0.95
L2D_CACHE_REFILL | 1827787623 | 1900351203 | 0.96 | 1975065788 | 1909551917 | 1.03 | 1.08 | 1.00
L2D_CACHE_REFILL_HWPRF | 1017222150 | 1024578925 | 0.99 | 963692147 | 897152822 | 1.07 | 0.95 | 0.88
L2D_CACHE_REFILL_PRF | 1017222150 | 1024578925 | 0.99 | 964186753 | 897632586 | 1.07 | 0.95 | 0.88
L2D_CACHE_REFILL_DM | 810565473 | 875772278 | 0.93 | 1010879035 | 1011919331 | 1.00 | 1.25 | 1.16
L2D_CACHE_WB | 474436183 | 510418295 | 0.93 | 481165262 | 469429725 | 1.02 | 1.01 | 0.92
LD_COMP_WAIT | 783957843900 | 670383891683 | 1.17 | 515460270334 | 521930831503 | 0.99 | 0.66 | 0.78
LD_COMP_WAIT_EX | 429528457728 | 323839573426 | 1.33 | 38884573461 | 37007309796 | 1.05 | 0.09 | 0.11
LD_COMP_WAIT_PFP_BUSY | 29585337 | 14584447 | 2.03 | 101373 | 182984 | 0.55 | 0.003 | 0.01
LD_COMP_WAIT_L1_MISS | 106374292021 | 113150011609 | 0.94 | 274005118311 | 274611290445 | 1.00 | 2.58 | 2.43
LD_COMP_WAIT_L2_MISS | 19383905950 | 19975680958 | 0.97 | 40911779437 | 41839552962 | 0.98 | 2.11 | 2.09
EU_COMP_WAIT | 625493183646 | 630338181042 | 0.99 | 340261838023 | 341315363810 | 1.00 | 0.54 | 0.54
BR_COMP_WAIT | 23654302037 | 21426208561 | 1.10 | 1701837812 | 1781324275 | 0.96 | 0.07 | 0.08
BR_MIS_PRED | 1941165909 | 3941304832 | 0.49 | 624100572 | 624524801 | 1.00 | 0.32 | 0.16
EFFECTIVE_INST_SPEC | 1641901478240 | 1346724114727 | 1.22 | 835329678475 | 830699232211 | 1.01 | 0.51 | 0.62
LD_SPEC | 326350027305 | 269139241392 | 1.21 | 195567194613 | 194711154297 | 1.00 | 0.60 | 0.72
BASE_LD_REG_SPEC | 215301026770 | 158090240857 | 1.36 | 68445331297 | 67589290981 | 1.01 | 0.32 | 0.43

Table 7. Derived rates and their ratios for the 3-d Hydro test on 1 core. The first 3 columns show the rates for the GCC compiler with and without hp, followed by the rate ratio. The next three columns show the same, but for the Fujitsu compiler. The last 2 columns show the ratio of rates between Fujitsu : GCC compiler. Rate | GCC | GCC | GCC | Fujitsu | Fujitsu | Fujitsu | Fuj : GCC | Fuj : GCC
---|---|---|---|---|---|---|---|---
| hp | no hp | hp : no hp | hp | no hp | hp : no hp | hp | no hp
Cycles per instruction (CPI) | 1.31 | 1.45 | 0.90 | 1.43 | 1.45 | 0.99 | 1.10 | 1.00
Branch misprediction rate | 0.001 | 0.003 | 0.40 | 0.0007 | 0.0008 | 0.99 | 0.63 | 0.26
L1-ITLB miss rate | 3.62E-06 | 0.0001 | 0.03 | 4.49E-06 | 6.97E-06 | 0.64 | 1.24 | 0.07
L1-DTLB miss rate | 0.0001 | 0.001 | 0.09 | 0.0006 | 0.002 | 0.31 | 4.96 | 1.39
L2-ITLB miss rate | 5.13E-08 | 1.05E-07 | 0.49 | 6.43E-08 | 3.98E-07 | 0.16 | 1.25 | 3.78
L2-DTLB miss rate | 1.36E-07 | 4.77E-06 | 0.03 | 2.49E-07 | 7.89E-06 | 0.03 | 1.84 | 1.65
L1D cache miss rate | 0.02 | 0.02 | 0.82 | 0.03 | 0.03 | 0.99 | 1.80 | 1.49
… attributable to demand access | 0.01 | 0.01 | 0.82 | 0.02 | 0.02 | 0.99 | 2.10 | 1.74
… attributable to prefetch access | 0.005 | 0.006 | 0.83 | 0.005 | 0.006 | 0.98 | 1.16 | 0.99
… attributable to software prefetch access | 0.0003 | 0.0004 | 0.82 | 0.0002 | 0.0002 | 0.92 | 0.51 | 0.45
L2 cache miss rate | 0.001 | 0.001 | 0.79 | 0.002 | 0.002 | 1.03 | 2.12 | 1.63
… attributable to demand access | 0.0005 | 0.0007 | 0.76 | 0.001 | 0.001 | 0.99 | 2.45 | 1.87
… attributable to prefetch access | 0.0006 | 0.0008 | 0.81 | 0.001 | 0.001 | 1.07 | 1.86 | 1.42
… attributable to software prefetch access | 0.0 | 0.0 | N/A | 5.92E-07 | 5.78E-07 | 1.03 | N/A | N/A
Average latency of |  |  |  |  |  |  |  |
… L1D cache miss processing | 40.82 | 40.97 | 1.00 | 42.27 | 42.16 | 1.00 | 1.04 | 1.03
… L2 cache miss processing | 1.26 | 1.23 | 1.03 | 1.12 | 1.16 | 0.96 | 0.89 | 0.94
Average number of outstanding misses |  |  |  |  |  |  |  |
… in L1D cache miss processing | 0.55 | 0.60 | 0.91 | 0.93 | 0.93 | 1.00 | 1.70 | 1.54
… in L2 cache miss processing | 0.001 | 0.001 | 0.90 | 0.002 | 0.002 | 1.00 | 1.71 | 1.55
Bidirectional effective BW between (GB/s) |  |  |  |  |  |  |  |
… L1D cache and L2 cache | 10.51 | 11.50 | 0.91 | 17.15 | 17.17 | 1.00 | 1.63 | 1.49
… L2 cache and memory | 0.49 | 0.57 | 0.87 | 0.95 | 0.91 | 1.04 | 1.91 | 1.61
LD_COMP_WAIT/CPU_CYCLES | 0.37 | 0.34 | 1.07 | 0.43 | 0.43 | 0.99 | 1.18 | 1.27
LD_COMP_WAIT_L1_MISS/CPU_CYCLES | 0.05 | 0.06 | 0.86 | 0.23 | 0.23 | 1.00 | 4.62 | 3.95
LD_COMP_WAIT_L2_MISS/CPU_CYCLES | 0.01 | 0.01 | 0.88 | 0.03 | 0.03 | 0.98 | 3.78 | 3.41
LD_COMP_WAIT_EX/CPU_CYCLES | 0.20 | 0.17 | 1.21 | 0.03 | 0.03 | 1.05 | 0.16 | 0.19
LD_COMP_WAIT_PFP_BUSY/CPU_CYCLES | 1.38E-05 | 7.46E-06 | 1.85 | 8.47E-08 | 1.52E-07 | 0.56 | 0.006 | 0.02


# Guiding Users to Where to Give Color Hints for Efficient Interactive Sketch
Colorization via Unsupervised Region Prioritization

Youngin Cho*1 Junsoo Lee*2 Soyoung Yang1 Juntae Kim3 Yeojeong Park1 Haneol
Lee4
Mohammad Azam Khan1 Daesik Kim2 Jaegul Choo1
1KAIST AI 2NAVER WEBTOON AI 3Korea University 4UNIST
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Existing deep interactive colorization models have focused on ways to utilize
various types of interactions, such as point-wise color hints, scribbles, or
natural-language texts, as methods to reflect a user’s intent at runtime.
However, another approach, which actively informs the user of the most
effective regions to give hints for sketch image colorization, has been under-
explored. This paper proposes a novel model-guided deep interactive
colorization framework that reduces the required amount of user interactions,
by prioritizing the regions in a colorization model. Our method, called
GuidingPainter, prioritizes these regions where the model most needs a color
hint, rather than just relying on the user’s manual decision on where to give
a color hint. In our extensive experiments, we show that our approach
outperforms existing interactive colorization methods in terms of the
conventional metrics, such as PSNR and FID, and reduces required amount of
interactions.

Figure 1:  Results of our proposed model on human faces and comics datasets.
Each column of (a)-(c) indicates the order of interactions as the $i$-th
priority. (a) visualizes masked regions which our model guides at the $i$-th
step. Given a region as (a), users select its representative color, and the
region is filled with the selected color. (c) shows intermediate colorization
results for given accumulated color hints as (b).  **footnotetext: Equal
contribution

##  1 Introduction

The colorization task in computer vision has received considerable attention
recently, since it can be widely applied in content creation. Most content
creation starts with drawn or sketch images, and these can be accomplished
within a reasonable amount of time, but fully colorizing them is a labor-
intensive task. For this reason, the ability to automatically colorize sketch
images has significant potential values. However, automatic sketch image
colorization is still challenging for the following reasons. (i) The
information provided by an input sketch image is extremely limited compared to
colored images or even gray-scale ones, and (ii) there can be multiple
possible outcomes for a given sketch image without any conditional input,
which tends to degrade the model performance and introduce bias toward the
dominant colors in the dataset.

To alleviate these issues, conditional image colorization methods take partial
hints in addition to the input image, and attempt to generate a realistic
output image that reflects the context of the given hints. Several studies
have leveraged user-guided interactions as a form of user-given conditions to
the model, assuming that the users would provide a desired color value for a
region as a type of point-wise color hint [40] or a scribble [28, 3]. Although
these approaches have made remarkable progress, there still exist nontrivial
limitations. First, existing approaches do not address the issue of estimating
semantic regions which indicate how far the user-given color hints should be
spread, and thus the colorization model tends to require lots of user hints to
produce a desirable output. Second, for every interaction at test time, the
users are still expected to provide a local-position information of color hint
by pointing out the region of interest (RoI), which increases the user’s
effort and time commitment. Lastly, since existing approaches typically obtain
the color hints on randomized locations at training time, the discrepancies
among intervention mechanisms for the training and the test phases need to be
addressed.

In this work, we propose a novel model-guided framework for the interactive
colorization of a sketch image, called GuidingPainter. A key idea behind our
work is to make a model actively seek for regions where color hints would be
provided, which can significantly improve the efficiency of interactive
colorization process. To this end, GuidingPainter consists of two modules:
active-guidance module and colorization module. Although colorization module
works similar to previous methods, our main contribution is a hint generation
mechanism in active-guidance module. The active-guidance module (Section
3.2-3.3) (i) divides the input image into multiple semantic regions and (ii)
ranks them in decreasing order of estimated model gains when the region is
colorized (Fig. 1(a)).

Since it is extremely expensive to obtain groundtruth for segmentation labels
or even their prioritization, we explore a simple yet effective approach that
identifies the meaningful regions in an order of their priority without any
manually annotated labels. In our active guidance mechanism (Section 3.3),
GuidingPainter can learn such regions by intentionally differentiating the
frequency of usage for each channel obtained from the segmentation network.
Also, we conduct a toy experiment (Section 4.5) to understand the mechanism,
and to verify the validity of our approach. We propose several loss terms,
e.g. smoothness loss and total variance loss, to improve colorization quality
in our framework (Section 3.5), and analyze its effectiveness for both
quantitatively and qualitatively (Section 4.6). Note that the only action
required of users in our framework is to select one representative color for
each region the model provides based on the estimated priorities (Fig. 1(b)).
Afterwards, the colorization network (Section 3.4) generates a high-quality
colorized output by taking the given sketch image and the color hints (Fig.
1(c)).

In summary, our contributions are threefold:

  * •

We propose a novel model-guided deep image colorization framework, which
prioritizes regions of a sketch image in the order of the interest of the
colorization model.

  * •

GuidingPainter can learn to discover meaningful regions for colorization and
arrange them in their priority just by using the groundtruth colorized image,
without additional manual supervision.

  * •

We demonstrate that our framework can be applied to a variety of datasets by
comparing it against previous interactive colorization approaches in terms of
various metrics, including our proposed evaluation protocol.

##  2 Related Work

###  2.1 Deep Image Colorization

Existing deep image colorization methods, which utilize deep neural networks
for colorization, can be divided into automatic and conditional approaches,
depending on whether conditions are involved or not. Automatic image
colorization models [39, 29, 36, 1] take a gray-scale or sketch image as an
input and generate a colorized image. CIC [39] proposed a fully automatic
colorization model using convolutional neural networks (CNNs), and Su et al.
[29] further improved the model by extracting the features of objects in the
input image. Despite the substantial performances of automatic colorization
models, a nontrivial amount of user intervention is still required in
practice.

Conditional image colorization models attempt to resolve these limitations by
taking reference images [16] or user interactions [40, 3, 38, 34, 37] as
additional input. For example, Zhang et al. [40] allowed the users to input
the point-wise color hint in real time, and AlacGAN [3] utilized stroke-based
user hints by extracting semantic feature maps. Although these studies
consider the results are improved by user hints, they generally require a
large amount of user interactions.

Figure 2:  Hint generation process of our proposed GuidingPainter model. The
segmentation network and the hint generation function renders colored hints
($C$) and condition masks ($M$). Based on the guidance results, our
colorization network colorizes the sketch image. The example illustrates the
hint generation process in the training phase where $N_{h}=3$ and $N_{c}=4$.
First, the groundtruth image is copied as $N_{c}$ times to consider each color
segment at each interaction step. After element-wise multiplication with
guided regions, (a) averages the color to decide representative colors for
each guided region. To restrict the number of hints, we mask out the segments
whose iteration step is larger than $N_{h}$, The masked results are (b). Based
on (a) and (b), our module generates the colored condition for each segment as
(c). In (d), we combine them into one partially-colorized image $C$. (e)
operates as the same manner with (d) and generates the condition mask $M$.

###  2.2 Interactive Image Generation

Beyond the colorization task, user interaction is utilized in numerous
computer vision tasks, such as image generation, and image segmentation. In
image generation, research has been actively conducted to utilize various user
interactions as additional input to GANs. A variety of GAN models employ
image-related features from users to generate user-driven images [7, 17] and
face images [26, 12, 31, 15, 30]. Several models generate and edit images via
natural-language text [35, 23, 42, 2]. In image segmentation, to improve the
details of segmentation results, recent models have utilized dots [27, 20] and
texts [9] from users. Although we surveyed a wide scope of interactive deep
learning models beyond sketch image colorization, there is no directly related
work with our approach, to the best of our knowledge. Therefore, the use of a
deep learning-based guidance system for interactive process can be viewed as a
promising but under-explored approach.

##  3 Proposed Approach

###  3.1 Problem Setting

The goal of the interactive colorization task is to train networks to generate
a colored image $\hat{Y}\in\mathbb{R}^{3\times H\times W}$ by taking as input
a sketch image $X\in\mathbb{R}^{1\times H\times W}$ along with user-provided
partial hints $U$, where $H$ and $W$ indicate the height and width of the
target image, respectively. The user-provided partial hints are defined as a
pair $U=(C,M)$ where $C\in\mathbb{R}^{3\times H\times W}$ is a sparse tensor
with RGB values, and $M\in\\{0,1\\}^{1\times H\times W}$ is a binary mask
indicating the region in which the color hints are provided. Our training
framework consists of two networks and one function: segmentation network
$\mathbf{f}$ (Section 3.2), colorization network $\mathbf{g}$ (Section 3.4),
and a hint generation function called $\mathbf{h}$ (Section 3.3), which are
trained in an end-to-end manner.

###  3.2 Segmentation Network

The purpose of segmentation network $\mathbf{f}(\cdot)$ is to divide the
sketch input $X$ into several semantic regions which are expected to be
painted in a single color, i.e.,

$S=\mathbf{f}(X;\theta_{f}),$ (1)

where $S=(S_{1},S_{2},...,S_{N_{c}})\in\\{0,1\\}^{N_{c}\times H\times W}$,
$S_{i}$ is the $i$-th guided region, and $N_{c}$ denotes the maximum number of
hints. Specifically, $\mathbf{f}$ contains an encoder-decoder network with
skip connections, based on U-Net [10] architecture, to preserve the spatial
details of given objects.

Since each guided region will be painted with a single color, we have to
segment the output of U-Net in a discrete form while taking advantages of end-
to-end learning. To this end, after obtaining an output tensor
$S_{logit}\in\mathbb{R}^{N_{c}\times H\times W}$ of U-Net, we discretize
$S_{logit}$ by applying straight-through (ST) gumbel estimator [11, 19] across
channel dimensions to obtain $S$ as a differentiable approximation. The result
$S$ satisfies $\sum_{i=1}^{N_{c}}S_{i}(j)=1$ where $S_{i}(j)$ indicates the
$i$-th scalar value of the $j$-th position vector, i.e., every pixel is
contained in only one guided region. Here, $S_{i}(j)=1$ indicates that the
$j$-th pixel is contained in the $i$-th guided region while $S_{i}(j)=0$
indicates that the pixel is not contained in the guided region.

###  3.3 Hint Generation

The hint generation function $\mathbf{h}(\cdot)$ is a non-parametric function
that plays the role of simulating $U$ based on $S$, a colored image $Y$, and
the number of hints $N_{h}$, i.e.,

$U=\mathbf{h}(S,Y,N_{h}).$ (2)

To this end, we first randomly sample $N_{h}$ from a bounded distribution
which is similar to a geometric distribution formulated as

$\mathcal{G}(N_{h}=i)=\begin{cases}(1-p)^{i}p&\mbox{if }i=0,1,...,N_{c}-1\\\
(1-p)^{N_{c}}&\mbox{if }i=N_{c},\end{cases}$ (3)

where $p<1$ is a hyperparameter indicating the probability that the user stops
adding a hint on each trial. We set $N_{c}=30$ and $p=0.125$ for the following
experiments.

Step1: building masked segments $\tilde{S}$. Given $N_{h}$, we construct a
mask vector $m\in\\{0,1\\}^{N_{c}}$ having each element with the following
rule:

$m_{i}=\begin{cases}1&\mbox{if }i\leq N_{h}\\\ 0&\mbox{otherwise},\end{cases}$
(4)

where $m_{i}$ indicates the $i$-th scalar value of the vector $m$. Afterwards,
we obtain a masked segment $\tilde{S}\in\mathbb{R}^{N_{c}\times H\times W}$ by
element-wise multiplying the $i$-th element of $m$ with the $i$-th channel of
$S$ as

$\tilde{S}_{i}=m_{i}S_{i},$ (5)

where $S_{i}$, $\tilde{S}_{i}\in\mathbb{R}^{1\times H\times W}$ denote the
$i$-th channel of $S$ and $\tilde{S}$, respectively.

Step2: building hint maps $C$. The goal of this step is to find the
representative color value of the activated region in each segment
$\tilde{S}_{i}$, and then to fill the corresponding region with this color. To
this end, we calculate a mean RGB color $\bar{c_{i}}\in\mathbb{R}^{3}$ as

$\bar{c_{i}}=\begin{cases}\frac{1}{N_{p}}\sum_{j}^{HW}{S_{i}(j)\odot
Y(j)}&\mbox{if }1\leq N_{p}\\\ 0&\mbox{otherwise},\end{cases}$ (6)

where $N_{p}=\sum_{j}{S_{i}(j)}$ indicates the number of activated pixels of
the $i$-th segment, $\odot$ denotes an element-wise multiplication, i.e., the
Hadamard product, after each element of $S_{i}$ is broadcast to the RGB
channels of $Y$, and both $S_{i}(j)$ and $Y(j)$ indicate the $j$-th position
vector of each map. Finally, we obtain hint maps $C\in\mathbb{R}^{3\times
H\times W}$ as

$C=\sum_{i=1}^{N_{c}}{\bar{c_{i}}\tilde{S}_{i}},$ (7)

where $\bar{c_{i}}$ is repeated to the spatial axis as the form of
$\tilde{S}_{i}\in\mathbb{R}^{1\times H\times W}$ similar to Eq. (5) and
$\tilde{S}_{i}$ is broadcast to the channel axis as the form of
$\bar{c_{i}}\in\mathbb{R}^{3}$ as in Eq. (6). In order to indicate the region
of given hints, we simply obtain a condition mask $M\in\mathbb{R}^{1\times
H\times W}$ as

$M=\sum_{i=1}^{N_{c}}\tilde{S}_{i}.$ (8)

Eventually, the output of this module $U=C\oplus M\in\mathbb{R}^{4\times
H\times W}$ where $\oplus$ indicates a channel-wise concatenation. Fig. 2
illustrates overall scheme of the hint generation process. At the inference
time, we can create $U$ similar to the hint generation process, but without an
explicit groundtruth image. Note that a sketch image is all we need to produce
$\tilde{S}$ at the inference time. We can obtain $C$ and $M$ by assigning a
color to each $S_{i}$ for $i=1,2,...,N_{h}$.

To understand how the hint generation module works, recall that $N_{h}$ is
randomly sampled from the bounded geometric distribution $\mathcal{G}$ (Eq.
(3)) per mini-batch at the training time. Since the probability that $i\leq
N_{h}$ is higher than the probability that $j\leq N_{h}$ for $i<j$, $S_{i}$ is
more frequently activated than $S_{j}$ during training the model. Hence, we
can expect the following effects via this module: i) $N_{h}$ affects in
determining how many segments starting from the first channel of $S$ as
computed in Eq. (4-5); therefore, this mechanism encourages the segmentation
network $\mathbf{f}(\cdot)$ to locate relatively important and uncertain
regions at the forward indexes of $S$. Section 4.5 shows this module behaves
as our expectation. ii) We can provide more abundant information for the
following colorization networks $\mathbf{g}(\cdot)$ than previous approaches
without requiring additional labels at training time or even interactions at
test time, helping to generate better results even with fewer hints than
baselines (Section 4.3).

###  3.4 Colorization Network

The colorization network $\mathbf{g}(\cdot)$ aims to generate a colored image
$\hat{Y}$ by taking all the information obtained from the previous steps,
i.e., a sketch image $X$, guided regions $S$, and partial hints $U$, as

$\hat{Y}=\mathbf{g}(X,S,U;\theta_{g}).$ (9)

The reason for using the segments as input is to provide information about the
color relationship, which the segmentation network infers. In order to capture
the context of the input and to preserve the spatial information of the sketch
image, our colorization networks also adopt the U-Net architecture, the same
as in the segmentation network. We then apply a hyperbolic tangent activation
function to normalize the output tensor of the U-Net.

###  3.5 Objective Functions

As shown in Fig. 2, our networks are trained using the combined following
objective functions. For simplicity, $\mathbf{G}$ denotes the generator of our
approach which contains all the procedures, i.e.,
$\mathbf{f},\mathbf{h},\mathbf{g}$, mentioned above while $\mathcal{D}$
denotes training datasets.

Smoothness loss. Although adjacent pixels in an image have similar RGB values,
our segment guidance networks do not have an explicit mechanism to generate
segments containing those locally continuous pixels. To improve the users’
ability to interpret the segments, we introduce smoothness loss, as

$\mathcal{L}_{smth}=\mathbb{E}\left[\sum_{i}^{HW}\sum_{j\in
N_{i}}||S_{logits}(i)-S_{logits}(j)||_{1}\right],$ (10)

where $N_{i}$ denotes a set of eight nearest neighbor pixels adjacent to the
$i$-th pixel, and $S_{logit}(i)$ indicates the $i$-th position vector of
$S_{logit}$.

Total variance loss. In our framework, the quality of segments from
$\mathbf{f}$ is important because the hints $U$ are built based on guided
regions $S=\mathbf{f}(X)$. Although the $f$ can be indirectly trained by the
colorization signal, we introduce a total variance loss in order to facilitate
this objective directly, i.e.,

$\mathcal{L}_{tv}=\mathbb{E}_{X,Y\sim\mathcal{D}}\left[\sum_{i=1}^{N_{c}}{||(Y-\bar{c}_{i})\odot
S_{i}||^{2}_{F}}\right],$ (11)

where $||\cdot||_{F}$ denotes a Frobenius norm. That is, $\mathcal{L}_{tv}$
attempts to minimize the color variance across pixels in each segment, which
helps pixels of similar color form into the same segment.

Reconstruction loss. Since both a sketch image $X$ and its corresponding
partial hint $U$ are built from a groundtruth image $Y$ in the training phase,
we can directly supervise our networks $\mathbf{G}$ so that it can generate an
output image close to the groundtruth $Y$. Following the previous work, we
select the $\mathbb{L}_{1}$ distance function as our reconstruction loss,
i.e.,

$\mathcal{L}_{rec}=\mathbb{E}_{X,Y\sim\mathcal{D},N_{h}\sim\mathcal{G}}\left[||\mathbf{G}(X,N_{h},Y)-Y||_{1}\right].$
(12)

Adversarial loss. As shown in the image generation work, we adopt an
adversarial training [5] strategy, in which our generator $\mathbf{G}$
produces a natural output image enough to fool a discriminator $D$, while $D$
attempts to classify whether the image is real or fake. During the image
colorization task, the original contents of a sketch input should be preserved
as much as possible. Therefore, we leverage the conditional adversarial [22]
loss, written as

$\begin{split}&\mathcal{L}_{adv}=\mathbb{E}_{X,Y\sim\mathcal{D}}\left[\log\mathbf{D}(Y,X)\right]\\\
&+\mathbb{E}_{X,Y\sim\mathcal{D},N_{h}\sim\mathcal{G}}\left[\log(1-\mathbf{D}(\mathbf{G}(X,N_{h},Y),X))\right].\end{split}$
(13)

Finally, our objective function is defined as

$\begin{split}\min_{\mathbf{G}}\max_{\mathbf{D}}\mathcal{L}=\lambda_{tv}\mathcal{L}_{tv}+\lambda_{smth}\mathcal{L}_{smth}\\\
+\lambda_{rec}\mathcal{L}_{rec}+\lambda_{adv}\mathcal{L}_{adv},\end{split}$
(14)

where each $\lambda$ indicates the weighting factor for each loss term. We
describe the implementation details in the supplementary material.

|  | $\mbox{PSNR}_{\uparrow}$ | $\mbox{FID}_{\downarrow}$ | $\mbox{NRI}_{\downarrow}$
---|---|---|---|---
Methods | Cond. | Yumi | Tag2pix | CelebA | Yumi | Tag2pix | CelebA | Yumi | Tag2pix | CelebA
CIC | ✗ | 15.17 | 13.99 | 17.01 | 137.35 | 167.88 | 79.05 | -
Pix2Pix | ✗ | 15.11 | 14.68 | 16.22 | 71.93 | 111.45 | 54.86 | -
AlacGAN | ✗ | 15.02 | 14.12 | 15.73 | 30.72 | 46.24 | 23.52 | -
RTUG | ✗ | 19.05 | 14.44 | 17.23 | 35.52 | 92.69 | 52.67 | -
Ours | ✗ | 18.63 | 15.19 | 16.53 | 34.07 | 55.31 | 42.46 | -
AlacGAN | ✓ | 15.68 | 14.53 | 16.52 | 29.74 | 46.52 | 22.83 | 31.00 | 31.00 | 31.00
RTUG | ✓ | 20.10 | 16.36 | 19.16 | 30.26 | 63.58 | 44.45 | 13.82 | 14.79 | 11.64
Ours | ✓ | 20.88 | 17.55 | 20.24 | 24.46 | 43.18 | 16.43 | 11.08 | 11.39 | 6.98
Table 1:  Quantitative comparisons in terms of PSNR, FID, and NRI (Section
4.2). For conditional cases, we compute the expected values of PSNR and FID
when the number of synthesized hints follows $\mathcal{G}$.

##  4 Experiments

###  4.1 Sketch Image Datasets

Yumi’s Cells [24] is composed of 10K images from 509 episodes of a web
cartoon, named Yumi’s Cells, where a small number of characters appear
repeatedly. Because it was published in a commercial industry, this dataset
includes not only character objects but also non-character objects, e.g., text
bubbles, letters, and background gradation. Therefore, we chose this dataset
to evaluate the practical effectiveness of our model.

Tag2pix [13] consists of over 60K filtered large-scale anime illustrations
from the Danbooru dataset [6]. While this dataset consists of images of a
single character and a simply colored background, the diversity of each
character in terms of pose and scale makes it challenging to generate
plausible colored outputs. We chose this dataset to verify that our model
reflects various user hints well.

CelebA [18] is a representative dataset which contains 203K human face images
from diverse races. We chose it to evaluate our model on real-world images
rather than artificial ones. We randomly divided each dataset into a training,
a validation, and a test set with the ratio of 81:9:10 and resize all images
to $256\times 256$. Referring to the recipe of Lee et al. (2020) [16], the
sketch images were extracted using the XDoG [33] algorithm.

###  4.2 Evaluation Metrics

Peak signal to noise ratio (PSNR) has been broadly used as a pixel-level
evaluation metric for measuring the distortion degree of the generated image
in the colorization tasks [39, 10]. The metric is computed as the logarithmic
quantity of the maximum possible pixel value of the image divided by the root
mean squared error between a generated image and its groundtruth.

Frechét inception distance (FID). We used FID [8] as an evaluation metric for
measuring the model performance by calculating the Wasserstein-2 distance of
feature space representations between the generated outputs and the real
images. A low FID score means that the generated image is close to the real
image distribution.

Number of required interactions (NRI). We propose a new evaluation metric to
measure how many user interactions are required for the model to produce an
image of a certain quality. To this end, we count the number of hints needed
by the model to reach a benchmark of PSNR. If the model cannot reach a certain
level of accuracy even with the maximum number of hints, we compute the count
as the maximum number of hints plus one. The benchmark can be set according to
the user’s tolerance or the purpose of a framework. We set 20.5, 17.5, and
19.5 as the benchmarks for Yumi, Tag2pix, and CelebA datasets, respectively.

Figure 3: Comparison to baselines on diverse datasets. We compare our model
with two conditional colorization baselines, AlacGAN and RTUG. From top to
bottom, conditional results on CelebA, Tag2pix, and Yumi’s Cells datasets are
presented.

###  4.3 Comparisons against Colorization Baselines

We compare our GuidingPainter against diverse baseline models for the deep
colorization tasks, including an image translation model Pix2Pix [10], an
automatic colorization model CIC [39], a point-based interactive colorization
model RTUG [40], and a scribble-based interactive colorization model AlacGAN
[3]. Since our main focus is interactive colorization, we primarily analyze
the performance of GuidingPainter using conditional cases. In order to analyze
the colorization efficiency of conditional models, we compute NRI and the
expected values of PSNR and FID when the number of color hints follows the
distribution $\mathcal{G}$. The color hints are synthesized by their own
method used for training, i.e., RTUG and AlacGAN provide hints in random
location while our model provides color hints to the regions obtained by the
active-guidance module, in order from the front channel ($S_{1},S_{2},...$).
Table 1 presents the quantitative results of our model and other baselines on
each dataset. Our model outperforms all of conditional baselines on all three
metrics. This reveals that our model can generate various realistic images,
reflecting the given conditions while reducing the interactions. Although our
framework is mainly designed to colorize sketch images when conditions are
given, our model shows the comparable performances across the automatic
colorization setting. We also analyze the effectiveness of our guidance
mechanism in situations where real users give hints in Section 4.4.

As shown in Fig. 3, our model colorizes each color within each segment by
successfully reflecting both the location and the color of hints. The results
show that ours is better than other conditional baselines. For a fair
qualitative comparison, we equalize the number of hints given to each method
and make the locations of the color hints for AlacGAN and RTUG similar to
ours, by sampling the points in the regions that our segmentation network
produces. The marks in the sketch image in Fig. 3 indicate where the hints are
provided for RTUG. Compared with the conditional baselines on the animation
dataset, our model reduces the color bleeding artifact, e.g., the second row
in Fig. 3, and generates the continuous colors for each segment, e.g., hair in
the first row, the sky and the ground in the third row in Fig. 3. This reveals
that our model can distinguish the semantic regions of character and
background and reflect the color hints into the corresponding regions.
Especially, for the last two rows of Fig. 3, our model is superior to colorize
the background region, while other baselines colorize the background across
the edges or only part of the object. Technically, our approach can be applied
to colorize not only a sketch image but also a gray-scale image. Additional
results for qualitative comparison and grayscale colorization are in the
supplementary materials.

|  $\mbox{TPI}_{\downarrow}$ / $\mbox{QS}_{\uparrow}$ | $\mbox{CS}_{\uparrow}$
---|---|---
| Yumi’s Cells | Tag2pix | CelebA | total
RTUG | 11.87 / 3.93 | 8.02 / 3.15 | 7.82 / 3.85 | 3.14
Ours | 7.80 / 4.07 | 7.22 / 4.00 |  7.13 / 3.81 | 4.07
Table 2:  User study results on three different datasets. Time per interaction
(TPI) is the average time (sec.) spent by a user before moving on to next
interaction. Quality score (QS) is the overall quality of a colorized image.
Convenience score (CS) denotes users’ convenience on the overall workflow. QS
and CS are measured from one to five, and all scores were surveyed by users.
Figure 4:  Dark snail example. A sketch image(top left) and a groundtruth
image (top right) on dark snail. (a) are prioritization results of
GuidingPainter, and (b) are when we fix $N_{h}=N_{c}$ of our model during the
training time.

###  4.4 User Study on Interactive Colorization Process

To validate the practical interactive process of our active-guidance
mechanism, we develop a straightforward user interface (UI) that control
peripheral variables except for our main algorithm. We conduct an in-depth
user evaluation, in which users directly participate in the process of our
framework. We then record various metrics to assess the practical usefulness
of our method. We choose RTUG as our baseline interactive method since its
interactive process is directly comparable to ours. As shown in Table 2, our
model shows better time-per-interaction (TPI) scores with less qualitative
degradation than RTUG model, confirming the superior time efficiency of our
model. The total colorization time is decreased by 14.2% on average compared
to RTUG. Furthermore, the improvement in the convenience score (CS) reveals
that our approach clearly reduces the users’ workload. For more details, e.g.,
our UI design, see the supplementary material.

###  4.5 Effectiveness of Active-Guidance Mechanism

To understand the effects of our active-guidance mechanism described in
Section 3.2-3.3, we design two sub-experiments as follows.

Dark Snail. The first one is a simulation to show that the proposed mechanism
works as we expected by using the toy example named Dark Snail. As shown in
the first row of Fig. 4, squares and rectangles are sequentially placed in a
clockwise direction, and a groundtruth is generated at every mini-batch by
having randomly sampled colors of red, green, and blue. In this setting, it is
impossible for a model to estimate the exact color of each object unless each
color hint is provided. Because the size of each rectangle is halved compared
to the previous one, querying the largest region first is an optimal choice in
terms of the information gain. In other words, this toy experiment is designed
to confirm whether our model can (i) divide the semantic regions with the same
color and (ii) ask for the color hints of objects in a descending order by
their size. Fig. 4 (a) shows the guided regions obtained from a model that is
trained by our original mechanism, GuidingPainter. Surprisingly, the original
model tends to build the semantic segments, which are i) bounded by only one
object and ii) placed in decreasing order based on the segment’s size, except
for the 4-th case. Alternately, Fig. 4 (b) is retrieved from a modified
version of our model that is trained by fixing $N_{h}=N_{c}$ during the
training time, i.e., we simply turn off the most critical role of hint
generation function. Fig. 4 (b) demonstrates that the modified model totally
loses its guiding function, implying that the active-guidance mechanism plays
a critical role in our framework.

Figure 5:  PSNR scores of GuidingPainter decrease when masking out all segment
information except for the $i$-th hint.

Importance of highly ranked segments. For every dataset described in Section
4.1, we test how each segment provided by the active-guidance module affects
the performance of colorization. To assess the importance of the $i$-th
segment, we put the map of the $i$-th channel in front of remaining channels
of $S$ and then give a hint only at the first segment. Fig. 5 shows the
tendency that the PSNR score decreases as a hint is given from the rear-ranked
segment, which shows that the active-guidance module encourages to locate the
important regions in the front channels of $S$.

While following the colorization order suggested by the model is an efficient
way to reduce loss at training time, it is also possible to change the
colorization order with additional learning. Detailed discussions on our
approach, including the learning method for changing the order and
limitations, are provided in the supplementary materials.

Figure 6:  Qualitative results of ablation study for the losses (a) $\mathcal{L}_{rec}$, (b) $\mathcal{L}_{rec}+\mathcal{L}_{adv}$, (c) $\mathcal{L}_{rec}+\mathcal{L}_{adv}+\mathcal{L}_{tv}$, and (d) $\mathcal{L}_{rec}+\mathcal{L}_{adv}+\mathcal{L}_{tv}+\mathcal{L}_{smth}$. Each column of images indicates the priority of guided regions. The first row shows regions guided by the model in order of their priority, and the second row represents the intermediate colorization results. | $\mathcal{L}_{rec}$ | $\mathcal{L}_{adv}$ | $\mathcal{L}_{tv}$ | $\mathcal{L}_{smth}$ | PSNR / FID / NRI
---|---|---|---|---|---
| Yumi’s Cells | Tag2pix | CelebA
(a) | ✓ |  |  |  |  21.04 / 28.30 / 9.95 |  18.11 / 66.21 / 8.86 |  20.76 / 39.73 / 5.58
(b) | ✓ | ✓ |  |  | 20.84 / 27.90 / 11.05 | 17.55 / 48.37 / 11.30 |  20.41 / 17.15 / 6.13
(c) | ✓ | ✓ | ✓ |  | 20.72 / 27.81 / 10.39 |  17.58 / 47.32 / 10.48 | 20.19 / 16.54 / 7.20
(d) | ✓ | ✓ | ✓ | ✓ |  20.88 / 24.46 / 11.08 | 17.55 / 43.18 / 11.39 | 20.24 / 16.43 / 6.98
Table 3:  Quantitative results of the ablation study for the losses (a)
$\mathcal{L}_{rec}$, (b) $\mathcal{L}_{rec}+\mathcal{L}_{adv}$, (c)
$\mathcal{L}_{rec}+\mathcal{L}_{adv}+\mathcal{L}_{tv}$, and (d)
$\mathcal{L}_{rec}+\mathcal{L}_{adv}+\mathcal{L}_{tv}+\mathcal{L}_{smth}$.

###  4.6 Effectiveness of Loss Functions

This section analyzes the effects of each loss function using both
quantitative measurements and qualitative results. In this ablation study, we
found a trade-off between the pixel-distance-based metric, i.e., PSNR, and the
feature-distribution-based metric, i.e., FID, according to the combination of
loss functions. Since $\mathcal{L}_{rec}$ exactly matches up to the PSNR,
Table 3 (a) shows the best score of the PSNR-related measurement. However, it
does not perform well in terms of FID especially in the Tag2pix and CelebA
datasets. This phenomenon can also be found in Fig. 6 (a). The character in
the first colorization result tends to be painted with grayish color, and
overall colorization results loss sharpness. After $\mathcal{L}_{adv}$ is
added, the FID scores in Table 3 (b) dramatically improve, along with the
qualitative results in Fig. 6 (b), but PSNR-based scores slightly decrease. As
discussed in a previous work [32], we guess that the PSNR score is not
sufficient to measure how naturally a model can generate if only partial
conditions are given. Although Fig. 6 (b) shows plausible images, the hair in
all the output images are slightly stained. By adding $\mathcal{L}_{tv}$,
these stains are removed, and the colors become clear, as shown in Fig. 6 (c).
After adding $\mathcal{L}_{smth}$, the guided regions become significantly
less sparse than before, and the strange colors on the sleeve of Fig. 6 (c)’s
character disappear, as shown in Fig. 6 (d). Table 3 shows the FID score
improves after adding $\mathcal{L}_{adv}$, $\mathcal{L}_{tv}$, and
$\mathcal{L}_{smth}$ one by one from $\mathcal{L}_{rec}$ on all datasets.
Despite the trade-off, we select (d) as our total loss function, considering
the qualitative improvements and the balance between the PSNR-based and FID
metrics.

##  5 Conclusions

This work presents a novel interactive deep colorization framework, which
enables the model to learn the priority regions of a sketch image that are
most in need of color hints. Experimental results show that our framework
improves the image quality of interactive colorization models, successfully
reflecting the color hints with our active guidance mechanism. Importantly,
our work demonstrates that GuidingPainter, without any manual supervision at
all, can learn the ability to divide the semantic regions and rank them in
decreasing order of priority by utilizing the colorization signal in an end-
to-end manner. We expect that our approach can be used to synthesize hints for
training other interactive colorization models. Developing a sophisticated UI
which integrates our region prioritization algorithm with diverse techniques,
such as region refinement, remains as our future work.

Acknowledgments This work was supported by the Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korean government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate
School Program (KAIST) and No. 2021-0-01778, Development of human image
synthesis and discrimination technology below the perceptual threshold) and
the National Research Foundation of Korea (NRF) grant funded by the Korean
government (MSIT) (No. NRF-2022R1A2B5B02001913).

## References

  * [1] Yun Cao, Zhiming Zhou, Weinan Zhang, and Yong Yu.  Unsupervised diverse colorization via generative adversarial networks.  In Joint European conference on machine learning and knowledge discovery in databases, pages 151–166. Springer, 2017.
  * [2] Yu Cheng, Zhe Gan, Yitong Li, Jingjing Liu, and Jianfeng Gao.  Sequential attention gan for interactive image editing, 2020.
  * [3] Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, and Zhongxuan Luo.  User-guided deep anime line art colorization with conditional adversarial networks.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1536–1544, 2018.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, 2014.
  * [6] Aaron Gokaslan Gwern Branwen.  Danbooru2017: A large-scale crowdsourced and tagged anime illustration dataset.  https://www.gwern.net/Danbooru2017, 2018.  [Online; accessed 22-03-2018].
  * [7] Eric Heim.  Constrained generative adversarial networks for interactive image generation, 2019.
  * [8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NIPS, 2017.
  * [9] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell.  Segmentation from natural language expressions, 2016.
  * [10] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, 2017.
  * [11] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In ICLR, 2017.
  * [12] Youngjoo Jo and Jongyoul Park.  Sc-fegan: Face editing generative adversarial network with user’s sketch and color, 2019.
  * [13] Hyunsu Kim, Ho Young Jhoo, Eunhyeok Park, and Sungjoo Yoo.  Tag2pix: Line art colorization using text tag with secat and changing loss.  In ICCV, 2019.
  * [14] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.
  * [15] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation, 2020\.
  * [16] Junsoo Lee, Eungyeup Kim, Yunsung Lee, Dongjun Kim, Jaehyuk Chang, and Jaegul Choo.  Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence, 2020.
  * [17] Yao Li, Xianggang Yu, Xiaoguang Han, Nianjuan Jiang, Kui Jia, and Jiangbo Lu.  A deep learning based interactive sketching system for fashion images design, 2020.
  * [18] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In ICCV, 2015.
  * [19] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  In ICLR, 2017.
  * [20] Soumajit Majumder and Angela Yao.  Localized interactive instance segmentation, 2020.
  * [21] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In ICCV, 2017.
  * [22] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [23] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.  Text-adaptive generative adversarial networks: Manipulating images with natural language, 2018.
  * [24] NaverWebtoon.  Yumi’s cells.  https://comic.naver.com/webtoon/list.nhn?titleId=651673, 2019.  [Online; accessed 22-11-2019].
  * [25] Weili Nie, Nina Narodytska, and Ankit Patel.  Relgan: Relational generative adversarial networks for text generation.  In ICLR, 2019.
  * [26] Tiziano Portenier, Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker.  Faceshop: Deep sketch-based face image editing, 2018.
  * [27] Holger R Roth, Dong Yang, Ziyue Xu, Xiaosong Wang, and Daguang Xu.  Going to extremes: Weakly supervised medical image segmentation, 2020\.
  * [28] Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays.  Scribbler: Controlling deep image synthesis with sketch and color.  In CVPR, 2017.
  * [29] Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang.  Instance-aware image colorization.  In CVPR, 2020.
  * [30] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Lu Yuan, Sergey Tulyakov, and Nenghai Yu.  Michigan: Multi-input-conditioned hair image generation for portrait editing, 2020.
  * [31] Hao Tang, Song Bai, Philip H. S. Torr, and Nicu Sebe.  Bipartite graph reasoning gans for person image generation, 2020.
  * [32] Zhou Wang, Eero P Simoncelli, and Alan C Bovik.  Multiscale structural similarity for image quality assessment.  In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398–1402. Ieee, 2003.
  * [33] Holger WinnemöLler, Jan Eric Kyprianidis, and Sven C Olsen.  Xdog: an extended difference-of-gaussians compendium including advanced image stylization.  Computers & Graphics, 36(6):740–753, 2012.
  * [34] Yi Xiao, Peiyao Zhou, and Yan Zheng.  Interactive deep colorization with simultaneous global and local inputs, 2018.
  * [35] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks, 2017.
  * [36] Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jaehyuk Chang, and Jaegul Choo.  Coloring with limited data: Few-shot colorization via memory augmented networks.  In CVPR, 2019.
  * [37] Lvmin Zhang, Chengze Li, Edgar Simo-Serra, Yi Ji, Tien-Tsin Wong, and Chunping Liu.  User-guided line art flat filling with split filling mechanism.  In CVPR, 2021.
  * [38] Lvmin Zhang, Chengze Li, Tien-Tsin Wong, Yi Ji, and Chunping Liu.  Two-stage sketch colorization.  ACM Transactions on Graphics (TOG), 37(6):1–14, 2018.
  * [39] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful image colorization.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016.
  * [40] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros.  Real-time user-guided image colorization with learned deep priors.  ACM Transactions on Graphics (TOG), 36(4):1–11, 2017.
  * [41] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, 2017.
  * [42] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros.  Generative visual manipulation on the natural image manifold, 2018.

##  Appendix A Overview

Figure 7: Qualitative results of gray-scale colorization on (a) CelebA, (b)
Imagenet Car, and (c) Summer2Winter Yosemite datasets. First column represents
gray-scale inputs. Second and fourth columns show first five guided regions
generated by the segmentation network and mark regions to change the color as
small squares. Third column shows the colorization results when reflecting
seven original color hints and sixth column shows the colorization results
when changing the color of the marked regions.

This document addresses with additional information that we does not cover in
our main paper due to the page limit. Section B show the applicability of our
approach to a gray-scale image on diverse datasets. Section C describes the
implementation details for the reproducibility, including the network
architectures, the hyperparameters of optimizers we used, and the training
details. Section D provides discussions on our approach, including our hinting
mechanism, a method to change colorization order, and limitation of our
method. Section E gives detailed descriptions of the user studies we conducted
in Section 4.4 of our main paper. At the rest of this document (Section F), we
demonstrate qualitative results of our proposed method. Note that an attached
video material could help to understand the behaviors of GuidingPainter in a
visual manner, and source codes are provided for needs of code-level details.

##  Appendix B Application to Gray-scale Colorization

We test our approach using the gray-scale input on CelebA [18], Imagenet Car
[4], and Summer2Winter Yosemite [41] dataset. To do this, we modify the
colorization model to take the L channel of an image and output the AB channel
of the image. Table 4 represents the quantitative results of RTUG and our
model on each dataset. Our model surpasses the baseline in terms of both PSNR
and FID. This demonstrates that our model can produce realistic images while
reflecting color hints in grayscale colorization. As shown in Fig. 7, our
model can guide the color hints to reasonably shaped regions and reflect the
color hints.

|  $\mbox{PSNR}_{\uparrow}$ / $\mbox{FID}_{\downarrow}$
---|---
| CelebA | Imagenet Car | Yosemite
RTUG | 29.92 / 3.61 | 26.40 / 31.90 | 27.58 / 57.97
Ours |  30.60 / 2.40 |  26.97 / 27.54 |  27.94 / 54.05
Table 4:  Quality comparison of grayscale colorization results in terms of PSNR, FID. We provide RTUG and our model the same amount of color hints which follows $\mathcal{G}$. Label | Layer
---|---
$E_{1}$ | DoubleConv($I:C_{in}$,$O:64$)
$E_{2}$ | DoubleConv($I:64$, $O:128$)
$E_{3}$ | DoubleConv($I:128$, $O:256$)
$E_{4}$ | DoubleConv($I:256$, $O:512$)
$D_{0}$ | DoubleConv($I:512$, $O:512$)
$D_{1}$ | DoubleConv($I:1024$, $O:256$)
$D_{2}$ | DoubleConv($I:512$, $O:128$)
$D_{3}$ | DoubleConv($I:256$, $O:64$)
$D_{4}$ | DoubleConv($I:128$, $O:64$)
Out | Conv($I:64$, $O:C_{out}$)
Table 5: The layer specification of the U-Net. DoubleConv denotes two
consecutive Conv-BatchNorm-ReLU blocks, and Conv denotes a convolution layer.
$I,O$ denote the size of input channels, the size of output channels,
respectively.

##  Appendix C Implementation Details

U-Net Architecture. We adopt the U-Net architecture in the segmentation
network and colorization network, except for the size of channels in the input
layer and output layer. The layer specification is shown in Table. 5, where
$(C_{in},C_{out})$ is equal to $(1,N_{c})$ for segmentation network and equal
to $(N_{c}+5,3)$ for colorization network. Maxpooling is applied to the front
of each layer $E_{2}$-$D_{0}$ to downsample the input tensor by a factor of 2.
For each $i=1...4$, the bilinear upsampled output of $D_{i-1}$ and the output
of $E_{5-i}$ are concatenated in the channel dimension, and then pass through
$D_{i}$. Every convolution in $E_{1}$-$D_{4}$ is applied with $3\times 3$
kernel, whereas the convolution in output layer is applied with $1\times 1$
kernel.

Discriminator. The discriminator $D$ is implemented with PatchGAN[10], which
outputs a $30\times 30$ tensor. We use the LSGAN[21] objective to train the
GAN architecture.

Training Details. We initialize the weight of networks from the normal
distribution with a mean of 0 and a standard deviation of 0.02. The Adam
optimizer [14] with $\beta_{1}=0.5,\beta_{2}=0.999$ is used to train our
networks on all datasets. The learning rate is fixed at 0.0002 for the first
half of epochs and linearly decays to zero for the remaining half of epochs.
We schedule the temperature $\tau$ of ST gumbel estimator with exponential
policy $\tau=0.1^{\text{current epoch}/\text{total epochs}}$, adopted from
RelGAN [25]. The total numbers of epochs for Yumi’s Cells, Tag2Pix, and CelebA
datasets are 500, 30, and 20, respectively. The optimization typically takes
about 1-2 days on 4 TITAN RTX GPUs.

##  Appendix D Discussions

###  D.1 RoI-based Hinting Mechanism

In this study, we mainly compare our RoI-based hinting mechanism with the
point and scribble-based hinting mechanisms. Note that there are trade-offs
between each method. Point or scribble-based approaches have their own
advantages in that they can utilize the location of the color hint. However,
our goal is to colorize images with a few hints, not to colorize perfectly
with plenty of time. We therefore focus on validating the effectiveness of our
region-based guidance system in terms of interaction efficiency. As shown in
the user study (Section 4.4 of our main paper), our region-based guidance
system can reduce the average time per interaction, resulting in the improved
convenience score.

Since artists spend a lot of time in adding a base color on a sketch image(s)
in real-world applications, it is valid work to find efficient method to
mitigate such labour-intensive process. In this context, our work will be able
to make the labour-intensive process significantly efficient. Although
accurately and efficiently colorizing more complex images is still difficult,
we expect that combining our RoI hinting mechanism with the point or scribble
hinting mechanism would be one of promising works to solve this problem.

###  D.2 How to Change Colorization Order

Figure 8: Colorization results of a Dark snail example in two different
colorization orders. (a) shows the colorization results when hinting regions
in decreasing order of its size. Conversely, (b) shows the colorization
results when hinting the small region first.

While GuidingPainter automatically guides color hints to regions in an
efficient order, the users may want to paint the regions regardless of this
fixed order. We found that it is possible to change the colorization order of
GuidingPainter through two-stage learning. After training GuidingPainter with
the ordinary learning process, the segmentation networks gain the ability to
estimate regions from a given sketch image. To make the colorization order
changeable, we train only the colorization networks by fixing $N_{h}=N_{c}$
and randomly dropping out some hints produced by the hint generation module,
i.e., letting each $m_{i}$ Bernoulli random variable with success probability
$p=0.125$, in a second learning phase. As a result of this learning, the
colorization networks can colorize the sketch image even if only some random
regions are given color hints in random order. As shown in Fig. 8, our
modified GuidingPainter can colorize a sketch image in different colorization
orders.

###  D.3 Effectiveness of the Number of Hints

We investigate how the performance of our model and baseline models changes as
the number of hints increases. Fig. 13 shows the change of PSNR and FID score
when each of 2,4,6,8,10,12 hints are provided to each model. We found that
GuidingPainter mostly surpasses performances of the baselines if the same size
of hints are given. If more than or equal to two color hints are given, our
model surpasses other baseline models in both PSNR and FID scores. The results
show that our hint guidance method enables the colorization module to
effectively reflect hints.

###  D.4 Limitation of Our Study

Figure 9: A failure case of the segmentation network.

In some cases, the quality of the result image is degraded due to wrong
prediction of the segmentation network. In Fig. 9, the region marked as red
shows a misaligned segment of the ‘sand’ region inside the ‘sea’ one. We found
that these minor errors can be slightly refined by the following colorization
network. Despite its self-correction, the stain still remains on the result,
which is marked as blue. Mitigating this problem through segmentation
correction techniques would be one of promising future works.

##  Appendix E User Study

This section describes details of the user study in Section 4.4 of our main
paper. In addition, we conduct user-perception study to evaluate whether our
model can reflect unusual color hints.

###  E.1 Efficiency of Interactive Process

Figure 10: User interface of user study. (1) Our model highlights recommended
region to colorize. (2) Considering the region, a user selects a color from
color palettes. (3) The result of reflecting the hint is displayed.

Since our framework aims to increase the interaction-efficiency on
colorization process, we conduct a user study to estimate how our model can
enhance the overall process when the users intervene in it. As a competitive
approach to our framework for estimating interaction-efficiency, we choose
RTUG since the interaction process presented in this original paper [40] is
most similar to ours and we can easily quantify the amount of interactions in
the process.

As shown in Fig. 10, we develop a straightforward user interface(UI) to rule
out peripheral variables except for the main algorithms as much as possible.
The UI consists of a color palette, screens for checking hints and
colorization results, and a few buttons to reflect hints or to select next
hints. The users test our method and RTUG on three datasets, Yumi’s Cells,
Tag2pix, and CelebA. If the user tests our method, the user can see a region
guided by our mechanism in the left image (Fig. 10 (1)), and choose a color
from the palette at the bottom of the screen (Fig. 10 (2)). After selecting
the color, the user can see an inference image on the right of the screen
(Fig. 10 (3)). This overall process is same as when the user tests RTUG,
except the facts that the location of the hint is displayed in the form of a
point on the left image and the user can move the point to the location by
clicking the left image. The user can click the next button to add another
color hint or click the finish button to end up the colorization process.

Before the evaluation of participants, we let the users freely use the UI for
about 5 minutes so that the users can be familiar with it. A total of 13
participants comprised of researchers or engineers related to computer science
and AI attend our user study. Each user is asked colorizing the given sketch
image as naturally as possible without a reference image. For each dataset and
method, the user completes three images using the UI. We guide the users to
finish each colorization task in roughly one minute, preventing the users from
spending too much time on a single task. The evaluation results of the user
study are shown in table 2 of our main paper.

Figure 11:  Results of reflecting diverse color hints. (a) represents accumulated hint images when giving diverse color hints to face, hair, and background. (b) shows the output images of our model for each hint image. |  $\mbox{Top-1}_{\uparrow}$ / $\mbox{Top-2}_{\uparrow}$
---|---
| Yumi’s Cells | Tag2pix | CelebA
AlacGAN | 2.7 / 12.7 | 0.0 / 10.0 | 1.4 / 7.3
RTUG | 20.0 / 90.0 | 13.6 / 91.8 | 29.1 / 97.7
Ours | 77.3 / 97.3 | 86.4 / 98.2 |  69.5 / 95.0
Table 6:  User-perception study results to evaluate how faithfully models
reflect user-provided conditions. ‘Top-k’ indicates how many the generated
images are ranked within the top-k among models over three datasets. All
numbers are in percentages.

###  E.2 Reflecting Unusual Color Hints

We also conduct a user study to evaluate how faithfully our model and the
baselines reflect the user interaction, even though the hints can contain
unusual colors. To be specific, we randomly select 500 images for each test
dataset and prepare images generated by each model with strongly perturbed
color hints as shown in Fig. 11 (a). The perturbed color is created by adding
random values between -64 and 64 to each RGB value of the groundtruth color.

For a fair comparison, we unify the locations of given hints to each model by
randomly sampling them within the guided regions produced by segmentation
network. Simply, the number of provided hints and their positions are similar
across the baselines and ours. The hint images have up to seven hints, and
each model generates images based on the same number of hints. With the
generated image and the hint map, the user is asked to rank the generated
images in the order of how much the hints are properly reflected. Table 6
shows the percentage of generated images within the top-$k$ of the rank over
all datasets. Our model does not only get the highest top-1 ratio over all
datasets, but also successfully reflect the diverse color conditions, as shown
in Fig. 11. This implies that our model can work robustly in terms of color
variations.

##  Appendix F Qualitative Results

This section provides additional qualitative results with the size of
$256\times 256$ over three different datasets. Fig. 13 compares qualitative
results for both automatic and conditional colorization models. Fig. 14, 15
and 16 show how the output images approach groundtruth when there are
interactions between the model and an user on the CelebA[18], Tag2pix[13], and
Yumi’s Cells[24], respectively. Fig. 17 represents diverse output images
according to the colors of given hints on each dataset.

Figure 12: Performance changes according to the number of user hints. Columns
mean the results of Yumi’s Cells [24], Tag2pix [13] and CelebA [18] over
baselines and GuidingPainter. The first row shows the PSNR scores and second
row presents its corresponding FID [8] scores.

Figure 13: Comparison to baselines on diverse datasets. We compare our model
in two approaches: automatic and conditional colorization ones. To assess the
performance of our model in an automatic setting, we choose CIC, Pix2Pix,
AlacGAN, and RTUG as baselines. For the condtional case, we equalize the
number of hints given to all baselines and our model. Our model successfully
colorizes each segment without color bleeding artifact, e.g., the third and
fourth rows, and generates the continuous colors for each segment, e.g., hair
in the first two rows, the sky and the ground in the fifth row.  Figure 14:
Qualitative results on CelebA dataset. For each interaction, which is noted as
number of hints, our model estimates a hint region which the model wants to
know first. Then, we select a representative color for the region and the
color is spread to the region as visualized in accumulated hints. Finally, a
colorization result is generated by our colorization network taking the guided
regions accumulated hints and the sketch image. In these examples, we do not
remove the noise used in the Gumbel Softmax operation to directly represent
the guided regions provided to the colorization network. We summarize the
interaction process in three rows for each four image. The results show how
the input images change along with the color hints at each interaction step.
In particular, the 6-th column shows that the model captures the shadow of
human face and colorizes it appropriate to the image. Figure 15: Qualitative
results on Tag2pix dataset. Each interaction reveals that our model recognizes
semantically related segments for each image, e.g., background, clothes, hair,
and face of a character. In the 1-st iteration, the model concentrates on the
background and adapts a color if the background color is inputted. Especially,
for the hair segment in the 3-rd iteration, our model successfully reflects
the color changes, not bleeding the color outside of the hair region.  Figure
16: Qualitative results on Yumi’s Cells dataset. Each intermediate iteration
shows that semantically meaningful segments are recommended and colorized,
such as, parts of background, speech balloon, clothes, face, and hair for an
image. As shown in the rows of colorization result, the automatically
colorized images become similar to the groundtruth image by adding each color
hint in only eight iteration. This demonstrates that our model not only
reflects the color condition in adequate location, but also improve the
quality of result images. Especially on the last two images, our model fixes
the key color errors, such as the purple night sky, the green clothes, and the
yellow speech bubble. Figure 17: Qualitative results of varying color hints on
diverse datasets. We sample two representative items from each three different
dataset of CelebA, Tag2pix, and Yumi’s cells. To confirm how well our model
can reflect user-interaction, we only vary the color of the hint after fixing
the input sketch image and its guided regions. The results show our model
covers a wide range of color palette, including the green and purple faces.