[ orcid=0000-0002-5263-8722 ]

<Credit authorship details>

[ orcid=0000-0002-0240-1065 ]

[cor]Corresponding author

tti]organization=Toyota Technological Institute, addressline=2-12-1 Hisakata,
Tempaku-ku, city=Nagoya, postcode=468-8511, state=Aichi, country=Japan

# Joint Learning of Blind Super-Resolution and Crack Segmentation for
Realistic Degraded Images

Yuki Kondo<EMAIL_ADDRESS>Norimichi Ukita<EMAIL_ADDRESS>[

###### Abstract

This paper proposes crack segmentation augmented by super resolution (SR) with
deep neural networks. In the proposed method, a SR network is jointly trained
with a binary segmentation network in an end-to-end manner. This joint
learning allows the SR network to be optimized for improving segmentation
results. For realistic scenarios, the SR network is extended from non-blind to
blind for processing a low-resolution image degraded by unknown blurs. The
joint network is improved by our proposed two extra paths that further
encourage the mutual optimization between SR and segmentation. Comparative
experiments with SoTA segmentation methods demonstrate the superiority of our
joint learning, and various ablation studies prove the effects of our
contributions.

###### keywords:

Crack segmentationImage processingBlind super resolutionJoint learning Multi-
task learning

##  1 Introduction

(a) Input LR (b) HR GT (c) Independent [12] (d) Multi-task [102] (e) Ours
(CSBSR)

Figure 1: Difficulty in real-world crack segmentation. From an input Low-Resolution (LR) image (a), High-Resolution (HR) segmentation results (c), (d), and (e) are acquired. (c) Independent and (d) Multi-task show the results on images enlarged by non-blind SR “trained independently of segmentation” and “trained with segmentation in a multi-task learning manner,” respectively. (b) is the manually-annotated ground-truth HR segmentation image. Table 1: Problems $\langle A\rangle$, $\langle B\rangle$, $\langle C\rangle$, and $\langle D\rangle$ and their solutions 1, 2, 3, and 4. If solution $S$ is for problem $\langle P\rangle$, column $P$ of row $S$ in the table is $\surd$. |  $\langle A\rangle$ Class imbalance |  $\langle B\rangle$ Fine cracks |  $\langle C\rangle$ LR cracks |  $\langle D\rangle$ Blur
---|---|---|---|---
1\. CSBSR |  |  | $\surd$ | $\surd$
2\. BC loss | $\surd$ | $\surd$ |  |
3\. Segmentation-aware SR-loss weights |  | $\surd$ | $\surd$ |
4\. Blur-reflected task learning |  |  |  | $\surd$

While many constructions and infrastructures such as buildings, pavements,
bridges, and tunnels are dilapidated in the world, it is difficult to always
manually inspect all of them. Instead of the manual inspection, automatic
inspection is one of the prospective solutions for efficiently diagnosing
these constructions. While such inspection can be achieved by several types of
sensors such as the Falling Weight Deflectometer, the Pavement Density
Profiler, and the Ground Penetrating Radar, this paper focuses on crack
segmentation on images captured by generic cameras for visual inspection.

Crack segmentation [31] is defined to be binary semantic segmentation in the
field of computer vision. While the number of classes in crack segmentation
(i.e., two classes) is much fewer than the one of recent generic segmentation
tasks, such as scene segmentation [53, 70] and aerial image segmentation [63],
real-world crack segmentation is not an easy problem even with recent powerful
deep neural networks. This is because of the following reasons:

  1. $\langle$A$\rangle$.

High class-imbalance: The number of crack pixels is much less than the number
of non-crack pixels (i.e., background pixels), as shown in Fig. 1 (b). In such
a problem, all pixels tend to be classified to background.

  2. $\langle$B$\rangle$.

Fine cracks: Cracks can be hairline, which are difficult to be segmented, as
shown in Fig. 1 (b).

  3. $\langle$C$\rangle$.

Low-Resolution (LR): For inspection of various structures such as tunnels [96,
11], pavements [32, 5], and bridges [84, 25], an inspection camera captures
cracks in LR (as shown in Fig. 1 (a)) because it cannot get close to the
structures for safety reasons.

  4. $\langle$D$\rangle$.

Cracks in blurred images: Since inspection images are usually captured from
moving vehicles such as cars and drones for efficient inspection, those images
can be blurred, as shown in Fig. 1 (a).

While even each of the aforementioned problems is not an easy problem, crack
segmentation is more challenging due to the combination of all of these
problems, even with SoTA methods, as shown in Fig. 1 (c) and (d). To cope with
these problems, this paper proposes a unified framework consisting of the
following novel contributions (Table 1):

  1. 1.

Crack Segmentation with Blind Super-Resolution (CSBSR): As with Crack
Segmentation with Super Resolution (CSSR) proposed in our earlier conference
paper [60], CSBSR proposed in this paper connects “a network for Super
Resolution (SR) accepting an input LR image” in series to “a segmentation
network” for end-to-end joint learning. We extend CSSR to CSBSR with blind SR
to handle realistically-blurred images. Our joint learning of blind SR and
segmentation allows us to optimize SR for improving segmentation (Fig. 1 (e))
more than similar methods [12, 102] using both SR and segmentation (Fig. 1 (c)
and (d)).

  2. 2.

Boundary Combo (BC) loss for segmentation: In addition to super-resolving tiny
cracks as mentioned above, fine boundaries are locally evaluated with global
constraints in the whole image for detecting fine cracks robustly to the
class-imbalance problem.

  3. 3.

Segmentation-aware SR-loss weights: While CSSR and CSBSR use BC loss to train
not only the segmentation network but also the SR network in and end-to-end
manner, the SR network is less optimized due to gradient vanishing through the
segmentation network. To train the SR network more for segmentation, BC loss
directly weights a loss for SR. For further improvement, the SR loss is also
weighted by additional weights based on fine-crack and hard-negative pixels.

  4. 4.

Blur skip for blur-reflected task learning: Since an SR image is imperfect,
blur effects remaining in the SR image give a negative impact on segmentation.
For segmentation more robustly to the blur effects, the blur estimated in SR
is provided to the segmentation network via a skip connection.

Our code is available at https://github.com/Yuki-11/CSBSR.

##  2 Related Work

###  2.1 Image Segmentation

Image segmentation techniques [78] are briefly divided into three categories,
namely semantic segmentation [90], instance segmentation [47], and panoptic
segmentation [59]. Crack segmentation is categorized into semantic
segmentation because it classifies all pixels into crack and background pixels
with no instance. That is, these crack pixels are not divided into crack
instances.

Class-imbalance Segmentation: As well as in various computer vision problems,
in image segmentation, class imbalance is a critical problem. Many approaches
for class imbalance are applicable to class-imbalance segmentation tasks. For
example, weighted loss such as the Weighted Cross Entropy (WCE) loss [26] and
the focal loss [67] for segmentation [64, 49, 109], re-sampling [126] for
segmentation [18], and hard mining [29] for segmentation [35].

Among all segmentation tasks, medical image segmentation has to cope with
highly-imbalanced classes (e.g., tiny tumors and background). Such difficult
medical image segmentation is tackled by a variety of loss functions such as
the Dice loss [77], the Generalized Dice loss [97], the Combo loss [98], the
Hausdorff loss [54], and the Boundary loss [55].

Crack Segmentation: Since the class-imbalance issue is important also for
crack segmentation as presented as Problem <A> in Table 1, the aforementioned
schemes proposed against class imbalance are useful for crack segmentation.
For example, in order to balance the number of samples between classes, Crack
GAN [119] oversamples crack images by using DC-GAN [85]. The Dice, Combo, and
WCE losses are employed for crack segmentation in [86], in [19], and in [71,
128, 69], respectively.

In addition to the class-imbalance issue, the fine boundaries of cracks are
not easy to be extracted and make crack segmentation difficult, as presented
as Problem <B> in Table 1. For such difficult fine crack segmentation, the
aforementioned schemes proposed against class imbalance (e.g., weighted loss,
re-sampling, class-imbalance-oriented loss) are also useful. Previous methods
for such fine cracks are divided into the following two approaches, namely
boundary-based and coarse-to-fine weighting.

In the boundary-based approach, the distance between the boundaries of ground-
truth and predicted cracks is minimized. In [54], the Hausdorff distance is
evaluated by using the distance transform. While its computational cost for
the exact solution is high, the sum of L2 distances is approximated by the sum
of regional integrals for efficiency in the Boundary loss [55].

Various coarse-to-fine weighting approaches such as [20] employ pyramid and
U-net like networks for weighting a fine but unreliable representation by more
reliable results in a coarse representation. The effectiveness of this
approach is validated also in crack segmentation [107, 128, 71, 69, 22, 66].

While the effectiveness of the both approaches is validated, the coarse-to-
fine weighting approach is applicable only to pyramid and U-net like
architectures. On the other hand, the boundary-based approach can be employed
with any other loss functions in any network architectures in general.

###  2.2 Super Resolution (SR)

Non-blind SR: SR reconstructs a High-Resolution (HR) image, $I^{H}$, from its
LR image, $I^{L}$. The image degradation process from HR to LR is modeled as
follows:

$I^{L}=(I^{H}\ast K)\downarrow_{s},$ (1)

where $K$, $\ast$, $\downarrow$, and $s$ denote a blur kernel, a convolution
operator, a downsampling process, and a scaling factor, respectively. By
downscaling HR training images to their LR images by a known downsampling
process, such as bicubic interpolation, we can have a set of $I^{H}$ and
$I^{L}$ for training a non-blind SR model.

Such non-blind SR is developed with various aspects [101, 17, 37, 116, 105]
such as arbitrary image degradations [117, 118, 115], attention mechanisms
[24, 81, 76], recurrent/iterative networks [65, 61, 43], stochastic generation
[89, 13, 73], and reference-based SR [122, 92, 72]. However, since the image
degradation process is assumed to be known in all of these non-blind SR
methods, their performance is decreased in real-world images with arbitrary
unknown degradations.

Blind SR: To apply SR to arbitrarily-degraded images, blur kernel $K$ is
employed in blind SR. Even without modeling $K$ in a SR network, blind SR can
be done by blurring training images [95, 125, 83, 48, 114] or by deblurring
input images [52] by $K$. In the kernel conditioning approach [23, 36, 74, 51,
103], a blur representation estimated from an input LR image is fed into a SR
network for conditioning the SR process by the blur. While this kernel
conditioning employs low-dimensional blur representations for efficiency and
stability in general, the original blur kernel, $K$, is modeled within a SR
network for further accuracy in [38, 57].

Since the blur kernel is more informative than its low-dimensional
representation, the blur kernel can be useful for additional tasks using a SR
image. As such an additional task, image segmentation is done in our work.

###  2.3 Joint Learning of SR and Other Tasks

(a) Independent learning with non-blind SR [12]

(b) Multi-task learning with non-blind SR [102]

(c) Joint learning with blind SR and extra paths (Ours)

Figure 2: Combinations of SR and segmentation. While orange arrows indicate
data flows, arrows leading out of the loss functions (i.e., $\mathcal{L}_{S}$
and $\mathcal{L}_{C}$) indicate the back-propagation paths for training. Blue
and green arrows indicate the back-propagations given by the SR and
segmentation tasks, respectively. Each ellipse indicates a loss or weights
given to a certain loss. Our CSBSR (c) is illustrated more in detail in Fig.
3.

With upscaled SR images, a variety of applications can be realized. For
example, distant-object detection [40, 50, 33] and segmentation [39], remote
sensing [10, 93], wide-angle image analysis[21, 8], and cell image analysis
[68, 79]. As with these examples, crack segmentation can be also supported by
SR [12] for detecting blurred LR cracks presented in Problems <C> and <D> in
Table 1.

While these methods have models for SR and another task (e.g., segmentation)
separately (Fig. 2 (a)), these tasks can be jointly trained in a single model
for supporting the additional task more explicitly. Such joint end-to-end
learning is also applicable in a variety of tasks such as classification [99,
94] and detection (e.g., faces [14], pedestrians [82, 121], vehicles [7], and
generic objects [62, 15, 44]).

Image segmentation can be also improved by combining with SR. As shown in Fig.
2 (b), DSRL [102] applies multi-task learning to the non-blind SR and
segmentation tasks so that a single feature extractor is shared by the
parallel SR and segmentation branches following the extractor. While multi-
task learning may improve both tasks, the SR and segmentation branches are
independently trained.

##  3 Joint Learning of Blind SR and Crack Segmentation

While methods using joint end-to-end learning with SR [99, 94, 14, 82, 121,
62, 15, 44, 102] mentioned in Sec. 2.3 are close to our work, it is difficult
to apply them to crack segmentation in realistic scenarios. This is because
these methods using non-blind SR cannot cope with unknown blurs observed in
images degraded by out-of-focus and motion blurs. Our CSBSR resolves this
problem by employing blind SR in joint learning (Sec. 3.1). For further coping
with the class-imbalance issue, this paper also proposes a new combination of
loss functions for class-imbalance fine segmentation (Sec. 3.2). In addition
to joint learning, we propose loss weighting for optimizing segmentation more
for SR in Sec. 3.3 and extra skip connection paths for optimizing SR more for
segmentation, as described in Secs. 3.4.

###  3.1 Joint Learning

Figure 3: Proposed joint learning network with blind SR and segmentation. See
the caption of Fig. 2 for the explanations of arrows and ellipses. $\odot$
indicates a pixelwise multiplication operator.

CSBSR consists of blind SR and segmentation networks, as shown in Fig. 2 (c).
Its detail is shown in Fig. 3. The blind SR network, $S(I^{L};\theta_{S})$
where $\theta_{S}$ denotes all the parameters of this SR network, maps $I^{L}$
to its SR image, $I^{S}$. The crack segmentation network, $C$, takes $I^{S}$
and outputs a crack segmentation image $I^{C}=C(I^{S};\theta_{C})$. Any
differentiable SR and crack segmentation networks can be employed as $S$ and
$C$, respectively. Let $\mathcal{L}_{S}$ and $\mathcal{L}_{C}$ denote loss
functions for $S$ and $C$, respectively. While $\mathcal{L}_{S}$ is back-
propagated through $S$, $\mathcal{L}_{C}$ is back-propagated through $S$ and
$C$ in an end-to-end manner. The whole network is trained by the following
loss with the task weight $\beta$ as a hyper-parameter:

$\displaystyle\mathcal{L}_{J}$ $\displaystyle=$
$\displaystyle(1-\beta)\mathcal{L}_{S}+\beta\mathcal{L}_{C}$ (2)

##### Implementation details

In our experiments, DBPN [45, 43] and its extension to blind SR [111], which
is called KBPN, are employed as $S$ for fair comparison between our proposed
methods with non-blind SR and blind SR (i.e., comparison between CSSR and
CSBSR). Different from DBPN as non-blind SR, KBPN also outputs its estimated
blur kernel. Loss functions used in [45, 43] and KBPN [111] are used as
$\mathcal{L}_{S}$ in our joint learning with no change.

$C$ is implemented with each of U-Net [87], PSPNet [124], CrackFormer [69],
and HRNet+OCR [112] for validating a wide applicability of our method111The
implementations of these SR and segmentation networks are publicly available
[41, 110, 1, 123, 2, 113].. Section 3.2 proposes a new general-purpose
segmentation loss, which is applicable to all of these networks as
$\mathcal{L}_{C}$.

###  3.2 Boundary Combo Loss

For suppressing class-imbalance difficulty in crack segmentation, we propose
the Boundary Combo (BC) loss that simultaneously achieves locally-fine and
globally-robust segmentation. Fine segmentation can be achieved by the
boundary-based approach such as the Boundary loss [55]. However, if only the
boundary-based approach is employed, the segmentation network is easy to fall
into local minima, as validated in [55]. This problem can be resolved by
employing the boundary-based approach simultaneously with a loss that
evaluates the whole image region. In [55], the Generalized Dice (GDice) loss
[97] is empirically demonstrated to be a good choice. However, it is reported
that the Sigmoid function included in the GDice loss and its original Dice
loss tends to cause the vanishing gradient problem [98].

This paper explores more appropriate losses combined with the Boundary loss
for stable learning as well as fine segmentation. We improve learning
stability by combining the GDice loss with the WCE loss that is expressed
without the derivative of the Sigmoid function, which tends to cause gradient
vanishing. Since the Dice loss and the WCE loss have different properties
(i.e., which are categorized to region-based and distribution-based losses,
respectively, as introduced in [75]), it is also validated that a pair of the
Dice and WCE losses, which is called the Combo loss [98], complementarily work
for better segmentation. Finally, we propose the following Boundary Combo (BC)
loss, $\mathcal{L}_{BC}$, as $\mathcal{L}_{C}$ for $C$ in our joint learning:

$\displaystyle\mathcal{L}_{BC}$ $\displaystyle=$
$\displaystyle\alpha\mathcal{L}_{B}+(1-\alpha)\left((1-\gamma)\mathcal{L}_{D}+\gamma\mathcal{L}_{WCE}\right),$
(3)

where $\mathcal{L}_{B}$, $\mathcal{L}_{D}$ and $\mathcal{L}_{WCE}$ denote the
Boundary, Dice, and WCE losses, respectively. $\alpha\in[0,1]$ and
$\gamma\in[0,1]$ are hyper-parameters. $\mathcal{L}_{BC}$ consists of the
region, distribution, and boundary-based losses. A combination of these three
loss categories are never evaluated according to the survey [75]. As a variant
of $\mathcal{L}_{BC}$, we also propose $\mathcal{L}_{GBC}$ in which the GDice
loss $\mathcal{L}_{GD}$ is used in $\mathcal{L}_{BC}$ instead of
$\mathcal{L}_{D}$.

While one may refer to the original papers of $\mathcal{L}_{B}$,
$\mathcal{L}_{D}$, $\mathcal{L}_{GD}$, and $\mathcal{L}_{WCE}$ for the
details, these losses are briefly explained in the following three paragraphs.

##### Boundary loss ($\mathcal{L}_{B}$)

The Boundary loss [55], computes the distance-weighted 2D area between the
ground-truth crack and its estimated one, which becomes zero in the ideal
estimation, as follows:

$\displaystyle D(\partial G,\partial S)$ $\displaystyle=$
$\displaystyle\int_{\partial G}||q_{\partial S}(p)-p||^{2}dp$ (4)
$\displaystyle\simeq$ $\displaystyle 2\int_{\Delta S}D_{G}(p)dp$
$\displaystyle=$ $\displaystyle
2\bigl{(}\int_{\Omega}\phi_{G}(p)s(p)-\int_{\Omega}\phi_{G}(p)g(p)dp\bigr{)},$

where $G$ and $S$ denote the pixel sets of the ground-truth crack and its
estimated one, respectively. $p$ and $q_{\partial S}(p)$ denote a point on
boundary $\partial G$ and its corresponding point on boundary $\partial S$,
respectively. $q_{\partial S}(p)$ is an intersection between $\partial S$ and
a normal of $\partial G$ at $p$. $\Delta S=(S/G)\cup(G/S)$ is the mismatch
part between $G$ and $S$. $D_{G}(p)$ is the distance map from $G$. $s(p)$ and
$g(p)$ are binary indicator functions, where $s(p)=1$ and $g(p)=1$ if $p\in S$
and $p\in G$, respectively. $\phi_{G}(q)$ is the level set representation of
boundary $\partial G$: $\phi_{G}=-D_{G}(q)$ if $q\in G$, and
$\phi_{G}=D_{G}(q)$ otherwise. $\Omega$ denotes a pixel set in the image. The
second term in Eq. (5) is omitted as it is independent of the network
parameters. By replacing $s(p)$ by the network softmax outputs
$s_{\theta}(p)$, we obtain the Boundary loss function below:

$\mathcal{L}_{B}=\int_{\Omega}\phi_{G}(p)s_{\theta}(p)dp$ (5)

##### Dice and GDice losses ($\mathcal{L}_{D}$ and $\mathcal{L}_{GD}$)

The Dice loss [77] is a harmonic mean of precision and recall as expressed as
follows:

$\displaystyle\mathcal{L}_{D}$ $\displaystyle=$
$\displaystyle\frac{2\sum^{M}_{j}\sum^{N}_{i}p_{ij}g_{ij}}{\sum^{M}_{j}\sum^{N}_{i}(p_{ij}^{2}+g_{ij}^{2})},$
(6)

where $M$ and $N$ denote the number of classes (i.e., $M=2$ in our problem)
and the number of all pixels in each image, respectively. $p_{ij}$ and
$g_{ij}$ are the classification probability ($0\leq p_{ij}\leq 1$) and its
ground truth ($g_{ij}\in\\{0,1\\}$).

Different from the Dice loss, the GDice loss [97] is weighted by the number of
pixels in each class as follows:

$\displaystyle\mathcal{L}_{GD}$ $\displaystyle=$
$\displaystyle\frac{2\sum^{M}_{j}w_{j}^{(GD)}\sum^{N}_{i}p_{ij}g_{ij}}{\sum^{M}_{j}w_{j}^{(GD)}\sum^{N}_{i}(p_{ij}+g_{ij})},$
(7)

where $w_{j}^{(GD)}=\frac{1}{\sum^{N}_{i}g_{ij}}$.

##### WCE loss ($\mathcal{L}_{WCE}$)

The WCE loss [26] is the Cross Entropy loss weighted by a hyper parameter,
$w_{j}^{(WCE)}$, which is determined based on the class imbalance (e.g.,
$w_{j}^{(WCE)}=\frac{1}{\sum^{N^{\prime}}_{i}g_{ij}}$ where
$N^{\prime}=NN_{I}$ and $N_{I}$ is the number of all training images):

$\displaystyle\mathcal{L}_{WCE}=\sum^{M}_{j}w_{j}^{(WCE)}\sum^{N}_{i}g_{ij}\log
p_{ij}$ (8)

###  3.3 Segmentation-aware Weights for SR

In addition to end-to-end learning with $\mathcal{L}_{C}$ (i.e., segmentation
loss in Eq. (2)), we propose to weight $\mathcal{L}_{S}$ by $\mathcal{L}_{C}$
for further optimizing the SR network $S$ for segmentation. This weighting is
achieved by pixelwise multiplying $\mathcal{L}_{S}$ by $\mathcal{L}_{C}$.

It is not yet easy to discriminate between crack and background pixels for
precisely detecting fine cracks. This difficulty arises especially around
crack pixels. For such difficult pixelwise segmentation, our method employs
the following two difficulty-aware weights:

  * •

For detecting all fine thin cracks, a segmentation loss function is weighted
so that pixels inside and around cracks are weighted higher. A weight given to
pixel $p$, $w^{C}_{p}$, is expressed as follows:

$\displaystyle w^{C}_{p}$ $\displaystyle=$ $\displaystyle\exp(-m^{C}D_{p})$
(9)

where $m^{C}$ and $D_{p}$ denote a weight constant and a distance between $p$
and its nearest crack pixel, respectively. $w^{C}_{p}$ is called the Crack-
Oriented (CO) weight.

  * •

For hard pixel mining, a segmentation loss function is weighted so that pixels
inside and around false-positive and false-negative pixels are weighted
higher. For such difficulty-aware segmentation, in our method, a weight given
to pixel $p$, $w^{F}_{p}$, is expressed as follows:

$\displaystyle w^{F}_{p}$ $\displaystyle=$
$\displaystyle\exp(m^{F}|T^{P}_{p}-T^{GT}_{p}|),$ (10)

where $0\leq T^{P}_{p}\leq 1$ and $T^{GT}_{p}\in\\{0,1\\}$ denote the value of
$p$-th pixel in predicted and ground-truth segmentation images, respectively.
$m^{F}$ is a weight constant. Our $w^{F}_{p}$ is applicable to any loss
function such as our BC loss, Eq. (3), consisting of multiple loss functions,
while the focal loss [67] and the anchor loss [88], both of which also
penalize hard samples, are based on a weighted cross entropy loss. $w^{F}_{p}$
is called the Fail-Oriented (FO) weight.

These two weights (9) and (10) are multiplied pixelwise by $\mathcal{L}_{S}$.

###  3.4 Blur Skip for Blur-reflected Task Learning

Figure 4:  The structure of our blur skip module using SFT [104]. Each 3D box
and rectangle depict a feature set and a process, respectively. $\odot$
indicates a pixelwise multiplication operator.

It is not easy for the blind SR network to perfectly predict the ground-truth
blur kernel $K$ and the ground-truth HR image $I^{H}$ so that $I^{S}=I^{H}$.
Let $K^{P}$ and $K^{S}$ denote the predicted kernel and the blur kernel that
remains in $I^{S}$ so that $K=K^{P}+K^{S}$ and $I^{S}=I^{H}\ast K^{S}$. We
assume that $K^{S}$ correlates with $K^{P}$.

Based on this assumption, this paper proposes blur-reflected segmentation
learning via a skip connection, which is called the blur skip, from the SR
network $S$ to the segmentation network $C$. This skip connection forwards
$K^{P}$ to the end of $C$ in order to condition features extracted by $C$ with
$K_{P}$. While this conditioning is achieved by the Spatial Feature Transform
(SFT) [104], SFT is marginally modified for CSBSR as follows. The detail of
the modified SFT layer is shown in Fig. 4. In the original SFT layer,
conditions are directly fed into conv layers for producing conditioning
features (which are depicted by red and yellow 3D boxes, respectively, in Fig.
4) for scaling and shifting. Different from this original SFT layer, target
features (“Segmentation features” in Fig. 4) are concatenated to the
conditions. It is empirically validated that this concatenation process
slightly improves the segmentation quality.

###  3.5 Training Strategy

Our joint learning has several loss functions, weights, and hyper-parameters.
They should be properly used for training our complex network consisting of
$S$ and $C$.

Step 1:

As with most tasks each of which has a limited amount of training data, $S$ is
pre-trained with general huge datasets for blind SR.

Step 2:

With a dataset for crack segmentation, only $S$ is initially finetuned with
$\beta=0$ in Eq .(2).

Step 3:

The whole network is finetuned so that $C$ is weighted by a constant (i.e.,
$\beta\neq 0$).

##  4 Experimental Results

###  4.1 Pre-training and Training Details

Figure 5: Sample images in the Khanhha dataset [56].

For pre-training the SR network $S$, 3,450 images in the DIV2K dataset [6]
(800 images) and the Flickr2K dataset [100] (2,650 images) were used. The
whole network for crack segmentation $C$ was not pre-trained but its feature
extractor was pre-trained with the ImageNet [28].

For pre-training $S$ (i.e., Step 1 in Sec. 3.5) and finetuning $S$ and $C$
(i.e., Steps 2 and 3), an image patch fed into each network is randomly
cropped with vertical and horizontal flips from each training image for data
augmentation. This patch is regarded as a HR image ($I^{H}$). From $I^{H}$,
its LR images ($I^{L}$) are generated with various blur kernels ($K$) and
bicubic downsampling ($\downarrow_{s}$), as expressed in Eq. (1). $K$ is
randomly sampled from the anisotropic 2D Gaussian blurs with variance
$\sigma_{a}^{2},\sigma_{b}^{2}\in[0.2,4.0]$ and angle
$\theta_{gaus}\in[0,\pi)$. The kernel size is $21\times 21$ pixels. The HR-LR
downscaling factor is $\frac{1}{4}$. The feature extractor of $C$ is pre-
trained depending on the segmentation network as follows. For U-Net and
PSPNet, VGG-16 is provided by torchvision [4]. For HRNet+OCR, the authors’
model [113] is used.

For pre-training of $S$ in Step 1, the number of iterations is 200,000. The
minibatch size is six. Adam [58] is used as an optimizer with
$\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$. The learning rate is
$2\times 10^{-4}$.

The number of iterations is 30,000 and 150,000 in Steps 2 and 3, respectively.
The minibatch size and the optimizer are equal to those in the aforementioned
pre-training. The learning rate is $2\times 10^{-5}$.

###  4.2 Synthetically-degraded Crack Images

Table 2:  Results on the Khanhha dataset. CSBSR is implemented with four different segmentation networks [124, 112, 87]. To validate the effect of our proposed joint learning, the SR and segmentation networks are also trained without joint learning in each pair of the SR and segmentation networks; see (d) vs. (e), (f) vs. (g), (h) vs. (i), and (j) vs. (k). For comparison, the results of SOTA methods with SR and segmentation are also shown in (b) and (c). For reference, instead of an LR image, its original HR image is directly fed into the segmentation network in “(a) Segmentation in HR” for the upper bound analysis. The best score in each column except for “(a) Segmentation in HR” is colored by red.  | Segmentation metrics | SR metrics
---|---|---
Model |  IoU${}_{max}\uparrow$ |  AIU$\uparrow$ |  HD95${}_{min}\downarrow$ |  AHD95$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
(a) Segmentation in HR | 0.616 | 0.559 | 6.20 | 11.89 | - | -
(b) SrcNet [12] | 0.368 | 0.320 | 95.16 | 130.47 | 27.82 | 0.639
(c) DSRL [102] | 0.391 | 0.285 | 44.23 | 148.97 | 20.16 | 0.501
(d) KBPN + PSPNet [124] | 0.548 | 0.524 | 28.45 | 31.62 | 28.62 | 0.706
(e) CSBSR w/ [124] ($\beta=0.3$) | 0.573 | 0.552 | 20.92 | 22.52 | 28.75 | 0.703
(f) KBPN + (HRNet+OCR [112]) | 0.522 | 0.501 | 26.45 | 28.74 | 28.68 | 0.706
(g) CSBSR w/ [112] ($\beta=0.9$) | 0.553 | 0.534 | 17.54 | 20.29 | 27.66 | 0.668
(h) KBPN + CrackFormer [69] | 0.447 | 0.424 | 46.86 | 58.91 | 28.68 | 0.706
(i) CSBSR w/ [69] ($\beta=0.9$) | 0.469 | 0.443 | 39.37 | 56.59 | 25.93 | 0.571
(j) KBPN + U-Net [87] | 0.470 | 0.455 | 45.26 | 45.94 | 28.68 | 0.706
(k) CSBSR w/ [87] ($\beta=0.3$) | 0.530 | 0.506 | 26.33 | 27.24 | 28.68 | 0.702

Figure 6:  IoU and HD95 comparison with SOTA methods on the Khanhha dataset.
(a) HR segmentation by PSPNet [124] (b) SR segmentation by SrcNet. (c) SR
segmentation by DSRL. (d) SR segmentation by CSBSR w/o joint learning. (e) SR
segmentation by CSSR. (f) SR segmentation by CSBSR.

(a) Input LR (b) HR GT (c) SrcNet [12] (d) DSRL [102] (e) Ours

Figure 7: Visual results of comparative experiments on the Khanhha dataset.

(a) Input LR (b) HR GT (c) SS in HR (d) SrcNet [12] (e) DSRL [102] (f) CSBSR

Figure 8:  Visual comparison on the Khanhha dataset. In the upper row of each
example: (a) Input LR image (enlarged by Bicubic interpolation for
visualization. (b, c) Ground-truth HR image. (d) SR image obtained by SrcNet.
(e) SR image obtained by DSRL. (f) SR image obtained by our CSBSR. In the
lower row of each example: (a) No image. (b) Ground-truth segmentation image
in HR. (c) HR segmentation image obtained by PSPNet [124] (d) SR segmentation
image obtained by SrcNet. (e) SR segmentation image obtained by DSRL. (f) SR
segmentation image obtained by our CSBSR.

(a) Input LR (b) HR GT (c) SS in HR (d) SrcNet [12] (e) DSRL [102] (f) CSBSR

Figure 9:  Examples where (f) the SR segmentation image obtained by our CSBSR
is better than (c) the HR segmentation image obtained in the ground-truth HR
image.

(a) Input LR (b) HR GT (c) SS in HR (d) SrcNet [12] (e) DSRL [102] (f) CSBSR

Figure 10:  Examples where there are no crack pixels in (a) input LR image.

####  4.2.1 Training

For experiments shown in Secs. 4.2 and 4.3, the Khanhha dataset [56] was used
to finetune the whole network for CSBSR. the SR and segmentation networks.
This dataset consists of CRACK500 [120], GAPs [30], CrackForest [91], AEL [9],
cracktree200 [127], DeepCrack [71], and CSSC [108] datasets. As shown in the
sample images of these datasets, (Fig. 5), the Khanhha dataset is challenging
so that a variety of structures are observed and the properties of annotated
cracks differ between the elemental datasets [120, 30, 91, 9, 127, 71, 108].
In the Khanhha dataset, the image size is $448\times 448$ pixels, which is
regarded as a HR image in our experiments. The dataset has 9, 122, 481, and
1,695 training, validation, and test images. These training and test sets were
used as training images for all experiments and test images in experiments
shown in Sec. 4.2, respectively.

####  4.2.2 Evaluation Metrics

Each SR image is evaluated with PSNR and SSIM [106]. Each segmentation image
is evaluated with Intersect of Union (IoU). While IoU is computed in a
binarized image, the output of CSBSR is a segmentation image in which each
pixel has a probability of being a crack or not a crack. Since IoU differs
depending on a threshold for binarization, the threshold for each method is
determined so that the mean IoU over all test images is maximized. This
maximized IoU is called IoUmax. For evaluation independently of thresholding,
IoUs are averaged over thresholds (AIU [107]). While IoU is a major metric for
segmentation, it is inappropriate for evaluating fine thin cracks because a
slight displacement makes IoU significantly small even if the structures of
ground-truth and estimated cracks are almost similar. For appropriately
evaluating such similar cracks, 95% Hausdorff Distance (HD95) [27] is
employed. As with IoU, the HD95 threshold for each method is also determined
so that the mean HD95 over all test images is minimized. This minimized HD95
is called HD95min. For evaluation independently of thresholding, HD95s are
also averaged over thresholds. This averaged HD95 is called AHD95.

####  4.2.3 Comparison with SOTA segmentation methods

For comparative experiments, 1,695 HR test images in the Khanhha dataset are
degraded to their LR images in the same manner as training image generation.

For validating the wide applicability of CSBSR, four SOTA segmentation
networks (i.e., PSPNet [124] for Table 2 (e), HRNet+OCR [112] for Table 2 (g),
CrackFormer [69] for Table 2 (i), and U-Net [87] for Table 2 (k)) are used as
a segmentation network in CSBSR, as described in Sec. 3.1. While CSBSR is
trained in a joint end-to-end manner (i.e., (e), (g), (i), (k) in Table 2),
the results of independent blind SR and segmentation networks (i.e., (d), (f),
(h), (j) in Table 2) are also shown for comparison. To focus on the difference
between the network architectures for segmentation, all of these segmentation
networks are trained with our BC loss in Eq. (3). In BC loss, $\alpha=0.5$ and
$\gamma=0.5$ were determined empirically. The task weight $\beta$ in Eq. (2)
is determined empirically for each method and fixed during Step 3 in the
training strategy (Sec. 3.5).

In addition, CSBSR is compared with SOTA methods in which non-blind SR and
segmentation are used (i.e., Table 2 (b) SrcNet [12] in which SR and
segmentation are trained independently and Table 2 (c) DSRL [102] in which SR
and segmentation are trained in a multi-task learning manner). The
segmentation network of SrcNet and DSRL is trained with the BCE loss. While
SrcNet is implemented by ourselves because its code is not available, we used
the publicly-available implementation of DSRL [3].

Quantitative Results: Table 2 shows quantitative results. In all metrics, all
variants of CSBSR are better than their original segmentation methods. That
is, (e), (g), (i), and (k) are better than (d), (f), (h), and (j),
respectively, in Table 2. As a result, CSBSR is the best in all segmentation
metrics (i.e., IoU, AIU, HD95, and AHD95).

Our proposed methods are also compared with SoTA segmentation methods using SR
(i.e., (b) and (c) in Table 2). The performance improvement of CSBSR compared
to SrcNet might be acquired by BC loss, joint learning, and/or blind SR. In
comparison between CSBSR and DSRL, we can see the effectiveness of serial
joint learning, as well as BC loss and blind SR.

Even in comparison with (a) segmentation in HR images (implemented by PSPNet
with BC loss), the segmentation scores of CSBSR get close to those of
segmentation in HR. For example, IoU and AIU of CSBSR with PSPNet are 93.0%
and 98.7% of those of segmentation in HR. In terms of HD95, on the other hand,
CSBSR is much inferior to segmentation in HR. This reveals that CSBSR should
be improved more in order to extract fine crack structures.

The IoU and HD95 scores of our proposed method with CSBSR are shown in Fig. 7.
For comparison, our method with non-blind SR (i.e., CSSR) and SOTA
segmentation methods using SR are compared with CSBSR. As the upper
limitation, the scores of segmentation on ground-truth HR images are also
shown as (a) in Fig. 7, while LR images are fed into all other methods (b),
(c), (d), (e), and (f) in Fig. 7.

It can be seen that (b) SrcNet and (c) DSRL are clearly inferior to others in
both IoU and HD95. In particular, the scores of DSRL are significantly changed
depending on a change in the threshold. This reveals that DSRL is sensitive to
a change in the threshold. The scores of all other methods accepting LR images
are close to those of (a) segmentation in HR images. In particular, (f) CSBSR
can get higher scores in a wide range of the thresholds. This stability
against a change in the threshold is crucial in applying CSBSR to a variety of
segmentation tasks.

Visual Results: Figure 7 shows visual results. In the upper row, from left to
right, the first and second images are an input LR image (enlarged by nearest
neighbor interpolation) and its ground-truth HR image. The remaining three
images are SR images of SrcNet, DSRL, and CSBSR. It can be seen that the SR
image of CSBSR is much sharper than those of SrcNet and DSRL. In terms of the
crack segmentation image also, CSBSR outperforms SrcNet and DSRL.

Figure 8 shows the examples of more complex cracks. Since such complex crack
pixels make it difficult to correctly detect these pixels, even segmentation
methods using SR reconstruction (i.e., SrcNet [12] and DSRL [102]) cannot
detect many crack pixels, as shown in Fig. 8 (d) and (e). As shown in Fig. 8
(f), on the other hand, our CSBSR can obtain crack segmentation images that
are similar to their corresponding segmentation images obtained in the
original HR images shown in Fig. 8 (c). It can also be seen that CSBSR can
reconstruct and detect even thin fine cracks in the SR image and segmentation
images, respectively. As a result, our results are similar to the ground-truth
segmentation images shown in Fig. 8 (b).

Figure 9 shows examples where (f) the SR segmentation image obtained by CSBSR
is better even than (c) the HR segmentation image obtained in the ground-truth
HR image. These images are characterized by low image-contrast around crack
pixels, thin cracks, and/or local illumination change around crack pixels.

We interpret that, while it is difficult for SR to reconstruct and for
segmentation to detect such high-frequency structures and low-contrast
structures shown in Figs. 8 and 9, our joint learning of SR and segmentation
with the segmentation-aware SR loss and the blur skip for blur-reflected
segmentation learning can achieve these difficult tasks.

Figure 10 shows sample test images where no crack pixels are observed. While
there are no crack pixels in these images, observed masonry joints tend to be
false-positives. For real applications using automatic image inspection, it is
important to successfully suppress such false-positives for avoiding false
alarms because most images have no crack pixels in real buildings. In Fig. 10,
it can be seen that (d) SrcNet and (e) DSRL detect false-positives around the
masonry joints, while (f) CSBSR successfully neglects all of these masonry
joint pixels.

####  4.2.4 Effects of $\beta$

Table 3:  Performance change depending on $\beta$. $\beta$ is fixed during Step 3 in the training strategy, except for “Increasing” shown in the bottom line in which $\beta$ is increased from 0 to 1 in proportion to iterations.  |  | Segmentation metrics | SR metrics
---|---|---|---
Model | $\beta$ |  IoU${}_{max}\uparrow$ |  AIU$\uparrow$ |  HD95${}_{min}\downarrow$ |  AHD95$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
CSBSR w/ PSPNet | w/o joint learning | 0.548 | 0.524 | 28.45 | 31.62 | 28.62 | 0.706
0.1 | 0.563 | 0.541 | 19.16 | 21.96 | 28.73 | 0.705
0.3 | 0.573 | 0.552 | 20.92 | 22.52 | 28.75 | 0.703
0.5 | 0.572 | 0.550 | 18.80 | 21.18 | 28.69 | 0.701
0.7 | 0.551 | 0.528 | 23.31 | 28.66 | 28.07 | 0.687
0.9 | 0.554 | 0.533 | 26.03 | 27.29 | 27.72 | 0.669
1.0 | 0.565 | 0.544 | 19.27 | 22.32 | 22.78 | 0.472
Increasing | 0.568 | 0.549 | 16.24 | 19.02 | 27.12 | 0.662
CSSR w/ PSPNet | w/o joint learning | 0.531 | 0.512 | 36.01 | 38.33 | 27.85 | 0.667
0.1 | 0.547 | 0.529 | 24.45 | 28.27 | 28.42 | 0.653
0.3 | 0.475 | 0.446 | 53.75 | 55.96 | 28.47 | 0.663
0.5 | 0.546 | 0.523 | 22.12 | 24.61 | 28.39 | 0.657
0.7 | 0.557 | 0.539 | 21.20 | 24.74 | 28.35 | 0.656
0.9 | 0.552 | 0.534 | 20.88 | 22.48 | 28.01 | 0.653
1.0 | 0.539 | 0.515 | 21.82 | 26.04 | 20.29 | 0.436
Increasing | 0.544 | 0.512 | 28.28 | 35.30 | 27.02 | 0.635
Table 4:  Comparison with other losses for class-imbalance segmentation. The best and second best scores are colored by red and blue, respectively.  | Segmentation metrics | SR metrics
---|---|---
Model |  IoU${}_{max}\uparrow$ |  AIU$\uparrow$ |  HD95${}_{min}\downarrow$ |  AHD95$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
BC loss (Ours) | 0.573 | 0.552 | 20.92 | 22.52 | 28.75 | 0.703
GBC loss (Ours) | 0.551 | 0.534 | 23.34 | 33.46 | 28.70 | 0.705
WCE [26] | 0.569 | 0.459 | 16.91 | 26.29 | 28.60 | 0.704
Dice [77] | 0.466 | 0.465 | 59.21 | 59.65 | 28.66 | 0.704
Combo [98] | 0.483 | 0.436 | 39.48 | 62.27 | 28.51 | 0.697
Boundary [55] \+ GDice [97] | 0.469 | 0.425 | 65.13 | 68.90 | 28.31 | 0.692

Figure 11:  Curves of IoU and HD95 scores varying with a change in the
threshold for segmentation image binarization.

Table 3 shows the evaluation results obtained in accordance with changes in
$\beta$. In all metrics of both SR and segmentation tasks, CSBSR outperforms
CSSR. Furthermore, in both CSSR and CSBSR, our proposed joint learning
acquires better results in all segmentation metrics.

More specifically, in terms of the segmentation results, IoUmax and AIU are
not so changed depending on $\beta$. On the other hand, the best HD95min and
AHD95 scores are better in the training strategy with increasing $\beta$
(i.e., “Increasing” in the table) and have a larger margin from the scores
obtained with any fixed $\beta$. Intuitively speaking, the segmentation score
should be best with $\beta=1$ so that the segmentation loss (i.e.,
$\mathcal{C}$ in Eq. 2) is fully weighted. We interpret that the segmentation
scores are not best with $\beta=1$ because it is difficult to fully optimize
the whole network directly from the pre-trained SR and segmentation networks.
That is why the training strategy with increasing $\beta$ is better than
$\beta=1$.

In terms of the SR image quality, While the best SSIM is acquired without
joint learning, the best PSNR is with $\beta=0.3$. Since the SR network is
trained without joint learning just to improve SR, it is expected that the
best SR results are obtained without joint learning. This expectation is
betrayed probably because of the feature extractor augmentation through the
training of the segmentation task. The features can be marginally augmented
also for SR as in multi-task learning if $\beta$ is smaller, while the
features are optimized for the segmentation task if $\beta$ is larger.

Table 5: Ablation study of weights given to $\mathcal{L}_{S}$, namely $\mathcal{L}_{C}$, $w^{C}$, and $w^{F}$. Scores better than the baseline (i.e., CSBSR w/o any weight) are underlined.  | Segmentation metrics | SR metrics
---|---|---
Model |  IoU${}_{max}\uparrow$ |  AIU$\uparrow$ |  HD95${}_{min}\downarrow$ |  AHD95$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
CSBSR | 0.573 | 0.552 | 20.92 | 22.52 | 28.75 | 0.703
w/ $L_{C}$ | 0.558 | 0.535 | 19.72 | 22.90 | 27.32 | 0.649
w/ $w^{C}$ ($m^{C}=8.0$) | 0.553 | 0.531 | 19.21 | 26.02 | 28.70 | 0.703
w/ $w^{F}$ ($m^{F}=1.0$) | 0.573 | 0.551 | 18.73 | 21.70 | 28.73 | 0.702
w/ $w^{F}$ ($m^{F}=0.5$) for $\mathcal{L}_{C}$ | 0.556 | 0.531 | 22.26 | 25.94 | 28.70 | 0.706

(a) Input LR (b) HR GT (c) w/o $w^{F}$ (d) w/ $w^{F}$ (b’) HR GT (c’) w/o
$w^{F}$ (d’) w/o $w^{F}$

Figure 12:  Visual comparison between CSBSR w/ and w/o the FO weight $w^{F}$. [Left part] In the upper row of each example: (a) Input LR image (enlarged by Bicubic interpolation for visualization). (b) Ground-truth HR image. (c) SR image obtained by CSBSR w/o $w^{F}$. (d) SR image obtained by CSBSR. In the lower row of each example: (a) No image. (b) Ground-truth segmentation image in HR. (c) SR segmentation image obtained by CSBSR w/o $w^{F}$. (d) SR segmentation image obtained by CSBSR. [Right part] Rectangle regions are cropped from the SR images shown in the left part, and their zoom-in images are shown. The boundary color of each cropped image shows the correspondence between the cropped images in the left and right parts. Differences between (c’) and (d’) are pointed by white arrows.  Table 6:  Ablation study of our blur skip process. Scores better than the baseline (i.e., CSBSR w/o any weight) are underlined.  | Segmentation metrics | SR metrics
---|---|---
Model |  IoU${}_{max}\uparrow$ |  AIU$\uparrow$ |  HD95${}_{min}\downarrow$ |  AHD95$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  Kernel PSNR$\uparrow$
CSBSR | 0.573 | 0.552 | 20.92 | 22.52 | 28.75 | 0.703 | 50.65
CSBSR w/ KS | 0.544 | 0.523 | 28.86 | 32.02 | 28.52 | 0.696 | 50.82
CSBSR w/ KS and $m^{F}=1.0$ | 0.550 | 0.528 | 18.06 | 19.10 | 28.65 | 0.702 | 50.91

(a) GT (b) Results without blur skip (c) Results with blur skip

Figure 13: Effectiveness of our proposed blur skip. The left and right images
show the HR/SR image and the segmentation image, respectively.

####  4.2.5 Effects of Segmentation losses

To verify the effectiveness of our BC and GBC losses, CSBSR is trained with
other losses for class-imbalance segmentation (i.e., WCE [26], Dice [77],
Combo [98], and GDice [97]). As shown in Table 4, BC loss gets the best scores
in four metrics (i.e., IoU, AIU, AHD95, and PSNR) and the second-best in HD95.
While it is the third place in SSIM, the gap from the best is tiny (0.705 vs
0.703).

Figure 11 shows IoU and HD95 scores varying with a change in a threshold for
binarizing the segmentation image. As shown in Table 4, GBC is inferior to BC.
However, GBC gets higher scores in a large range of thresholds in both IoU and
HD95. This property might be given by GDice, included in GBC, which works
robustly to class imbalance. On the other hand, while WCE gets better results
in a few metrics in Table 4, its performance drop depending on the threshold
is significant. This performance drop makes it difficult to apply WCE loss to
a variety of scenarios. As with GBC, the curves of BC are also not so
decreased.

Based on the aforementioned observations, we conclude that our BC and GBC
loesses are superior to other SOTA losses in terms of the max performance (as
shown in Table 4) and stability (as shown in Fig. 11).

####  4.2.6 Effects of Segmentation-aware SR-loss Weights

The effects of additional weights given to $\mathcal{L}_{S}$, which are
proposed in Sec. 3.3, are evaluated in Table 5. Since $w^{C}$ and $w^{F}$ have
hyper parameters (i.e., $m^{C}$ and $m^{F}$, respectively), the best results
among
$\left\\{m^{C},m^{F}\right\\}=\left\\{2^{-3},2^{-2},2^{-1},2^{0},2^{1},2^{2},2^{3}\right\\}$
are shown in Table 5. We can see the following observations:

  * •

All weights given to $\mathcal{L}_{S}$ improve HD95.

  * •

Conversely, all weights given to $\mathcal{L}_{S}$ decrease IoU and AIU, while
the performance drops are not so significant. In particular, IoU and AIU
provided by $w^{F}$ given to $\mathcal{L}_{S}$ are almost equal to those of
the baseline CSBSR (i.e., 0.573 vs 0.573 in IoU and 0.551 vs 0.552 in AIU).

  * •

While $w^{F}$ weights the segmentation loss ($\mathcal{L}_{C}$), the results
are inferior to the baseline in most metrics, as shown in the bottom row of
Table 5.

In addition to the quantitative comparison shown in Table 5, Fig. 12 visually
shows the effect of the FO weight. All images are the results obtained with
$w^{F}=1.0$. In the left part of Fig. 12, we can see that $w^{F}$ allows CSBSR
to detect thin crack pixels in segmentation images. In order to see the
results of SR image enhancement by $w^{F}$, the zoom-in images of several
regions in the SR images are shown in the right part of Fig. 12.

In (c) images obtained without $w^{F}$, detected crack pixels are broken. In
(d) images obtained with $w^{F}$, on the other hand, cracks are more
continuously detected, though it is difficult to visually see any significant
difference between zoom-in SR images shown in (c’) and (d’). In an opposite
way, background textures enclosed by the purple dashed ellipse are falsely
detected in CSBSR without $w^{F}$, as shown in (c) of the lower example.
However, these background pixels reconstructed by CSBSR without and with
$w^{F}$ (enclosed by the purple dashed ellipses in (c’) and (d’)) are also
almost the same as each other. These results demonstrate the effectiveness of
$w^{F}$ for discriminating between remarkably-similar crack and background
pixels in the segmentation network of CSBSR.

####  4.2.7 Effects of Blur Skip

Input image Input image DSRL [102] SrcNet [12] Ours

Figure 14: Visual results of comparative experiments on real blurred crack
images. In (c), (d), and (e), detected crack pixel are colored by red. True-
positive, false-negative, and false-positive cracks are enclosed by blue,
green, and yellow ellipses, respectively.

The effects of the proposed blur skip process are shown in Table 6. Since the
quality of the estimated kernel is high enough (e.g., above 50 dB in PSNR),
our kernel skip should have the potential to support the segmentation task.
While the single usage of the blur skip cannot work well for all metrics, the
blur skip used with $w^{F}$ improves HD95 and AHD95. The typical examples are
shown in Fig. 13. While the results without the blur skip are much inferior to
their ground truths, the blur skip can improve the performance, as shown in
the rightmost image in Fig. 13.

###  4.3 Crack Images with Real Degradations

For experiments with real images, we captured 809 wall images ($1280\times
720$ pixels) with a flying drone (DJI MAVIC MINI). This dataset includes out-
of-focus images as well as motion-blurred images. By using all the images in
this dataset as test images, we visually verify the effectiveness of CSBSR for
realistically-blurred images. Since it is essentially difficult to annotate
severely-blurred cracks correctly, only qualitative comparison is done with
this dataset.

In the first row of Fig. 14, cracks are very thin. DSRL and SrcNet cannot
detect any crack pixels. In addition, false-positive cracks (enclosed by
yellow ellipses) are detected. CSBSR, on the other hand, can detect most crack
pixels, as depicted by superimposed red pixels.

The second row of Fig. 14 shows the segmentation results detected on the image
of complex cracks observed on a building wall. While DSRL detects no crack
pixels, SrcNet and CSBSR successfully detect several crack pixels. CSBSR can
detect more true-positive crack pixels, in particular, along a crack located
in the upper part of the image (enclosed by blue ellipses). However, there are
also many false-negative crack pixels (enclosed by green ellipses) even in the
segmentation image of CSBSR.

In the input image shown in the third row of Fig. 14, there are thin
electrical wires as well as thin cracks (enclosed by blue and green ellipses).
A crack segmentation method is required to detect only real cracks without
being disturbed by the wires. DSRL detects several wire pixels (enclosed by
the yellow ellipse) and crack pixels, while SrcNet detects nothing. While
CSBSR detects only crack pixels, even CSBSR fails to detect blurry cracks
observed in the lower part of the image (enclosed by green ellipses).

As mentioned above, while our CSBSR outperforms SOTA segmentation methods
using SR, it also fails to detect severely-degraded cracks. Improving crack
segmentation in such severely-degraded images is important for future work.

##  5 Concluding Remarks

This paper proposes an end-to-end joint learning network consisting of blind
SR and segmentation networks. Blind SR allows us to apply the proposed method
to realistically-blurred images. The information exchange between the SR and
segmentation networks (i.e., segmentation-aware SR-loss weights and blur skip
for blur-reflected task learning) enables further improvement. For better
segmentation in class-imbalance fine crack images, BC loss is proposed.

Future work includes quantitative evaluation on real-image datasets in which
ground-truth segmentation pixels are manually given. It is also interesting to
apply CSBSR to other segmentation tasks such as medical imaging. An essential
difficulty in SR is that SR is an ill-posed problem in which a larger number
of pixels are reconstructed from a smaller number of pixels. In order to
relieve this difficulty, multiple LR images are used as a set of input images
in video SR [34, 80, 46, 42] and burst SR [16]. Our proposed method can also
be extended to the one with time-series images.

###  5.1 Acknowledgments

This work was partly supported by JSPS KAKENHI Grant Numbers 19K12129 and
22H03618.

## References

  * [1] Crack segmentation.  https://github.com/khanhha/crack_segmentation.
  * [2] Crackformer-ii.  https://github.com/LouisNUST/CrackFormer-II.
  * [3] Dual super-resolution learning for semantic segmentation.  https://github.com/Dootmaan/DSRL.
  * [4] Torchvision.models.  https://pytorch.org/vision/stable/models.html.
  * [5] Allen Zhang abd Kelvin C. P. Wang, Yue Fei, Yang Liu, Siyu Tao, Cheng Chen, Joshua Q. Li, and Baoxian Li.  Deep learning–based fully automated pavement crack detection on 3d asphalt surfaces with an improved cracknet.  Journal of Computing in Civil Engineering, 32(5), 2018.
  * [6] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In CVPRW, 2017.
  * [7] Kazutoshi Akita, Muhammad Haris, and Norimichi Ukita.  Region-dependent scale proposals for super-resolution in object detection.  In IPAS, 2020.
  * [8] Kazutoshi Akita, Masayoshi Hayama, Haruya Kyutoku, and Norimichi Ukita.  AVM image quality enhancement by synthetic image learning for supervised deblurring.  In MVA, 2021.
  * [9] Rabih Amhazand, Sylvie Chambon, Jérôme Idier, and Vincent Baltazart.  Automatic crack detection on two-dimensional pavement images: An algorithm based on minimal path selection.  TITS, 17(10):2718–2729, 2016.
  * [10] Md Rifat Arefin, Vincent Michalski, Pierre-Luc St-Charles, Alfredo Kalaitzis, Sookyung Kim, Samira Ebrahimi Kahou, and Yoshua Bengio.  Multi-image super-resolution for remote sensing using deep recurrent networks.  In CVPR Workshops, 2020.
  * [11] Leanne Attard, Carl James Debono, Gianluca Valentino, and Mario Di Castro.  Tunnel inspection using photogrammetric techniques and image processing: A review.  ISPRS Journal of Photogrammetry and Remote Sensing, 140:180–18, 2018.
  * [12] Hyunjin Bae, Keunyoung Jang, and Yun-Kyu An.  Deep super resolution crack network (srcnet) for improving computer vision–based automated crack detectability in in situ bridges.  Structural Health Monitoring, 20(4):1428–1442, 2021.
  * [13] Yuval Bahat and Tomer Michaeli.  Explorable super resolution.  In CVPR, 2020.
  * [14] Yancheng Bai, Yongqiang Zhang, Mingli Ding, and Bernard Ghanem.  Finding tiny faces in the wild with generative adversarial network.  In CVPR, 2018.
  * [15] Yancheng Bai, Yongqiang Zhang, Mingli Ding, and Bernard Ghanem.  SOD-MTGAN: small object detection via multi-task generative adversarial network.  In ECCV, 2018.
  * [16] Goutam Bhat, Martin Danelljan, Radu Timofte, Kazutoshi Akita, Wooyeong Cho, Haoqiang Fan, Lanpeng Jia, Daeshik Kim, Bruno Lecouat, Youwei Li, Shuaicheng Liu, Ziluan Liu, Ziwei Luo, Takahiro Maeda, Julien Mairal, Christian Micheloni, Xuan Mo, Takeru Oba, Pavel Ostyakov, Jean Ponce, Sanghyeok Son, Jian Sun, Norimichi Ukita, Rao Muhammad Umer, Youliang Yan, Lei Yu, Magauiya Zhussip, and Xueyi Zou.  NTIRE 2021 challenge on burst super-resolution: Methods and results.  In CVPR Workshop, 2021.
  * [17] Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor.  2018 pirm challenge on perceptual image super-resolution.  In ECCV Workshop, 2018.
  * [18] Samuel Rota Bulò, Gerhard Neuhold, and Peter Kontschieder.  Loss max-pooling for semantic image segmentation.  In CVPR, 2017.
  * [19] Hanshen Chen, Yishun Su, and Wei He.  Automatic crack segmentation using deep high-resolution representation learning.  Applied Optics, 60(21):6080–6090, 2021.
  * [20] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834–848, 2018\.
  * [21] Dong-Yoon Choi, Ji Hoon Choi, Jin Wook Choi, and Byung Cheol Song.  Sharpness enhancement and super-resolution of around-view monitor images.  IEEE Trans. Intell. Transp. Syst., 19(8):2650–2662, 2018.
  * [22] Wooram Choi and Young-Jin Cha.  Sddnet: Real-time crack segmentation.  IEEE Trans. Ind. Electron., 67(9):8016–8025, 2020.
  * [23] Victor Cornillère, Abdelaziz Djelouah, Yifan Wang, Olga Sorkine-Hornung, and Christopher Schroers.  Blind image super-resolution with spatially variant degradations\̇lx@bibnewblockACM Trans. Graph., 38(6):166:1–166:13, 2019.
  * [24] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In CVPR, 2019.
  * [25] Dimitris Dais, Ihsan Engin Bal, Eleni Smyrou, and Vasilis Sarhosis.  Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning.  Automation in Construction, 25, 2021.
  * [26] Dimitris Dais, İhsan Engin Bal, Eleni Smyrou, and Vasilis Sarhosis.  Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning.  Automation in Construction, 125:103606, 2021.
  * [27] DeepMind.  Surface distance metrics.  https://github.com/deepmind/surface-distance.
  * [28] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [29] Qi Dong, Shaogang Gong, and Xiatian Zhu.  Class rectification hard mining for imbalanced deep learning.  In ICCV, 2017.
  * [30] Markus Eisenbach, Ronny Stricker, Daniel Seichter, Karl Amende, Klaus Debes, Maximilian Sesselmann, Dirk Ebersbach, Ulrike Stoeckert, and Horst-Michael Gross.  How to get pavement distress detection ready for deep learning? a systematic approach.  In IJCNN, 2017.
  * [31] Faris Elghaish, Saeed Talebi, Essam Abdellatef, Sandra T Matarneh, M Reza Hosseini, Song Wu, Mohammad Mayouf, Aso Hajirasouli, et al.  Developing a new deep learning cnn model to detect and classify highway cracks.  Journal of Engineering, Design and Technology, 2021.
  * [32] Yue Fei, Kelvin C. P. Wang, Allen Zhang, Cheng Chen, Joshua Q. Li, Yang Liu, Guangwei Yang, and Baoxian Li.  Pixel-level cracking detection on 3d asphalt pavement images through deep-learning- based cracknet-v.  IEEE Trans. Intell. Transp. Syst., 21(1):273–284, 2020.
  * [33] Sanae Fujii, Kazutoshi Akita, and Norimichi Ukita.  Distant bird detection for safe drone flight and its dataset.  In MVA, 2021.
  * [34] Dario Fuoli, Zhiwu Huang, Shuhang Gu, Radu Timofte, Arnau Raventos, Aryan Esfandiari, Salah Karout, Xuan Xu, Xin Li, Xin Xiong, Jinge Wang, Pablo Navarrete Michelini, Wenhao Zhang, Dongyang Zhang, Hanwei Zhu, Dan Xia, Haoyu Chen, Jinjin Gu, Zhi Zhang, Tongtong Zhao, Shanshan Zhao, Kazutoshi Akita, Norimichi Ukita, Hrishikesh P. S, Densen Puthussery, and C. V. Jiji.  AIM 2020 challenge on video extreme super-resolution: Methods and results.  In ECCV Workshop, 2020.
  * [35] Lixue Gong, Yiqun Zhang, Yunke Zhang, Yin Yang, and Weiwei Xu.  Erroneous pixel prediction for semantic image segmentation.  Comput. Vis. Media, 8(1):165–175, 2022.
  * [36] Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong.  Blind super-resolution with iterative kernel correction.  In CVPR, 2019.
  * [37] Shuhang Gu, Hanwen Liu, Dan Zhu, Tangxin Xie, Xin Yang, Chen Zhu, Jia Yu, Wenyu Sun, Xin Tao, Zijun Deng, Liying Lu, Martin Danelljan, Wenbo Li, Taian Guo, Xiaoyong Shen, Xuemiao Xu, Yu-Wing Tai, Jiaya Jia, Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, Radu Timofte, Jiayi Ma, Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, Wan-Chi Siu, Yui-Lam Chan, Ruofan Zhou, Majed El Helou, Kuldeep Purohit, Praveen Kandula, Muhammad Haris, Maitreya Suin, A. N. Rajagopalan, Kazutoshi Akita, Greg Shakhnarovich, Norimichi Ukita, Pablo Navarrete Michelini, and Wenbin Chen.  Aim 2019 challenge on image extreme super-resolution: Methods and results.  In AIM (ICCVW), 2019.
  * [38] Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan.  Closed-loop matters: Dual regression networks for single image super-resolution.  In CVPR, 2020.
  * [39] Zhiling Guo, Guangming Wu, Xiaoya Song, Wei Yuan, Qi Chen, Haoran Zhang, Xiaodan Shi, Mingzhou Xu, Yongwei Xu, Ryosuke Shibasaki, and Xiaowei Shao.  Super-resolution integrated building semantic segmentation for multi-source remote sensing imagery.  IEEE Access, 7:99381–99397, 2019.
  * [40] Zekun Hao, Yu Liu, Hongwei Qin, Junjie Yan, Xiu Li, and Xiaolin Hu.  Scale-aware face detection.  In CVPR, 2017.
  * [41] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Deep back-projection networks for super-resolution.  https://github.com/alterzero/DBPN-Pytorch.
  * [42] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Space-time-aware multi-resolution video enhancement.  In CVPR, 2020.
  * [43] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Deep back-projectinetworks for single image super-resolution.  IEEE Trans. Pattern Anal. Mach. Intell., 43(12):4323–4337, 2021\.
  * [44] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Task-driven super resolution: Object detection in low-resolution images.  In ICONIP, 2021.
  * [45] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita.  Deep back-projection networks for super-resolution.  In CVPR, 2018.
  * [46] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita.  Recurrent back-projection network for video super-resolution.  In CVPR, 2019.
  * [47] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  IEEE Trans. Pattern Anal. Mach. Intell., 42(2):386–397, 2020\.
  * [48] Majed El Helou, Ruofan Zhou, and Sabine Süsstrunk.  Stochastic frequency masking to improve super-resolution and denoising networks.  In ECCV, 2020.
  * [49] Md Sazzad Hossain, John M. Betts, and Andrew P. Paplinski.  Dual focal loss to address class imbalance in semantic segmentation.  Neurocomputing, 462:69–87, 2021.
  * [50] Peiyun Hu and Deva Ramanan.  Finding tiny faces.  In CVPR, 2017.
  * [51] Zheng Hui, Jie Li, Xiumei Wang, and Xinbo Gao.  Learning the non-differentiable optimization for blind super-resolution.  In CVPR, 2021.
  * [52] Shady Abu Hussein, Tom Tirer, and Raja Giryes.  Correction filter for single image super-resolution: Robustifying off-the-shelf deep super-resolvers.  In CVPR, 2020.
  * [53] Jianbo Jiao, Yunchao Wei, Zequn Jie, Honghui Shi, Rynson W. H. Lau, and Thomas S. Huang.  Geometry-aware distillation for indoor semantic segmentation.  In CVPR, 2019.
  * [54] Davood Karimi and Septimiu E. Salcudean.  Reducing the hausdorff distance in medical image segmentation with convolutional neural networks.  IEEE Trans. Medical Imaging, 39(2):499–513, 2020.
  * [55] Hoel Kervadec, Jihene Bouchtiba, Christian Desrosiers, Eric Granger, Jose Dolz, and Ismail Ben Ayed.  Boundary loss for highly unbalanced segmentation.  Medical Image Anal., 67:101851, 2021.
  * [56] Khanhha.  Crack segmentation, 2020.  https://github.com/khanhha/crack_segmentation.
  * [57] Soo Ye Kim, Hyeonjun Sim, and Munchurl Kim.  Koalanet: Blind super-resolution using kernel-oriented adaptive local adjustment.  In CVPR, 2021.
  * [58] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [59] Alexander Kirillov, Kaiming He, Ross B. Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In CVPR, 2019.
  * [60] Yuki Kondo and Norimichi Ukita.  Crack segmentation for low-resolution images using joint learning with super-resolution.  In MVA, 2021.
  * [61] Junyeop Lee, Jaihyun Park, Kanghyu Lee, Jeongki Min, Gwantae Kim, Bokyeung Lee, Bonhwa Ku, David K. Han, and Hanseok Ko.  FBRNN: feedback recurrent neural network for extreme image super-resolution.  In CVPR Workshop, 2020.
  * [62] Jianan Li, Xiaodan Liang, Yunchao Wei, Tingfa Xu, Jiashi Feng, and Shuicheng Yan.  Perceptual generative adversarial networks for small object detection.  In CVPR, 2017.
  * [63] Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, and Zhouchen Lin.  Pointflow: Flowing semantics through points for aerial image segmentation.  In CVPR, 2021.
  * [64] Zeju Li, Konstantinos Kamnitsas, and Ben Glocker.  Analyzing overfitting under class imbalance in neural networks for image segmentation.  IEEE Trans. Medical Imaging, 40(3):1065–1077, 2021.
  * [65] Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu.  Feedback network for image super-resolution.  In CVPR, 2019.
  * [66] Zhihang Li, Huamei Zhu, and Mengqi Huang.  A deep learning-based fine crack segmentation network on full-scale steel bridge images with complicated backgrounds.  IEEE Access, 9:114989–114997, 2021.
  * [67] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [68] Zhipeng Lin, Zhi Gao, Hong Ji, Ruifang Zhai, Xiaoqing Shen, and Tiancan Mei.  Classification of cervical cells leveraging simultaneous super-resolution and ordinal regression.  Appl. Soft Comput., 115:108208, 2022.
  * [69] Huajun Liu, Xiangyu Miao, Christoph Mertz, Chengzhong Xu, and Hui Kong.  Crackformer: Transformer network for fine-grained crack detection.  In ICCV, 2021.
  * [70] Shice Liu, Yu Hu, Yiming Zeng, Qiankun Tang, Beibei Jin, Yinhe Han, and Xiaowei Li.  See and think: Disentangling semantic scene completion.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, NeurIPS, 2018.
  * [71] Yahui Liu, Jian Yao, Xiaohu Lu, Renping Xie, and Li Li.  Deepcrack: A deep hierarchical feature learning architecture for crack segmentation.  Neurocomputing, 338:139–153, 2019.
  * [72] Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia.  MASA-SR: matching acceleration and spatial adaptation for reference-based image super-resolution.  In CVPR, 2021.
  * [73] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In ECCV, 2020.
  * [74] Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang, and Tieniu Tan.  Unfolding the alternating optimization for blind super resolutio n.  In NeurIPS, 2020.
  * [75] Jun Ma, Jianan Chen, Matthew Ng, Rui Huang, Yu Li, Chen Li, Xiaoping Yang, and Anne L. Martel.  Loss odyssey in medical image segmentation.  Medical Image Anal., 71:102035, 2021.
  * [76] Yiqun Mei, Yuchen Fan, and Yuqian Zhou.  Image super-resolution with non-local sparse attention.  In CVPR, 2021.
  * [77] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 3DV, 2016.
  * [78] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  arXiv, 2001.05566, 2020.
  * [79] Masashi Nagaya and Norimichi Ukita.  Embryo grading with unreliable labels due to chromosome abnormalities by regularized PU learning with ranking.  IEEE Trans. Medical Imaging, 41(2):320–331, 2022.
  * [80] Seungjun Nah, Radu Timofte, Shuhang Gu, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Kyoung Mu Lee, Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, Chen Change Loy, Yuchen Fan, Jiahui Yu, Ding Liu, Thomas S. Huang, Xiao Liu, Chao Li, Dongliang He, Yukang Ding, Shilei Wen, Fatih Porikli, Ratheesh Kalarot, Muhammad Haris, Greg Shakhnarovich, Norimichi Ukita, Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, Jiayi Ma, Hang Dong, Xinyi Zhang, Zhe Hu, Kwan-Young Kim, Dong Un Kang, Se Young Chun, Kuldeep Purohit, A. N. Rajagopalan, Yapeng Tian, Yulun Zhang, Yun Fu, Chenliang Xu, A. Murat Tekalp, M. Akin Yilmaz, Cansu Korkmaz, Manoj Sharma, Megh Makwana, Anuj Badhwar, Ajay Pratap Singh, Avinash Upadhyay, Rudrabha Mukhopadhyay, Ankit Shukla, Dheeraj Khanna, A. S. Mandal, Santanu Chaudhury, Si Miao, Yongxin Zhu, and Xiao Huo.  NTIRE 2019 challenge on video super-resolution: Methods and results.  In CVPR Workshop, 2019.
  * [81] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  In ECCV, 2020.
  * [82] Yanwei Pang, Jiale Cao, Jian Wang, and Jungong Han.  Jcs-net: Joint classification and super-resolution network for small-scale pedestrian detection in surveillance images.  IEEE Trans. Inf. Forensics Secur., 14(12):3322–3331, 2019.
  * [83] Seobin Park, Jinsu Yoo, Donghyeon Cho, Jiwon Kim, and Tae Hyun Kim.  Fast adaptation to super-resolution networks via meta-learning.  In ECCV, 2020.
  * [84] Prateek Prasanna, Kristin J. Dana, Nenad Gucunski, Basily B. Basily, Hung Manh La, Ronny Salim Lim, and Hooman Parvardeh.  Automated crack detection on concrete bridges.  IEEE Trans Autom. Sci. Eng., 13(2):591–599, 2016.
  * [85] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In ICLR, 2016.
  * [86] Amir Rezaie, Radhakrishna Achanta, Michele Godio, and Katrin Beyer.  Comparison of crack segmentation using digital image correlation measurements and deep learning.  Construction and Building Materials, 261(20):120474, 2020.
  * [87] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * [88] Serim Ryou, Seong-Gyun Jeong, and Pietro Perona.  Anchor loss: Modulating loss scale based on prediction difficulty.  In ICCV, 2019.
  * [89] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  Singan: Learning a generative model from a single natural image.  In ICCV, 2019.
  * [90] Evan Shelhamer, Jonathan Long, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  IEEE Trans. Pattern Anal. Mach. Intell., 39(4):640–651, 2017\.
  * [91] Yong Shi, Limeng Cui, Zhiquan Qi, Fan Meng, and Zhensong Chen.  Automatic road crack detection using random structured forests.  TITS, 17(12):3434–3445, 2016.
  * [92] Gyumin Shim, Jinsun Park, and In So Kweon.  Robust reference-based super-resolution with similarity-aware deformable convolution.  In CVPR, 2020.
  * [93] Kodai Shimosato and Norimichi Ukita.  Multi-modal data fusion for land-subsidence image improvement in psinsar analysis.  IEEE Access, 9:141970–141980, 2021.
  * [94] Maneet Singh, Shruti Nagpal, Richa Singh, and Mayank Vatsa.  Dual directed capsule network for very low resolution image recognition.  In ICCV, 2019.
  * [95] Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho.  Meta-transfer learning for zero-shot super-resolution.  In CVPR, 2020.
  * [96] Simon Stent, Riccardo Gherardi, Björn Stenger, Kenichi Soga, and Roberto Cipolla.  An image-based system for change detection on tunnel linings.  In MVA, 2013.
  * [97] Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M. Jorge Cardoso.  Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations.  In MICCAI, 2017.
  * [98] Saeid Asgari Taghanaki, Yefeng Zheng, S Kevin Zhou, Bogdan Georgescu, Puneet Sharma, Daguang Xu, Dorin Comaniciu, and Ghassan Hamarneh.  Combo loss: Handling input and output imbalance in multi-organ segmentation.  Comput Med Imaging Graph, 75:24–33, 2019.
  * [99] Hossein Talebi and Peyman Milanfar.  Learning to resize images for computer vision tasks.  In ICCV, 2021.
  * [100] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang.  Ntire 2017 challenge on single image super-resolution: Methods and results.  In CVPRW, 2017.
  * [101] Radu Timofte, Shuhang Gu, Jiqing Wu, and Luc Van Gool.  Ntire 2018 challenge on single image super-resolution: Methods and results.  In NTIRE (CVPRW), 2018.
  * [102] Li Wang, Dong Li, Yousong Zhu, Lu Tian, and Yi Shan.  Dual super-resolution learning for semantic segmentation.  In CVPR, 2020.
  * [103] Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo.  Unsupervised degradation representation learning for blind super-resolution.  In CVPR, 2021.
  * [104] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In CVPR, 2018.
  * [105] Zhihao Wang, Jian Chen, and Steven C. H. Hoi.  Deep learning for image super-resolution: A survey.  TPAMI, 43(10):3365–3387, 2021.
  * [106] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on image processing, 13(4):600–612, 2004.
  * [107] Fan Yang, Lei Zhang, Sijia Yu, Danil Prokhorov, Xue Mei, and Haibin Ling.  Feature pyramid and hierarchical boosting network for pavement crack detection.  TITS, 21(4):1525–1535, 2020.
  * [108] Liang Yang, Bing Li, Wei Li, Liu Zhaoming, Guoyong Yang, and Jizhong Xiao.  Deep concrete inspection using unmanned aerial vehicle towards cssc database.  In IROS, 2017.
  * [109] Michael Yeung, Evis Sala, Carola-Bibiane Sch’́onlieb, and Leonardo Rundo.  Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation.  Computerized Medical Imaging and Graphics, 95:102026, 2022.
  * [110] Tomoki Yoshida, Yuki Kondo, Takahiro Maeda, Kazutoshi Akita, and Norimichi Ukita.  Kernelized back-projection networks for blind super resolution.  https://github.com/Yuki-11/KBPN.
  * [111] Tomoki Yoshida, Yuki Kondo, Takahiro Maeda, Kazutoshi Akita, and Norimichi Ukita.  Kernelized back-projection networks for blind super resolution.  arXiv, 2023.
  * [112] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In ECCV, 2020.
  * [113] Yuhui Yuan et al.  Ocnet series.  https://github.com/openseg-group/openseg.pytorch/blob/master/MODEL_ZOO.md.
  * [114] Kai Zhang, Jingyun Liang Luc Van Gool, and Radu Timofte.  Designing a practical degradation model for deep blind image super-resolution.  In ICCV, 2021.
  * [115] Kai Zhang, Luc Van Gool, and Radu Timofte.  Deep unfolding network for image super-resolution.  In CVPR, 2020.
  * [116] Kai Zhang, Shuhang Gu, Radu Timofte, Taizhang Shang, Qiuju Dai, Shengchen Zhu, Tong Yang, Yandong Guo, Younghyun Jo, Sejong Yang, Seon Joo Kim, Lin Zha, Jiande Jiang, Xinbo Gao, Wen Lu, Jing Liu, Kwangjin Yoon, Taegyun Jeon, Kazutoshi Akita, Takeru Ooba, Norimichi Ukita, Zhipeng Luo, Yuehan Yao, Zhenyu Xu, Dongliang He, Wenhao Wu, Yukang Ding, Chao Li, Fu Li, Shilei Wen, Jianwei Li, Fuzhi Yang, Huan Yang, Jianlong Fu, Byung-Hoon Kim, JaeHyun Baek, Jong Chul Ye, Yuchen Fan, Thomas S. Huang, Junyeop Lee, Bokyeung Lee, Jungki Min, Gwantae Kim, Kanghyu Lee, Jaihyun Park, Mykola Mykhailych, Haoyu Zhong, Yukai Shi, Xiaojun Yang, Zhijing Yang, Liang Lin, Tongtong Zhao, Jinjia Peng, Huibing Wang, Zhi Jin, Jiahao Wu, Yifu Chen, Chenming Shang, Huanrong Zhang, Jeongki Min, Hrishikesh P. S, Densen Puthussery, and C. V. Jiji.  Ntire 2020 challenge on perceptual extreme super-resolution: Methods and results.  In NTIRE (CVPRW), 2020.
  * [117] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Learning a single convolutional super-resolution network for multiple degradations.  In CVPR, 2018.
  * [118] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Deep plug-and-play super-resolution for arbitrary blur kernels.  In CVPR, 2019.
  * [119] Kaige Zhang, Yingtao Zhang, and Heng-Da Cheng.  Crackgan: Pavement crack detection using partially accurate ground truths based on generative adversarial learning.  TITS, 22(2):1306–1319, 2020.
  * [120] Lei Zhang, Fan Yang, Yimin Daniel Zhang, and Ying Julie Zhu.  Road crack detection using deep convolutional neural network.  In ICIP, 2016.
  * [121] Yongqiang Zhang, Yancheng Bai, Mingli Ding, Shibiao Xu, and Bernard Ghanem.  Kgsnet: Key-point-guided super-resolution network for pedestrian detection in the wild.  IEEE Trans. Neural Networks Learn. Syst., 32(5):2251–2265, 2021\.
  * [122] Zhifei Zhang, Zhaowen Wang, Zhe L. Lin, and Hairong Qi.  Image super-resolution by neural texture transfer.  In CVPR, 2019.
  * [123] Hengshuang Zhao.  Pytorch semantic segmentation.  https://github.com/hszhao/semseg/blob/master/model/pspnet.py.
  * [124] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * [125] Ruofan Zhou and Sabine Süsstrunk.  Kernel modeling super-resolution on real low-resolution images.  In ICCV, 2019.
  * [126] Zhi-Hua Zhou and Xu-Ying Liu.  Training cost-sensitive neural networks with methods addressing the class imbalance problem.  IEEE Trans. Knowl. Data Eng., 18(1):63–77, 2006.
  * [127] Qin Zou, Yu Cao, Qingquan Li, Qingzhou Mao, and Song Wang.  Cracktree: Automatic crack detection from pavement images.  Pattern Recognition Letters, 33(3):227–238, 2012.
  * [128] Qin Zou, Zheng Zhang, Qingquan Li, Xianbiao Qi, Qian Wang, and Song Wang.  Deepcrack: Learning hierarchical convolutional features for crack detection.  IEEE Trans. Image Process., 28(3):1498–1512, 2019.

##  6 Biography Section

photos/kondo.pdf Yuki Kondo received the bachelor degree in engineering from
Toyota Technological Institute in 2022. Currently, he is a researcher with
Toyota Technological Institute. His research interests include low-level
vision including image and video super-resolution and its application to tiny
image analysis such as crack detection. His award includes the best practical
paper award in MVA2021.

photos/ukita.pdf Norimichi Ukita received the B.E. and M.E. degrees in
information engineering from Okayama University, Japan, in 1996 and 1998,
respectively, and the Ph.D. degree in Informatics from Kyoto University,
Japan, in 2001. From 2001 to 2016, he was an assistant professor (2001 to
2007) and an associate professor (2007-2016) with the graduate school of
information science, Nara Institute of Science and Technology, Japan. In 2016,
he became a professor with Toyota Technological Institute, Japan. He was a
research scientist of Precursory Research for Embryonic Science and
Technology, Japan Science and Technology Agency, during 2002–2006, and a
visiting research scientist at Carnegie Mellon University during 2007–2009.
Currently, he is also an adjunct professor at Toyota Technological Institute
at Chicago. Prof. Ukita’s awards include the excellent paper award of IEICE
(1999), the winner award in NTIRE 2018 challenge on image super-resolution,
the 1st place in PIRM 2018 perceptual SR challenge, the best poster award in
MVA2019, and the best practical paper award in MVA2021.