# Variational AutoEncoder for Reference based Image Super-Resolution

Zhi-Song Liu1
1Caritas Institute of Higher Education
<EMAIL_ADDRESS>Wan-Chi Siu1,2 Li-Wen Wang 2
1,2The Hong Kong Polytechnic University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a novel reference based image super-resolution
approach via Variational AutoEncoder (_RefVAE_). Existing state-of-the-art
methods mainly focus on single image super-resolution which cannot perform
well on large upsampling factors, e.g., 8$\times$. We propose a reference
based image super-resolution, for which any arbitrary image can act as a
reference for super-resolution. Even using random map or low-resolution image
itself, the proposed RefVAE can transfer the knowledge from the reference to
the super-resolved images. Depending upon different references, the proposed
method can generate different versions of super-resolved images from a hidden
super-resolution space. Besides using different datasets for some standard
evaluations with PSNR and SSIM, we also took part in the NTIRE2021 SR Space
challenge [28] and have provided results of the randomness evaluation of our
approach. Compared to other state-of-the-art methods, our approach achieves
higher diverse scores.

##  1 Introduction

Image Super-Resolution (SR) is a fundamental problem in image processing.
Given a low-resolution (LR) image, the objective is to upsample the LR image
by $\alpha\times$ to obtain the SR image. In other words, each LR pixel is
used for predicting $\alpha\times\alpha$ HR pixels. In real applications,
super-resolution is useful for images/videos display, storage, broadcasting
and transmission. In particular, it can provide better visual experience for
watching movies and playing video games. [2, 3]

Figure 1: Visualization of 8$\times$ SR using our proposed _RefVAE_. The
visualization of the SR images given different references (right upper corner
of the SR images), like random noise, random images or LR image itself.

Most works for image super-resolution are limited as they focus on single
image super-resolution by optimizing the mean squared errors (MSE) between HR
and SR pixels. Due to the ill-posed nature of super-resolution, using MSE
usually leads to the blurry effect on edges [6, 19]. One solution is to use
Generative Adversarial Network to implicitly align the distribution between SR
and HR images. Despite recent development on GAN based image SR [19, 38, 32],
they usually focus on smaller upsampling factors, like 4$\times$ [38]. For
larger upsampling factors, like 16$\times$ SR, researchers either use
4$\times$ SR model twice, or use simple interpolation to upsample the LR image
by 4$\times$ first and then use SR model for another 4$\times$ upsampling
[20]. Our concern is that the single image super-resolution has reached its
limit on large upsampling factors even with most advanced deep learning
techniques. Hence, we propose a reference based image super-resolution
(_RefVAE_), which can search similar patterns from the reference to guide the
super-resolution. It is an online searching but fast processing because of the
propsoed Conditional Variational Inference (CVAE). It is not an uncommon
procedure in image SR using online approach to search similar patches from
external [45] or internal [44] datasets for SR. The difference is that we
compress various reference images into a compact hidden space by CVAE. Instead
of learning complex regression models [14], we propose to use CVAE to learn
explicit distribution from reference images, then we sample the patterns from
the distribution as a condition or prior to super-resolve LR image. Different
from reference based image super-resolution,hence it is fast for real-time
applications. Our proposed method is also different from single image SR. Once
the training is done, we can use different “references” to generate different
SR candidates, hence, we can expand the SR space to allow users to pick one of
the SR candidates for applications. As shown in Figure 1, the “reference”
(right-hand upper sub-images) can be any images, even a random Gaussian map or
the LR image itself. Our approach can generate a SR image with good visual
quality but with different perceptual details around the edges and textures
(The differences are marked in boxes with different colors).

To sum up, our contributions include:

  * •

We propose _RefVAE_ to explicitly discover image distribution using the
proposed Conditional Variational AutoEncoder for image super-resolution.

  * •

Instead of learning directly the data distribution, we introduce references as
a condition to guide super-resolution. To transfer the features from style
images, we propose to combine both pixel loss and style loss for training.
(See Section )

  * •

We provide experimental results on different datasets and show analysis and
comparison on both single image SR and reference based image SR results.

##  2 Related work

In this section, we give a brief review on related previous works on single
image super-resolution. To introduce our proposed _RefVAE_ method, we also
revisit some representative works on generative learning approaches for image
SR. Finally, we also introduce related works on reference based image super-
resolution.

###  2.1 Single image super-resolution

Single image super-resolution uses one single LR image to produce the
corresponding SR image. It is a classic topic in image processing and a lot of
works have been proposed to resolve it. With the development of learning
approaches, learning based image SR dominates this field. We can categorize
learning based SR into two groups: conventional learning based approaches [35,
22, 21] and deep learning based approaches [27, 42, 20, 26, 17].

Let us focus on some representative deep learning based approaches. Dong et
al. [9] proposed the first CNN based image SR using only three layers of
convolutions to learn the mapping relations between LR and HR images. Later
on, VDSR [17], LapSRN [18], SRResNet [19], EDSR [20] and many other works [42,
43] use a deeper and wider (number of filters) CNN model to learn the mapping
functions for super-resolution. Back projection in CNN [12, 26] also sets a
new path for image SR for which iterative residual update can minimize the
feature distance for super-resolution. Recent studies [27, 8, 30] show that
attention can learn non-local features for super-resolution, especially useful
for larger images. Liu et al. [27] proposed an attention based back projection
network for large-scale super-resolution. Dai et al. [8] proposed higher-order
attention using channel and spatial attention for image. To fully explore the
correlations in the channel dimension, Niu et al. [30] proposed a holistic
attention network for image super-resolution. The above mentioned approaches
train different convolutional neural network end-to-end to optimize mean
squared errors between SR and HR pixels. The problem is that using MSE to give
equal weights to different pixels can cause blurry effect. To address this
problem, many researchers have proposed generative learning approaches for
image SR.

###  2.2 Generative learning approaches for SR

Using generative learning approaches for image SR is a popular topic. We can
consider using a single CNN model for SR as discriminative learning that does
not explore the data distribution for modelling. On the other hand, generative
learning approaches explicitly or implicitly study the data distribution for
modelling. Generative Adversarial Network [11] is one of the popular implicit
generative approaches. It is also used in image SR. For example, SRGAN [19]
uses GAN to train 4$\times$ SR. In order to generate more photo-realistic
features, VGG based feature loss is used to minimize the deep feature
distances between SR and HR images. ESRGAN [38] proposes to use relativistic
GAN [16] to let the discriminator predict relative realness instead of the
absolute value. ESRGAN+ [32] further improves ESRGAN by introducing noise for
to explore stochastic variations. Vartiaonal AutoEncoder is another choice for
SR.

Different from GAN, VAE explicitly explores the data distribution for
modelling. Liu et al. [24] first proposed VAE for image super-resolution that
achieves comparable performance as CNN based approaches. Furthermore, [23]
introduces a reference based face SR that uses the conditional VAE to achieve
domain-specific super-resolution. A sampling generator is proposed to
constraint the choice of samples for better face reconstruction. To avoid the
blurry effect [10] caused by the sampling process in VAE, [25] uses VAE for
both image super-resolution and denoising by combining GAN and VAE together so
that it can use adversarial loss to encourage photo-realistic feature
reconstruction. Most recently, Lugmayr et al. [29] proposed to use conditional
normalizing flow to model image distribution for super-resolution and achieved
good performance in both quantitative and qualitative results. Normalizing
flow based approaches can also be extended to image manipulation [4, 7].
However, the whole normalizing flow structure is over-complex (approx.
50,000,000 parameters) which could be difficult for training.

###  2.3 Reference based Super-Resolution (RefSR)

The development of reference based image super-resolution (RefSR) is not a
surprise to researchers. Originally, it comes from the non-local filtering.
The conventional learning based approaches are patch based process. In other
words, the LR patches can be reconstructed by learning the mapping models from
images/videos with similar contents [45, 46, 34, 44, 14]. For example, k
Nearest Neighbor (kNN) [14] can be used for online search so it limits the big
data search. Random Forests [21] is a fast algorithm that can classify
training patches into different groups for diverse regression modelling.

Figure 2: The training and testing processes of the proposed RefVAE. It
consists of 1) VGG Encoder (the green box), 2) Conditional Variational
Variational AutoEncoder (CVAE) (the pink box), 3) Image Decoder (the dark
green box), and Discriminator (Discrim.) for loss computation. Detailed
structures of the Image Decoder and Feature Encoder/Decoder are also shown on
the right bottom corner.

Reference based SR can also be achieved by using deep learning. [45] proposes
to use an extra HR reference to guide SR. The idea is to use CNN to extract
multi-scale HR features to fill the missing information for LR images.
Similarly, Zheng et al. [46, 34] proposed a similar reference based SR
approach using CNN. The difference is that it uses flow warping operation to
align LR and reference features for super-resolution. However, it still
requires a pre-aligned reference for SR, or the warping operation cannot well
match the feature maps. Most recently, Tan et al. [34] proposed a reference
pool to expand the searching region for multi-reference based super-
resolution. Not only CNN can be used for RefSR, Liu et al. [23] proposed a
face SR that uses an image from the same identity as reference. The idea is to
train a conditional Variational AutoEncoder to learn conditional distribution
between the reference and LR image. The limitation is that it only
investigates the facial image SR and the reference image has to be the face
with the same identity. It is not suitable for general image SR.

The study of RefSR also reveals the ill-posed nature of SR that more than one
SR image can be formed from a LR image. Depending on different external or
internal images as reference, we can transfer reference features to fill out
the missing information for LR image enlargement. As described in the Learning
the Super-Resolution Space Challenge [28], we can formulate the SR problem as
learning a stochastic mapping, capable of sampling from the space of plausible
high-resolution images given a low-resolution image. By using an arbitrary
reference image to expand data diversity, RefSR is one way to resolve SR
space. The advantage is that 1) multiple predictions can be generated and
compared and 2) we allow users to choose references with desired patterns for
interactive SR.

##  3 Method

In this section, we will give detailed introduction on the proposed Reference
based image super-resolution approach via Variational AutoEncoder (_RefVAE_).
Let us formally define the image SR. Mathematically, given a LR image
$\mathbf{X}\in\mathbb{R}^{\mathit{m}\times\mathit{n}\times 3}$ which has been
down-sampled from the corresponding HR image
$\mathbf{Y}\in\mathbb{R}^{\mathit{\alpha m}\times\mathit{\alpha n}\times 3}$,
where ($\mathit{m}$, $\mathit{n}$) is the dimension the image and $\alpha$ is
the up-sampling factor. They are related by the following degradation model,

$\mathbf{X}=\mathbf{D}\otimes\mathbf{Y}+\mu$ (1)

where $\otimes$ represents the down-sampling convolution, $\mu$ is the
additive white Gaussian noise and D denotes the down-sampling kernel. The goal
of image SR is to resolve Equation 1 as the Maximum A Posterior (MAP) problem
as,

$\mathbf{\hat{Y}}=\underset{\mathbf{Y}}{\arg\max}\,logP(\mathbf{X|Y})+logP(\mathbf{Y})$
(2)

where $\mathbf{\hat{Y}}$ is the predicted SR image.
log$P$($\mathbf{X}|\mathbf{Y}$) represents the log-likelihood of LR images
given HR images and log$P$($\mathbf{Y}$) is the prior of HR images that is
used for model optimization. Note that Equation 2 generally describes the
optimization process that multiple SR $\mathbf{\hat{Y}}$ can be found. We can
rewrite Equation 2 as Reference based SR,

$\begin{matrix}\begin{split}&\mathbf{\hat{Y}_{i}}=\underset{\mathbf{Y}}{\arg\max}\,logP(\mathbf{X},\mathbf{R}_{i}|\mathbf{Y})+logP(\mathbf{Y})\\\
&where\ \mathbf{R}_{i}\in\mathbb{S}_{\mathbf{R}}\ and\
\mathbf{\hat{Y}_{i}}\in\mathbb{S}_{\mathbf{Y}}\end{split}\end{matrix}$ (3)

and we define the SR space as $\mathbb{S}_{\mathbf{Y}}$ and the reference
space as $\mathbb{S}_{\mathbf{R}}$. The desired SR image can be randomly
sampled from $\mathbb{S}_{\mathbf{R}}$ space given arbitrary reference image.

The complete architecture of the proposed _RefVAE_ is shown in Figure 2, which
includes the training (upper half of the figure) and testing (lower half of
the figure) stages. Our proposed _RefVAE_ takes arbitrary references
$\mathbf{R}_{i}$ and LR images $\mathbf{X}$ for training and testing. It
includes three components: 1) VGG Encoder, 2) CVAE (in the pink box), and 3)
Image Decoder. The VGG Encoder is used to extract feature maps from LR and
reference images. CVAE is used to transfer reference features to a hidden
space for conditional feature sampling. In other words, CVAE generates a
candidate feature pool where random feature vectors can be sampled as
conditional features. The LR feature maps are split into mean and variance
maps, then they compute with the conditional features to obtain the estimated
features. Finally, the Image Decoder takes the estimated features for image
reconstruction.

###  3.1 VGG Encoder

The VGG Encoder follows the structure of VGG-19 by keeping all convolution
layers and discarding the fully connection layers. We directly use pre-trained
VGG19 [33] to extract feature maps from references
($F_{\mathbf{R}}=G(\mathbf{R})$) and the bicubic upsampled LR images
($F_{\mathbf{X}}=G(\mathbf{X})$), where G stands for the process of VGG
feature extraction. Inside the VGG Encoder, there are three maxpooling layers
to downsample the input image by 8$\times$. Note that we resize arbitrary
reference images to $256\times 256$ before passing it through the VGG Encoder.
Hence we have fixed reference feature maps as
$F_{\mathbf{R}}\in\mathbb{R}^{32\times 32\times 512}$. For LR images, we
initially upsample them by Bicubic interpolation to the desired size. The
reason of using pre-trained VGG Encoder is that: 1) VGG was trained for
general image classification that the extracted feature maps are generalized
to images with different contents and 2) we want to project the LR and
reference images to a same feature domain such that we can fuse their features
together for super-resolution.

###  3.2 Conditional Variational AutoEncoder

We have the Conditional Variational AutoEncoder (CVAE) that projects the
reference feature maps to a latent space to learn the hidden distribution via
Feature Encoder. The Feature Decoder learns to transfer the reference features
as conditions $C_{R}$ for LR feature maps. The combination of Feature Encoder
and Decoder (for detailed structures: see the right of Figure 2) forms the
Variational Inference process. The idea of Variational Inference is to learn
the generative model for the reference images that can be represented by a
Gaussian model as $P(z|\mathbf{R})=N\sim(z;\
\mu(\mathbf{R}),\Sigma(\mathbf{R}))$, where $\mu$ and
$\Sigma=diag(\sigma_{1}^{2},...,\sigma_{n}^{2})$ are the mean and variance of
the learned Gaussian model. In other words, we learn the pixel inter-
correlations and represent them as a probability model. We introduce
randomness by the sampling process. As the Gaussian curve shown in Figure 2,
we use a normal distribution $Q(z)=N\sim(0,I)$ to sample from the reference
model as $z=\mu+\epsilon\cdot\sigma$. To ensure the learned probability model
close to a normal distribution, we use KL divergence to optimize the model as,

$\begin{matrix}\begin{split}D_{KL}(P(z|\mathbf{R})&||Q(z))=E[logP(z|\mathbf{R})-Q(z)]\\\
&=\frac{1}{2}\left(-\sum_{i}(log\sigma^{2}_{i}+1)+\sum_{i}\sigma^{2}_{i}+\sum_{i}\mu_{i}^{2}\right)\end{split}\end{matrix}$
(4)

During the testing, we can discard the Feature Encoder as many existing VAE
based image reconstruction [31, 15]. We can use vectors sampled from the
normal distribution as conditional priors for super-resolution. We can also
keep the Feature Encoder to allow user to define their own reference image for
customized super-resolution. That is, we keep the Feature Encoder to extract
specific prior distribution $P(z|\mathbf{R})=N\sim(z;\
\mu(\mathbf{R}),\Sigma(\mathbf{R}))$ as prior conditions for super-resolution.
Next, to obtain the conditional feature maps $C_{\mathbf{R}}$, we use one
convolution block to project the learned distribution back to the spacial
domain (we also resize it to the same size as the LR feature map via simple
interpolation).

In order to transfer the conditional features to the LR feature map, we use a
convolution block to learn the mean and variance (note that the mean and
variance are the spatial statistics of the feature maps, rather than the
variables of the Gaussian distribution) for the LR feature maps as $F_{\mu}$
and $F_{\Sigma}$. We then have the fused features as
$F_{X|R}=C_{R}\cdot(1+F_{\Sigma})+F_{\mu}$.

###  3.3 Image Decoder

Finally, the Image Decoder learns to reconstruct the SR image from the fused
feature maps $\mathbf{\hat{Y}}$. The structure of Image Decoder (dark green
boxes in Figure 2) has a similar structure as the VGG Encoder stacking three
convolution layers followed by a simple bilinear interpolation.

###  3.4 Training losses

To train the proposed _RefVAE_ to generate SR results with photo-realistic
visual quality, we suggest to use a discriminator to reduce the perceptual
distance between SR and ground truth images. We design the same discriminator
as PatchGAN [37] and the adversarial loss is defined as,

$L_{adv}=log[1-D(\mathbf{\hat{Y}})]$ (5)

The idea of using style and content losses for style transfer [31, 15] is an
efficient approach to transfer the reference style to the target image while
preserving the content information. It is also suitable for reference based
SR. We not only want the SR image close to the ground truth, but we also want
it close to the reference image in terms of style similarity. In other words,
we want to ensure the reference features to be transferred to the LR images.

Content loss For content loss, we extract features for the HR image Y and the
SR $\mathbf{\hat{Y}}$ using VGG-19 [33] (we take the feature map at relu4_1
layer). We refer these features to as $W_{\mathbf{Y}}$ and
$W_{\mathbf{\hat{Y}}}$, respectively. For pixel-wised difference, we also have
the $L_{1}$ loss between SR and HR and their down-sampled versions. Totally,
we have the content loss as,

$L_{\text{content}}=\left\|W_{\mathbf{Y}}-W_{\mathbf{\mathbf{\hat{Y}}}}\right\|^{1}+\left\|\mathbf{Y}-\mathbf{\hat{Y}}\right\|^{1}+\left\|Down(\mathbf{Y})-Down(\mathbf{\hat{Y}})\right\|^{1}~{}~{}.$
(6)

where Down is the $\alpha\times$ bicubic down-sampling operator.

Style loss. To measure the style similarity, we use VGG-19 to extract feature
maps (relu1_2, relu2_2, relu3_4, relu4_1) for the reference image R and the SR
image $\mathbf{\hat{Y}}$ as $V_{\mathbf{R}}$ and $V_{\mathbf{\hat{Y}}}$.
Similar to [31], we measure their mean and variance to align the SR features
close to the reference features as:

$L_{\text{style}}=\sum_{i}\|mean(V^{i}_{\mathbf{Y}}-V^{i}_{\mathbf{\hat{Y}}})\|^{1}+\|var(V^{i}_{\mathbf{Y}}-V^{i}_{\mathbf{\hat{Y}}})\|^{1}~{}~{}.$
(7)

where mean and std are the operations for calculating the mean and variance of
the feature maps. In order to have SR images visually close to the HR image,
the LPIPS loss [41] is used to measure perceptual differences as
$L_{LPIPS}(\mathbf{\hat{Y}},\mathbf{Y})$. The Total Variation loss
$L_{TV}(\mathbf{\hat{Y}})$ is used to encourage smooth quality, for which we
calculate the first-order horizontal and vertical pixel gradients as

$L_{TV}=\sum_{i,j}\left((\mathbf{\hat{Y}}_{i,j-1}-\mathbf{\hat{Y}}_{i,j})^{2}+(\mathbf{\hat{Y}}_{i-1,j}-\mathbf{\hat{Y}}_{i,j})^{2}\right)^{\beta}~{}~{}.$
(8)

Total loss. We have the total loss as,

$\begin{matrix}\begin{split}L_{\text{style}}=&\lambda_{content}L_{\text{content}}+\lambda_{style}L_{\text{style}}+\lambda_{LPIPS}L_{\text{LPIPS}}\\\
&+\lambda_{TV}L_{\text{TV}}+\lambda_{KL}L_{\text{KL}}~{}~{}.\end{split}\end{matrix}$
(9)

where $\lambda_{content}$, $\lambda_{style}$, $\lambda_{LPIPS}$,
$\lambda_{TV}$ and $\lambda_{KL}$ are the weighting parameters for content
loss, style loss, LPIPS loss, TV loss and KL loss. Figure 3 shows a summary of
all the loss terms for readers’ reference.

Figure 3: Training Losses in the proposed _RefVAE_. It consists of 1) Content
loss, 2) Style loss, 3) KL loss, 4) TV loss, 5) LPIPS loss and 6) Adversarial
loss.

##  4 Experiments

Figure 4: Visualization of 8$\times$ SR on different datasets using different
SR methods. (a) is from Set14, (b) is from Urban100, (c) and (d) are from
DIV2K validation.  Figure 5: Visualization of 8$\times$ SR on using different
reference images. To visualize the differences, we subtract the SR to the
ground truth HR image to obtain the residual map. Then we multiple the
residual map by 5 to enhance the differences. We also show two examples of
using LR and HR images as references for SR. The significant differences are
marked in yellow circles.

###  4.1 Data Preparation and Network Details

We trained our model on DIV2K [36] and Flickr2K [20] datasets. They both
contain images with resolution larger than 1000$\times$1000\. We extracted LR
and HR patches from the training dataset with the size of 32$\times$32 and
32$\alpha\times$32$\alpha$, respectively, where $\alpha$=8 is the upsampling
factor. The down-sampling process is done by bicubic interpolation. For the
reference image, we used Wikiart [1] that is widely used in style transfer.
The testing datasets included Set5 [5], Set14 [40], Urban100 [13] and DIV2K
validation [36].

We conducted our experiments using Pytorch 1.7 on a PC with one NVIDIA
GTX1080Ti GPU. During the training, we set the learning rate to 0.0001 for all
layers. The batch size was set to 16 and we trained it for 5$\times 10^{4}$
iterations. For optimization, we used Adam with the momentum equal to 0.9 and
the weight decay of 0.0001. During the training, we set $\lambda_{content}=1$,
$\lambda_{style}=10$, $\lambda_{LPIPS}=1$, $\lambda_{TV}=1$ and
$\lambda_{KL}=1$, which are the weighting parameters for content loss, style
loss, LPIPS loss, TV loss and KL loss. The executive codes and more
experimental results can be found in: https://github.com/Holmes-Alan/RefVAE

Table 1: Comparison with state-of-the-art methods. We compare with SRNTT and
SRFlow on 8$\times$ super-resolution on PSNR (dB), SSIM and LPIPS.

Methods | Set5 | Set14 | BSD100 | Urban100 | DIV2K validation
---|---|---|---|---|---
PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
Bicubic | 24.39 | 0.657 | 0.537 | 23.19 | 0.568 | 0.630 | 23.67 | 0.547 | 0.713 | 21.24 | 0.516 | 0.686 | 25.17 | 0.664 | 0.584
SRNTT(HR) | 24.69 | 0.674 | 0.492 | 23.16 | 0.564 | 0.613 | 23.63 | 0.542 | 0.687 | - | - | - | - | - | -
SRNTT(LR) | 24.36 | 0.655 | 0.522 | 23.16 | 0.564 | 0.613 | 23.65 | 0.543 | 0.691 | - | - | - | - | - | -
SRFlow | 24.18 | 0.650 | 0.237 | 22.13 | 0.513 | 0.331 | 22.96 | 0.499 | 0.425 | 20.676 | 0.531 | 0.300 | 24.53 | 0.616 | 0.272
Our(LR) | 25.87 | 0.723 | 0.214 | 24.15 | 0.598 | 0.323 | 24.15 | 0.558 | 0.376 | 21.74 | 0.575 | 0.334 | 24.75 | 0.641 | 0.232
Our(HR) | 25.92 | 0.724 | 0.218 | 24.20 | 0.601 | 0.327 | 24.18 | 0.561 | 0.386 | 21.80 | 0.575 | 0.332 | 24.94 | 0.650 | 0.201
Our(random) | 25.94 | 0.721 | 0.224 | 24.28 | 0.601 | 0.328 | 24.25 | 0.561 | 0.384 | 21.85 | 0.576 | 0.358 | 25.00 | 0.650 | 0.202

#### Metrics.

For evaluation, we have used several metrics: PSNR measures the average pixel
differences between ground truth and estimation. SSIM [39] measures the
structural similarity between ground truth and estimation. 111To calculate
PSNR and SSIM, we first convert the SR and ground truth images from RGB to
YCbCr and take the Y channel for computation. LPIPS [41] measures the
perceptual similarity between ground truth and estimation. Diverse score
measures the spanning of the SR space. We follow the same measurement defined
by NTIRE2021 SR space challenge [28]. We sampled 10 images, and densely
calculated a metric between the samples and the ground truth. To obtain the
local best we pixel-wisely selected the best score out of the 10 samples and
took the full image’s average. The global best was obtained by averaging the
whole image’s score and selecting the best. Finally, we calculated the score
as: (global best - local best)/(global best) $\times$ 100.

###  4.2 Compare with state-of-the-art methods

Our proposed _RefVAE_ is for diverse image super-resolution that is able to
generate many SR results. We consider two types of related SR methods for
comparison: reference based image SR and generative model based image SR. The
SR approaches we compare with are: SRNTT [44], reference based SR using
similar image as reference for online warping, and SRFlow [29], normalizing
flow based SR that predicts multiple SR results. 222Note that since most of
state-of-the-art SR approaches are dedicated for producing one prediction with
high PSNR and do not consider the possibility of multiple SR results, it is
not our intention to compete with those approaches in terms of PSNR and SSIM.

In Table 1, we show the overall comparison with SRNTT and SRFlow on 8$\times$
image super-resolution on Set5, Set14, BSD100, Urban100 and DIV2K validation
datasets. It is seen that our proposed _RefVAE_ can achieve the best PSNR,
SSIM and LPIPS by about 2 dB, 0.1 and 0.01, respectively. Note that SRNTT [44]
was originally trained and tested on specific CUFED5 dataset [44] for which
each image has five images with similar contents. For a fair comparison, we
tested SRNTT by using LR and HR images as reference for SR, which are
SRNTT(LR) and SRNTT(HR). 333Note also that SRNTT requires a lot of memory for
online warping. Larger image SR cannot be done on one GPU, hence we did not
have the results for Urban100 and DIV2K. Similarly, we also tested our
proposed _RefVAE_ by using LR and HR images as Our(LR) and Our(HR),
respectively. Visually, we show four examples in Figure 4. It can be seen that
our proposed method can restore the details better than others. Especially, it
is seen that SRFlow collapses on several cases. For example, SRFlow generated
a large black region on ppt3. It also produces holes on img076, 0816 and 0897
with noises. For SRNTT, it generated blurry results without restoring sharp
edges. On the other hand, our proposed _RefVAE_ can generate plausible SR
results with sharp edges and textures.

More importantly, we are interested in the ability of expansion of SR space.
Since our proposed _RefVAE_ can take arbitrary references for super-
resolution, we can measure the diversity by using Diverse score defined in
NTIRE2021 SR space challenge [28].

Table 2: Analysis on SR space of _RefVAE_ on DIV2K dataset. The PSNR, SSIM and
LPIPS on SR estimation are indicated as SR PSNR, SR SSIM and SR LPIPS, while
for LR estimation they are respectively indicated as LR PSNR and LR SSIM. The
overall estimation using SRFlow and our methods are shown in the 2nd half of
the table.

Reference |  SR PSNR $\uparrow$ |  SR SSIM $\uparrow$ |  SR LPIPS $\downarrow$ |  LR PSNR $\uparrow$ |  LR SSIM $\uparrow$
---|---|---|---|---|---
Ref. 1 | 24.27 | 0.616 | 0.310 | 42.90 | 0.996
Ref. 2 | 25.18 | 0.661 | 0.306 | 44.55 | 0.997
Ref. 3 | 25.03 | 0.654 | 0.304 | 44.37 | 0.997
Ref. 4 | 24.76 | 0.644 | 0.302 | 43.93 | 0.997
Ref. 5 | 25.28 | 0.664 | 0.314 | 44.80 | 0.997
Ref. 6 | 25.17 | 0.662 | 0.306 | 44.81 | 0.997
Ref. 7 | 24.62 | 0.627 | 0.298 | 43.43 | 0.996
Ref. 8 | 25.04 | 0.659 | 0.307 | 44.26 | 0.997
Ref. 9 | 24.27 | 0.611 | 0.320 | 42.86 | 0.996
Ref. 10 | 24.42 | 0.615 | 0.310 | 42.30 | 0.996
LR | 25.31 | 0.664 | 0.306 | 46.21 | 0.997
HR | 25.40 | 0.665 | 0.308 | 45.63 | 0.997
Overall Estimation
| SRFlow
---
Div. score: 10.07
24.46 | 0.615 | 0.328 | 51.18 | 0.996
| Our (Reference)
---
Div. score: 14.80
24.89 | 0.641 | 0.308 | 44.17 | 0.997
| Our (Random)
---
Div. score: 14.91
24.91 | 0.642 | 0.321 | 44.77 | 0.997

As discussed in Section III, our proposed _RefVAE_ can take random noise or
external images as references. In Table 2, we randomly chose 10 images from
WikiArt as references for SR. We also use LR image itself and HR image as
references. We list the results on SR estimation as SR PSNR, SR SSIM and SR
LPIPS. It can be seen that the proposed _RefVAE_ can generate different SR
images (different PSNR and SSIM) given different references. It demonstrates
that the proposed _RefVAE_ can expand the SR space by a large margin. In the
meantime, we do not want to distort SR images to be far away from the ground
truth LR images. Hence, we down-sample SR images by bicubic to estimate PSNR
and SSIM on the LR space as LR PSNR and LR SSIM, respectively. It can be seen
that different SR estimation can well preserve LR information with PSNR over
40 dB.

Overall, we have calculated the Diverse score of SRFlow and our approach in
Table 2. Given different references, we have two results as entitled
Our(Reference) and Our(Random). It can be seen that our approach can
outperform SRFlow in terms of PSNR, SSIM and Diverse score. We also show SR
results of using 12 different references in Figure 5. To visualize the
differences among multiple SR images, we subtracted the SR by the ground truth
HR images to obtain the residual maps, then we multiplied the residual values
by 5 to show the differences. We can observe the differences around the edge
and texture regions from the residual maps (marked in yellow circles), e.g.,
the patterns of the T-shirt and the roof of the building. In the meantime, we
also visualize the SR results using LR and HR images. It can be seen that
using LR or HR as references can better reduce the residues.

Table 3: Ablation study of _RefVAE_ on DIV2K dataset on 8$\times$ super-
resolution. We report the average across datasets.

Model | Components | Eval.
---|---|---
CVAE | SC | Dicri. |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Div. $\uparrow$
Our | - | - | - | 25.23 | 0.650 | 0.412 | 0
✓ | - | - | 24.82 | 0.641 | 0.338 | 12.326
✓ | ✓ | - | 24.84 | 0.643 | 0.400 | 14.142
Final | ✓ | ✓ | ✓ | 24.75 | 0.640 | 0.272 | 14.992

###  4.3 Ablation study

Let us evaluate various key components of the _RefVAE_ : (1) the Conditional
Variational AutoEncoder (CVAE), (2) the use of style and content losses (SC
loss), (3) the use of discriminator (Discri.), To examine their impact, we
started with a simple network structure consisting Image Encoder and Decoder
only, and we progressively added all components. Table 3 reports the results
of 8$\times$ SR on DIV2K validation datasets. We have included PSNR, SSIM,
LPIPS and Div. (Diverse score) in the table. From the values of Div., we can
find that using CVAE module can introduce randomness into the network to
generate multiple SR results. When the style and content losses were used for
training, we can further maximize the Diverse score to expand the SR space.
From the values of PSNR, SSIM and LPIPS, using CVAE, SC loss and Discri can
reduce the PSNR and SSIM, but they can improve the LPIPS value, which
indicates better quality.

Figure 6: Weak cases of using proposed _RefVAE_. We show two 8$\times$ SR
examples from BSD100 dataset. It can be found that the facial details cannot
be preserved.

###  4.4 Weak cases

In terms of perceptual quality, our proposed method does not perform too well
on images with smaller size. We show two examples in Figure 6. It can be seen
that the fine facial expression cannot be ideally restored by our approach. To
resolve this problem, we could further study in depth of the latent
representation of the Variational AUtoEncoder. One other solution is to use
hierarchical latent space manipulation. This is really a fruitful direction
for research.

##  5 Conclusion

In this paper, we introduce a novel approach for reference based image super-
resolution _RefVAE_. Unlike other reference based SR approaches, our proposed
method can take any image with arbitrary content or even random noise as
reference for image super-resolution. It is also unlike other generative model
based SR approaches, _RefVAE_ can expand the SR space, so that multiple unique
SR images can be generated. Furthermore, _RefVAE_ does not suffer from the
model collapse or generate bizarre patterns like other generative model based
approaches (examples are shown in Figure 4). Our analysis shows that _RefVAE_
can better preserve the LR information and generate SR image with better
perceptual quality. Possible future work includes the super-resolution space
exploration by probing the latent vector or by image quantization.

## References

  * [1] K. nichol. painter by numbers, wikiart.  https://www.kaggle.com/c/painter-by-numbers, 2016.
  * [2] Nvidia dlss 2.0: A big leap in ai rendering.  https://www.nvidia.com/en-gb/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/, March, 2020.
  * [3] Amd fidelityfx.  https://www.amd.com/en/technologies/radeon-software-fidelityfx, October, 2020.
  * [4] Yuval Bahat and Tomer Michaeli.  Explorable super resolution.  In CVPR, pages 2713–2722. IEEE, 2020.
  * [5] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie line Alberi Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In Proceedings of the British Machine Vision Conference, pages 135.1–135.10. BMVA Press, 2012.
  * [6] Y. Blau and T. Michaeli.  The perception-distortion tradeoff.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6228–6237, 2018.
  * [7] Marcel C. Bühler, Andrés Romero, and Radu Timofte.  Deepsee: Deep disentangled semantic explorative extreme super-resolution.  In ACCV, volume 12625 of Lecture Notes in Computer Science, pages 624–642. Springer, 2020.
  * [8] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11065–11074, 2019.
  * [9] C. Dong, C. C. Loy, K. He, and X. Tang.  Image super-resolution using deep convolutional networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, Feb 2016.
  * [10] Jesse Engel, Matthew Hoffman, and Adam Roberts.  Latent constraints: Learning to generate conditionally from unconditional generative models.  CoRR, abs/1711.05772, 2017.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672–2680. Curran Associates, Inc., 2014.
  * [12] M. Haris, G. Shakhnarovich, and N. Ukita.  Deep back-projection networks for single image super-resolution.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
  * [13] J. Huang, A. Singh, and N. Ahuja.  Single image super-resolution from transformed self-exemplars.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5197–5206, June 2015.
  * [14] Kwok-Wai Hung and Wan-Chi Siu.  Robust soft-decision interpolation using weighted least squares.  IEEE Transactions on Image Processing, 21(3):1061–1069, 2012.
  * [15] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Eur. Conf. Comput. Vis., 2016.
  * [16] Alexia Jolicoeur-Martineau.  The relativistic discriminator: a key element missing from standard gan.  arXiv preprint arXiv:1807.00734, 2018.
  * [17] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  CoRR, abs/1511.04587, 2015.
  * [18] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Deep laplacian pyramid networks for fast and accurate super-resolution.  CoRR, abs/1704.03915, 2017.
  * [19] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105–114, 2017.
  * [20] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee.  Enhanced deep residual networks for single image super-resolution.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132–1140, 2017.
  * [21] Zhi-Song Liu and Wan-Chi Siu.  Cascaded random forests for fast image super-resolution.  In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 2531–2535, Oct 2018.
  * [22] Zhi-Song Liu, Wan-Chi Siu, and Yui-Lam Chan.  Fast image super-resolution via randomized multi-split forests.  In 2017 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–4, May 2017.
  * [23] Zhi-Song Liu, Wan-Chi Siu, and Yui-Lam Chan.  Reference based face super-resolution.  IEEE Access, 7:129112–129126, 2019.
  * [24] Zhi-Song Liu, Wan-Chi Siu, and Yui-Lam Chan.  Photo-realistic image super-resolution via variational autoencoders.  IEEE Transactions on Circuits and Systems for Video Technology, pages 1–1, 2020.
  * [25] Zhi-Song Liu, Wan-Chi Siu, Wang Li-Wen, Li Chu-Tak, Marie-Paule Cani, and Yui-Lam Chan.  Unsupervised real image super-resolution via generative variational autoencoder.  In IEEE International Conference on Computer Vision and Pattern Recognition Workshop(CVPRW), June 2020.
  * [26] Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, and Wan-Chi Siu.  Hierarchical back projection network for image super-resolution.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2041–2050, 2019.
  * [27] Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, Wan-Chi Siu, and Yui-Lam Chan.  Image super-resolution via attention based back projection networks.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3517–3525, 2019.
  * [28] Andreas Lugmayr, Martin Danelljan, and et al. Radu Timofte.  Ntire 2021 learning the super-resolution space challenge.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [29] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In ECCV, pages 715–732. Springer, 2020.
  * [30] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 191–207, Cham, 2020. Springer International Publishing.
  * [31] Dae Young Park and Kwang Hee Lee.  Arbitrary style transfer with style-attentional networks.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [32] N. C. Rakotonirina and A. Rasoanaivo.  Esrgan+ : Further improving enhanced super-resolution generative adversarial network.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3637–3641, 2020.
  * [33] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Int. Conf. Learn. Represent., 2015.
  * [34] Y. Tan, H. Zheng, Y. Zhu, X. Yuan, X. Lin, D. Brady, and L. Fang.  Crossnet++: Cross-scale large-parallax warping for reference-based super-resolution.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
  * [35] Radu Timofte, Vincent De Smet, and Luc Van Gool.  A+: Adjusted anchored neighborhood regression for fast super-resolution.  volume 9006, pages 111–126, 04 2015.
  * [36] Radu Timofte and et al.  Ntire 2017 challenge on single image super-resolution: Methods and results.  In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1110–1121. IEEE Computer Society, 8 2017.
  * [37] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [38] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang.  ESRGAN: enhanced super-resolution generative adversarial networks.  CoRR, abs/1809.00219, 2018.
  * [39] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, April 2004\.
  * [40] Roman Zeyde, Michael Elad, and Matan Protter.  On single image scale-up using sparse-representations.  volume 6920, pages 711–730, 06 2010.
  * [41] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [42] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  CoRR, abs/1807.02758, 2018.
  * [43] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  CoRR, abs/1802.08797, 2018.
  * [44] Z. Zhang, Z. Wang, Z. Lin, and H. Qi.  Image super-resolution by neural texture transfer.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7974–7983, 2019.
  * [45] Haitian Zheng, Mengqi Ji, Lei Han, Ziwei Xu, Haoqian Wang, Yebin Liu, and Lu Fang.  Learning cross-scale correspondence and patch-based synthesis for reference-based super-resolution.  In Gabriel Brostow Tae-Kyun Kim, Stefanos Zafeiriou and Krystian Mikolajczyk, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 138.1–138.13. BMVA Press, September 2017.
  * [46] Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang.  Crossnet: An end-to-end reference-based super resolution network using cross-scale warping.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 88–104, 2018.


# On nearly free arrangements of lines with nodes and triple points

Jakub Kabat

###### Abstract

We provide a classification result on nearly free arrangements of lines in the
complex projective plane with nodes and triple points.

Keywords 14N20, 52C35, 32S22

Mathematics Subject Classification (2020) nearly free curves, line
arrangements

##  1 Introduction

In the recent years there is a great interest on line arrangements in the
complex projective plane that are free. This is due to the reason that the
celebrated Terao’s conjecture predicts that the freeness of complex line
arrangements in the projective plane is determined by the combinatorics. In
the context of a potential counterexample to Terao’s conjecture, Dimca and
Sticlaru in [4] defined a new class of line arrangements that is called nearly
free. Roughly speaking, if Terao’s conjecture fails, then it is expected that
we have two combinatorially equivalent line arrangements, one is free and the
second is nearly free. Here our aim is to understand nearly free complex line
arrangement that are nearly free, and they have only nodes and triple
intersection points. This setting is motivated by a very recent paper by Dimca
and Pokora [6] where the authors study free conic-line arrangement with nodes,
tacnodes, and ordinary triple points. Based on their ideas, we perform our
classification procedure and we obtain the following result.

######  Main Theorem.

Let $\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be an arrangement of $d$
lines with only nodes and triple intersection points. Suppose that
$\mathcal{L}$ is nearly free, then $d\in\\{4,5,6,7,8\\}$.

Here is the structure of the paper. In Section 2, we recall basics on nearly
free reduced plane curves in the plane. In Section 3, we provide some
combinatorial constraints on the number of lines of nearly free line
arrangements with nodes and triple points. In Section 4, using deformation
arguments applied on free line arrangements with nodes and triple points, we
perform a classification procedure that will eventually lead to Main Theorem.

##  2 Introduction to nearly free curves

Let $C$ be a reduced curve $\mathbb{P}^{2}_{\mathbb{C}}$ of degree $d$ given
by $f\in S:=\mathbb{C}[x,y,z]$. We denote by $J_{f}$ the Jacobian ideal
generated by the partials derivatives
$\partial_{x}f,\,\partial_{y}f,\,\partial_{z}f$. Moreover, we denote by
$r:={\rm mdr}(f)$ the minimal degree of a relation among the partial
derivatives, i.e., the minimal degree $r$ of a triple $(a,b,c)\in S_{r}^{3}$
such that

$a\cdot\partial_{x}f+b\cdot\partial_{y}f+c\cdot\partial_{z}f=0.$

We denote by $\mathfrak{m}=\langle x,y,z\rangle$ the irrelevant ideal.
Consider the graded $S$-module $N(f)=I_{f}/J_{f}$, where $I_{f}$ is the
saturation of $J_{f}$ with respect to $\mathfrak{m}=\langle x,y,z\rangle$.

######  Definition 2.1.

We say that a reduced plane curve $C$ is _nearly free_ if $N(f)\neq 0$ and for
every $k$ one has ${\rm dim}\,N(f)_{k}\leqslant 1$.

To complete the picture, we need to define free curves.

######  Definition 2.2.

We say that a reduced plane curve $C$ is _free_ if $N(f)=0$.

In order to study the nearly freeness of a reduced plane curve $C\,:\,f=0$,
$f\in S_{d}$, Dimca and Sticlaru provided a homological criterion on the
Milnor algebra $M(f):=S/J_{f}$.

######  Theorem 2.3 (Dimca-Sticlaru).

If $C$ is a nearly free curve of degree $d$ given by $f\in S$, then the
minimal free resolution of the Milnor algebra $M(f)$ has the following form:

$\begin{split}0\rightarrow S(-b-2(d-1))\rightarrow S(-d_{1}-(d-1))\oplus
S(-d_{2}-(d-1))\oplus S(-d_{3}-(d-1))\\\ \rightarrow S^{3}(-d+1)\rightarrow
S\rightarrow M(f)\rightarrow 0\end{split}$

for some integers $d_{1},d_{2},d_{3},b$ such that $d_{1}+d_{2}=d$,
$d_{2}=d_{3}$, and $b=d_{2}-d+2$. In that case, the pair $(d_{1},d_{2})$ is
called the set of exponents of $C$.

######  Example 2.4.

Let us consider the rational cuspidal curve $C\,:\,y^{2}z=x^{3}$. It is known
that $C$ is a nearly free curve. We can compute the minimal free resolution of
$M(f)$ which has the following form:

$0\rightarrow S(-5)\rightarrow S(-4)^{2}\oplus S(-3)\rightarrow
S(-2)^{3}\rightarrow S\rightarrow M(f)\rightarrow 0.$

It means that the exponents are $(1,2)$.

From now on we stick to line arrangements in $\mathbb{P}^{2}_{\mathbb{C}}$. In
order to study their nearly freeness, we will use [3, Theorem 1.3] which turns
out to be a vital technical tool.

######  Theorem 2.5 (Dimca).

Let $\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be an arrangement of $d$
lines and let $f=0$ be its defining equation. Denote by $r:={\rm mdr}(f)$.
Assume that $r\leqslant d/2$, then $\mathcal{L}$ is nearly free if and only if

$r^{2}-r(d-1)+(d-1)^{2}=\mu(\mathcal{L})+1,$ (1)

where $\mu(\mathcal{L})$ is the total Milnor number of $\mathcal{L}$, i.e.,

$\mu(\mathcal{L})=\sum_{p\in{\rm Sing}(\mathcal{L})}({\rm mult}_{p}-1)^{2}.$

######  Example 2.6.

Let us consider the line arrangement
$\mathcal{A}\subset\mathbb{P}^{2}_{\mathbb{C}}$ defined by the following
equation

$Q(x,y,z)=(x^{2}+xy+y^{2})(y^{3}-z^{3})(z^{3}-x^{3}).$

This arrangement consists of $8$ lines and delivers $4$ nodes and $8$ triple
intersection points. Using `Singular` [2] we can compute ${\rm mdr}(Q)$ which
is equal to $4$. Since $r\leqslant d/2=4$, we can use the above criterion,
namely

$37=4^{2}-4\cdot 7+7^{2}=\mu(\mathcal{A})+1=4+4\cdot 8+1,$

so $\mathcal{A}$ is nearly free.

##  3 Nearly free arrangements of line arrangements with nodes and triple
points

In order to provide a lower bound on the number of lines of nearly free line
arrangements with nodes and triple points, we recall the following result by
Dimca and Pokora [6, Proposition 4.7] which is adjusted to our purposes.

######  Proposition 3.1.

Let $C\,:\,f=0$ be an arrangement of $d$ lines in
$\mathbb{P}^{2}_{\mathbb{C}}$ such that it has only nodes and triple
intersection points. Then one has

${\rm mdr}(f)\geqslant\frac{2}{3}d-2.$

If $C\,:f=0$ is a nearly free arrangement of $d$ lines with nodes and triple
intersection points with the exponents $(d_{1},d_{2})$, $d_{1}\leqslant
d_{2}$, then ${\rm mdr}(f)=d_{1}$, and since

$2d_{1}\leqslant d_{1}+d_{2}=d$

we obtain that ${\rm mdr}(f)\leqslant d/2$. Combining it with the above
proposition, we arrive at

$\frac{2}{3}d-2\leqslant{\rm mdr}(f)\leqslant d/2.$

It gives us the following result.

######  Proposition 3.2.

If $\mathcal{A}\subset\mathbb{P}^{2}_{\mathbb{C}}$ is a nearly free
arrangement of $d$ lines with nodes and triple intersection points, then
$d\leqslant 12$.

Based on the above proposition, our goal is the following.

######  Problem 3.3.

Classify all weak combinatorics of line arrangements with nodes and triple
points in $\mathbb{P}^{2}_{\mathbb{C}}$ which are nearly free.

Here by the weak combinatorics, for an arrangement of lines $\mathcal{L}$, we
mean the vector $(d;t_{2},t_{3})$, where $d$ is the number of lines, $t_{2}$
is the number of nodes, and $t_{3}$ is the number of triple points. Sometimes
we will write $t_{i}(\mathcal{L})$ with $i\in\\{2,3\\}$ in order to emphasize
the underlying arrangement of lines $\mathcal{L}$. The first step towards the
classification is the following proposition.

######  Proposition 3.4.

Let $\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be a nearly free
arrangement of $d$ lines with nodes and triple intersection points. Then

$t_{3}\geqslant\frac{1}{4}\bigg{(}d^{2}-4d-1\bigg{)}.$ (2)

###### Proof.

If $\mathcal{L}$ is nearly free with $r={\rm mdr}(f)$, where
$f\in\mathbb{C}_{d}[x,y,z]$ is the defining equation, then by Theorem 2.5 one
has

$r^{2}-r(d-1)+(d-1)^{2}=\mu(\mathcal{L})+1.$

Let us recall that we have the following combinatorial count

$\binom{d}{2}=\frac{d(d-1)}{2}=t_{2}+3t_{3}.$

Since $\mu(C)=t_{2}+4t_{3}=\binom{d}{2}+t_{3}$, we obtain

$r^{2}-r(d-1)+(d-1)^{2}=r^{2}-r(d-1)+d^{2}-2d+1=\binom{d}{2}+t_{3}+1.$

After simple manipulations, we arrive at

$r^{2}-r(d-1)+\frac{d^{2}-3d-2t_{3}}{2}=0.$

The above equation can have integer roots if
$\triangle_{r}=(d-1)^{2}-2d^{2}+6d+4t_{3}\geqslant 0$. This leads us to

$t_{3}\geqslant\frac{1}{4}\bigg{(}d^{2}-4d-1\bigg{)},$

which complete the proof. ∎

In the next step, let us recall the following bound on the number of triple
points for line arrangements which is due to Schönheim [9]. Define

$U_{3}(d):=\bigg{\lfloor}\bigg{\lfloor}\frac{d-1}{2}\bigg{\rfloor}\cdot\frac{d}{3}\bigg{\rfloor}-\varepsilon(d),$

where $\varepsilon(d)=1$ if $d=5\,{\rm mod}(6)$ and $\varepsilon(d)=0$
otherwise. Then

$t_{3}\leqslant U_{3}(d).$ (3)

If $\mathcal{L}$ is a nearly free arrangement with only nodes and triple
intersection points, then

$\frac{1}{4}\bigg{(}d^{2}-4d-1\bigg{)}\leqslant t_{3}\leqslant U_{3}(d).$ (4)

Observe that for $d\in\\{12,11,10\\}$ the chain of inequalities in (4) leads
us to a contradiction.

######  Corollary 3.5.

Let $\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be a nearly free
arrangement of $d$ lines with only nodes and triple intersection points. Then

$d\leqslant 9.$

In the next chapter, we are going to indicate those values of
$d\in\\{4,5,6,7,8,9\\}$ for which there exists a line arrangement with nodes
and triple points that is nearly free. In order to do so, we are going to use
some tricks regarding deletion and deformation-type arguments.

##  4 Classification

One of the tools that will help us is the following deformation type result
which has a general meaning, and that is the reason why we formulate it for
all reduced plane curves. Before we formulate it, let us present the following
definition.

######  Definition 4.1.

Let $\mathcal{C}\,:f=0$ be a reduced curve in $\mathbb{P}^{2}_{\mathbb{C}}$ of
degree $d$. For $r={\rm mdr}(f)$ we define the number

$\eta(C)=r^{2}-r(d-1)+(d-1)^{2}.$

Moreover, we define the total Tjurina number of $C$ by

$\tau(C):=\sum_{p\in{\rm Sing}(C)}\tau_{p},$

where $\tau_{p}$ denotes the (local) Tjurina number of $C$ at the singular
point $p$.

######  Remark 4.2.

Observe that if $C$ is an arrangement of $d$ lines in
$\mathbb{P}^{2}_{\mathbb{C}}$, then

$\mu(C)=\tau(C),$

and it follows from the fact that all the singular points for line
arrangements are _quasi-homogeneous_.

######  Proposition 4.3.

Let $C:f=0$ be a free reduced plane curve of degree $d$ having only nodes and
$n_{3}\geqslant 1$ ordinary triple points. Let $C^{\prime}$ be a reduced curve
obtained by the following deformation performed on $C$:

$(\star)$: a triple intersection point $p$ is deformed into three nodes.

Denote by $f^{\prime}=0$ the defining equation of $C^{\prime}$. Assume that
$\eta(C)=\eta(C^{\prime})$ and ${\rm mdr}(f^{\prime})\leqslant d/2$, then
$C^{\prime}$ is nearly free.

###### Proof.

If $C$ is free, then by [3, Corollary 1.2] we have

$\eta(C)=r^{2}-r(d-1)+(d-1)^{2}=\tau(C)=n_{2}+4n_{3}.$

Observe that

$\tau(C^{\prime})=(n_{2}+3)+4(n_{3}-1)=n_{2}+4n_{3}-1,$

so $\tau(C^{\prime})+1=\tau(C)$. Since ${\rm mdr}(f^{\prime})\leqslant d/2$,
then in the light of [3, Theorem 1.3] $C^{\prime}$ is nearly free, and it
finishes the proof. ∎

Let us explain this idea in detail by the forthcoming example.

######  Example 4.4.

Consider $A_{1}(6)$ arrangement defined by the equation

$f(x,y,z)=xyz(x-y)\cdot(y-z)\cdot(x-z).$

It is well-known that $t_{2}(\mathcal{A}_{1}(6))=3$,
$t_{3}(\mathcal{A}_{1}(6))=4$, ${\rm mdr}(f)=2$, and
$\eta(\mathcal{A}_{1}(6))=19$. Now we consider the following deformation of
$\mathcal{A}_{1}(6)$, denoted here by $\mathcal{A}_{6}$, given by the
following equation

$f^{\prime}(x,y,z)=xyz(y-z)\cdot(x-z)\cdot\bigg{(}x-\frac{1}{2}y\bigg{)}.$

It has $t_{2}(\mathcal{A}_{6})=6$ and $t_{3}(\mathcal{A}_{6})=3$, ${\rm
mdr}(f^{\prime})=3$, and $\eta(A_{6})=19$.

Observe that $\mathcal{A}_{6}$ is nearly free, namely

$\eta(\mathcal{A}_{6})=19=t_{2}(\mathcal{A}_{6})+4t_{3}(\mathcal{A}_{6})+1=\mu(\mathcal{A}_{6})+1=\mu(\mathcal{A}_{1}(6)),$

which completes our justification.

In this way, we have constructed a nearly free arrangement of $d=6$ lines. Now
we are going to use results from [7] in order to construct examples of nearly
free arrangement of lines using our deformation argument, and at the end the
deletion procedure.

  1. $d=4$:

It is known that the maximal number of triple points for $4$ lines is equal to
$1$. Moreover, this arrangement consisting of $t_{3}=1$, $t_{2}=3$, and $d=4$
is free since it is a supersolvable line arrangement. In this situation, as it
turns out, we can apply our deformation argument above and conclude that an
arrangement of $d=4$ lines and $6$ nodes is nearly free.

  2. $d=5$:

It is also known that the maximal number of triple points for $5$ lines is
equal to $2$. Moreover, an arrangement with $d=5$, $t_{3}=2$, and $t_{2}$ is
supersolvable, and thus it is free arrangement. Again, we can perform our
deformation argument at one triple point, we obtain in that way an arrangement
with $d=5$, $t_{3}=1$, and $t_{2}=7$, and this arrangement is nearly free.

  3. $d=6$:

This case is covered by Example 4.4.

  4. $d=7$:

We know that over the complex numbers the maximal number of triple points for
$7$ lines is equal to $6$. Consider the arrangement

$Q(x,y,z)=z\cdot(x^{2}-z^{2})\cdot(y^{2}-z^{2})\cdot(y^{2}-x^{2}).$

We can check, using `Singular`, that the arrangement $\mathcal{A}$ defined by
$Q$ is free with the exponent $(3,3)$. Now we can deform arrangement
$\mathcal{A}$ at one of the triple intersection points. Consider the following
deformation

$G(x,y,z)=z\cdot(y^{2}-z^{2})\cdot(x^{2}-z^{2})\cdot(y+x)\cdot(2y-x+2z).$

The arrangement $\mathcal{A}^{\prime}$ defined by $G$ delivers $t_{3}=5$ and
$t_{2}=6$. Using Proposition 4.3, we can conclude that indeed
$\mathcal{A}^{\prime}$ is nearly free. We can check this also directly, since
${\rm mdr}(G)=3$, we have

$r^{2}-6\cdot r+36=9-18+36=\mu(\mathcal{A}^{\prime})+1=6+4\cdot 5+1=27.$

  5. $d=8$:

This case is covered by Example 2.6. However, we want to explain a bit how
this arrangements can be derived. Recall that the dual Hesse arrangement
$\mathcal{H}$ of $d=9$ lines and $t_{3}=12$ is given by the following defining
equation.

$Q(x,y,z)=(x^{3}-y^{3})\cdot(y^{3}-z^{3})\cdot(z^{3}-x^{3}).$

Now we are going to remove one line. Since the picture is completely
symmetric, we remove a line $\ell$ given by $x-y=0$. Then

$\widetilde{Q}(x,y,z)=Q(x,y,z)/(x-y)=(x^{2}+xy+y^{2})\cdot(y^{3}-z^{3})\cdot(z^{3}-x^{3}),$

so we arrive at the situation of Example 2.6. The procedure above is called in
the literature as the deletion procedure. It is well-known that the above
arrangement is nothing else than Mac Lane arrangement of $8$ lines, and it
realizes the maximal number of triple points among line arrangements with $8$
lines, according to [7].

The last case boils down to decide whether $d=9$ can occur, and it has a
different flavour comparing with the previous cases. Using our bound (2) we
see that for $d=9$ our nearly free arrangement should have $t_{3}\geqslant 11$
triple points. If $t_{3}=12$, then this is the dual Hesse arrangement which is
unique up to a projective equivalence, and the dual Hesse arrangement is free.
Due to this reason, our problem reduces to decide whether there exists and
arrangement of $d=9$ lines with $t_{3}=11$ and $t_{2}=3$. In order to approach
this problem, we use database described in [8]. It turns out that there exists
the only one matorid of $d=9$ lines and $11$ triple points. However, this
matroid cannot be realized as a line arrangement over any field due to method
explained in [1, Section 4] \- in fact in this case there is failure of Dress-
Wentzl’s valuation criterion. This observation finishes our classification
procedure. Based on that, we are ready to formulate our main result.

######  Theorem 4.5.

Let $\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be an arrangement of $d$
lines with nodes and triple intersection points that is nearly free. Then
$d\in\\{4,5,6,7,8\\}$.

Now our aim is to find all weak combinatorics $\mathcal{C}=(d;t_{2},t_{3})$
with $d\in\\{4,5,6,7,8\\}$ such that $\mathcal{C}$ can be geometrically
realized over the complex numbers as an arrangement of $d$ lines with
prescribed $t_{2}$ and $t_{3}$ such that $\mathcal{C}$ is nearly free. We can
use a classification result that comes from [8] and Proposition 3.4.

  1. $d=4$:

We should have $t_{3}\geqslant 0$, and the only possibility to a have nearly
free and not free arrangement is $(4;6,0)$.

  2. $d=5$:

We should have $t_{3}\geqslant 1$, and the only possibility to have a nearly
free and not free arrangement is $(5;7,1)$.

  3. $d=6$:

We should have $t_{3}\geqslant 3$, and the only possibility to have a nearly
free and not free arrangement is $(6;6,3)$.

  4. $d=7$:

We should have $t_{3}\geqslant 5$, and the only possibility to have a nearly
free and not free arrangement is $(7;6,5)$.

  5. $d=8$:

We should have $t_{3}\geqslant 7$ and it turns out that we have two
possibilities, namely $(8;4,8)$ or $(8;7,7)$. We need to decide whether an
arrangement with the weak combinatorics $\mathcal{C}=(8;7,7)$ is nearly free –
of course such an arrangement can be constructed over the real numbers. If
such an arrangement would be nearly free, then by Theorem 2.5 the following
polynomial must have integer roots

$r^{2}-7r+49=r^{2}-r(d-1)+(d-1)^{2}=\mu(\mathcal{C})+1=t_{2}+4t_{3}+1=7+4\cdot
7+1=36.$

Of course equation $r^{2}-7r+13=0$ does not have integer roots, so the weak
combinatorics $(8;7,7)$ does not rise to a nearly free arrangement.

Based on the above considerations, there are exactly $5$ weak combinatorics
$(d;t_{2},t_{3})$ leading to nearly free arrangements of $d$ lines with
$t_{2}$ double and $t_{3}$ triple intersection points.

## Acknowledgments

The author would like to thank Piotr Pokora for his guidance. We want to thank
Lukas Kühne for his explanations regarding the non-existence of an arrangement
with $d=9$ lines and $11$ triple points, and to Alexandru Dimca for helpful
suggestions that allowed to improve the note.

The author was partially supported by the National Science Center (Poland)
Preludium Grant Nr UMO 2018/31/N/ST1/02101.

## References

  * [1] M. Barakat, R. Behrends, Ch. Jefferson, L. Kühne, M. Leuner, On the generation of rank 3 simple matroids with an application to Terao’s freeness conjecture. SIAM J. Discrete Math. 35(2): 1201 – 1223 (2021).
  * [2] W. Decker, G.-M. Greuel, G. Pfister, and H. Schönemann,  Singular 4-1-1 — A computer algebra system for polynomial computations.  http://www.singular.uni-kl.de, 2018.
  * [3] A. Dimca, Freeness versus Maximal Global Tjurina Number for Plane Curves. Math. Proc. Camb. Philos. Soc. 163(1): 161 –172 (2017).
  * [4] A. Dimca, G. Sticlaru, Free divisors and rational cuspidal plane curves. Math. Res. Lett. 24(4): 1023 – 1042 (2017).
  * [5] A. Dimca and G. Sticlaru, Free and Nearly Free Curves vs. Rational Cuspidal Plane Curves. Publ. Res. Inst. Math. Sci. 54(1): 163 – 179 (2018).
  * [6] A. Dimca and P. Pokora, On conic-line arrangements with nodes, tacnodes, and ordinary triple points. J. Algebraic Combin., arXiv:2111.12349.
  * [7] M. Dumnicki, Ł. Farnik, A. Główka, M. Lampa-Baczyńska, G. Malara, T. Szemberg, J. Szpond, H. Tutaj-Gasińska, Line arrangements with the maximal number of triple points. Geom. Dedicata 180: 69 – 83 (2016).
  * [8] Y. Matsumoto, S. Moriyama, H. Imai, D. Bremner, Matroid enumeration for incidence geometry. Discrete Comput. Geom. 47(1): 17 – 43 (2012).
  * [9] J. Schönheim, On Maximal Systems of $k$-Tuples, Stud. Sci. Math. Hungar. 1: 363 – 368 (1966).

Jakub Kabat
Department of Mathematics, Pedagogical University of Krakow, ul. Podchorazych
2, PL-30-084 Kraków, Poland.
E-mail address<EMAIL_ADDRESS>