# Dendrogram distance: an evaluation metric for generative networks using
hierarchical clustering

Gustavo Sutter P. Carvalho, Moacir A. Ponti

###### Abstract

We present a novel metric for generative modeling evaluation, focusing
primarily on generative networks. The method uses dendrograms to represent
real and fake data, allowing for the divergence between training and generated
samples to be computed. This metric focus on mode collapse, targeting
generators that are not able to capture all modes in the training set. To
evaluate the proposed method it is introduced a validation scheme based on
sampling from real datasets, therefore the metric is evaluated in a controlled
environment and proves to be competitive with other state-of-the-art
approaches.

##  1 Introduction

Generative modeling is a task that aims to estimate the generation process of
a given source dataset. Models obtained as a result of this approach can be
used to sample novel data points that follow the distribution of the source
training set, allowing for different applications in machine learning.
Performing generative modeling using neural networks has become very popular
mainly because of the success of Generative Adversarial Networks (GANs)
(Goodfellow et al., 2014) and later with Diffusion models (Luo, 2022). The GAN
framework relies on two different networks, a generator and a discriminator,
that compete against their selves to perform the generative task, as shown in
Figure 1.

Figure 1: Diagram that illustrates the different components of GAN. The
generator network $G$ transforms a random input $z$ into samples that should
be realistic, while the discriminator network $D$ tells apart which samples
came from the training data.

More specific, a GAN is composed by a generator network $G$ and a
discriminator network $D$. The goal of the generator is to produce realistic
samples, while the discriminator acts as a binary classifier telling apart
samples from the training set and created by the generator. That is to say
that $G$ receives a random vector $z$ as input and tries to approximate the
real data distribution $p_{data}$ using a distribution $p_{model}$. The whole
system is trained together, optimizing the following mini-max objective:

$\min_{G}\max_{D}L(G,D)=\mathbb{E}_{x\sim p_{data}}[\log
D(x))]+\mathbb{E}_{\tilde{x}\sim p_{model}}[\log(1-D(G(\tilde{x}))],$ (1)

where $x\in X$ is sampled from a training set $X$ containing $n$ examples,
while $\tilde{x}$ comes from the generator.

The field is prolific, which can be seen by an exponential number of published
papers in the last years (Faria and Carneiro, 2020). Its approach, which uses
two neural networks trained against each other, demonstrated to produce great
results even in hard high dimensional settings. Because of that, many other
methods built on top of the traditional adversarial setup were developed
improving its results and contribution to its use.

Even though GANs have achieved success in different applications, such as
image super-resolution (Ledig et al., 2017), style transfer (Zhu et al.,
2017), image synthesis (Wang et al., 2018), sketch synthesis (Sampaio Ferraz
Ribeiro et al., 2022) and tracking (Comes et al., 2020), the difficulty of
training and evaluating a generative model remains its main limitations (Ponti
et al., 2021). With respect to training, instability of the generator results
and the high sensitivity to network architecture and hyper-parameters are
among the relevant issues. Even when the model converges, there are notably
two possible undesired effects: low sample quality and mode collapse (Adiga et
al., 2018). Keeping the problem of generating images in mind, low sample
quality happens when the generated images are blurry, noisy or with
undesirable visual artifacts. Mode collapse, on the other hand, has to do with
how diverse the generated data is. When the data produced by the generative
model fails to cover the modes of the training data it said that it suffers
from mode collapse.

The substantial progress of applications and generative approaches are not
equally followed by efforts on evaluating them (Borji, 2019, Ribeiro et al.,
2020). However, even those effects are not easy to be measured. In contrast
with supervised learning methods, such as classification, developing metrics
for generative modeling is not as straightforward because of the ill-posed
nature of the task. More specifically, as the task is to produce samples that
follow the training data distribution, there is no direct and trivial way to
determine how good a sampled instance is (Parimala and Channappayya, 2019)
(Shmelkov et al., 2018). This issue is amplified when dealing with high
dimensional data as images.

Several metrics have been proposed on the last years, notably the Inception
Score (Salimans et al., 2016) (Gurumurthy et al., 2017) and the Frechet
Inception Distance (Heusel et al., 2017), which are widely used and currently
the de-facto evaluation references. Both metrics make use of a pre-trained
convolutional network to extract a latent representation in order to deal with
high-dimensional image data. Some problems regarding this metrics are well
known in the literature (Barratt and Sharma, 2018) (Xu et al., 2018) (Borji,
2019) as we present evidence that both can fail to detected mode collapse on
different datasets.

We propose the Dendrogram Distance (DD), a novel evaluation metric for
generative models which relies on hierarchical clustering to evaluate the
generated data. Using results from clustering theory, we demonstrate the
equivalence between dendrograms and ultrametric spaces to develop a metric
that has theoretical foundations (Carlsson and MÃŠmoli, 2010) and that is
sensitive to variations in the generated data, in particular to detect
generated modes. Our method captures the ordered hierarchical clustering via
dendrograms and is guaranteed to detect differences when they exist. This
guarantee is not present in previous evaluation methods, therefore it
represents a relevant contribution.

Our experiments demonstrate that Dendrogram Distance is capable of detecting
different number of modes on generated data. In particular, it was shown to be
sensible to nuances of the space for which the competing methods fail to
capture, such as when the modes’ means are not equally spaced. Therefore, this
makes DD a superior choice to 2D benchmark, but also useful to evaluate
directly the generated data, such as image pixels or image embeddings, as well
as capturing convergence of models.

##  2 Related work

Generative deep learning has became very popular since the introduction of
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014). This
framework, composed of two neural networks that compete against each other,
rapidly became the state-of-the-art in various tasks over the past years,
being applied to a variety of tasks. However, the evaluation of generative
models is a challenging task since there is no explicit label or category to
compare with, such as in supervised learning. Grasping the quality and
diversity of generated data is often performed visually, in particular when it
comes to early studies on deep generative methods (Radford et al., 2016).

Evaluating generative methods is a challenge and, as of yet, there is no
consensus on the best way to to it (Borji, 2019). Notably the most widely used
evaluation method for image generation is the Inception Score (IS) (Salimans
et al., 2016), which uses a pre-trained Inception-V3 (Szegedy et al., 2016) to
verify if the samples produced by the generative model present high quality
and diversity. To this end the IS uses the class distribution $p(y|x)$
produced by the network when passing through it the generated samples. The
main point is that high quality samples must imply that images should be
easily identifiable, therefore $p(y|x)$ must have low entropy, and given a set
a images its marginal distribution $p(y)$ must have high entropy, meaning that
there must diversity. Some variations on the IS were proposed but are not as
widely used, for example: the Mode Score (Che et al., 2016), AM Score (Zhou et
al., 2017) and Modified Inception Score (Gurumurthy et al., 2017).

Another metric popularly used for evaluating generative modeling on images is
the Fréchet Inception Distance (FID) (Heusel et al., 2017). Simliarly to the
the Inception Score the FID also makes use of a pre-trained Inception-V3 as a
feature extractor, but it used the activations from the last convolutional
layer to represent the data points. It assumes that these representations, for
the training and generated sets, follow a multivariate normal and compute the
Fréchet Distance between the two. That is done by simple estimating their mean
vector and covariance matrices. Thus, FID operates on the feature spaces and
therefore can be applied to non-image data.

One important drawback of IS and FID scores is that they make more sense in
large scale natural scene datasets, while for other tasks one often has to
design specific evaluation protocols (Yang and Lerch, 2020). Both the IS and
FID can be categorized as extrinsic evaluation metrics, given that they
measure the performance of the generator based on a downstream task, namely
how good are the representations produced by a network pre-trained with data
from the same domain. Other approaches for image data rely on image quality
metrics such as MS-SSIM (Odena et al., 2017), that uses structural-similarity
for image pixels. On the other hand there is a set of metrics that do not rely
on external models or data, which are referred intrinsic evaluation metrics
(Theis et al., 2016).

A standard approach to perform intrinsic evaluation of a generative model is
to estimate its density, via Kernel Density Estimation, and compute classical
metrics such as the Kullback–Leibler or Jensen–Shannon divergences. However,
this method requires too much data when applied to high-dimensional spaces and
relies on the Euclidean distance between data points due the kernel
probability, which can be harmful when dealing with images (Theis et al.,
2016).

All the aforementioned metrics result in a number that allow to compare
different generative results, however, specific characteristics such as mode
collapse and sample quality are not explicitly addressed. Therefore, 2D
benchmarks became a common approach, i.e. the use of synthetic datasets that
allow for rapid investigation of the results. The 2D Ring and the 2D Grid
datasets can be used to visualize such aspects (Srivastava et al., 2017) and
makes it particularly easier verify how well a generative model does in terms
of mode collapse. It provides useful feedback when iterating over ideas and a
controlled and explainable benchmark. Although it may seen to be an easy task
because of its low dimensionality, 2D datasets represent a great challenge to
generative models such as GANs. Those are a way to clearly visualize how the
model covers the target modes.

##  3 Method

We propose an evaluation method based on the divergence between dendrograms.
Our approach follows from the natural assumption that if the generated data is
similar to the real data, the clustering of their samples must also be
similar. The hypothesis is that a dendrogram allows the metric to better
capture the distributions than the first and second moments.

###  3.1 Hierarchical Clustering and Dendrograms

###### Definition 1

If $X$ is a finite set, $\mathcal{C}(X)$ is the collection of all non-empty
subsets of $X$

###### Definition 2

If $X$ is a finite set, $\mathcal{P}(X)$ is the collection of all partitions
of $X$

A dendrogram over a dataset $X$ is a representation of the hierarchical
clustering of the data, often represented as a tree. Formally, dendrograms are
described as a pair $(X,\theta)$, where $X$ is the dataset and
$\theta:[0,\infty]\rightarrow\mathcal{P}(X)$ is a function that maps every
distance to the set of clusters at that point, that holds the following
properties (Costa, 2017):

  1. 1.

$\theta(0)=\left\\{\left\\{x_{0}\right\\},\left\\{x_{1}\right\\},\ldots\left\\{x_{n}\right\\}\right\\}$:
the initial level of the dendrogram is the most fine grained representation as
possible;

  2. 2.

There is tome $t_{0}$ such that $\theta(t)$ is a unique block for every $t\geq
t_{0}$, so that for every value $t>t_{0}$ dendrograms will be the same and
contain all elements, i.e. $\theta(t)=\left\\{\left\\{x_{0},x_{1},\ldots
x_{n}\right\\}\right\\}$;

  3. 3.

If $r\leq s$ then $\theta(s)$ is a more fine grained dendrogram than
$\theta(r)$, which ensures a sense of scale for $\theta(.)$ and suggests
nested partitions;

  4. 4.

For all $r$ there is an $\varepsilon>0$ such that $\theta(r)=\theta(t)$ for
$t\in[r,r+\varepsilon]$.

The agglomerative distances $d$ of a dendrogram are the points where two
branches merge, as illustrated in Figure 2.

Figure 2: A dendrogram built over the set $X=\\{x_{1},x_{2},x_{3},x_{4}\\}$.
In this example the agglomerative distances are $d=[2,4,6]$.

###  3.2 Distance between dendrograms

The equivalence between dendrograms and ultrametric spaces was demonstrated in
Carlsson and MÃŠmoli (2010). Ultrametric space satisfies a stronger type of
the triangular inequality. It uses the fact that, for every $r\geq 0$ there is
a block $\mathcal{B}$ of dendrogram $\theta(r)$ in which elements $x_{i}$ and
$x_{j}$ are agglomerated, if and only if there is an ultrametric space in
which the distance between those elements is smaller than or equal to $r$.
Therefore there is a bijective function
$\Psi:\mathcal{D}(X)\rightarrow\mathcal{U}(X)$ mapping the collection
$\mathcal{D}(X)$ of all produced dendrograms of $X$ into the collection
$\mathcal{U}(X)$ of all ultrametric spaces of $X$.

Given a finite set $X$, for each produced dendrogram $\theta\in\mathcal{D}(X)$
there is a corresponding ultrametric space. The conversion between a
dendrogram $(X,\theta)$ and a ultrametric space $(X,u)$, with a ultrametric
distance $u$, is (Carlsson and MÃŠmoli, 2010):

$u(x_{i},x_{j})=\min\\{r\geq 0|x_{i}\textrm{ and }x_{j}\textrm{belong to the
same cluster if }\theta(r)\\}.$ (2)

That is to say that the distance between two points in the ultrametric space
is the point in the dendrogram where the branches coming from the two points
meet.

The equivalence in Equation 2 allows us to use methods from the metric space
literature to analyze and better understand dendrograms for machine learning
applications. For example, a method proposed by Costa (2017) uses the
divergence between dendrograms in order to detect concept drift on data
streams. That is done by computing a approximation of the Gramov-Hausdorff
divergence between the equivalents ultrametric spaces. Considering two
dendrograms $(X_{\alpha},\theta_{\alpha})$ and $(X_{\beta},\theta_{\beta})$
with sorted agglomerative distances $d^{\alpha}$ and $d^{\beta}$, the distance
between the dendrograms is:

$\textrm{distance}(\theta_{\alpha},\theta_{\beta})=\max_{i}|d_{i}^{\alpha}-d_{i}^{\beta}|$
(3)

Notice that we are using the sorted agglomerative distances as a way of
aligning the clusters in the two datasets. It was proposed by Costa (2017) and
serves as a lower bound to the real distance between the dendrograms. For this
reason we do not use an explicit assignment algorithm.

Figure 3: Step-by-step representation of our method. (I) The set of generated
images is produced by the generator. (II) A dendrogram is built using each
set. (III) The metric is computed as the divergence between the dendrograms.

###  3.3 Dendrogram Distance for Evaluation of Generative Models

The method, illustrated in Figure 3, works by computing the distance between
two dendrograms, one of them constructed using samples from the real
(training) data and the other from samples produced by the generator.
Formally, our metric takes a set of real data $X_{data}$ and a set of
generated data $X_{model}$, using them to construct their correspondent
dendrograms $(X_{real},\theta_{data})$ and $(X_{model},\theta_{model})$ using
a single-linkage clustering algorithm. The real and generated data dendrograms
have agglomerative distances $d^{data}$ and $d^{model}$, respectively.
Therefore, the divergence between them is computed by the following equation,
which comes from Equation 3 but uses the average instead of the maximum value:

$DD(X_{data},X_{model})=\frac{1}{n}\sum_{i=1}^{n}|d_{i}^{data}-d_{i}^{model}|,$
(4)

the use of average representing a relaxation of the original formulation that
allows to smooth out the results.

####  3.3.1 2D Benchmarks

Our main contribution is a metric to verify GANs abilities to deal with mode
collapse in a fast and easy to interpret manner using 2D datasets. Such 2D
datasets are widely used to visualize the mode coverage of generative models
(Borji, 2019). This evaluation procedure as it is, however, relies heavily on
human interpretation. Also, the data distributions are often simplistic. We
propose an extension to this method, by adding more nuance to the datasets and
a novel reliable quantitative measure.

Figure 4: The two bidimensional datasets: 2D Grid and 2D Ring. They are
specified by determining the mean of each mode, which are normal
distributions.

The main idea behind using synthetic 2D datasets is to easily iterate and
verify the results of a model in terms of mode coverage. Two datasets are used
for this purpose: the 2D Ring and 2D Grid, demonstrated in Figure 4. However,
both the ring and the grid, as often employed to evaluate generative models,
assume that distances between different pairs of modes are the same, which may
not reflect real scenarios, such as unstructured data embeddings.

Note that the Fréchet Inception Distance (FID) between the real and generated
samples makes use of mean and variance of each distribution, which we claim
may not be able to properly detect the presence of mode collapse in datasets
where the position of each mode is random. Our Dendrogram Distance (DD) can be
used as an additional quantitative analysis of models performance on the
aforementioned datasets. That is a better alternative because the DD is
agnostic to the position of each mode; it only cares about the relative
distance between samples. For the same reason it has a lower variance in such
cases, as we further demonstrate in our experiments.

####  3.3.2 Generated Image Embeddings

Beyond its applicability on 2D datasets, the Dendrogram Distance can also be
applied to other types of data, such as images. However, applying it directly
to the pixel space can unfeasible due to high dimensionality. Therefore,
inspired in the Inception Score (IS), which uses a pre-trained network to
obtain the class distributions, we first extract a lower dimensional
representation of the data points also using a pre-trained network.

In fact, both IS and FID uses this approach, which allows for the metric to
invariant to certain changes in the images while preserving more information
than the class distribution outputs. The architecture used as a feature
extractor in this paper is a Inception-V3 pre-trained on ImageNet, in order to
make our results comparable to the IS and the FID.

In summary, the Dendrogram Distance applied to unstructured data involves
extracting representations for the training and generated sets, building the
correspondent dendrograms with such representations and computing the
divergence between the two dendrograms to get the final score. Next, we
demostrate that the Dendrogram Distance produces competitive results and
additional insights on image data when compared to FID and IS.

##  4 Experiments and Results

This paper includes experiments with the Dendrogram Distance (DD) in three
main parts: evaluation with 2D benchmarks, with image pixels and with image
embeddings.

###  4.1 2D benchmark for generative models

In this part, we employ the 2D Ring and the 2D Grid datasets. We also
generated data in which the location, i.e. the mean, of each mode is
perturbed, so that the relative distances between modes is not fixed.

In order to evaluate how well the metrics captured the issues in the generated
distributions, we simulated generated sets with increasingly different number
of modes, in order to check if the metrics reflect the changes. More
concretely, the generated fake datasets that contained from 1 up to the number
of modes in the true dataset. The datasets are described in Table 1.

Dataset | Target # of Modes | Radius | Grid length
---|---|---|---
2D Grid | 9 | – | 100
2D Ring | 7 | 50 | –
Table 1: Description of the 2D datasets

Our first experiment consisted of adding different levels of noise to the
position of each mode and checking if the metrics were still able to correctly
quantify the number of modes in the datasets. In practice, for each mode we
generated a displacement value sampled from $\textrm{Unif}(-\alpha L,\alpha
L)$, where $\alpha$ is the noise scale that we varied and $L$ is the length of
the grid and the diameter of the ring.

In Figure 5 we show a simulation with 2D Grid in which we add modes
iteratively and compute both FID and DD. In a noise-free scenario, both DD and
FID captures well the number of modes, but, as the mean of each mode is
perturbated, the FID suffers from instability, while the DD is more stable and
smooth, therefore demonstrating to be more robust. Similarly, in Figure 6 the
simulation with 2D Ring is shown, with similar outcomes: DD grows more
smoothly as the number of modes increase.

Figure 5: Results of FID and DD on 2D Grid. The first row shows the dataset;
the second and third row shows, respectively, the FID and DD results when
different number of modes of the dataset are generated. For the perturbed
datasets we also show in red the curve relative to the noise-free scenario.
Figure 6: Results of FID and DD on 2D Ring. The first row shows the dataset;
the second and third row shows, respectively, the FID and DD results when
different number of modes of the dataset are generated. For the perturbed
datasets we also show in red the curve relative to the noise-free scenario

We also performed experiments to check if the Dendrogram Distance has a lower
variance when compared to the FID. For that we continue to use the strategy
with different noise factors, but we executed the experiment ten times for
each factor and mode. By doing that we are able to verify if the metrics
assign different scores for the same number of modes if their positions
change.

Figure 7: 2D Grid stability analysis for different perturbations of the modes.
Figure 8: 2D Ring stability analysis for different perturbations of the modes

As shown in Figure 7 for the 2D Grid and Figure 8 for the 2D Ring, in the
noise-free scenario the DD still captures better the presence of new modes,
with both DD and FID showing small standard deviations. As the noise
increases, the FID becomes unstable with respect to the value of the metric in
particular when more modes are present, while the DD metric is less affected
by the number of modes.

Those results indicate the advantages of the Dendrogram Distance for capturing
mode collapse in 2D datasets. It presents lower standard deviation as the
number of modes dropped get higher and the randomness in the position of the
modes increase, both cases in which a good evaluation becomes even more
important. Conversely, that is not the case for the Frechet Inception
Distance. The results, overall also demonstrate that the Dendrogram Distance
is more sensitive with respect to the number of modes than the FID. It also
shows that, if the representation of the data is such that the modes are well
defined in the features space, then the DD suits very well the evaluation
task.

###  4.2 Image Pixels

This part investigates the use of DD directly in image pixels. We used MNIST
dataset due to its relatively lower dimensionality and discriminative
properties of its pixel space.

####  4.2.1 Mode collapse detection

The intuition for using Dendrogram Distance is to better capture the
underlying distribution of the data, therefore making it suitable to detect
mode collapse. In order to assess this ability on image datasets, we first
create a proof-of-concept by sampling from a real dataset to simulate training
and generated data points.

More precisely, we start by sampling the training set which contains all the
modes. Then, using the same source dataset, we sample an increasing number of
modes, by taking into account the labels of the images. For this reason we use
a classification dataset. We drop one mode at a time, going from ten to one
modes in each simulated generated set. This procedure is performed ten times
to better understand the method’s variance.

Figure 9: Behavior of the Dendrogram Distance as the number of modes in the
data changes. It is possible to notice that as the number of modes gets lower
the scores became worse.

The results, presented in Figure 9, demonstrate that the Dendrogram Distance
is able to correctly detected mode collapse using MNIST’s pixel space. The
presented curve shows that the metric is able to distinguish between different
levels of mode collapse properly. It is important to point out that the metric
does have a high standard deviation, which is a consequence of working
directly with pixels.

####  4.2.2 Metric as convergence proxy

We also examine if the Dendrogram Distance is able to detect the convergence
of a generative model during training. That is relevant given that GANs are
optimizing two objectives at the same time and to analyze convergence is not
as easy as in other learning tasks.

To this end, we trained a WGAN (Arjovsky and Bottou, 2017) on MNIST for 7000
iterations using RMSprop with learning rate $10^{-5}$. During training we kept
track of its generated samples, computing the DD during training to see how it
relates with overall sample quality. Our results are shown in Figure 10.

Figure 10: Dendrogram Distance changes during training and a corresponding
sample at each iteration.

These results demonstrate that the Dendrogram Distance is able to indicate the
generator convergence. The metric correctly follows the quality of the
generated samples. Some fluctuations in the metric are a result of the
instability during the training process of GANs, due to the two adversarial
objectives.

###  4.3 Generated Image Embeddings

Finally, we studied the use of image embeddings, obtained from pre-trained
CNNs, to evaluate generated data, making it viable to assess datasets with
higher dimensionality. In particular, we focus on detection of mode collapse,
as it is the main focus of the proposed metric.

We adopt the same sampling strategy used for MNIST on three image
classification datasets: CIFAR-10, STL-10 and CelebA. On the first two each
class was said to be a different mode of the data distributing, therefore on
both datasets ten modes are considered. For the simulated generated sets we
drop the number of modes from ten to one. On CelebA there is no explicit class
label for the images, thus we constructed four sets using the attributes:
dark-haired men, blonde men, dark-haired women and blonde women. We drop one
mode at a time, going from four to one in the order just presented.

We compare the Dendrogram Distance (higher value is better) with the Fréchet
Inception Distace (higher value is better) and the Inception Score (lower
value is better). As mentioned before we used a Inception-V3 as a feature
extractor, resizing all images to $299\times 299$ before passing them through
the network. The Dendrogram Distance is using the output of the final
convolutional layer of the network to represent the data points.

The results for CIFAR-10 can be seen in Figure 11. The DD was capable of
capturing the mode collapse, outputting high values for 1, 2 and 3 modes, and
increasingly lower values when more modes were present. The FID was more
sensitive on higher number of modes, while the IS failed to detect new modes
up until 7 modes appeared in the generated dataset.

When analyzing the results on STL-10 in Figure 12 is possible to see that the
DD had more difficult to distinguish between a higher number of modes when
compared to the FID. On this dataset the IS presented a better performance,
being able to distinguish between the different number of modes.

All of the metrics were able to perform well on CelebA, as can be seen in
Figure 13. The results do not differ by much in terms of mode collapse
detection. We hypothesize that it happens because the different between the
modes are presented very explicitly in the image, given that relevant features
for, like hair color and length, represent a large portion of the images.

The overall result of the Dendrogram Distance demonstrates that despite not
being developed with image evaluation in mind, the metric is capable of
dealing with such type of data. It is worth mentioning that the DD depends on
the formation of clusters, so we believe that, with better feature spaces, we
can achieve even better results.

Figure 11: Results on CIFAR-10 Figure 12: Results on STL-10 Figure 13: Results
on CelebA

##  5 Conclusions

The Dendrogram Distance has demonstrated to be a stable approach for
quantifying mode collapse on synthetic bi-dimensional datatets. The use of
dendrograms makes the metric agnostic to the position of the modes, allowing
it to deal with more complex distributions where the distance between groups
of data points may vary. By providing a more reliable quantitative measure for
the 2D Ring and 2D Grid the Dendrogram Distance can be established as a useful
benchmark when developing novel generative models. Also, the Dendrogram
Distance is capable of dealing with image data, both in pixel and feature
spaces. The quality of the results does confirm the capabilities of the
method. The DD allows for easy verification of the results while offering
interpretability and theoretical foundations linking clusters and ultrametric
spaces.

Considering the performance on the bi-dimensional regime, it indicates that
modification in the metric pipeline might lead to better results when dealing
with images. For instance, improving the feature extraction to make it work
better for the clustering task should yield better results. The results also
suggest that is worth investigating the use of the Dendrogram Distance as an
additional training objective, instead of being used just as an evaluation
metric. This and other improvements, such as using DD as a general metric for
comparison between datasets, are beyond the scope of this paper, but can be a
matter of investigation in future work.

## Acknowledgements

This work is supported by FAPESP grants #2018/22482-0 and #2019/07316-0; CNPq
fellowship #304266/2020-5.

## References

  * Adiga et al. (2018) S. Adiga, M. A. Attia, W.-T. Chang, and R. Tandon.  On the tradeoff between mode collapse and sample quality in generative adversarial networks.  In _2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP)_ , pages 1184–1188. IEEE, 2018.
  * Arjovsky and Bottou (2017) M. Arjovsky and L. Bottou.  Towards principled methods for training generative adversarial networks.  _Proceedings of the International Conference on Learning Representations_ , 2017.
  * Barratt and Sharma (2018) S. Barratt and R. Sharma.  A note on the inception score.  _arXiv preprint arXiv:1801.01973_ , 2018.
  * Borji (2019) A. Borji.  Pros and cons of gan evaluation measures.  _Computer Vision and Image Understanding_ , 179:41–65, 2019\.
  * Carlsson and MÃŠmoli (2010) G. Carlsson and F. MÃŠmoli.  Characterization, stability and convergence of hierarchical clustering methods.  _Journal of machine learning research_ , 11(Apr):1425–1470, 2010.
  * Che et al. (2016) T. Che, Y. Li, A. P. Jacob, Y. Bengio, and W. Li.  Mode regularized generative adversarial networks.  _arXiv preprint arXiv:1612.02136_ , 2016.
  * Comes et al. (2020) M. Comes, J. Filippi, A. Mencattini, P. Casti, G. Cerrato, A. Sauvat, E. Vacchelli, A. De Ninno, D. Di Giuseppe, M. D’Orazio, et al.  Multi-scale generative adversarial network for improved evaluation of cell–cell interactions observed in organ-on-chip experiments.  _Neural Computing and Applications_ , pages 1–19, 2020.
  * Costa (2017) F. G. d. Costa.  _Employing nonlinear time series analysis tools with stable clustering algorithms for detecting concept drift on data streams_.  PhD thesis, Universidade de São Paulo, 2017.
  * Faria and Carneiro (2020) F. A. Faria and G. Carneiro.  Why are generative adversarial networks so fascinating and annoying?  In _2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_ , pages 1–8. IEEE, 2020.
  * Goodfellow et al. (2014) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , pages 2672–2680, 2014.
  * Gurumurthy et al. (2017) S. Gurumurthy, R. Kiran Sarvadevabhatla, and R. Venkatesh Babu.  Deligan: Generative adversarial networks for diverse and limited data.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 166–174, 2017.
  * Heusel et al. (2017) M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _arXiv preprint arXiv:1706.08500_ , 2017.
  * Ledig et al. (2017) C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 105–114, 2017.
  * Luo (2022) C. Luo.  Understanding diffusion models: A unified perspective.  _arXiv preprint arXiv:2208.11970_ , 2022.
  * Odena et al. (2017) A. Odena, C. Olah, and J. Shlens.  Conditional image synthesis with auxiliary classifier gans.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 2642–2651, 2017.
  * Parimala and Channappayya (2019) K. Parimala and S. Channappayya.  Quality aware generative adversarial networks.  In _Advances in Neural Information Processing Systems_ , pages 2948–2958, 2019.
  * Ponti et al. (2021) M. A. Ponti, F. P. dos Santos, L. S. Ribeiro, and G. B. Cavallari.  Training deep networks from zero to hero: avoiding pitfalls and going beyond.  In _2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_ , pages 9–16. IEEE, 2021.
  * Radford et al. (2016) A. Radford, L. Metz, and S. Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In _Proceedings of the International Conference on Learning Representations_ , 2016.
  * Ribeiro et al. (2020) L. S. F. Ribeiro, T. Bui, J. Collomosse, and M. Ponti.  Sketchformer: Transformer-based representation for sketched structure.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14153–14162, 2020.
  * Salimans et al. (2016) T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen.  Improved techniques for training gans.  In _Advances in neural information processing systems_ , pages 2234–2242, 2016.
  * Sampaio Ferraz Ribeiro et al. (2022) L. Sampaio Ferraz Ribeiro, T. Bui, J. Collomosse, and M. Ponti.  Scene designer: compositional sketch-based image retrieval with contrastive learning and an auxiliary synthesis task.  _Multimedia Tools and Applications_ , pages 1–23, 2022.
  * Shmelkov et al. (2018) K. Shmelkov, C. Schmid, and K. Alahari.  How good is my gan?  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 213–229, 2018.
  * Srivastava et al. (2017) A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton.  Veegan: Reducing mode collapse in gans using implicit variational learning.  In _Advances in Neural Information Processing Systems_ , pages 3308–3318, 2017.
  * Szegedy et al. (2016) C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2818–2826, 2016.
  * Theis et al. (2016) L. Theis, A. van den Oord, and M. Bethge.  A note on the evaluation of generative models.  In _International Conference on Learning Representations (ICLR 2016)_ , pages 1–10, 2016.
  * Wang et al. (2018) T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * Xu et al. (2018) Q. Xu, G. Huang, Y. Yuan, C. Guo, Y. Sun, F. Wu, and K. Weinberger.  An empirical study on evaluation metrics of generative adversarial networks.  _arXiv preprint arXiv:1806.07755_ , 2018.
  * Yang and Lerch (2020) L.-C. Yang and A. Lerch.  On the evaluation of generative models in music.  _Neural Computing and Applications_ , 32(9):4773–4784, 2020.
  * Zhou et al. (2017) Z. Zhou, H. Cai, S. Rong, Y. Song, K. Ren, W. Zhang, Y. Yu, and J. Wang.  Activation maximization generative adversarial nets.  _arXiv preprint arXiv:1703.02000_ , 2017.
  * Zhu et al. (2017) J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.


# Empirical evaluation of full-reference image quality metrics on MDID
database

Domonkos Varga
Department of Networked Systems and Services
Budapest University of Technology and Economics

###### Abstract

In this study, our goal is to give a comprehensive evaluation of 32 state-of-
the-art FR-IQA metrics using the recently published MDID. This database
contains distorted images derived from a set of reference, pristine images
using random types and levels of distortions. Specifically, Gaussian noise,
Gaussian blur, contrast change, JPEG noise, and JPEG2000 noise were
considered.

_K_ eywords Full-reference image quality assessment

##  1 Introduction

The goal of objective image quality assessment is to design mathematical
models that are able to predict the perceptual quality of digital images. The
classification of objective image quality assessment algorithms is based on
the accessibility of the reference image. In the case of reference image is
unavailable image quality assessment is considered as a no-reference (NR) one.
Reduced-reference (RR) methods have only partial information about the
reference image, while full-reference (FR) algorithms have full access to the
reference image.

The research of objective image quality assessment demands databases that
contain images with the corresponding MOS values. To this end, a number of
image quality databases have been made publicly available. Roughly speaking,
these databases can be categorized into three groups. The first one contains a
smaller set of pristine, reference digital images and artificially distorted
images derived from the pristine images considering different artificial
distortions at different intensity levels. The second group contains only
digital images with authentic distortions collected from photographers, so
pristine images cannot be found in such databases. Virtanen et al. [1] were
first to introduce this type of database for images by releasing CID2013. As a
consequence, the development of FR methods is connected to the first group of
databases. In contrast Waterloo Exploration [2] and KADIS-700k [3] databases
are meant to provide an alternative evaluation of objective image quality
assessment models, by means of paired comparisons. That is why, they contain a
set of reference (pristine) images, distorted images, and distortion levels.
In contrast to other databases, they do not provide MOS values. Information
about major publicly available image quality assessment databases are
summarized in Table 1.

In this study, we provide a comprehensive evaluation of 32 full-reference
image quality assessment (FR-IQA) algorithms on MDID database. In contrast to
other available image quality databases, the images in MDID contain multiple
types of distortions simultaneously.

The rest of this study is organized as follows. There are a number of publicly
available image quality databases, such as IVC [4], LIVE IQA [5], A57 [6],
Toyoma [7], TID2008 [8], CSIQ [9], IVC-LAR [10], MMSP 3D [11], IRSQ [12],
[13], TID2013 [14], CID2013 [1], LIVE In the Wild [15], Waterloo Exploration
[2], MDID [16], KonIQ-10k [17], KADID-10k [18], and KADIS-700k [18]. In
Section 2, we give a brief introduction to each of them. In Section 3, we give
a comprehensive evaluation of 31 full-reference image quality assessment (FR-
IQA) algorithms on MDID database. Finally, a conclusion is drawn in Section 4.

##  2 Image quality databases

IVC111http://www2.irccyn.ec-nantes.fr/ivcdb/ [4] database consists of 10
pristine images, and 235 distorted images, including four types of distortions
(JPEG, JPEG2000, locally adaptive resolution coding, blurring). Quality score
ratings (1 to 5) are provided in the form of MOS.

LIVE Image Quality
Database222http://www.live.ece.utexas.edu/research/quality/subjective.htm
(LIVE IQA) [5] has two releases, Release 1 and Release 2. Laboratory for Image
and Video Engineering (University of Texas at Austin) conducted an extensive
experiment to obtain scores from human subjects for a number of images
distorted with different distortion types. Release 2 has more distortion types
— JPEG (169 images), JPEG2000 (175 images), Gaussian blur (145 images, White
noise (145 images), bit errors in JPEG2000 bit stream (145 images). The
subjective quality scores in this database are DMOS (Differential MOS),
ranging from 0 to 100.

A57 Database333http://vision.eng.shizuoka.ac.jp/mod/page/view.php?id=26 [6]
has 3 pristine images, and 54 distorted images, including six types of
distortions (JPEG, JPEG2000, JPEG2000 with dynamic contrast-based
quantization, quantization of the LH subbands of DWT, additive Gaussian white
noise, Gaussian blurring). Quality score ratings (0 to 1) are provided in the
form of DMOS.

Toyoma Database [7] consists of 14 pristine images, and 168 distorted images,
including two types of distortions (JPEG, JPEG2000). Quality score ratings (1
to 5) are provided in the form of MOS.

Tampere Image Database 2008444http://www.ponomarenko.info/tid2008.htm
(TID2008) [8] contains 25 reference images and 1,700 distorted images (25
reference images $\times 17$ types of distortions $\times 4$ levels of
distortions). The MOS was obtained from the results of 838 experiments carried
out by observers from three countries. 838 observers have performed 256,428
comparisons of visual quality of distorted images or 512,856 evaluations of
relative visual quality in image pairs. Higher value of MOS (0 - minimal, 9 -
maximal, MSE of each score is 0.019) corresponds to higher visual quality of
the image. A file enclosed “mos.txt” contains the Mean Opinion Score for each
distorted image.

Computational and Subjective Image
Quality555http://vision.eng.shizuoka.ac.jp/mod/page/view.php?id=23 (CSIQ) [9]
database consists of 30 original images, each distorted using one of six types
of distortions, each at four to five different levels of distortion. The
images were subjectively rated based on a linear displacement of the images
across four calibrated monitors placed side-by-side with equal viewing
distance to the observer. The database contains 5,000 subjective ratings from
35 different — both male and female — observers. Quality score ratings (0 to
1) are provided in the form of DMOS.

IVC-LAR666http://ivc.univ-nantes.fr/en/databases/LAR/ [10] database contains 8
pristine images (4 natural images and 4 art images), and 120 distorted images,
consisting of three types of distortions (JPEG, JPEG2000, locally adaptive
resolution coding). Quality score ratings (1 to 5) are provided in the form of
MOS.

Wireless Imaging Quality777https://computervisiononline.com/dataset/1105138665
(WIQ) Database [19], [20] consists of 7 reference images and 80 distorted
images. The subjective quality scores are given in DMOS, ranging from 0 to
100.

In contrast to other publicly available image quality databases MMSP 3D Image
Quality Assessment Database888https://mmspg.epfl.ch/downloads/3diqa/ [11]
consists of stereoscopic images with a resolution of $1,920\times 1,080$
pixels. Specifically, 10 indoor and outdoor scenes were captured with a wide
variety of colors, textures, and depth structures. Furthermore, 6 different
stimuli have been considered corresponding to different camera distances (10,
20, 30, 40, 50, and 60 cm) for each scene.

Image Retargeting Subjective
Quality999http://ivp.ee.cuhk.edu.hk/projects/demo/retargeting/index.html
(IRSQ) Database [12], [13] consists of 57 reference images grouped into four
attributes, specfically face and people, clear foreground object, natural
scenery, and geometric structure. Moreover, ten different retargeting methods
(cropping, seam carving, scaling, shift-map editing, scale and stretch, etc.)
are applied to generate retargeted images. In total, 171 test images can be
found in this database.

Tampere Image Database 2013101010http://www.ponomarenko.info/tid2013.htm
(TID2013) [14] contains 25 reference images and 3,000 distorted images (25
reference images $\times 24$ types of distortions $\times 5$ levels of
distortions). MOS (Mean Opinion Score) is provided as subjective score,
ranging from 0 to 9.

The CID2013111111http://www.helsinki.fi/psychology/groups/visualcognition/ [1]
database contains 474 images with authentic distortions captured by 79 imaging
devices, such as mobile phones, digital still cameras, and digital single-lens
reflex cameras.

LIVE In the Wild Image Quality Challenge
Database121212http://live.ece.utexas.edu/research/ChallengeDB/ [15] contains
widely diverse authentic image distortions on a large number of images
captured using a representative variety of modern mobile devices. The LIVE In
the Wild Image Quality Database has over 350,000 opinion scores on 1,162
images evaluated by over 8,100 unique human observers.

Waterloo Exploration131313https://ece.uwaterloo.ca/ k29ma/exploration/ [2]
database consists of 4,744 reference images and 94,880 distorted images
created from them. Instead of collecting MOS for each test image, the authors
introduced three alternative test criteria to evaluate the performance of IQA
models, such as discriminability test (D-test), listwise ranking consistency
test (L-test), and pairwise preference consistency test (P-test).

In contrast to other databases considering artificial distortions,
MDID141414https://www.sz.tsinghua.edu.cn/labs/vipl/mdid.html [16] obtains
distorted images from reference images with random types and levels of
distortions. In this way, each distorted image contains multiple types of
distortions simultaneously. Gaussian noise, Gaussian blur, contrast change,
JPEG noise, and JPEG2000 noise were considered.

The main challenge in applying state-of-the-art deep learning methods to
predict image quality in-the-wild is the relatively small size of existing
quality scored datasets. The reason for the lack of larger datasets is the
massive resources required in generating diverse and publishable content. In
KonIQ-10k151515http://database.mmsp-kn.de/koniq-10k-database.html [17] a new
systematic and scalable approach is presented to create large-scale, authentic
image datasets for image quality assessment. KonIQ-10k [17] consists of 10,073
images, on which large scale crowdsourcing experiments has been carried out in
order to obtain reliable quality ratings from 1,467 crowd workers (1.2 million
ratings) [21]. During the test users exhibiting unusual scoring behavior were
removed.

KADID-10k161616http://database.mmsp-kn.de/kadid-10k-database.html [18]
consists of 81 pristine images and $10,125$ distorted images derived from the
pristine images considering $25$ different distortion types at 5 intensity
levels ($10,125=81\times 25\times 5$). In contrast, KADIS-700k [18] contains
$140,000$ pristine images and distorted images were derived using $25$
different distortion types at 5 intensity levels but MOS values are not given
in this database.

Table 1: Major publicly available image quality assessment databases. Publicly available image quality databases can be divided into three groups. The first one contains a smaller set of reference images and artificially distorted images are derived from them using different noise types at different intensity levels. There are also databases which contains only pristine images, distorted images, and distortion levels without MOS.  Database | Year | Reference images | Test images | Distortion type | Subjective score
---|---|---|---|---|---
IVC [4] | 2005 | 10 | 235 | artificial | MOS (1-5)
LIVE IQA [5] | 2006 | 29 | 779 | artificial | DMOS (0-100)
A57 [6] | 2007 | 3 | 54 | artificial | DMOS (0-1)
Toyoma [7] | 2008 | 14 | 168 | artificial | MOS (1-5)
TID2008 [8] | 2008 | 25 | 1,700 | artificial | MOS (0-9)
CSIQ [9] | 2009 | 30 | 866 | artificial | DMOS (0-1)
IVC-LAR [10] | 2009 | 8 | 120 | artificial | MOS (1-5)
WIQ [19], [20] | 2009 | 7 | 80 | artificial | DMOS (0-100)
MMSP 3D [11] | 2009 | 9 | 54 | artificial | MOS (0-100)
IRSQ [12], [13] | 2011 | 57 | 171 | artificial | MOS (0-5)
TID2013 [14] | 2013 | 25 | 3,000 | artificial | MOS (0-9)
CID2013 [1] | 2013 | 8 | 474 | authentic | MOS (0-9)
LIVE In the Wild [15] | 2016 | - | 1,162 | authentic | MOS (1-5)
Waterloo Exploration [2] | 2016 | 4,744 | 94,880 | artificial | -
MDID [16] | 2017 | 20 | 1600 | artificial | MOS (0-8)
KonIQ-10k [17] | 2018 | - | 10,073 | authentic | MOS (1-5)
KADID-10k [3] | 2019 | 81 | 10,125 | artificial | MOS (1-5)
KADIS-700k [3] | 2019 | 140,000 | 700,000 | artificial | -

##  3 Experimental results

Table 2: Performance comparison of 31 FR-IQA algorithms on MDID database. Method | Year | PLCC | SROCC | KROCC
---|---|---|---|---
BLeSS-SR-SIM [22] | 2016 | 0.7535 | 0.8148 | 0.6258
BLeSS-FSIM [22] | 2016 | 0.8193 | 0.8467 | 0.6576
BLeSS-FSIMc [22] | 2016 | 0.8527 | 0.8827 | 0.7018
CBM [23] | 2005 | 0.7367 | 0.7212 | 0.5306
CSV [24] | 2016 | 0.8785 | 0.8814 | 0.6998
CW-SSIM [25] | 2009 | 0.5900 | 0.6148 | 0.4450
DSS [26] | 2015 | 0.8714 | 0.8661 | 0.6793
ESSIM [27] | 2013 | 0.6694 | 0.8253 | 0.6349
FSIM [28] | 2011 | 0.8591 | 0.8870 | 0.7074
FSIMc [28] | 2011 | 0.8639 | 0.8902 | 0.7122
GMSD [29] | 2013 | 0.8544 | 0.8617 | 0.6797
HaarPSI [30] | 2018 | 0.9051 | 0.9028 | 0.7340
MAD [9] | 2010 | 0.7439 | 0.7243 | 0.5327
MCSD [31] | 2016 | 0.8386 | 0.8457 | 0.6622
MDSI (’mult’) [32] | 2016 | 0.8130 | 0.8278 | 0.6441
MDSI (’sum’) [32] | 2016 | 0.8249 | 0.8363 | 0.6527
MS-SSIM [33] | 2003 | 0.7884 | 0.8292 | 0.6360
MS-UNIQUE [34] | 2017 | 0.8604 | 0.8712 | 0.6893
NQM [35] | 2000 | 0.6177 | 0.5869 | 0.4143
PerSIM [36] | 2015 | 0.8282 | 0.8196 | 0.6296
PSNR-HVS [37] | 2006 | 0.679 | 0.6637 | 0.4845
PSNR-HVS-M [38] | 2007 | 0.6875 | 0.6739 | 0.4944
QILV [39] | 2006 | 0.3296 | 0.4592 | 0.3214
QSSIM [40] | 2011 | 0.8022 | 0.8014 | 0.6074
RFSIM [41] | 2010 | 0.7035 | 0.6758 | 0.4884
SCIELAB [42] | 1997 | 0.2552 | 0.1232 | 0.0824
SR-SIM [43] | 2012 | 0.7948 | 0.8517 | 0.6683
SSIM [44] | 2004 | 0.5798 | 0.5761 | 0.4105
SSIM CNN [45] | 2018 | 0.8706 | 0.8804 | 0.6992
SUMMER [46] | 2019 | 0.7427 | 0.7343 | 0.5434
UQI [47] | 2002 | 0.2175 | 0.3608 | 0.2476
VSI [48] | 2014 | 0.7883 | 0.8570 | 0.6710

The evaluation of objective visual quality assessment is based on the
correlation between the predicted and the ground-truth quality scores.
Pearson’s linear correlation coefficient (PLCC) and Spearman’s rank order
correlation coefficient (SROCC) are widely applied to this end. Furthermore,
some authors give the Kendall’s rank order correlation coefficient as well.

The PLCC between data set $A$ and $B$ is defined as

$PLCC(A,B)=\frac{\sum_{i=1}^{n}(A_{i}-\overline{A})(B_{i}-\overline{B})}{\sqrt{\sum_{i=1}^{n}(A_{i}-\overline{A})^{2}}\sqrt{\sum_{i=1}^{n}(B_{i}-\overline{B})^{2}}},$
(1)

where $\overline{A}$ and $\overline{B}$ stand for the average of set $A$ and
$B$, $A_{i}$ and $B_{i}$ denote the $i$th elements of set $A$ and $B$,
respectively. For two ranked sets A and B SROCC is defined as

$SROCC(A,B)=\frac{\sum_{i=1}^{n}(A_{i}-\hat{A})(B_{i}-\hat{B})}{\sqrt{\sum_{i=1}^{n}(A_{i}-\hat{A})^{2}}\sqrt{\sum_{i=1}^{n}(B_{i}-\hat{B})^{2}}},$
(2)

where $\hat{A}$ and $\hat{B}$ are the middle ranks of set A and B. KROCC
between dataset $A$ and $B$ can be calculated as

$KROCC(A,B)=\frac{n_{c}-n_{d}}{\frac{1}{2}n(n-1)},$ (3)

where $n$ is the length of the input vectors, $n_{c}$ is the number of
concordant pairs between $A$ and $B$, and $n_{d}$ is the number of discordant
pairs between $A$ and $B$.

We collected 31 FR-IQA metrics whose source codes are available online.
Furthermore, we reimplemented SSIM
CNN171717https://github.com/Skythianos/Pretrained-CNNs-for-full-reference-
image-quality-assessment [45] in MATLAB R2019a. In Table 2, we present PLCC,
SROCC, and KROCC values measured over the MDID database. It can be clearly
seen from the results that there is still a lot of space for the improvement
of FR-IQA algorithms because only HaarPSI [30] was able to produce PLCC and
SROCC values higher than 0.9. Furthermore, only three methods — FSIM [28],
FSIMc [28], HaarPSI [30] — were able to produce KROCC values higher than 0.7.

##  4 Conclusion

First, we gave information about the mostly applied image quality databases.
Subsequently, we extensively evaluated 32 state-of-the-art FR-IQA methods on
MDID database whose images contain multiple types of distortions
simultaneously. We dmonstrated that there is still a lot of space for the
improvement of FR-IQA algorithms because only HaarPSI [30] was able to produce
PLCC and SROCC values higher than 0.9.

## References

  * [1] Toni Virtanen, Mikko Nuutinen, Mikko Vaahteranoksa, Pirkko Oittinen, and Jukka Häkkinen.  Cid2013: A database for evaluating no-reference image quality assessment algorithms.  IEEE Transactions on Image Processing, 24(1):390–402, 2014.
  * [2] Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang.  Waterloo exploration database: New challenges for image quality assessment models.  IEEE Transactions on Image Processing, 26(2):1004–1016, 2017\.
  * [3] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Kadid-10k: A large-scale artificially distorted iqa database.  In 2019 Tenth International Conference on Quality of Multimedia Experience (QoMEX), pages 1–3. IEEE, 2019.
  * [4] Patrick Le Callet and Florent Autrusseau.  Subjective quality assessment irccyn/ivc database, 2005.  http://www.irccyn.ec-nantes.fr/ivcdb/.
  * [5] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik.  A statistical evaluation of recent full reference image quality assessment algorithms.  IEEE Transactions on image processing, 15(11):3440–3451, 2006.
  * [6] Damon M Chandler and Sheila S Hemami.  Vsnr: A wavelet-based visual signal-to-noise ratio for natural images.  IEEE transactions on image processing, 16(9):2284–2298, 2007.
  * [7] Sylvain Tourancheau, Florent Autrusseau, Parvez Sazzad, and Yuukou Horita.  Impact of the subjective dataset on the performance of image quality metrics.  In IEEE International Conference on Image Processing 2008. ICIP 2008., 2008.
  * [8] Nikolay Ponomarenko, Vladimir Lukin, Alexander Zelensky, Karen Egiazarian, Marco Carli, and Federica Battisti.  Tid2008-a database for evaluation of full-reference visual quality assessment metrics.  Advances of Modern Radioelectronics, 10(4):30–45, 2009.
  * [9] Eric Cooper Larson and Damon Michael Chandler.  Most apparent distortion: full-reference image quality assessment and the role of strategy.  Journal of Electronic Imaging, 19(1):011006, 2010.
  * [10] Florent Autrusseau and Marie Babel.  Subjective quality assessment of lar coded art images, 2009.  http://www.irccyn.ec-nantes.fr/ autrusse/Databases/.
  * [11] Lutz Goldmann, Francesca De Simone, and Touradj Ebrahimi.  Impact of acquisition distortion on the quality of stereoscopic images.  In Proceedings of the International Workshop on Video Processing and Quality Metrics for Consumer Electronics, number MMSPL-CONF-2009-022, 2010\.
  * [12] Lin Ma, Weisi Lin, Chenwei Deng, and King Ngi Ngan.  Image retargeting quality assessment: A study of subjective scores and objective metrics.  IEEE Journal of Selected Topics in Signal Processing, 6(6):626–639, 2012.
  * [13] Lin Ma, Weisi Lin, Chenwei Deng, and King N Ngan.  Study of subjective and objective quality assessment of retargeted images.  In Circuits and Systems (ISCAS), 2012 IEEE International Symposium on, pages 2677–2680. IEEE, 2012.
  * [14] Nikolay Ponomarenko, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Lina Jin, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al.  Color image database tid2013: Peculiarities and preliminary results.  In Visual Information Processing (EUVIP), 2013 4th European Workshop on, pages 106–111. IEEE, 2013.
  * [15] Deepti Ghadiyaram and Alan C Bovik.  Massive online crowdsourced study of subjective and objective picture quality.  IEEE Transactions on Image Processing, 25(1):372–387, 2015.
  * [16] Wen Sun, Fei Zhou, and Qingmin Liao.  Mdid: A multiply distorted image database for image quality assessment.  Pattern Recognition, 61:153–168, 2017.
  * [17] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Koniq-10k: Towards an ecologically valid and large-scale iqa database.  arXiv preprint arXiv:1803.08489, 2018.
  * [18] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Kadid-10k: A large-scale artificially distorted iqa database.  In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pages 1–3. IEEE, 2019.
  * [19] Ulrich Engelke, Maulana Kusuma, Hans-Jürgen Zepernick, and Manora Caldera.  Reduced-reference metric design for objective perceptual quality assessment in wireless imaging.  Signal Processing: Image Communication, 24(7):525–547, 2009.
  * [20] Ulrich Engelke, Hans-Jürgen Zepernick, and Tubagus Maulana Kusuma.  Subjective quality assessment for wireless image communication: The wireless imaging quality database.  In International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM). VPQM, 2010.
  * [21] Dietmar Saupe, Franz Hahn, Vlad Hosu, Igor Zingman, Masud Rana, and Shujun Li.  Crowd workers proven useful: A comparative study of subjective video quality assessment.  In QoMEX 2016: 8th International Conference on Quality of Multimedia Experience, 2016.
  * [22] Dogancan Temel and Ghassan AlRegib.  Bless: Bio-inspired low-level spatiochromatic similarity assisted image quality assessment.  In 2016 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2016.
  * [23] Xinbo Gao, Tao Wang, and Jie Li.  A content-based image quality metric.  In International Workshop on Rough Sets, Fuzzy Sets, Data Mining, and Granular-Soft Computing, pages 231–240. Springer, 2005.
  * [24] Dogancan Temel and Ghassan AlRegib.  Csv: Image quality assessment based on color, structure, and visual system.  Signal Processing: Image Communication, 48:92–103, 2016.
  * [25] Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey.  Complex wavelet structural similarity: A new image similarity index.  IEEE transactions on image processing, 18(11):2385–2401, 2009.
  * [26] Amnon Balanov, Arik Schwartz, Yair Moshe, and Nimrod Peleg.  Image quality assessment based on dct subband similarity.  In 2015 IEEE International Conference on Image Processing (ICIP), pages 2105–2109. IEEE, 2015.
  * [27] Xuande Zhang, Xiangchu Feng, Weiwei Wang, and Wufeng Xue.  Edge strength similarity for image quality assessment.  IEEE Signal processing letters, 20(4):319–322, 2013.
  * [28] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang.  Fsim: A feature similarity index for image quality assessment.  IEEE transactions on Image Processing, 20(8):2378–2386, 2011.
  * [29] Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik.  Gradient magnitude similarity deviation: A highly efficient perceptual image quality index.  IEEE Transactions on Image Processing, 23(2):684–695, 2013.
  * [30] Rafael Reisenhofer, Sebastian Bosse, Gitta Kutyniok, and Thomas Wiegand.  A haar wavelet-based perceptual similarity index for image quality assessment.  Signal Processing: Image Communication, 61:33–43, 2018.
  * [31] Tonghan Wang, Lu Zhang, Huizhen Jia, Baosheng Li, and Huazhong Shu.  Multiscale contrast similarity deviation: An effective and efficient index for perceptual image quality assessment.  Signal Processing: Image Communication, 45:1–9, 2016.
  * [32] Hossein Ziaei Nafchi, Atena Shahkolaei, Rachid Hedjam, and Mohamed Cheriet.  Mean deviation similarity index: Efficient and reliable full-reference image quality evaluator.  IEEE Access, 4:5579–5590, 2016.
  * [33] Zhou Wang, Eero P Simoncelli, and Alan C Bovik.  Multiscale structural similarity for image quality assessment.  In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398–1402. Ieee, 2003.
  * [34] Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib.  Ms-unique: Multi-model and sharpness-weighted unsupervised image quality estimation.  Electronic Imaging, 2017(12):30–35, 2017.
  * [35] Niranjan Damera-Venkata, Thomas D Kite, Wilson S Geisler, Brian L Evans, and Alan C Bovik.  Image quality assessment based on a degradation model.  IEEE transactions on image processing, 9(4):636–650, 2000.
  * [36] Dogancan Temel and Ghassan AlRegib.  Persim: Multi-resolution image quality assessment in the perceptually uniform color domain.  In 2015 IEEE International Conference on Image Processing (ICIP), pages 1682–1686. IEEE, 2015.
  * [37] Karen Egiazarian, Jaakko Astola, Nikolay Ponomarenko, Vladimir Lukin, Federica Battisti, and Marco Carli.  New full-reference quality metrics based on hvs.  In Proceedings of the Second International Workshop on Video Processing and Quality Metrics, volume 4, 2006.
  * [38] Nikolay Ponomarenko, Flavia Silvestri, Karen Egiazarian, Marco Carli, Jaakko Astola, and Vladimir Lukin.  On between-coefficient contrast masking of dct basis functions.  In Proceedings of the third international workshop on video processing and quality metrics, volume 4, 2007.
  * [39] Santiago Aja-Fernandez, Raul San Jose Estepar, Carlos Alberola-Lopez, and Carl-Fredrik Westin.  Image quality assessment based on local variance.  In 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pages 4815–4818. IEEE, 2006.
  * [40] Amir Kolaman and Orly Yadid-Pecht.  Quaternion structural similarity: a new quality index for color images.  IEEE Transactions on Image Processing, 21(4):1526–1536, 2011.
  * [41] Lin Zhang, Lei Zhang, and Xuanqin Mou.  Rfsim: A feature based image quality assessment metric using riesz transforms.  In 2010 IEEE International Conference on Image Processing, pages 321–324. IEEE, 2010.
  * [42] Xuemei Zhang, D Amnon Silverstein, Joyce E Farrell, and Brian A Wandell.  Color image quality metric s-cielab and its application on halftone texture visibility.  In Proceedings IEEE COMPCON 97. Digest of Papers, pages 44–48. IEEE, 1997.
  * [43] Lin Zhang and Hongyu Li.  Sr-sim: A fast and high performance iqa index based on spectral residual.  In 2012 19th IEEE international conference on image processing, pages 1473–1476. IEEE, 2012.
  * [44] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [45] Seyed Ali Amirshahi, Marius Pedersen, and Azeddine Beghdadi.  Reviving traditional image quality metrics using cnns.  In Color and Imaging Conference, volume 2018, pages 241–246. Society for Imaging Science and Technology, 2018.
  * [46] Dogancan Temel and Ghassan AlRegib.  Perceptual image quality assessment through spectral analysis of error representations.  Signal Processing: Image Communication, 70:37–46, 2019.
  * [47] Zhou Wang and Alan C Bovik.  A universal image quality index.  IEEE signal processing letters, 9(3):81–84, 2002.
  * [48] Lin Zhang, Ying Shen, and Hongyu Li.  Vsi: A visual saliency-induced index for perceptual image quality assessment.  IEEE Transactions on Image Processing, 23(10):4270–4281, 2014.