# Score-Based Generative Classifiers

Roland S. Zimmermann
<EMAIL_ADDRESS>
&Lukas Schott1
<EMAIL_ADDRESS>
&Yang Song2
<EMAIL_ADDRESS>
&Benjamin A. Dunn3
<EMAIL_ADDRESS>
&David A. Klindt3
<EMAIL_ADDRESS>
University of Tübingen, 2 Stanford University, 3 Norwegian University of
Science and Technology.

###### Abstract

The tremendous success of generative models in recent years raises the
question whether they can also be used to perform classification. Generative
models have been used as adversarially robust classifiers on simple datasets
such as MNIST, but this robustness has not been observed on more complex
datasets like CIFAR-10. Additionally, on natural image datasets, previous
results have suggested a trade-off between the likelihood of the data and
classification accuracy. In this work, we investigate score-based generative
models as classifiers for natural images. We show that these models not only
obtain competitive likelihood values but simultaneously achieve state-of-the-
art classification accuracy for generative classifiers on CIFAR-10.
Nevertheless, we find that these models are only slightly, if at all, more
robust than discriminative baseline models on out-of-distribution tasks based
on common image corruptions. Similarly and contrary to prior results, we find
that score-based are prone to worst-case distribution shifts in the form of
adversarial perturbations. Our work highlights that score-based generative
models are closing the gap in classification accuracy compared to standard
discriminative models. While they do not yet deliver on the promise of
adversarial and out-of-domain robustness, they provide a different approach to
classification that warrants further research.

##  1 Introduction

Figure 1: Model Comparison. Previous approaches [8, 1, 27] have demonstrated a
trade-off between accuracy and likelihoods of generative classifiers on
CIFAR-10. The black lines show current SOTA discriminative [horizontal, 7] and
generative [vertical, 20] models.

There exist two fundamentally distinct ways of performing classification. The
standard way is to train a classifier which discriminates between classes by
modeling the conditional probability $p(y|\mathbf{x})$ of labels $y$ given
some input $\mathbf{x}$. These models are called discriminative classifiers
and have been extremely successful in supervised learning [24, 22]. The
alternative is to model the conditional likelihood $p(\mathbf{x}|y)$ of an
image for each label and predict the label that maximizes this likelihood
[39]. This approach is much less common because it requires solving the more
complicated task of learning conditional generative models of the data; these
models are referred to as generative classifiers.

A line of research has been discussing the advantages and disadvantages
between using discriminative or generative models for classification [39, 33,
36, 2, 26]. They found generative models to be more data efficient, but
discriminative models to be asymptotically better. However, these analyses
were mostly limited to shallow models [but see 12] and performance evaluation
within the same data domain. On MNIST [24] and SVHN [32], previous results
have demonstrated that generative classifiers are robust even to worst-case
distribution shifts in the form of adversarial perturbations [40, 17, 35].
Furthermore, it was found that generative classifiers have the highest
alignment with humans on _controversial_ stimuli [11].

However, so far, these results have not transferred to any natural datasets of
higher visual complexity [25, 8, 1, 27]. Fetaya et al. [8] argue that
‘obtaining strong classification accuracy without harming likelihood
estimation is still a challenging problem’. This is empirically supported in
their paper as well as a number of related works [13, 1, 27] who all
demonstrate a trade-off between good likelihoods and high classification
accuracy, suggesting that to better capture what is unique for a label, some
performance of the generative model needs to be sacrificed [8]. We challenge
these ideas and add a new datapoint to the ongoing debate around generative
classifiers demonstrating that near state-of-the-art likelihoods can be
combined with classification accuracy at the level of discriminative models
[45].

More specifically, score-based generative models have been proposed in recent
years as a promising approach towards modeling complex high-dimensional data
distributions [41, 16, 42]. They are built on the idea of gradually diffusing
a complex data distribution with a parameterized stochastic differential
equation into a tractable noise distribution. Previous work has suggested the
effectiveness of these diffusion-based models for out-of-domain (OOD) tasks
such as adversarially robust classification [51] and as a general prior for
natural images [48, 19].

In this work, we propose score-based generative classifiers (SBGC) and show
that while these models overcome previous limitations — such as trading off
likelihood for accuracy — they still do not solve the problem of worst-case
distribution shifts on natural images. More precisely, using our SBGC model,
we report state-of-the-art classification accuracy for generative models on
CIFAR-10 [22], along with highly competitive likelihood values. Nonetheless,
we find that score-based generative classifiers show only slight improvements,
if at all, in OOD benchmarks and are vulnerable to adversarial examples.

##  2 Methods

###  2.1 Score-Based Generative Models

This work builds on recent advances in score-based generative models of
natural images [41, 16, 42, 43, 20]. The general idea behind this approach is
to define a stochastic differential equation (SDE) describing the temporal
transformation of the complex data distribution ($p$) into a tractable noise
distribution ($p_{T}$, isotropic Gaussian). This dynamical process can then be
reversed by learning a time-dependent score function
$\mathbf{s}_{\bm{\theta}}(\mathbf{x}(t),t)$ modeled by a neural network with
parameters $\bm{\theta}$. Intuitively, this score function acts as an
infinitesimal denoiser [19]. To generate new samples from the data
distribution, one starts with random noise $\mathbf{x}(T)$ and follows the
dynamics of the reverse SDE to produce a sample $\mathbf{x}(0)$ from the data
distribution [see 42, for more details].

Moreover, by removing the stochasticity, the SDE becomes a (neural) ordinary
differential equation [42, 5]. Using a continuous time analog of the change of
variables formula, one can compute the likelihood ($p_{0}$) of an input image
$\mathbf{x}(0)$ under the model [42] as

$\displaystyle\log p_{0}(\mathbf{x}(0))=\log
p_{T}(\mathbf{x}(T))+\int_{0}^{T}\nabla\cdot\widetilde{\mathbf{f}}_{\bm{\theta}}(\mathbf{x}(t),t)\,\mathop{}\\!\mathrm{d}t$
(1)
$\displaystyle\text{with}\quad\widetilde{\mathbf{f}}_{\bm{\theta}}(\mathbf{x}(t),t)=\mathbf{f}(\mathbf{x},t)-\frac{1}{2}g(t)^{2}\mathbf{s}_{\bm{\theta}}(\mathbf{x},t).$
(2)

For the drift and diffusion coefficients $\mathbf{f}(\mathbf{x},t)$ and
$g(t)$, we used the same functions as the sub-VP model of Song et al. [42]. We
use the training objective suggested in Song et al. [43] which maximizes the
likelihood of the training data under the model. This is equivalent to
minimizing the Kullback-Leibler divergence (KL) between the model and the data
distribution $\min_{\theta}\operatorname{KL}(p_{0}|p)$ [29].

###  2.2 Score-Based Generative Classifiers

The most straightforward application of score-based generative models as
classifiers would be to train one model per class as in Schott et al. [40].
However, this produces very unstable training and considerable overfitting
because the dataset size is, for each model, effectively reduced by one order
of magnitude. Instead, we add the image label $y\in\\{1,...,10\\}$ as a
conditioning variable to the score function $f_{\theta}(\mathbf{x}(t),t,y)$;
note that this also reduces the required amount of memory for multiple models.
In terms of the network architecture, we integrate this additional input by
first converting it into a one-hot vector and then adding it to the model’s
representation in the same way as the time variable [see 42]. This is in line
with concurrent work of Nichol and Dhariwal [34] and loosely inspired by Jun
et al. [18]. For classification of a given input $\mathbf{x}(0)$, we can then
condition the score function on each label $y$

$\log p_{0}(\mathbf{x}(0)|y)=\log
p_{T}\left(\mathbf{x}(0)+\int_{0}^{T}\widetilde{\mathbf{f}}_{\bm{\theta}}(\mathbf{x}(t),t,y)\mathop{}\\!\mathrm{d}t\right)+\int_{0}^{T}\nabla\cdot\widetilde{\mathbf{f}}_{\bm{\theta}}(\mathbf{x}(t),t,y)\mathop{}\\!\mathrm{d}t,$
(3)

where $\widetilde{\mathbf{f}}_{\bm{\theta}}$ is defined as above in Eq. (1).
For any input image $\mathbf{x}$, the model then predicts the label that
yields the highest conditional likelihood $\max_{y}\log
p_{0}(\mathbf{x}(0)|y)$. To get the unconditional likelihood of samples we
marginalize over the classes, i.e.
$p(\mathbf{x})=\sum_{y_{i}}p_{0}(\mathbf{x}|y=y_{i})$.

##  3 Results

###  3.1 Classification Accuracy and Data Likelihood

Table 1: Model Comparison. Accuracy (in %) and negative log-likelihoods (NLL) in bits per dimension [47] on the CIFAR-10 test set. The lower half of the table shows a baseline discriminative model [45] and the current state-of-the-art discriminative [7] and generative [20] models.  Model approach | Accuracy [%] $\uparrow$ | NLL [bits/dim.] $\downarrow$
---|---|---
Invertible Network (Mackowiak et al. [27]) | 67.30 | 4.34
GLOW (Fetaya et al. [8]) | 84.00 | 3.53
Normalizing flow (Ardizzone et al. [1]) | 89.73 | 5.25
Energy model (Grathwohl et al. [13]) | 92.90 | N/A
SBGC (ours) | 95.04 | 3.11
WideResNet-28-12 (Targ et al. [45]) | 95.42 | N/A
ViT-H/14 (Dosovitskiy et al. [7]) | 99.50 | N/A
VDM (Kingma et al. [20]) | N/A | 2.49

First of all, we note that previous methods [8, 1, 27] have discussed a trade-
off between classification accuracy and likelihoods (see Figure 1). This is
further supported by observations on MNIST, where increasing the latent
dimensionality of a variational autoencoder increases the likelihood at the
cost of decreasing classification accuracy and robustness [6]. Older work has
focused on hybrid models that combine the data efficiency of (shallow)
generative models with the asymptotic performance of discriminative models.
However, this comes with a cost in model likelihood [10, 36].

In contrast to these previous methods, we find that our model achieves both
state-of-the-art accuracy and likelihoods for generative classifiers on
CIFAR-10 (Table 1). We also test the deeper architecture proposed in Song et
al. [43]: While this does not change the classification accuracy, it slightly
increases the NLL to $3.08$ bits/dim. We leave further exploration of this
direction to future research.

###  3.2 Out-of-Distribution Robustness: Common Image Corruptions

Table 2: Performance on Common Image Corruptions. Accuracy (in %) on clean CIFAR-10 test set and the mean accuracy on CIFAR-10-C (considering a random subset with 10% of the original size for SBGC because of computational limitations). We refer to random flips, random crops and uniform noise as simple augmentations.  |  | CIFAR-10-C
---|---|---
Models w/ data augm. | CIFAR-10 | all corruptions | w/o noises
ResNeXt29 + AugMix (Hendrycks et al. [15]) | 95.83 | 89.09 | 90.51
ResNet-50 + adv. augm. (Calian et al. [4]) | 94.93 | 92.17 | 92.53
Models w/ simple data augm. |  |  |
WideResNet-28-12 (Targ et al. [45]) | 95.42 | 74.42 | 80.21
SBGC (ours) | 95.04 | 76.24 | 75.71

Most previous approaches towards improving out-of-distribution performance on
common image corruptions [14] have relied on some form of data augmentation in
the form of carefully hand-crafted transformations [15] or adversarial
training [4]. Here, we propose an orthogonal approach that is based on
different modeling assumptions. Specifically, we are interested in seeing
whether the inductive bias implicit in score-based generative classifiers
improves their classification accuracy when generalizing to the image domains
of CIFAR-10-C [14]. We find that our model performs better than some previous
models that are solely trained on the original data (Table 2).

Since score-based models are trained like denoisers, they might work
specifically well on noisy data. Thus, just comparing the mean accuracy over
all corruption types, could give them an unfair advantage over the baselines.
Hence, we also compare the mean accuracy for all corruptions excluding noises
111Gaussian noise, shot nose, impulse noise (Table 2). Here, we see that our
SBGC model does not perform better than the baselines. We conclude that the
improvements of the SBGC model in terms of out-of-distribution robustness are
limited to noise corruptions.

When integrating data augmentation methods, discriminative models achieve
higher performance (Table 2, top). However, we note that the approach of
augmenting datasets is independent of our suggested changes, with the caveat
that changing the dataset needs to be done carefully for generative models
[18]. Preliminary experiments indicate that we can increase our performance on
CIFAR-10-C further by leveraging more data augmentations [15] (accuracy $\sim
83\%$). However, there is still a large gap towards discriminative models with
data augmentation.

###  3.3 Out-of-Distribution Robustness: Adversarial Perturbations

Despite their small magnitude, adversarial perturbations can cause a drastic
change in the behavior of a neural network [44] and can be seen as a worst-
case distribution shift. To assess the robustness of our SBGC model against
such perturbations, we first generate them using a projected gradient descent
(PGD) [28] on the negative cross-entropy of the model’s prediction using
Foolbox [37]. For this, we estimate the gradients of the model likelihoods
with respect to the inputs
$\mathop{}\\!\mathrm{d}p_{0,y}(\mathbf{x})/\mathop{}\\!\mathrm{d}\mathbf{x}$
using the adjoint sensitivity method [5]. This requires second-order
optimization because of the Jacobian term in equation 3. We implement this
efficiently with Hessian-vector-products in JAX [3] and verify the
implementation by comparing the calculated gradient with numerical estimations
based on finite differences. Since computing gradients is still
computationally expensive (a single forward and backward pass take
approximately 90 minutes on an NVIDIA Tesla V100 GPU), we follow common
practice in this domain and perform the adversarial evaluation on a reduced
sample size of $N=216$.

While our approach partially improves robustness against common image
corruptions, it strikingly fails in the presence of adversarial perturbations
(Tab. 3). Within both standard $\ell_{\infty}$ perturbation norms ($8/255$) as
well as standard $\ell_{2}$ perturbation norms ($0.5$), our model miss-
classifies every adversarially perturbed input image.

Table 3: Performance on Adversarial Perturbations. Accuracy (in %) of different models against adversarial perturbations generated by a norm-bounded $\ell_{\infty}$ and $\ell_{2}$ PGD [28] attack (against each specific model) with a bound of $\epsilon=8/255$ and $\epsilon=0.5$, respectively. The bottom row shows the current state-of-the-art model for adversarially robust classification on CIFAR-10 as a reference value.  Model | Clean | $\ell_{\infty}$ | $\ell_{2}$
---|---|---|---
SBGC (ours) | 95.04 | $0.00$ | $0.00$
WideResNet-70-16 + augm. (Rebuffi et al. [38]) | 92.23 | 82.32 | 66.56

###  3.4 Likelihood Estimation on Interpolated Samples

Figure 2: Interpolation Experiment. We linearly interpolate between dataset
samples and asses the shape of the log-likelihood interpolation function
(equation 4). While Fetaya et al. [8] found a concave shape, indicating high
likelihoods outside the data distribution, we find a convex function
suggesting that the score based generative model does assign lower likelihoods
outside the training data distribution. The black line shows the mean and the
shaded area indicates the standard deviation of the estimated log-likelihood
over $500$ image pairs.

In a previous study, Fetaya et al. [8] highlighted a problem with generative
classifiers. They found that the likelihood of an interpolated image on
CIFAR-10 is actually higher than the likelihoods of both the start $x_{A}$ and
endpoint $x_{B}$ of the interpolation. More precisely, they demonstrated that
the function

$f(t)=p_{0}\left(t\cdot x_{A}+(1-t)\cdot x_{B}\right),\quad t\in[0,1]$ (4)

is concave, which is counter-intuitive because a generative model should
assign low likelihood to inputs outside of its training distribution such as
nonsensical interpolations in pixel space.

To test this for our SBGC model, we measure the function $f(t)$ for $24$
values for $t\in[0,1]$ and average this over $500$ image pairs. As depicted in
Figure 2, we find that the interpolation of the likelihood is generally
convex. Thus, in contrast to the analysis of previous models by [8], our SBGC
model assigns lower likelihood values to interpolated images that are outside
the (training) data distribution.

##  4 Conclusion

We have shown that the latest advances in score-based generative modeling of
natural images translate into generative classifiers which have highly
competitive classification accuracies as well as likelihoods. However, in the
search for further benefits, we found that these models show only minor or no
improvements on common image corruptions. Similarly, but in contrast to
previous results [51], they spectacularly fail on worst-case distribution
shifts. This highlights the necessity to perform extensive and, if at all
possible, gradient-based adversarial attacks. The lack of adversarial
robustness aligns with previous results [21, 25, 1, 27]. Interestingly, this
vulnerability remains _despite_ resolving the problem of high likelihood on
interpolated images [8], and so we can exclude this as a possible explanation
for the lacking robustness.

Thus, we are left wondering why generative classification of natural images
does not show the same robustness to worst-case distribution shifts as
observed on MNIST [40]. A possible explanation for the positive results on
MNIST might be that Euclidean distances (on which the likelihood of
variational autoencoders with a Gaussian likelihood is based) are a useful
metric on MNIST but not on real images. This also led to the provable
robustness on MNIST [40]. However, on natural images Euclidean distances are
badly aligned with human perception [46, 50, 23] and we have no guarantee
about the model behavior even in small $\ell_{p}$ epsilon balls around the
training data. Further, the results of Chen et al. [6] offer an additional
explanation as they hypothesize that a low dimensional latent space is a
necessary property of a robust generative classifier. Since the latent space
of our SBGC models has the same dimensionality as the images - and hence
cannot be described as low dimensional - this might explain the lack of
robustness to worst-case distribution shifts.

In a nutshell, while we achieve almost the same accuracy on clean data as a
baseline discriminative classifier and show high data likelihoods, we find no
advantage over discriminative classifiers regarding worst-case distribution
shifts and common corruptions.

## Acknowledgments and Funding Sources

We thank Matthias Bethge, Wieland Brendel, Will Grathwohl, Zahra Kadkhodaie,
Dylan Paiton, Ben Poole, Yash Sharma and Eero Simoncelli for their feedback.
Further, we thank the International Max Planck Research School for Intelligent
Systems (IMPRS-IS) for supporting RSZ. This work was partially supported by a
Research Council of Norway FRIPRO grant (90532703) and the German Federal
Ministry of Education and Research (BMBF) through the Competence Center for
Machine Learning (TUE.AI, FKZ 01IS18039A).

## References

  * Ardizzone et al. [2020] Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe.  Training normalizing flows with the information bottleneck for competitive generative classification.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Bouchard and Triggs [2004] Guillaume Bouchard and Bill Triggs.  The tradeoff between generative and discriminative classifiers.  In _16th IASC International Symposium on Computational Statistics (COMPSTAT’04)_ , pages 721–728, 2004.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.
  * Calian et al. [2021] Dan A Calian, Florian Stimberg, Olivia Wiles, Sylvestre-Alvise Rebuffi, Andras Gyorgy, Timothy Mann, and Sven Gowal.  Defending against image corruptions through adversarial augmentations.  _ArXiv preprint_ , abs/2104.01086, 2021.
  * Chen et al. [2018] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud.  Neural ordinary differential equations.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 6572–6583, 2018.
  * Chen et al. [2020] Yanzhi Chen, Renjie Xie, and Zhanxing Zhu.  On breaking deep generative model-based defenses and beyond.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1736–1745. PMLR, 2020\.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ArXiv preprint_ , abs/2010.11929, 2020.
  * Fetaya et al. [2020] Ethan Fetaya, Jörn-Henrik Jacobsen, Will Grathwohl, and Richard S. Zemel.  Understanding the limitations of conditional generative models.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * Field [1987] David J Field.  Relations between the statistics of natural images and the response properties of cortical cells.  _Josa a_ , 4(12):2379–2394, 1987.
  * Fujino et al. [2005] Akinori Fujino, Naonori Ueda, and Kazumi Saito.  A hybrid generative/discriminative approach to semi-supervised classifier design.  In _AAAI_ , pages 764–769, 2005.
  * Golan et al. [2020] Tal Golan, Prashant C Raju, and Nikolaus Kriegeskorte.  Controversial stimuli: Pitting neural networks against each other as models of human cognition.  _Proceedings of the National Academy of Sciences_ , 117(47):29330–29337, 2020.
  * Goodfellow et al. [2013] Ian J. Goodfellow, Mehdi Mirza, Aaron C. Courville, and Yoshua Bengio.  Multi-prediction deep boltzmann machines.  In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , pages 548–556, 2013.
  * Grathwohl et al. [2020] Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky.  Your classifier is secretly an energy based model and you should treat it like one.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas G. Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * Hendrycks et al. [2020] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Ju and Wagner [2020] An Ju and David Wagner.  E-abs: extending the analysis-by-synthesis robust classification model to more complex image domains.  In _Proceedings of the 13th ACM Workshop on Artificial Intelligence and Security_ , pages 25–36, 2020.
  * Jun et al. [2020] Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever.  Distribution augmentation for generative modeling.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5006–5019. PMLR, 2020\.
  * Kadkhodaie and Simoncelli [2020] Zahra Kadkhodaie and Eero P Simoncelli.  Solving linear inverse problems using the prior implicit in a denoiser.  _ArXiv preprint_ , abs/2007.13640, 2020.
  * Kingma et al. [2021] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _ArXiv preprint_ , abs/2107.00630, 2021.
  * Kos et al. [2018] Jernej Kos, Ian Fischer, and Dawn Song.  Adversarial examples for generative models.  In _2018 ieee security and privacy workshops (spw)_ , pages 36–42. IEEE, 2018.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.
  * Laparra et al. [2016] Valero Laparra, Johannes Ballé, Alexander Berardino, and Eero P Simoncelli.  Perceptual image quality assessment using a normalized laplacian pyramid.  _Electronic Imaging_ , 2016(16):1–6, 2016.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Li et al. [2019] Yingzhen Li, John Bradshaw, and Yash Sharma.  Are generative classifiers more robust to adversarial attacks?  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 3804–3814. PMLR, 2019\.
  * Liang and Jordan [2008] Percy Liang and Michael I. Jordan.  An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators.  In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, _Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008_ , volume 307 of _ACM International Conference Proceeding Series_ , pages 584–591. ACM, 2008.  doi: 10.1145/1390156.1390230.
  * Mackowiak et al. [2021] Radek Mackowiak, Lynton Ardizzone, Ullrich Kothe, and Carsten Rother.  Generative classifiers as a basis for trustworthy image classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2971–2981, 2021.
  * Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.
  * Murphy [2013] Kevin P. Murphy.  _Machine learning : a probabilistic perspective_.  MIT Press, Cambridge, Mass. [u.a.], 2013.  ISBN 9780262018029 0262018020.
  * Nalisnick et al. [2019a] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan.  Detecting out-of-distribution inputs to deep generative models using a test for typicality.  _ArXiv preprint_ , abs/1906.02994, 2019a.
  * Nalisnick et al. [2019b] Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019b.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning, 2011\.
  * Ng and Jordan [2001] Andrew Y. Ng and Michael I. Jordan.  On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes.  In Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani, editors, _Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada]_ , pages 841–848. MIT Press, 2001.
  * Nichol and Dhariwal [2021] Alex Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  _ArXiv preprint_ , abs/2102.09672, 2021.
  * Paiton et al. [2020] Dylan M Paiton, Charles G Frye, Sheng Y Lundquist, Joel D Bowen, Ryan Zarcone, and Bruno A Olshausen.  Selectivity and robustness of sparse coding networks.  _Journal of Vision_ , 20(12):10–10, 2020.
  * Raina et al. [2003] Rajat Raina, Yirong Shen, Andrew Y. Ng, and Andrew McCallum.  Classification with hybrid generative/discriminative models.  In Sebastian Thrun, Lawrence K. Saul, and Bernhard Schölkopf, editors, _Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada]_ , pages 545–552. MIT Press, 2003.
  * Rauber et al. [2020] Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel.  Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax.  _Journal of Open Source Software_ , 5(53):2607, 2020.
  * Rebuffi et al. [2021] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann.  Fixing data augmentation to improve adversarial robustness.  _ArXiv preprint_ , abs/2103.01946, 2021.
  * Revow et al. [1996] Michael Revow, Christopher KI Williams, and Geoffrey E Hinton.  Using generative models for handwritten digit recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 18(6):592–606, 1996.
  * Schott et al. [2019] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel.  Towards the first adversarially robust neural network model on MNIST.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Francis R. Bach and David M. Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 2256–2265. JMLR.org, 2015.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _ArXiv preprint_ , abs/2011.13456, 2020.
  * Song et al. [2021] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.  Maximum likelihood training of score-based diffusion models.  _arXiv e-prints_ , pages arXiv–2101, 2021.
  * Szegedy et al. [2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In Yoshua Bengio and Yann LeCun, editors, _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.
  * Targ et al. [2016] Sasha Targ, Diogo Almeida, and Kevin Lyman.  Resnet in resnet: Generalizing residual architectures.  _ArXiv preprint_ , abs/1603.08029, 2016.
  * Teo and Heeger [1994] Patrick C Teo and David J Heeger.  Perceptual image distortion.  In _Proceedings of 1st International Conference on Image Processing_ , volume 2, pages 982–986. IEEE, 1994.
  * Theis et al. [2016] Lucas Theis, Aäron van den Oord, and Matthias Bethge.  A note on the evaluation of generative models.  In Yoshua Bengio and Yann LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.
  * Venkatakrishnan et al. [2013] Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg.  Plug-and-play priors for model based reconstruction.  In _2013 IEEE Global Conference on Signal and Information Processing_ , pages 945–948. IEEE, 2013.
  * Virtanen et al. [2020] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272, 2020.  doi: 10.1038/s41592-019-0686-2.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Yoon et al. [2021] Jongmin Yoon, Sung Ju Hwang, and Juho Lee.  Adversarial purification with score-based generative models.  _ArXiv preprint_ , abs/2106.06041, 2021.

##  Appendix A Appendix

###  A.1 Training and Implementation Details

We build on the architecture (baseline model with $4$ residual blocks)
proposed by Song et al. [43] and extend it to condition the score function
also on the class label. All in all, our model has $67.19$ M parameters.
Furthermore, we use the same SDE (variance preserving) and maximum likelihood
training procedure (trained for $950,000$ steps with a batch size of $128$ and
importance sampling) as Song et al. [43]. The training takes approximately
five days on two NVIDIA Tesla V100 GPUs.

For the likelihood inference, i.e., for solving the SDE, we use the Runge-
Kutta 4(5) solver of SciPy (_scipy.integrate.solve_ivp_) [49] with absolute
and relative precision equal to $10^{-5}$. We perform uniform dequantization
(i.e., adding uniform noise with a magnitude of $1/256$) and integrate the SDE
up to a minimal $\epsilon=10^{-5}$ for numerical stability [42]. Furthermore,
the final likelihood is computed by taking the average over $n=30$ random
draws for the epsilon trace estimator [42]. For the OOD tasks (CIFAR-10-C and
adversarial perturbations), we lower this to $n=10$ to reduce the compute
time, thus giving a more conservative lower estimate of the performance on
these datasets (see also section A.2.1).

For the WideResNet-28-12 baseline we use an existing
implementation222https://github.com/meliketoy/wide-resnet.pytorch in PyTorch.
To make this baseline more comparable with our SBGC, we also use
dequantization noise for augmenting the training data of the WideResNet.

###  A.2 Additional Analyses

####  A.2.1 Convergence Analysis

Figure 3: Convergence Analysis. The left plot shows the classification
accuracy as a function of the number of repetitions for the trace estimator
(error bars indicate two standard deviations over random subsamples). The
right plot shows the estimate of the log-likelihood as a function of the
number of repetitions for the trace estimator (error bars indicate standard
error of the mean) for both the correct class (black) as well the wrong class
with the highest likelihood (orange).

Here, we study the effect of varying the number of repetitions of the trace
estimator (see previous section). In Figure. 3 on the left we see that the
classification accuracy improves monotonically with the number of repetitions.
A number of $n=10$ gives nearly asymptotic performance and is, thus, a good
trade-off for our OOD tasks which are computationally intensive. For
asymptotic performance, we see that the accuracy rises above $95\%$.

On the right side of Figure. 3, we study the convergence of the trace
estimator which is part of the likelihood (equation 3. We can see that the
standard error of the mean (errorbars) converges very slowly. We therefore
report the likelihood at $n=30$ repetitions. Note also that the likelihood of
the correct class converges (averaged across all test images) to a higher
value as that of the wrong class with the highest likelihood. This is expected
for a model with good classification accuracy. Thus, provided labels, our
model attains slightly higher likelihoods than in the unconditioned case.

Figure 4: Detailed Results on CIFAR-10-C. We show the log-likelihood (orange,
left axis) and the accuracy (black, right axis) across all image corruption
types and severities (horizontal axis) for the CIFAR-10-C benchmark [14]. Note
that the severity level _Cl._ corresponds to clean samples.

####  A.2.2 CIFAR-10-C Performance

Here, we further resolve the model performance on CIFAR-10-C for all different
corruption types and strengths in Figure 4. Interestingly, the results
highlight that likelihood and accuracy need not always be aligned. For the
different corruptions based on blurring or some sort of low-pass filtering
(e.g., fog or JPEG compression), we observe that while the accuracy decreases
(as signal is lost) the likelihood under the model actually increases.

This observation is in line with previous studies [31]. We hypothesize that
this highlights the fact that natural images follow a $1/f$ power spectrum [9]
and, thus, low-pass filtering will increase the likelihood of any image. This
also explains previous observations of non-natural images obtaining high
likelihoods under generative models: Starting from a non-natural image we can
always blur the image to obtain high likelihoods under our model (see, e.g.,
our Figure 4 zoom blur and Table 3 in [8]). Similarly, we can start from a
natural image (e.g., from the training set) and add white noise. The high-
frequency perturbations will quickly deteriorate the likelihoods (see Figure 4
Gaussian noise). Relatedly, Nalisnick et al. [30] found that a measure of
dataset complexity can be leveraged in conjunction with likelihood estimates
to perform OOD detection.


# Feature Pyramid Network for Multi-Class Land Segmentation

Selim S. Seferbekov
Veeva Systems
Frankfurt am Main, 60314, Germany
<EMAIL_ADDRESS>Vladimir I. Iglovikov
Level5 Engineering Center, Lyft Inc
Palo Alto, CA 94304, USA
<EMAIL_ADDRESS>Alexander V. Buslaev
Mapbox R$\&$D Center
Minsk, 220030, Belarus
<EMAIL_ADDRESS>Alexey A. Shvets
Massachusetts Institute of Technology
Cambridge, MA 02139, USA
<EMAIL_ADDRESS>
###### Abstract

Semantic segmentation is in-demand in satellite imagery processing. Because of
the complex environment, automatic categorization and segmentation of land
cover is a challenging problem. Solving it can help to overcome many obstacles
in urban planning, environmental engineering or natural landscape monitoring.
In this paper, we propose an approach for automatic multi-class land
segmentation based on a fully convolutional neural network of feature pyramid
network (FPN) family. This network is consisted of pre-trained on ImageNet
Resnet50 encoder and neatly developed decoder. Based on validation results,
leaderboard score and our own experience this network shows reliable results
for the DEEPGLOBE - CVPR 2018 land cover classification sub-challenge.
Moreover, this network moderately uses memory that allows using GTX 1080 or
1080 TI video cards to perform whole training and makes pretty fast
predictions.

##  1 Introduction

The advent of high-resolution optical satellite imagery opens new
possibilities to monitor changes on the earth’s surface. The advantages of
this data compared to aerial imagery are the almost worldwide availability,
and sometimes the imagery data contains additional spectral channels [18, 9].
The geometric resolution with 0.5-1.0 m is worse than for aerial imagery, but
for the land cover categorization, it is sufficient [1, 5]. The worldwide
availability of the data makes it possible to produce topographic databases
for nearly any region of the earth. It, in turn, can help in various
industries to enhance their productivity and quality of work whether it is for
military purposes of disaster prevention or relief.

In the last years, different methods have been proposed to tackle the problem
of creating convolutional neural networks (CNN) that can produce a
segmentation map for an entire input image in a single forward pass. One of
the most successful state-of-the-art deep learning method is based on the
Fully Convolutional Networks (FCN) [17]. The main idea of this approach is to
use CNN as a powerful feature extractor by replacing the fully connected
layers by convolution one to output spatial feature maps instead of
classification scores. Those maps are further upsampled to produce dense
pixel-wise output. This method allows training CNN in the end to end manner
for segmentation with input images of arbitrary sizes. Moreover, this approach
achieved an improvement in segmentation accuracy over common methods on
standard datasets like PASCAL VOC [6].

FCN has been further improved and now known as U-Net and Feature Pyramid (FPN)
neural networks [19, 14, 13]. We approach the problem of multi-class land
segmentation using FPN. FPN uses a pyramidal hierarchy of deep convolutional
networks to construct feature pyramids with marginal extra cost. It consisted
of bottom-up and top-down pathways. For the bottom-up feature encoder we have
chosen to use ResNet50 [8] pre-trained on ImageNet. A top-down pathway with
lateral connections is developed for building high-level semantic feature maps
at all scales. This architecture shows significant improvement as a generic
feature extractor in several applications such as object detection and
instance object segmentation [15, 7, 21, 20]. Specific applications of
convolutional networks on remote sensing image segmentation have been recently
studied [10, 11, 16, 23, 4]. These applications are based mostly on aerial
imagery data and contain pretty simple network architectures. In this work, we
generalize many ideas and provide a neural network for semantic land
segmentation that can work with high-resolution satellite imagery and approach
problem of multi-class segmentation.

##  2 Dataset

The training data for land cover challenge contains 803 satellite imagery in
RGB format. Each image has a size of 2448x2448 pixels. These images have 50cm
pixel resolution, collected by DigitalGlobe’s satellite [2, 5]. Moreover, each
image in the training dataset contains a paired mask for land cover
annotations. The mask is given as a RGB image with 7 classes of labels, using
color-coding (R, G, B) as follows: 1) Urban land: (0, 255, 255) - man-made,
built up areas with human artifacts (without roads); 2) Agriculture land:
(255, 255, 0) - farms, any plantation, cropland, orchards, vineyards,
nurseries, and ornamental horticultural areas; confined feeding operations; 3)
Rangeland: (255, 0, 255) - any non-forest, non-farm, green land, grass; 4)
Forest land: (0, 255, 0) - any land with certain tree crown density plus
clear-cuts; 5) Water: (0, 0, 255) - rivers, oceans, lakes, wetland, ponds; 6)
Barren land: (255, 255, 255) - mountain, land, rock, dessert, beach, no
vegetation; 7) Unknown: (0, 0, 0) - clouds and others. A satellite image and
corresponding mask from the training dataset is shown in Fig.1.

It is worth to mention that the values of the mask image may not be pure 0 and
255. As a result, the recommended threshold for binarization is 128 for each
label. Moreover, the labels are not perfect due to the high cost of annotating
multi-class segmentation mask. In addition, small objects are not annotated
consciously. To measure the performance of our model we also provided by 171
validation images that do not contain masks. The predicted masks for the
validation images should be upload to deepglobe website [2].

Figure 1: A satellite image (left) taken from the dataset and corresponding
multi-channel mask (right). Each color on the mask indicate different class of
objects.

##  3 Model

Objects segmentation in different scales is challenging in particular for
small objects. For this problem, we use feature pyramid network (FPN) to
implement land segmentation. The general scheme for FPN is shown in Fig. 2.
FPN composes of a bottom-up and top-down pathways. The bottom-up pathway is
the typical convolutional network for feature extraction. In our particular
case, we have chosen to use pre-trained ResNet50 as a feature encoder. It
composes of many convolution modules ($conv_{i}$ for $i$ equals 1 to 5) each
has many convolution layers. As we move up, the spatial dimension is reduced
by 1/2 (i.e., double the stride). The output of each convolution module is
labeled as $C_{i}$ and later used in the top-down pathway.

As we go up, the spatial resolution decreases. With more high-level structures
extracted, the semantic value for each layer increases. On the top of the
bottom-up pathway, we apply a 1x1 convolution filter to reduce $C_{5}$ channel
depth to 256 to create $M_{5}$ module. Then we apply successively two times
3x3 convolution to create $P_{5}$ which becomes the first feature map layer
used for object segmentation. $P_{5}$ has the same spatial resolution as
$conv_{5}$ and 128 channels. As we go down in the top-down path, we upsample
the previous layer by 2 using nearest neighbors upsampling. We again apply a
1x1 convolution to the corresponding feature maps in the bottom-up pathway.
Then we add them element-wise. After that, we apply successively two times 3x3
convolution to output the next feature map layers for object segmentation.
This filter reduces the aliasing effect of upsampling.

In the final step, we concatenate all $P_{i}$ modules that have 1/4 of the
input image resolution and 128 channels each. As a result, we have a module
with 512 channels. Then, we apply 512 3x3 convolution filters, batch
normalization, and ReLU activation. To reduce over-fitting we apply spatial
dropout operation [22]. As a next step, we reduce the number of output
channels utilizing 1x1 convolutions. In this problem, we have seven classes so
that the output feature map has seven channels. Then up-sample output to the
original image size using bi-linear interpolation.

Figure 2: Feature pyramid network with Resnet50 encoder pre-trained on
ImageNet. As an input, we have an RGB image. The number of channels increases
stage by stage on the left part of the scheme while the size of the feature
maps decreases stage by stage. The arrows on top show transformations
implemented between the layers. In the final step, feature maps upsample to
the same size and concatenated. Then, the number of channels decreases to the
number of classes, and the resulting image is upsampled to the original image
size.

##  4 Training

As the first step in training, we prepared masks as seven channel images using
one hot encoding. Then, as the evaluation metric, we use Jaccard index
(Intersection Over Union or IoU). It can be interpreted as a similarity
measure between a finite number of sets. For two sets $A$ and $B$, it can be
defined as following:

$J(A,B)=\frac{|A\cap B|}{|A\cup B|}=\frac{|A\cap B|}{|A|+|B|-|A\cap B|}$ (1)

Since an image consists of pixels, the last expression can be adapted for
discrete objects in the following way:

$J=\frac{1}{n}\sum_{c=1}^{7}w_{c}\sum\limits_{i=1}^{n}\left(\frac{y_{i}^{c}\hat{y}^{c}_{i}}{y_{i}^{c}+\hat{y}^{c}_{i}-y_{i}^{c}\hat{y}_{i}^{c}}\right)$
(2)

where $y_{i}^{c}$ and $\hat{y}_{i}^{c}$ are a binary values (label) and
corresponding predicted probability for the pixel $i$ of the class $c$. In
addition, we introduce the class weights $w_{c}$ that could help us to prevent
difficulties with classes imbalance. For simplicity, in this problem we set
$w_{c}=1$ for $c\in[1,\dots 7]$.

Since image segmentation task can also be considered as a pixel classification
problem, we additionally use common classification loss functions, denoted as
$H$ that for a multi-class segmentation problem is a categorical cross
entropy.

The final expression for the generalized loss function is obtained by
combining Eq. (2) and $H$ as following:

$L=\alpha H+\beta(1-J)$ (3)

By minimizing this loss function, we simultaneously maximize probabilities for
right pixels to be predicted and maximize the intersection $J$ between masks
and corresponding predictions. For the land classification problem, using
validation technique, we found $\alpha=1$ and $\beta=0.5$ that provided the
best performance.

For training our network, we split our dataset using 1/4 hold out values for
validation. Then, on the fly, we make several augmentations to increase the
diversity of our data artificially. For spatial augmentation, we use scale
transformations $0.6-1.4$ of the original image and mask. Then, we randomly
rotate the image and mask by 30 degrees. From the resulting image and mask, we
take random crops with size 448x448 pixels. These images are subject to color
transformation such as random contrast/brightness/HSV. One video card
GTX1080$Ti$ with 11 Gb of memory allows using the batch size of 8 images.

We train our network using Adam optimizer with learning rate 1e-4 and decay
1e-4 [12]. The training is done for 20k iterations (batches) saving weights
from several best iterations. Since the dataset is fairly limited in its size
and labels of train images are not robust the predicted value for IoU is
varied significantly between iterations as well as between classes. To reduce
the effect of over-fitting, we use spatial dropout operation on the output of
our network with $p=0.5$ [22]. After that, the training and validation IoU
became close to each other at value $0.55$ with a standard deviation around
$0.13$ between classes and standard deviation $0.03$ between iterations.

We made predictions on the whole image with 2448x2448 pixels padding it by 8
pixels so that the side is divisible by $32=2^{5}$. It helped to prevent
artifacts related to the bottom-up pathway of the network. To improve the
robustness of our predictions, we also implemented test time augmentation
(TTA) that composed of averaging of 4 predictions that correspond to 90
degrees rotation each.

##  5 Conclusions

We developed multi-class land segmentation algorithms using feature pyramid
network with ResNet50 network pre-trained on Imagenet in the bottom-up pathway
and a neatly designed loss function. The main difficulty in this multi-class
problem come from inaccurate labeling of classes in the training dataset. To
prevent over-fitting, we used pretty strong spatial dropout on the last layer
of the network as well as test time augmentation technique. The best public
score of our model on the public leaderboard is 0.493.

## Acknowledgment

The authors would like to thank Open Data Science community [3] for many
valuable discussions and educational help in the growing field of machine/deep
learning.

## References

  * [1] https://spacenetchallenge.github.io/.
  * [2] http://deepglobe.org/.
  * [3] http://ods.ai/.
  * [4] K. Chen, K. Fu, M. Yan, X. Gao, X. Sun, and X. Wei.  Semantic segmentation of aerial images with shuffling convolutional neural networks.  IEEE Geoscience and Remote Sensing Letters, 15(2):173–177, 2018\.
  * [5] I. Demir, K. Koperski, D. Lindenbaum, G. Pang, J. Huang, S. Basu, F. Hughes, D. Tuia, and R. Raskar.  Deepglobe 2018: A challenge to parse the earth through satellite images.  arXiv preprint arXiv:1805.06561, 2018.
  * [6] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [7] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] V. Iglovikov, S. Mushinskiy, and V. Osin.  Satellite imagery feature detection using deep convolutional neural network: A kaggle competition.  arXiv preprint arXiv:1706.06169, 2017.
  * [10] V. Iglovikov and A. Shvets.  Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation.  arXiv preprint arXiv:1801.05746, 2018.
  * [11] V. I. Iglovikov, S. Seferbekov, A. V. Buslaev, and A. Shvets.  Ternausnetv2: Fully convolutional network for instance segmentation.  arXiv preprint arXiv:1806.00844, 2018.
  * [12] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [13] A. Kirillov, K. He, R. Girshick, and P. Dollár.  A unified architecture for instance and semantic segmentation.  http://presentations.cocodataset.org/COCO17-Stuff-FAIR.pdf.
  * [14] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In CVPR, volume 1, page 4, 2017.
  * [15] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  arXiv preprint arXiv:1708.02002, 2017.
  * [16] Y. Liu, D. Minh Nguyen, N. Deligiannis, W. Ding, and A. Munteanu.  Hourglass-shapenetwork based semantic segmentation for high resolution aerial imagery.  Remote Sensing, 9(6):522, 2017.
  * [17] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [18] M. Pesaresi and J. A. Benediktsson.  A new approach for the morphological segmentation of high-resolution satellite imagery.  IEEE transactions on Geoscience and Remote Sensing, 39(2):309–320, 2001.
  * [19] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241, 2015.
  * [20] A. Shvets, V. Iglovikov, A. Rakhlin, and A. A. Kalinin.  Angiodysplasia detection and localization using deep convolutional neural networks.  arXiv preprint arXiv:1804.08024, 2018.
  * [21] A. Shvets, A. Rakhlin, A. A. Kalinin, and V. Iglovikov.  Automatic instrument segmentation in robot-assisted surgery using deep learning.  arXiv preprint arXiv:1803.01207, 2018.
  * [22] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler.  Efficient object localization using convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 648–656, 2015.
  * [23] M. Volpi and D. Tuia.  Dense semantic labeling of subdecimeter resolution images with convolutional neural networks.  IEEE Transactions on Geoscience and Remote Sensing, 55(2):881–893, 2017.