# Hamming Distributions of Popular Perceptual Hashing Techniques

Sean McKeown  William J. Buchanan

###### Abstract

Content-based file matching has been widely deployed for decades, largely for
the detection of sources of copyright infringement, extremist materials, and
abusive sexual media. Perceptual hashes, such as Microsoft’s PhotoDNA, are one
automated mechanism for facilitating detection, allowing for machines to
approximately match visual features of an image or video in a robust manner.
However, there does not appear to be much public evaluation of such
approaches, particularly when it comes to how effective they are against
content-preserving modifications to media files. In this paper we present a
million-image scale evaluation of several perceptual hashing archetypes for
popular algorithms (including Facebook’s PDQ, Apple’s Neuralhash, and the
popular pHash library) against seven image variants. The focal point is the
distribution of Hamming distance scores between both unrelated images and
image variants to better understand the problems faced by each approach.

###### keywords:

Perceptual Hashing, Fuzzy Hashing, Hash Matching, CSAM, Image Forensics

††journal: DFRWS EU 2023

[inst1]organization=School of Computing, Engineering, and the Built
Environment,addressline=Edinburgh Napier University, city=Edinburgh,
country=UK

##  1 Introduction

High-speed broadband has facilitated a transition from a text-based Internet
to one with a media-heavy landscape, which is capitalised upon by platforms
such as Facebook, YouTube, Instagram, TikTok, and others. Unfortunately, as
with any technology, multimedia can be used for illegal and abusive purposes,
necessitating that law enforcement agencies, social media platforms, cloud
providers, and so on, engage in some form of content moderation. At Web scale
the manual evaluation of multimedia becomes infeasible, resulting in the use
of automatic detection technologies. Content-based detection is implemented
for a variety of reasons, such as for the detection of extremist and violent
material [1], copyright infringing material [2], and the detection of Child
Sexual Abuse Material (CSAM) [3]. The CSAM use-case is particularly important,
as the Internet Watch Foundation reports a sharp increase in the detection and
distribution of CSAM media - taking action against 250,000 websites in 2021
alone [4]. Content matching technologies have already been deployed for many
cloud-based services, backed by Microsoft’s PhotoDNA [5] and Facebook’s PDQ
[6], with Apple looking to introduce client-side device scanning [7] to
further combat the growing problem.

For large-scale deployment, it is imperative that these detection technologies
are well understood, both in terms of their strengths and weaknesses, in order
to avoid improper configuration/deployment and incorrect actionable
intelligence. The primary contribution of this work is an analysis of
contemporary algorithms in still-image content-based perceptual hash matching,
with analysis of their robustness in the face of content-preserving attacks,
with experiments being conducted at the million-image scale.

##  2 Background and Related Work

Content-based file matching can take a variety of forms, with the approach
archetypes corresponding to: i) Exact matching with Cryptographic hashes; ii)
Approximate similarity binary-level matching; and iii) Semantic approximate
matching of file contents [8].

In each case, a hash digest is created which serves as a fingerprint/signature
for the file, which can then be compared to the hashes of other files to
detect identical or similar file content. Cryptographic approaches are
commonly used in CSAM detection in digital forensics analysis, however, they
are easily defeated by modifying a single-bit in a file. Approximate matching
at the binary level is suitable for some file types, but is generally a poor
choice for media files as simply changing encoding parameters will result in
completely different binary content [9]. We, therefore, choose to focus on the
semantic domain, which, for still-images, is the matching of visually similar
content (as opposed to similarly sounding audio, or similar semantic content
in text). For visual content, semantic domain matching is referred to as
Perceptual Image Matching.

###  2.1 Perceptual Image Matching

Perceptual hashing [9, 10] approaches are inspired by the domain of Content-
Based Image Retrieval (CBIR) [11], with the goal of providing the ability to
measure visual image similarity and leverage this to return matches for
visually identical, or similar, images. Ideally, this process should be robust
to content-preserving attacks, such as adding noise, filters, cropping, and so
on, but also to some degree, content modifying manipulations, perhaps
involving the addition or removal of objects.

Hadmi et al. [10] identified a typical pipeline for the creation of such
robust hashes: _(i)_ Transformation, e.g., spatial transformations, or
transformation to another domain, such as the frequency domain, with Discrete
Cosine Transform (DCT) and Discrete Wavelet Transform (DWT) commonly being
used; _(ii)_ Feature Vector Extraction; _(iii)_ Quantization and Feature
Reduction; and _(iv)_ Compression or Encryption to generate a fixed-length
hash.

These hashes are then compared using a variety of similarity metrics (Hamming
distance, Euclidean distance, Earthmover distance, L2 distance, and so on),
typically normalised to a value between zero and unity, with unity
representing complete similarity.

A wide array of features have been used as the basis for generating robust
image signatures. Histograms and statistical information about the entire
image may be used, providing a high-level representation which is insensitive
to small, localised, changes in the image. This can take the form of colour
histograms [12], texture and edge histograms [13], or frequency domain
statistics [14]. Properties of human vision may also be exploited, such as
insensitivity to high-frequency changes in an image over a small area - a
property which is exploited by JPEG compression [15]. Low frequency properties
of an image may be used to derive a perceptual hash [16, 17], which provides
robustness to compression artefacts and other content-preserving
modifications. In a similar vein, coarse image representations may be used,
such as low-resolution versions of the image, or the average colour value of
sub-blocks in the image [18]. Alternatively, invariant relationships in the
image may be exploited, such as those found on radial lines projected out from
the centre of the image [19], which is particularly effective against re-
scaled images.

It should also be noted that there is a distinction between hashes which are
generated using a _shallow_ , heuristic, approach and those derived from a
deep learning, data-driven, approach [20]. In the former case, hashes are
generated in a deterministic manner from a set of operations (such as
statistical features of the image), while the learning approaches will
generate different hashes based on the dataset used to train the corresponding
model.

###  2.2 Evaluating Perceptual Image Matching Against Attack

Despite the popularity and wide spread use of perceptual hashing, the
robustness of these algorithms is not well understood for malicious and
abusive content detection [21], as adversaries can produce image variants that
may hinder detection and identification.

One of the most popular, and easily available, perceptual hashing algorithms
is pHash (DCT-based) [22]. This hashing method is available in the Python
ImageHash library [23], and is often evaluated together with the other
algorithms in the library: ahash (average block colour); dhash (adjacent block
colour); and whash (DWT-based) [20, 21, 24, 25, 26].

Hamadouche et al. [24] studied the ImageHash algorithms together with SVD-Hash
(Singular Value Decomposition) against a variety of image-wide filters, noise,
and scaling for a small dataset of 800 images. While ahash typically produced
the smallest distances between original and modification, pHash and dhash were
the only algorithms to produce normally distributed Hamming distances - a
property which will be discussed further in Section 3.3. Jain et al. [26]
performed more sophisticated perturbation attacks against ImageHash algorithms
and Facebook’s DCT-based PDQ on over one million images. The authors noted
that by using black-box attacks, it was possible to manipulate modified image
distances to the extent that the False Positive Rate would become unacceptably
large in all cases. Similar work by Hao et al. [21] for ImageHash and
Blockhash (average block colour) allowed for large distances to be achieved
for each algorithm using noise, cropping, rotation and scaling attacks. Dmric
et al. [25] evaluated the ImageHash library, with 1,480 images, against user-
level modifications, such as resizing, rotation, adding borders, etc.,
together with social media post-processing tests. pHash generated the best F1
score (weighted combination of precision/recall) when aggregated across
attacks, with social media manipulations disturbing performance less than the
more direct manipulations.

McKeown and Russell [27] explored the use of pHash and Blockhash for matching
originals to thumbnail cache entries in Windows Vista, 7, and 10, noting that
neither algorithm could achieve an acceptable False Positive Rate to False
Negative Rate trade-off for forensics purposes. However, the combination of
both algorithms may be sufficient. The authors also noted particular
weaknesses in the algorithms, such as fractal/patterned images for pHash and
solid colour background/smooth gradients for Blockhash.

Looking more widely, Zauner [22] and Breitinger et al. [8] compared DCT, Block
Mean Based, Radial (e.g. Radon projection) and Marr-Hildreth (MH) operator
hashes. Particular attention was paid to the strengths and weaknesses of each
algorithm for each modification attack. JPEG compression did not have much
effect on any tested algorithm, while resizing only particularly affected MH
(most algorithms downscale images in pre-processing). Mirroring, rotating, and
cropping all have much larger effects on performance across attacks, except
when algorithms build in specific handling (such as for rotation in rHash
[18]).

Machine learning approaches have also been shown to be susceptible to trivial
content-preserving manipulations, with Struppek et al. [28] demonstrating that
Apple’s Neuralhash is not robust to gradient and familiar transformation-based
attacks, with hash collision attacks also being possible. Dohlansky et al.
[20] found similar results for AlexNet, ResNet, and EfficientNet, although the
authors noted that cross-attacks between shallow and deep approaches do not
work well, such that combining algorithms from each class may lead to
additional robustness against attack.

##  3 Methodology

While there is a body of existing work in the evaluation of perceptual
hashing, there is generally a focus on aggregating statistics across multiple
attack types, or in generating specific attack scenarios, rather than
understanding the behavioural properties of the approaches against common
user-level attacks. With a similar approach to Hamadouche et al. [24], we set
out to understand the Hamming distance distribution of perceptual hashing
algorithms in order to better quantify behaviour for not only the
aggregate/average case, but also the best and worst case scenarios. To achieve
this, we make use of the Flickr 1 Million dataset [29], removing SHA256 hash
duplicates, which allows us to scale experiments to large numbers of natural
images. The original hashes, Hamming scores, and additional statistical data
used in this paper are available online with DOI: 10.5281/zenodo.7426035.

###  3.1 Selection of Hashing Algorithms

Our perceptual hash selection takes into account a wide range of feature
extraction techniques, but also focuses on popular algorithms. The chosen hash
algorithms are listed in Table 1. The list covers: frequency transforms;
colour histograms; block mean colour; and deep learning-derived approaches.
Aside from additional deep learning approaches, one notable omission of the
shallow approach to hashing in this testing is Microsoft’s PhotoDNA [5] as it
is a controlled technology and not readily available. While such evaluations
may already be available to those with access, it would be beneficial for
future work to make evaluations open-source, particularly as it is the de-
facto standard for the perceptual detection of CSAM [30]. The details of its
implementation are not widely detailed, however it appears that PhotoDNA uses
an edge-based method [31], such that it is unclear what particular weaknesses
it may share with the algorithms presented in this paper. Another possible
algorithm class to include in future testing is the Scale Invariant Feature
Transform (SIFT), a feature-point based approach, which is more commonly used
for image forgery detection [32]. ForBild/rHash [18] was also considered,
though it was excluded early on as the sourced implementation generated many
hash collisions due to black backgrounds, or similar colour compositions.

Blockhash |  An implementation of the Block Mean Based approach [33]. It can be found, written in C, on the Commons Machinery Github [34]. Generates 256-bit hashes.
---|---
ColourHash |  Part of the Python ImageHash Library [23]. Images are matched based on colour distributions. Generates 44-bit hashes.
NeuralHash (Apple) |  Apple’s CSAM hashing scheme. This is a machine-learned approached, with the neuralhash_128x96_seed1.dat model being extracted from an iPhone14,5_15.3.1_19D52 IPSW firmware image. The model extraction process and code for generating hashes, can be found on Github [35]. Generates 96-bit hashes.
PDQ (Facebook) |  Facebook’s improved version of the pHash (DCT) algorithm, with optimisations for downsampling, and larger hash output size by default (for better Web-scale accuracy). The Python implementation - available on Facebook’s Threat Exchange Github [6] \- was used. Generates 256-bit hashes.
pHash |  DCT-based, as implemented in the ImageHash Python library [23]. Generates 64-bit hashes.
Wavehash |  A hash based on the Discrete Wavelet Transform (DWT) (as opposed to the DCT based approaches above). Also part of the Python ImageHash library [23]. Generates 64-bit hashes.
Table 1: Perceptual Hashes used in experiments.

###  3.2 Image Modifications

A set of six image modifications were chosen as _attacks_ against the
perceptual hashing algorithms. Each modification was implemented using the
Python PIL library [36]. This means that all images were modified using the
same tool, introducing some limitations in terms of the diversity of the
images produced, but we do not expect this to have a significant effect on the
outcomes.

To reduce the total processing time across all hash and modification
permutations, a random subset of 250,000 images was selected for the creation
of modifications. Table 2 describes the chosen modifications, which are
intended to reflect the low-barrier to entry, unsophisticated, content-
preserving black-box attacks that a typical user may employ which do not
compromise viewability (such as for evading copyright detection [37]), with
the addition of Windows 10 generated thumbnails as a realistic downscaling
exemplar. Examples of the visually distinct modifications are provided in
Figure 1.

Border (30px) |  Add a fixed size 30-pixel black border to the outside of the image, extending it, rather than overwriting any of the existing content.
---|---
Compression (Q30) |  Reduce the JPEG image quality to 30%, scaling down the default quantization tables. This simulates lower-quality images (but not lower resolution), which can affect perceptual hashing performance. All other modifications passed the quantization tables through from the original without manipulation, which is on average 96% quality for images in the Flickr 1 Million dataset.
Crop (5%) |  Remove 5% of the image from the top, left, right, and bottom of the image, in effect reducing the overall pixel count to 81% of the original ($0.9height\times 0.9width$ ).
Mirror (x-axis) |  Flip the image on its x-axis, preserving viewability, but generating significant pixel/binary level changes.
Scale (1.5x) |  Scale the image up to 150% the size of the original, using the PIL [36] Image.resize function.
Thumbs96 (Windows) |  Generate legitimate Windows 10 $96\times 96$ pixel thumbnails as per the methodology in McKeown and Russel [27], which generate a larger Hamming distance in this cited work for pHash and Blockhash than their $256\times 256$ thumbnail counterpart.).
Watermark |  Add a watermark to the bottom right of the image. The watermark consists of a logo, text and a URL. The watermark was scaled to 10% of the image height, with a minimum of 40 pixels.
Table 2: Image modifications applied to each image in the dataset. Figure 1:
Visibly distinct modification examples for 50.jpg in the dataset.

###  3.3 Evaluation Process and Criteria

There are several things to consider when evaluating perceptual hashing
systems. The first is the relative discrimination behaviour between unrelated
images (inter-image). When measuring the distance between two hashes (usually
via normalised Hamming distance), unrelated images should be essentially
tossing a coin for each bit, resulting in a distance of around 0.5, on average
(mean and median). Additionally, we should expect that very close hashes, and
particularly exact matches, should not occur in large groups, and the
occurrences should be reasonable and expected, that is, highly similar images.

In order to characterise inter-score  behaviour for the original (unmodified)
images, each image in the Flickr 1 Million dataset was compared to a random
selection of 50 other images, for a total of approximately 50 million
comparisons111There are a possible total of 500 billion comparisons
($\frac{1million\times 1million}{2}$) for 1 million images, which is
unnecessary to acquire the understanding we need here. to generate statistics
from. To better understand the impact of the various image modifications on
the behaviour of each hash, inter-scores were also analysed for each of the
seven modifications, resulting in 12.5 million additional samples per
modification for each hashing algorithm. Results for these comparisons (via
normalised Hamming distances) are reported in Section 4.1.

The second main characteristic is the performance of a perceptual hashing
algorithm when comparing an image to its variant. This second image could be a
pixel perfect copy, a re-encoded version of the image into a different file
format, a thumbnail, cropped manipulation, etc. Ideally the perceptual hashing
algorithm generates the same, or similar, hash for each of these images, such
that the Hamming distance is zero, or very small. The distance distributions
of inter- and intra-scores for each algorithm then allow for a wider
understanding of potential false positive and false negative rates for a given
threshold distance, as discussed in prior work [21, 27].

We explore the intra-score characteristics of the various algorithms and image
modifications in Section 4.2. Each modified image in the dataset was hashed
and compared to its original version, generating 250,000 Hamming distance
comparisons for each hash algorithm/modification pair. Results present not
only measures of distribution (range, mean, median, and standard deviation),
but also exact hash match percentages to demonstrate particularly strong
matching performance.

##  4 Findings

###  4.1 Inter-score (Different Images)

The findings for inter-image distributions (i.e., between unrelated images)
are presented first, with Section 4.1.1 describing each algorithm’s behaviour
for the original Flickr 1 Million image set. This essentially acts as a
baseline for each algorithm’s behaviour when images are not expected to match.
Section 4.1.2 compares images in each modification class (e.g., cropped to
cropped) in order to determine if there is any bias introduced by the
modifications themselves, even if the images are still assumed to be
unrelated. This is followed by Section 4.2 which moves on to examine the
impact that the modifications have when comparing the same image to versions
of itself, which provides a different window into the biases and difficulties
produced by the modified versions of the images.

hash algorithm | mean | median | stdev | range
---|---|---|---|---
blockhash | 0.4923 | 0.4922 | 0.0785 | 0.0078–1.0000
colourhash | 0.1601 | 0.1591 | 0.0547 | 0.0000–0.3636
neuralhash | 0.4973 | 0.5000 | 0.0600 | 0.0521–0.7917
pdq | 0.5000 | 0.5000 | 0.0321 | 0.0391–0.6953
phash | 0.4904 | 0.5000 | 0.0649 | 0.0938–0.8438
wavehash | 0.4854 | 0.5000 | 0.1241 | 0.0000–1.0000
Table 3: Inter-score normalised Hamming distances between random images in the
Flickr 1 Million dataset. Each image was compared to 50 random images. The
ideal value is 0.5 distance to make best use of the Hamming space.

Figure 2: Normalised Hamming inter-scores with Normal Probability Density
Function (NPDF) (50 million pair-wise samples) for the original Flickr 1
Million dataset. a) Blockhash, b) Colourhash, C) NeuralHash, d) PDQ, e) pHash,
f) Wavehash

####  4.1.1 Original to Original

The results for the inter-score comparisons between original images in the
dataset, for each hash, are presented in Table 3, with distributions depicted
in Figure 2.

While almost all of the algorithms do well to centre inter-image scores around
0.5, which is clearly a design goal (particularly achieved by PDQ), Colourhash
behaves quite differently. Colourhash has much lower scores for unrelated
images, which is not necessarily detrimental in itself, however, it does mean
that it makes less uniform use of the Hamming space. This will result in a
lack of diversity, and more hash collisions due to chance.

When comparing the equivalence classes (distance of zero), the shortcomings of
Colourhash are clear, with three equivalence classes containing over 20,000
images each, with many more classes with members in the thousands, or tens of
thousands, making it functionality indiscriminate. Wavehash, despite its
reasonable mean/median scores, produced a single class of 5,527 items, and
several classes in the hundreds, again making it a poor discriminator at
scale. Of the remaining algorithms, the only one which had a class size higher
than size 5 is Blockhash, with aberrant classes of size 52 and 54.

In terms of their distributions, the algorithms which use lower hash sizes by
default generate sparse distributions, with some distance intervals being
unused, essentially quantizing the output across the Hamming space. This is
particularly clear for Colourhash, pHash and Wavehash, where the sparseness
creates a large mismatch with the Normal Probability Density Function (NPDF)
in Figure 2. NeuralHash has an intermediate hash length, but fits the NPDF
much better, despite a notable gap in the centre of the distribution. The
spikes produced by Blockhash and PDQ, which both use large hashes, are
therefore likely algorithmic, making certain hash codes and distances more
likely to occur by chance.

Of the hashes which conform to 0.5 as the median/mean, Wavehash has the widest
distribution (highest standard deviation), with tails making it reasonably
likely that it generates a zero distance match by chance at any reasonable
scale, as is evident from the previously noted equivalence classes of over
20,000 items. This makes it unsuitable at large scale, even prior to attack.
PDQ, on the other hand, has a very tight distribution, which is likely by
design for scaling purposes to greatly reduce false positives.

####  4.1.2 Modified to Modified

From the equivalence class analysis alone, the Colourhash and Wavehash
algorithms should be discounted as potential solutions. However, Wavehash
remains in the following tables to provide insight into how a non-DCT based
frequency domain transform is impacted by the various modifications.

Table 4 represents inter-score distances for each algorithm where the images
in each modification class are compared to each other (e.g. cropped-to-
cropped). The perceptual hashes largely remain stable across the modified
image comparisons, though adding in repeated content in the form of a border
has had a larger than expected impact for some hashes. Blockhash, pHash and
Wavehash all have sizable reductions in mean distance between unrelated images
as a result of the shared border. This means that they are more likely to
match images with borders to each other, regardless of the content residing
within the border. NeuralHash appears to be the only algorithm which is not
particularly affected by the addition of a border. The watermark has a similar
effect, though it is not nearly as pronounced.

hash algorithm | modification | mean | median | stdev | range
---|---|---|---|---|---
blockhash | border | 0.3284 | 0.3281 | 0.0549 | 0.0391–0.8047
blockhash | compression | 0.4922 | 0.4922 | 0.0782 | 0.0391–0.9805
blockhash | crop | 0.4941 | 0.4922 | 0.0761 | 0.0234–0.9766
blockhash | mirror | 0.4921 | 0.4922 | 0.0784 | 0.0312–0.9609
blockhash | scale | 0.4922 | 0.4922 | 0.0783 | 0.0391–0.9492
blockhash | thumbs96 | 0.4921 | 0.4922 | 0.0787 | 0.0234–0.9609
blockhash | watermark | 0.4789 | 0.4766 | 0.0758 | 0.0391–0.9531
neuralhash | border | 0.4969 | 0.5000 | 0.0601 | 0.0000–0.7812
neuralhash | compression | 0.4974 | 0.5000 | 0.0600 | 0.0729–0.7812
neuralhash | crop | 0.4976 | 0.5000 | 0.0599 | 0.0833–0.7812
neuralhash | mirror | 0.4974 | 0.5000 | 0.0600 | 0.0521–0.8021
neuralhash | scale | 0.4973 | 0.5000 | 0.0600 | 0.0417–0.7812
neuralhash | thumbs96 | 0.4969 | 0.5000 | 0.0602 | 0.0312–0.7812
neuralhash | watermark | 0.4969 | 0.5000 | 0.0602 | 0.0521–0.7812
pdq | border | 0.4794 | 0.4766 | 0.0363 | 0.2109–0.6719
pdq | compression | 0.5000 | 0.5000 | 0.0321 | 0.2500–0.6797
pdq | crop | 0.5000 | 0.5000 | 0.0318 | 0.2891–0.7109
pdq | mirror | 0.5000 | 0.5000 | 0.0321 | 0.2969–0.6797
pdq | scale | 0.5000 | 0.5000 | 0.0321 | 0.2812–0.7188
pdq | thumbs96 | 0.5000 | 0.5000 | 0.0321 | 0.2812–0.7031
pdq | watermark | 0.4941 | 0.4922 | 0.0333 | 0.2031–0.7344
phash | border | 0.4283 | 0.4375 | 0.0727 | 0.0625–0.8438
phash | compression | 0.4904 | 0.5000 | 0.0649 | 0.1562–0.8438
phash | crop | 0.4907 | 0.5000 | 0.0641 | 0.0625–0.8125
phash | mirror | 0.4904 | 0.5000 | 0.0649 | 0.1562–0.8438
phash | scale | 0.4904 | 0.5000 | 0.0649 | 0.1250–0.8438
phash | thumbs96 | 0.4904 | 0.5000 | 0.0650 | 0.1250–0.8125
phash | watermark | 0.4704 | 0.4688 | 0.0674 | 0.0000–0.8125
wavehash | border | 0.3013 | 0.2812 | 0.1167 | 0.0000–0.9688
wavehash | compression | 0.4854 | 0.5000 | 0.1240 | 0.0000–1.0000
wavehash | crop | 0.4869 | 0.5000 | 0.1212 | 0.0000–1.0000
wavehash | mirror | 0.4854 | 0.5000 | 0.1239 | 0.0000–1.0000
wavehash | scale | 0.4854 | 0.5000 | 0.1240 | 0.0000–1.0000
wavehash | thumbs96 | 0.4854 | 0.5000 | 0.1241 | 0.0000–1.0000
wavehash | watermark | 0.4631 | 0.4688 | 0.1174 | 0.0000–1.0000
Table 4: Inter-score normalised Hamming distances between random images within each modification category for the Flickr 1 Million dataset. Each image was compared to 50 random images (e.g. each cropped image was compared to 50 random cropped images for the same algorithm). The ideal mean/median is a distance of 0.5, demonstrating that the modification does not bias the algorithm. hash algorithm | modification | mean | median | stdev | range | % exact matches
---|---|---|---|---|---|---
blockhash | border | 0.2783 | 0.2734 | 0.0851 | 0.0000–0.7188 | 0.0000
blockhash | compression | 0.0095 | 0.0078 | 0.0142 | 0.0000–0.4492 | 36.4310
blockhash | crop | 0.1668 | 0.1641 | 0.0610 | 0.0000–0.7344 | 0.0180
blockhash | mirror | 0.4450 | 0.4531 | 0.1410 | 0.0000–1.0000 | 0.0460
blockhash | scale | 0.0013 | 0.0000 | 0.0040 | 0.0000–0.1719 | 85.4360
blockhash | thumbs96 | 0.0254 | 0.0234 | 0.0179 | 0.0000–0.4961 | 5.5890
blockhash | watermark | 0.0504 | 0.0469 | 0.0305 | 0.0000–0.3672 | 2.9450
neuralhash | border | 0.0763 | 0.0729 | 0.0381 | 0.0000–0.5312 | 0.3670
neuralhash | compression | 0.0082 | 0.0104 | 0.0105 | 0.0000–0.3333 | 49.1630
neuralhash | crop | 0.0605 | 0.0521 | 0.0332 | 0.0000–0.4688 | 1.1540
neuralhash | mirror | 0.2823 | 0.2812 | 0.1251 | 0.0000–0.6979 | 0.0960
neuralhash | scale | 0.0036 | 0.0000 | 0.0069 | 0.0000–0.1458 | 73.6020
neuralhash | thumbs96 | 0.0809 | 0.0729 | 0.0445 | 0.0000–0.4792 | 0.9690
neuralhash | watermark | 0.0551 | 0.0417 | 0.0424 | 0.0000–0.5729 | 3.6560
pdq | border | 0.3949 | 0.3984 | 0.0599 | 0.0547–0.7422 | 0.0000
pdq | compression | 0.0094 | 0.0078 | 0.0091 | 0.0000–0.4453 | 23.4350
pdq | crop | 0.3255 | 0.3281 | 0.0564 | 0.0234–0.6719 | 0.0000
pdq | mirror | 0.4975 | 0.5000 | 0.0226 | 0.0000–0.9844 | 0.0020
pdq | scale | 0.0237 | 0.0234 | 0.0132 | 0.0000–0.4609 | 1.6320
pdq | thumbs96 | 0.0721 | 0.0703 | 0.0281 | 0.0000–0.4844 | 0.0060
pdq | watermark | 0.1029 | 0.0938 | 0.0489 | 0.0000–0.5469 | 0.0020
phash | border | 0.2656 | 0.2500 | 0.0745 | 0.0000–0.6562 | 0.0000
phash | compression | 0.0053 | 0.0000 | 0.0138 | 0.0000–0.4688 | 83.9040
phash | crop | 0.1686 | 0.1562 | 0.0586 | 0.0000–0.6562 | 0.0430
phash | mirror | 0.4904 | 0.5000 | 0.0339 | 0.1562–0.6875 | 0.0000
phash | scale | 0.0020 | 0.0000 | 0.0091 | 0.0000–0.4688 | 94.0050
phash | thumbs96 | 0.0245 | 0.0312 | 0.0250 | 0.0000–0.5938 | 39.0830
phash | watermark | 0.1227 | 0.1250 | 0.0824 | 0.0000–0.6875 | 4.5350
wavehash | border | 0.2744 | 0.2500 | 0.1333 | 0.0000–0.8750 | 0.2570
wavehash | compression | 0.0029 | 0.0000 | 0.0106 | 0.0000–0.5312 | 91.4070
wavehash | crop | 0.1049 | 0.0938 | 0.0649 | 0.0000–0.8750 | 3.5740
wavehash | mirror | 0.3474 | 0.3125 | 0.1698 | 0.0000–1.0000 | 1.2670
wavehash | scale | 0.0007 | 0.0000 | 0.0052 | 0.0000–0.8438 | 97.9830
wavehash | thumbs96 | 0.0158 | 0.0000 | 0.0214 | 0.0000–0.5938 | 57.3560
wavehash | watermark | 0.0451 | 0.0312 | 0.0483 | 0.0000–0.4688 | 39.6030
Table 5: Intra-score normalised Hamming distances between the original image
and various modifications of the original for the Flickr 1 Million dataset.
The ideal distance value is 0, which is reflected in the ‘% matches’ column.
Bolded items are very good performances, while items in red are bad, to the
point of being as bad, or worse, than selecting a random, unrelated, image.

###  4.2 Intra-score (Versions of the Same Image)

Intra-score values, comparing the original image to its modifications, are
depicted in Table 5. These scores give us an opportunity to explore how well
the various techniques can deal with attempts to circumvent content-based
detection.

All algorithms stumble into very high Hamming distances at some point,
represented by the high maximum distance values, which simply suggests that
there is at least one image in the dataset that troubles each algorithm for
almost every modification. In some cases, the distance approaches one, which
may actually be worthy of considering to be a match, as given a normal
distribution around 0.5, a distance of one is just as likely as a distance of
zero. These outliers are often hidden when considering aggregate behaviour,
and they do not always occur, but particularly troublesome ranges are present
for _Blockhash-Mirror_ , _PDQ-Border_ , _PDQ-Mirror_ , and _pHash-Mirror_.

Most algorithms coped well with image scaling, likely because one of the main
steps in generating perceptual hashes is to downscale the image to a
manageable size/complexity. Most algorithms have a very high exact match
percentage for scaling222Curiously, the more nuanced downsampling approach
taken by PDQ seems to hinder it in this case., peaking at 97.9% with Wavehash,
meaning that only 2.1% of upscaled images would not have an exact hash match.
Interestingly, while thumbnail performance is often almost as good, it does
not produce anywhere near the same percentage of exact hash matches. All
algorithms also coped very well with poor quality JPEGs.

NeuralHash produces consistently good results, though it has a very wide
distribution (almost normal around 0.3) for mirroring attacks, when ideally
the distribution should be very long tailed with most distances around zero.
On paper, Wavehash also falls into this category, but the false positive rate
is likely too high for most use cases due to the inter-score distribution.
Blockhash, pHash and PDQ all perform well on average, however, each of them
have cases where the distribution is very wide due to a high standard
deviation (border, crop and mirror for all three, and watermark for PDQ and
pHash). Generally, mirroring an image seems to cause the most disturbance,
with many algorithms essentially distributing scores as if it was an unrelated
image. Borders appear to be second most impactful due to the introduction of
common content.

##  5 Conclusion and Future Work

In this work, we have explored several popular and widely used perceptual
hashing algorithms in order to understand the distributions of their Hamming
distances when discriminating between unrelated images, and versions of the
same image. The raw data for this paper are available online with DOI:
10.5281/zenodo.7426035. Of the tested algorithms, all but ColourHash and
Wavehash were shown to have reasonable inter-score distributions, resulting in
few false positives by complete chance. PDQ is particularly strong in this
regard. Surprisingly, adding a border or watermark is enough to substantially
throw off the distances between unrelated images for most algorithms, simply
due to a small portion of shared content being introduced, biasing them
towards similarity.

When comparing the images to various content-preserving modifications,
mirroring the image on the x-axis was found to be particularly destructive
across the board, with the addition of a bordering being a distant second.
While distances should be clustered around 0 in these intra-score cases,
distributions for unfavourable modifications often resulted in relatively wide
spreads, sometimes centred around 0.4 and 0.5. Additionally, most techniques
had at least one poorly handled image for each modification. Overall,
NeuralHash appears to be the least affected by the modifications, in addition
to having a desirable inter-score distribution, suggesting that deep, learned,
approaches may be strong against naive user-level attack.

The understanding generated above allows for further insights to be mined in
terms of setting appropriate thresholds (given some level of false
positive/false negative trade-off) [27], but also to allow us to consider
black-box mitigations against common problems in matching. Pre-processing
approaches for mirrored and bordered images would help reduce negative impacts
on scoring. This could be handled in a similar manner to how ForBild mitigates
its algorithmic weakness to rotated images, by rotating the image such that
the darkest corner is always in the same position [18].

The analysis above is also largely content-independent. While prior work has
acknowledged that there are types of image which cause issues with certain
algorithms [27, 38], these particular weaknesses could be better documented at
the archetype/algorithm level. This is particularly important for the impact
it may have on varying corpora, such as when comparing animated images, or
exported video frames using still image hashing formats.

## References

  * [1] Microsoft Corporate Blog, “The Christchurch Call and steps to tackle terrorist and violent extremist content,” May 2019\. [Online]. Available: https://blogs.microsoft.com/on-the-issues/2019/05/15/the-christchurch-call-and-steps-to-tackle-terrorist-and-violent-extremist-content/
  * [2] P. Saadatpanah, A. Shafahi, and T. Goldstein, “Adversarial attacks on copyright detection systems,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 8307–8315.
  * [3] H.-E. Lee, T. Ermakova, V. Ververis, and B. Fabian, “Detecting child sexual abuse material: A comprehensive survey,” _Forensic Science International: Digital Investigation_ , vol. 34, p. 301022, 2020.
  * [4] The Internet Watch Foundation, “IWF: The Annual Report 2021,” 2021. [Online]. Available: https://annualreport2021.iwf.org.uk/pdf/IWF-Annual-Report-2021.pdf
  * [5] N. Krawetz. Photodna and limitations. Not known. [Online]. Available: https://www.hackerfactor.com/blog/index.php?archives/931-PhotoDNA-and-Limitations.html
  * [6] Facebook. Pdq. [Online]. Available: https://github.com/facebook/ThreatExchange/tree/main/pdq/python
  * [7] Apple, “Csam detection technical summary,” https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf, 2022, [Online; accessed 2-Feb-2022].
  * [8] F. Breitinger, H. Liu, C. Winter, H. Baier, A. Rybalchenko, and M. Steinebach, “Towards a process model for hash functions in digital forensics,” in _International Conference on Digital Forensics and Cyber Crime_. Springer, 2013, pp. 170–186.
  * [9] H. Farid, “An Overview of Perceptual Hashing,” _Journal of Online Trust and Safety_ , vol. 1, no. 1, Oct. 2021.
  * [10] A. Hadmi, W. Puech, B. A. E. Said, and A. A. Ouahman, “Perceptual image hashing,” in _Watermarking-Volume 2_. INTECH Open Access Publisher, 2012.
  * [11] G. Rafiee, S. S. Dlay, and W. L. Woo, “A review of content-based image retrieval,” in _Communication Systems Networks and Digital Signal Processing (CSNDSP), 2010 7th International Symposium on_. IEEE, 2010, pp. 775–779.
  * [12] M. J. Swain and D. H. Ballard, “Color indexing,” _International journal of computer vision_ , vol. 7, no. 1, pp. 11–32, 1991.
  * [13] B. S. Manjunath, J.-R. Ohm, V. V. Vasudevan, and A. Yamada, “Color and texture descriptors,” _IEEE Transactions on circuits and systems for video technology_ , vol. 11, no. 6, pp. 703–715, 2001.
  * [14] R. Venkatesan, S.-M. Koon, M. H. Jakubowski, and P. Moulin, “Robust image hashing,” in _Image Processing, 2000. Proceedings. 2000 International Conference on_ , vol. 3. IEEE, 2000, pp. 664–666.
  * [15] G. K. Wallace, “The JPEG still picture compression standard,” _Consumer Electronics, IEEE Transactions on_ , vol. 38, no. 1, pp. xviii–xxxiv, 1992.
  * [16] J. Fridrich, “Robust bit extraction from images,” in _Multimedia Computing and Systems, 1999. IEEE International Conference on_ , vol. 2, Jul. 1999, pp. 536–540 vol.2.
  * [17] J. Fridrich and M. Goljan, “Robust hash functions for digital watermarking,” in _Information Technology: Coding and Computing, 2000. Proceedings. International Conference on_. IEEE, 2000, pp. 178–183.
  * [18] M. Steinebach, “Robust hashing for efficient forensic analysis of image sets,” in _International Conference on Digital Forensics and Cyber Crime_. Springer, 2011, pp. 180–187.
  * [19] F.-X. Standaert, E. Lefebvre, G. Rouvroy, B. Macq, J.-J. Quisquater, and J.-D. Legat, “Practical evaluation of a radial soft hash algorithm,” in _International Conference on Information Technology: Coding and Computing (ITCC’05)-Volume II_ , vol. 2. IEEE, 2005, pp. 89–94.
  * [20] B. Dolhansky and C. C. Ferrer, “Adversarial collision attacks on image hashing functions,” Nov. 2020, arXiv:2011.09473 [cs]. [Online]. Available: http://arxiv.org/abs/2011.09473
  * [21] Q. Hao, L. Luo, S. T. Jan, and G. Wang, “It’s Not What It Looks Like: Manipulating Perceptual Hashing based Applications,” in _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_. Virtual Event Republic of Korea: ACM, Nov. 2021, pp. 69–85.
  * [22] C. Zauner, “Implementation and benchmarking of perceptual image hash functions,” 2010.
  * [23] J. Buchner. Image hash. [Online]. Available: {https://github.com/JohannesBuchner/imagehash}
  * [24] M. Hamadouche, K. Zebbiche, M. Guerroumi, H. Tebbi, and Y. Zafoune, “A comparative study of perceptual hashing algorithms: Application on fingerprint images,” in _2nd International Conference on Computer Science’s Complex Systems and their Application_ , Algeria, May 2021, p. 12.
  * [25] A. Drmic, M. Silic, G. Delac, K. Vladimir, and A. S. Kurdija, “Evaluating robustness of perceptual image hashing algorithms,” in _2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. Opatija, Croatia: IEEE, May 2017, pp. 995–1000.
  * [26] S. Jain, A.-M. Crețu, and Y.-A. de Montjoye, “Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning,” in _31st USENIX Security Symposium (USENIX Security 22)_ , 2022, pp. 2317–2334.
  * [27] S. McKeown, G. Russell, and P. Leimich, “Fast Forensic Triage Using Centralised Thumbnail Caches on Windows Operating Systems,” _Journal of Digital Forensics, Security and Law_ , vol. 14, no. 3, 2019.
  * [28] L. Struppek, D. Hintersdorf, D. Neider, and K. Kersting, “Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash,” in _2022 ACM Conference on Fairness, Accountability, and Transparency_ , Jun. 2022, pp. 58–69.
  * [29] “MIRFLICKR Download.” [Online]. Available: http://press.liacs.nl/mirflickr/mirdownload.html
  * [30] H. Farid, “Reining in online abuses,” _Technology & Innovation_, vol. 19, no. 3, pp. 593–599, 2018.
  * [31] J. Prokos, T. M. Jois, N. Fendley, R. Schuster, M. Green, E. Tromer, and Y. Cao, “Squint hard enough: Evaluating perceptual hashing with machine learning,” Cryptology ePrint Archive, Paper 2021/1531, 2021. [Online]. Available: https://eprint.iacr.org/2021/1531
  * [32] S. Bourouis, R. Alroobaea, A. M. Alharbi, M. Andejany, and S. Rubaiee, “Recent advances in digital multimedia tampering detection for forensics analysis,” _Symmetry_ , vol. 12, no. 11, p. 1811, 2020.
  * [33] B. Yang, F. Gu, and X. Niu, “Block mean value based image perceptual hashing,” in _Intelligent Information Hiding and Multimedia Signal Processing, 2006. IIH-MSP’06. International Conference on_. IEEE, 2006, pp. 167–172.
  * [34] Commonsmachinery, “Contribute to blockhash development by creating an account on Github,” Jul. 2018, original-date: 2014-09-02T17:46:34Z. [Online]. Available: https://github.com/commonsmachinery/blockhash
  * [35] A. Ygvar, “AppleNeuralHash2ONNX,” Oct. 2022, original-date: 2021-08-15T19:52:47Z. [Online]. Available: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX
  * [36] S. L. AB (PythonWare), “PIL: Python Imaging Library.” [Online]. Available: http://www.pythonware.com/products/pil
  * [37] V. Jabade, “Modelling of Geometric Attacks for Digital Image Watermarking,” _International Journal of Innovations in Engineering Research and Technology_ , vol. 3, p. 8, Mar. 2016.
  * [38] L. Struppek, D. Hintersdorf, D. Neider, and K. Kersting, “Learning to break deep perceptual hashing: The use case neuralhash,” in _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 2022, pp. 58–69.


# Towards Natural Robustness Against
Adversarial Examples

Haoyu Chu, Shikui Wei & Yao Zhao

Institute of Information Science
Beijing Key Laboratory of Advanced Information Science and Network Technology
Beijing Jiaotong University, Beijing 100044, China
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper.

###### Abstract

Recent studies have shown that deep neural networks are vulnerable to
adversarial examples, but most of the methods proposed to defense adversarial
examples cannot solve this problem fundamentally. In this paper, we
theoretically prove that there is an upper bound for neural networks with
identity mappings to constrain the error caused by adversarial noises.
However, in actual computations, this kind of neural network no longer holds
any upper bound and is therefore susceptible to adversarial examples.
Following similar procedures, we explain why adversarial examples can fool
other deep neural networks with skip connections. Furthermore, we demonstrate
that a new family of deep neural networks called Neural ODEs (Chen et al.,
2018) holds a weaker upper bound. This weaker upper bound prevents the amount
of change in the result from being too large. Thus, Neural ODEs have natural
robustness against adversarial examples. We evaluate the performance of Neural
ODEs compared with ResNet under three white-box adversarial attacks (FGSM,
PGD, DI2-FGSM) and one black-box adversarial attack (Boundary Attack).
Finally, we show that the natural robustness of Neural ODEs is even better
than the robustness of neural networks that are trained with adversarial
training methods, such as TRADES and YOPO.

##  1 Introduction

Deep neural networks have made great progress in numerous domains of machine
learning, especially in computer vision. But Szegedy et al. (2013) found that
most of the existing state-of-the-art neural networks are easily fooled by
adversarial examples that generated by putting only very small perturbations
to the input images. Since realizing the unstability of deep neural networks,
researchers have proposed different kinds of methods to defense adversarial
examples, such as adversarial training (Goodfellow et al., 2014), data
compression (Dziugaite et al., 2016), and distillation defense (Papernot et
al., 2016). But each of these methods is a remedy for the original problem,
and none of these methods can solve it fundamentally. For example, Moosavi-
Dezfooli et al. (2016) showed that no matter how much adversarial examples are
added to training sets, there are new adversarial examples that can
successfully attack the adversarial trained deep neural network.

So, avoiding adversarial examples technically cannot solve the most essential
problem: why such subtle change in adversarial examples can beat deep neural
networks? Meanwhile, it leads to a more important question: how to make deep
neural networks have natural robustness so that they can get rid of malicious
adversarial examples.

Early explanations for adversarial examples considered that a smoothness prior
is typically valid for kernel methods that imperceptibly tiny perturbations of
a given image do not normally change the underlying class, while the
smoothness assumption does not hold for deep neural networks due to its high
non-linearity (Szegedy et al., 2013). This analysis underlies plain deep
neural networks like AlexNet (Krizhevsky et al., 2012). But later than that,
Goodfellow et al. (2014) claim adversarial examples are a result of models
being too linear rather than too non-linear, they can be explained as a
property of high-dimensional dot products. Unfortunately, both of these
explanations seem to imply that adversarial examples are inevitable for deep
neural networks.

On the other hand, we notice that skip connections are widely used in current
deep neural networks after the appearance of Highway Network (Srivastava et
al., 2015) and ResNet (He et al., 2016). It turns out that the identity
mapping in ResNet is formally equivalent to one step of Euler’s method which
has been used to solve ordinary differential equations (Weinan, 2017). More
than that, other kinds of skip connections used by different network
architectures can be considered as different numerical methods for solving
ordinary differential equations. The link between numerical ordinary
differential equations with deep neural networks can bring us a whole new
perspective to explain adversarial examples through the numerical stability
analysis.

In this paper, we attempt to utilize the natural property of neural networks
to defense adversarial examples. We first analyze how adversarial examples
affect the output of neural networks with identity mappings, obtain an upper
bound for this kind of neural networks, and find that this upper bound is
impractical in actual computations. In the same way, we figure out why
adversarial examples can fool commonly used deep neural networks with skip
connections. Then, we demonstrate that Neural ODEs hold a weaker upper bound
and verify the natural robustness of Neural ODEs under four types of
perturbations. Finally, we compare Neural ODEs with three types of adversarial
training methods to show that the natural robustness of Neural ODEs is better
than the robustness of neural networks that are trained with adversarial
training. The main contributions of our work are as follows:

  * •

We introduce and formalize the numerical stability analysis for deep neural
networks with identity mappings, prove that there is an upper bound for neural
networks with identity mappings to constrain the error caused by adversarial
noises.

  * •

We provide a new reason why commonly used deep neural networks with skip
connections cannot resist adversarial examples.

  * •

We demonstrate that Neural ODEs hold a weaker upper bound which limits the
amount of change in the result from being too large. Compare with ResNet and
three types of adversarial training methods, we show the natural robustness of
Neural ODEs.

##  2 Related works

###  2.1 Adversarial Defense

Adversarial training typically uses a robust optimization to generate
adversarial examples for training deep neural networks. Madry et al. (2017)
take the optimization as a saddle point problem. Zhang et al. (2019a) cast
adversarial training as a discrete time differential game. Adversarial
training can be seen as a data augmentation particularly enhance the
robustness to white-box attacks (Tramèr et al., 2017). Zantedeschi et al.
(2017) augmented the training sets with examples perturbed using Gaussian
noises which can also enhance the robustness to black-box attacks. Lee et al.
(2017) proposed a novel adversarial training method using a generative
adversarial network framework. Besides, Finlay et al. (2018) augmented
adversarial training with worst case adversarial training which improves
adversarial robustness in the $\ell_{2}$ norm on CIFAR10.

Modifying the neural networks by using auto encoders, input gradient
regularization and distillation can result in robustness against adversarial
attacks (Bai et al., 2017; Ross & Doshi-Velez, 2017; Papernot et al., 2016).
Besides, there are some biologically inspired deep learning models designed to
have natural robustness against adversarial examples. Nayebi & Ganguli (2017)
developed a scheme similar to nonlinear dendritic computation to train deep
neural networks to make them robust to adversarial attacks. Krotov & Hopfield
(2018) proposed Dense Associative Memory (DAM) models and suggested that DAM
with higher order energy functions are closer to human visual perception than
deep neural networks with ReLUs.

In addition to augmenting datasets or modifying original neural networks,
there exist adversarial defense methods that rely on using external models and
detecting adversarial examples. Akhtar et al. (2018) presented Perturbation
Rectifying Network (PRN) as ‘pre-input’ layers to a targeted model, if a
perturbation is detected, the output of the PRN is used for label prediction
instead of the actual image. Xu et al. (2017) proposed a strategy called
feature squeezing to reduce the search space available to an adversary by
coalescing samples that correspond to many different feature vectors in the
original space into a single sample.

##  3 Numerical stability analysis for DNNs

###  3.1 ResNet and Euler’s Method

The building block of ResNet is defined as

$\mathbf{y}_{n+1}=\mathbf{y}_{n}+f(\mathbf{y}_{n};\mathbf{\theta}_{n})$ (1)

Where $n\in\\{0...N(h)-1\\}$, $\mathbf{y}_{n}$ and $\mathbf{y}_{n+1}$ are
input and output vectors of the layers considered. The function
$f(\mathbf{y}_{n};\theta_{n})$ represents the residual mapping and
$\theta_{n}$ are weights to be learned.

In numerical methods for solving ordinary differential equations, Euler’s
method is defined by taking this to be exact:

$\mathbf{y}_{n+1}=\mathbf{y}_{n}+hf(t_{n},\mathbf{y}_{n};\mathbf{\theta}_{n})$
(2)

It can be easily seen that Eqn.(1) is a special case of Euler’s method when
the step size $h=1$. The iterative updates of ResNet can be seen as an Euler
discretization of a continuous transformation (Lu et al., 2018; Haber &
Ruthotto, 2017; Ruthotto & Haber, 2019). When the input data is perturbed by
adversarial noises $\mathbf{\epsilon}$, we denote the adversarial example by

$\mathbf{z}_{0}=\mathbf{y}_{0}+\epsilon$ (3)

To perform stability analysis for ResNet with adversarial example
$\mathbf{z}_{0}$ as input, we define a numerical solution $\mathbf{z}_{n}$ by

$\mathbf{z}_{n+1}=\mathbf{z}_{n}+hf(t_{n},\mathbf{z}_{n};\mathbf{\theta}_{n})$
(4)

and provide following theorem to show that the amount of change between
$\mathbf{y}_{n}$ and $\mathbf{z}_{n}$ holds an upper bound by some
assumptions.

######  Theorem 3.1.

Let $f(t,\mathbf{y};\mathbf{\theta})$ be a continuous function for $t_{0}\leq
t\leq b$ and $-\infty<\mathbf{y}<\infty$, and further assume that
$f(t,\mathbf{y};\mathbf{\theta})$ satisfies the Lipschitz condition. Then
compare the two numerical solutions $\mathbf{y}_{n}$ and $\mathbf{z}_{n}$ as
$h\rightarrow 0$, there is a constant $c\geq 0$, such that amount of change
between $\mathbf{y}_{n}$ and $\mathbf{z}_{n}$ satisfies

$\max_{0\leq n\leq N(h)}|\mathbf{z}_{n}-\mathbf{y}_{n}|\leq c|\epsilon|$ (5)

###### Proof.

Let $\mathbf{e}_{n}=\mathbf{z}_{n}-\mathbf{y}_{n},n\geq 0$. Then
$\mathbf{e}_{0}=\mathbf{\epsilon}$, and subtracting Eqn.(2) from Eqn.(4), we
obtain

$\mathbf{e}_{n+1}=\mathbf{e}_{n}+h[{f(t_{n},\mathbf{z}_{n};\theta_{n})}-f(t_{n},\mathbf{y}_{n};\theta_{n})]$
(6)

Assume that the derivative function $f(t,\mathbf{y};\mathbf{\theta})$
satisfied the following Lipschitz condition: there exists $K\geq 0$ such that

$|f(t,\mathbf{y}_{1};\mathbf{\theta})-f(t,\mathbf{y}_{2};\mathbf{\theta})|\leq
K|\mathbf{y}_{1}-\mathbf{y}_{2}|$ (7)

Taking bounds using Eqn.(7), we obtain

$|\mathbf{e}_{n+1}|\leq|\mathbf{e}_{n}|+hK|\mathbf{z}_{n}-\mathbf{y}_{n}|$ (8)
$|\mathbf{e}_{n+1}|\leq(1+hK)|\mathbf{e}_{n}|$ (9)

Apply this recursively to obtain

$|\mathbf{e}_{n}|\leq(1+hK)^{n}|\mathbf{e}_{0}|$ (10)

Using the inequality $(1+t)^{m}\leq e^{mt}$, for any $t\geq-1$, any $m\geq 0$,
we obtain

$(1+hK)^{n}\leq\mathbf{e}^{nhK}=\mathbf{e}^{(b-t_{0})K}$ (11)

and this implies the main result Eqn.(5). ∎

Roughly speaking, theorem 3.1 means that a small adversarial perturbation
initial value of the problem leads to a small change in the solution, provided
that the function $f(t,\mathbf{y};\mathbf{\theta})$ is continuous and the step
size $h$ is sufficiently small.

Obviously, ResNet does not satisfied the assumption of continuity and its step
size $h$ always equal to 1. Particularly, the functions
$f(t,\mathbf{y};\mathbf{\theta})$ in ResNet are composite function which
contains ReLU activation functions. Although ReLU activation functions
$g(x)=max(0,x)$ break the continuity, due to their contractive property, i.e.
satisfies $\|g(x)-g(x+\epsilon)\|\leq|\epsilon|$ for all $x,\epsilon$; it
follows that

$\displaystyle\|g_{n}(x;\theta_{n})-g_{n}(x+\epsilon;\theta_{n})\|$ (12)
$\displaystyle=\|max(0,\theta_{n}x+b_{n})-max(0,\theta_{n}(x+\epsilon)+b_{n}))\|$
$\displaystyle\leq\|\theta_{n}\epsilon\|\leq\|\theta_{n}\|\|\epsilon||$

This provides a weaker upper bound for ReLU to mitigate the loss of
continuity.

Even if the step size $h$ in ResNet can be adjusted by multiply outputs of
convolution layers by a chosen constant, but unfortunately, the step size $h$
in ResNet cannot be too small since a very small step size decreases the
efficiency in actual computations. We can experimentally show that when step
size $h$ is small (such as $10^{-1}$, $10^{-2}$ and $10^{-3}$), ResNet has no
obvious robustness against adversarial examples, and when step size is very
small (such as $10^{-8}$, $10^{-9}$, $10^{-10}$), ResNet is difficult to
train, so not only the classification accuracy of adversarial examples but
also the accuracy of clean examples is quite low.

To summarize, the main reason for ResNet’s failure in adversarial examples is
the step size $h=1$ destroys the upper bound given by Eqn.(5) and we cannot
find a proper way to deal with it. Thus, when the input of ResNet is perturbed
by adversarial noises, the amount of change in the result will become
unpredictably large so that ResNet can no longer correctly classify the input.

###  3.2 Neural ODEs

The residual block can be described as
$\mathbf{y}_{n+1}=\mathbf{y}_{n}+hf(\mathbf{y}_{n};\mathbf{\theta}_{n})$ with
step size $h=1$. When taking smaller steps and add more layers, which in the
limit, Neural ODEs parameterize the continuous dynamics of hidden units using
an ordinary differential equation specified by a neural network

$\lim_{h\rightarrow
0}\frac{\mathbf{y}_{n+h}-\mathbf{y}_{n}}{h}=\frac{d\mathbf{y}}{dh}=f(t,\mathbf{y};\theta)$
(13)

The solution of Eqn.(13) can be computed using modern ODE solvers such as
Runge–Kutta (Runge, 1895; Kutta, 1901). Euler’s method only uses
$f(t_{n},\mathbf{y}_{n};\mathbf{\theta}_{n})$, which means from any point on a
curve, we can find an approximation of a nearby point on the curve by moving a
short distance along a tangent line to the curve. Instead of using a single
tangent line, modern numerical ODE methods often use lots of different tangent
lines and weighted sum all of them to generate a super tangent line, we denote
the super tangent line by $F(t_{n},\mathbf{y}_{n};\theta_{n},f)$.

We can show that modern ODE solvers based on Runge–Kutta are stable under some
assumptions, the amount of change between $\mathbf{y}_{n}$ and
$\mathbf{z}_{n}$ holds an upper bound similar to theorem 3.1.

######  Theorem 3.2.

Let $f(t,\mathbf{y};\mathbf{\theta})$ be a continuous function for $t_{0}\leq
t\leq b$ and $-\infty<\mathbf{y}<\infty$, and further assume that
$f(t,\mathbf{y};\mathbf{\theta})$, $F(t,\mathbf{y};\theta,f)$ satisfies the
Lipschitz condition. Then compare the two numerical solutions $\mathbf{y}_{n}$
and $\mathbf{z}_{n}$ as $h\rightarrow 0$, there is a constant $\widehat{c}\geq
0$, such that amount of change between $\mathbf{y}_{n}$ and $\mathbf{z}_{n}$
satisfies

$\max_{0\leq n\leq
N(h)}|\mathbf{z}_{n}-\mathbf{y}_{n}|\leq\widehat{c}|\epsilon|$ (14)

###### Proof.

This result can be obtained in analogy with Eqn.(5) in theorem 3.1. ∎

The neural network in Neural ODEs also use ReLU activation functions. And we
already show that Eqn.(12) provides a weaker upper bound for ReLU.

Besides, modern ODE solvers, such as Fehlberg method (Fehlberg, 1969) and
DOPRI method (Dormand & Prince, 1980), can adaptively adjust the step size
until that the desired tolerance is reached. At each step, two different
approximations (for example, a fourth order Runge–Kutta and a fifth order
Runge–Kutta) for the solution are made and compared. If the two answers are in
close agreement, the approximation is accepted. If the two answers do not
agree to a specified accuracy, the step size is reduced. If the answers agree
to more significant digits than required, the step size is increased. So
modern ODE solvers provide a weaker upper bound for step size when it does not
agree with $h\rightarrow 0$.

To summarize, although Neural ODEs cannot hold the strong upper bound provided
by Eqn.(14), it still holds a weaker upper bound to constrain the error caused
by adversarial noises. We will experimentally show that Neural ODEs is more
robust on adversarial examples compared with ResNet.

###  3.3 Other Deep Neural Networks with Skip Connections

Following similar procedures, we can figure out the reason why other deep
neural networks with skip connections cannot resist adversarial examples.

####  3.3.1 PolyNet

PolyNet (Zhang et al., 2017) introduced a family of models called
PolyInception. Each PolyInception module is a polynomial combination of
Inception units that can be described as

$\left(I+F+F^{2}\right)\cdot\mathbf{y}=\mathbf{y}+F(\mathbf{y})+F(F(\mathbf{y}))$
(15)

Where $\mathbf{y}$ denotes the input, $I$ the identity operator, and $F$ the
nonlinear transform carried out by the residual block, which can also be
considered as an operator.

It has been shown that Eqn.(15) can be interpreted as an approximation to one
step of the backward (or implicit) Euler method (Lu et al., 2018):

$\mathbf{y}_{n+1}=\mathbf{y}_{n}+hf(t_{n+1},\mathbf{y}_{n+1})$ (16)

And from Eqn.(16) we can get

$\mathbf{y}_{n+1}=(I-hf)^{-1}\mathbf{y}_{n}$ (17)

Where $(I-hf)^{-1}$ can be formally rewritten as

$I+hf+(hf)^{2}+\cdots+(hf)^{n}+\cdots$ (18)

So, there are two differences between the polynomial combination with the
backward Euler method:

  * •

The polynomial combination is only a second order approximation to the
backward Euler method, the truncation error generated when high order terms
are ignored.

  * •

PolyNet faces the same problem with ResNet that the step size $h=1$. Thus,
PolyNet cannot holds the upper bound given by the backward Euler method.

And step size $h=1$ is the main reason that PolyNet fails to resist
adversarial examples.

####  3.3.2 FractalNet

FractalNet (Larsson et al., 2016) introduced a design strategy for neural
network macro-architecture based on self-similarity. The expansion rule
generates a fractal architecture with $C$ intertwined columns can be described
as

$f_{C+1}(\mathbf{y})=\frac{1}{2}(f_{C}\circ
f_{C})(\mathbf{y})+\frac{1}{2}f_{1}(\mathbf{y})$ (19)

Where $f_{1}(\mathbf{y})=\operatorname{conv}(\mathbf{y})$ and $\circ$ denotes
composition.

This expansion rule resembles to the Runge-Kutta method of order 2 (also known
as Heun’s method)(Lu et al., 2018)

$\mathbf{y}_{n+1}=\mathbf{y}_{n}+\frac{1}{2}h(f(t_{n},\mathbf{y}_{n})+f(t_{n+1},\mathbf{y}_{n}+hf(t_{n},\mathbf{y}_{n})))$
(20)

So, there are two differences between the expansion rule with Heun’s method:

  * •

The expansion rule losses two identity mapping, namely $\mathbf{y}_{n}$,
compared to Heun’s method.

  * •

FractalNet faces the same problem with ResNet that the step size $h=1$. Thus,
FractalNet cannot holds the upper bound given by Heun’s method.

And step size $h=1$ is the main reason that FractalNet fails to resist
adversarial examples.

####  3.3.3 RevNet

RevNet (Gomez et al., 2017) introduced a variant of ResNet where each layer’s
activations can be reconstructed exactly from the next layer’s. Each
reversible block takes inputs and produces outputs according to the following
additive coupling rules

$\begin{array}[]{l}X_{n+1}=X_{n}+f_{1}\left(Y_{n}\right)\\\
Y_{n+1}=Y_{n}+f_{2}\left(X_{n+1}\right)\end{array}$ (21)

This additive coupling rules can be interpreted as Euler’s method applies to
the following systems of differential equations (Lu et al., 2018)

$\begin{array}[]{l}\dot{X}=f_{1}(Y,t)\\\ \dot{Y}=f_{2}(X,t)\end{array}$ (22)

RevNet faces the same problem with ResNet that the step size $h=1$. Thus,
RevNet cannot holds the upper bound given by Euler’s method for the dynamic
systems.

##  4 Experimental results

###  4.1 ResNet with step size $h\neq 1$

We first evaluate the performance of ResNets with different step sizes on
adversarial examples. Adopting the same hyperparameter settings for ResNet56
that presented in He et al. (2016), we train 11 ResNet56s on CIFAR10 datasets
with the step size decreases from $10^{-0}$ to $10^{-10}$.

Figure 1: The classification accuracy of 11 ResNet56s both on clean CIFAR10
images and three kinds of adversarial CIFAR10 images(on the log scale). The
adversarial CIFAR10 images are generated by Fast Gradient Sign Method with
$\epsilon=0.1,0.3,0.5$ on another neural network which has three residual
blocks.

Through the numerical stability analysis of ResNet, we know that the
classification result should be well behaved when considering small
adversarial noises, provided that the step size of $h$ is sufficiently small.
In actual computations, however, when we decrease the step size from $10^{-1}$
to $10^{-10}$, ResNet has no obvious robustness against adversarial examples.
And when step size is very small, ResNet is difficult to train. Therefore, the
accuracy of ResNet56 is considerably low both on clean and adversarial inputs.
We consider that ResNet is impossible to have natural robustness against
adversarial examples.

###  4.2 The natural robustness of Neural ODEs

As we mentioned in 3.2, Neural ODEs define a continuous dynamic system, which
is equivalent to ResNet with step size $h\rightarrow 0$. And Neural ODEs hold
a weaker upper bound to constrain the error caused by adversarial noises. In
this section, we evaluate the performance of Neural ODEs under three white-box
attacks, FGSM(Goodfellow et al., 2014), PGD(Madry et al., 2017), DI2-FGSM(Xie
et al., 2019) and one black-box adversarial attack, Boundary Attack(Brendel et
al., 2017).

Without data augmentation, the accuracy of ResNet on clean MNIST and CIFAR10
datasets is 99.33% and 89.60%, the accuracy of Neural ODEs on clean MNIST and
CIFAR10 datasets is 99.05% and 69.94%. Experiment details can be seen in
appendix A.

####  4.2.1 The evaluation on white-box adversarial attacks

Table 1 shows that Neural ODEs have natural robustness under FGSM and PGD
attacks. The classification accuracy of adversarial MNIST images generated by
FGSM only drops by less than $2\%$ for all $\epsilon$. Even the strongest
first order attack method PGD can hardly beat Neural ODEs on MNIST. Although
the performance of Neural ODEs on clean CIFAR10 images is worse than state-of-
the-art networks like ResNet, it still works well on adversarial CIFAR10
images generated by PGD.

Datasets | Attack | Models | $\epsilon=0.2$ | $\epsilon=0.3$ | $\epsilon=0.4$ | $\epsilon=0.5$
---|---|---|---|---|---|---
MNIST | FGSM | ResNet50 | 25.20 | 10.95 | 9.80 | 9.75
Neural ODEs | 97.50 | 97.46 | 97.52 | 97.48
PGD | ResNet50 | 0 | 0 | 0 | 0
Neural ODEs | 93.62 | 90.55 | 84.64 | 75.23
Datasets | Attack | Models | $\epsilon=8/255$ | $\epsilon=12/255$ | $\epsilon=16/255$ | $\epsilon=20/255$
CIFAR | FGSM | ResNet50 | 4.65 | 4.16 | 3.60 | 3.26
Neural ODEs | 60.70 | 60.65 | 60.67 | 60.67
PGD | ResNet50 | 0 | 0 | 0 | 0
Neural ODEs | 59.15 | 56.93 | 53.58 | 49.53
Table 1: The classification accuracy of Neural ODEs and ResNet50 under FGSM
and PGD attacks. For both MNIST and CIFAR10, we set the size of perturbation
$\epsilon$ of PGD in an infinite norm sense, the size of PGD step is set to
0.01, the number of PGD steps is set to 40 and a uniform random perturbation
is added before performing PGD.

Figure 2: The evaluation of ResNet and Neural ODEs under DI2-FGSM.
$\epsilon=0$ represents the accuracy of ResNet and Neural ODEs without the
adversarial attack applied. For MNIST, the maximum perturbation $\epsilon$ is
set to 0.5 among all experiments, with pixel value in [0,1]. For CIFAR10, the
maximum perturbation $\epsilon$ is set to 50 among all experiments, with pixel
value in [0,255].

As shown in Figure 2, when facing adversarial noises generated by DI2-FGSM,
for MNIST, Neural ODEs remains strongly resistant to perturbations while the
accuracy of ResNet drops sharply with larger $\epsilon$. For CIFAR10, the
reduction in accuracy of Neural ODEs is significantly less than ResNet50 with
an increase of $\epsilon$. Overall, Neural ODEs is more stable when facing
white-box adversarial noises compared with ResNet.

####  4.2.2 The evaluation on black-box adversarial attacks

Boundary Attack is a decision-based attack that starts from a large
adversarial perturbation and then seeks to reduce the perturbation while
staying adversarial(Brendel et al., 2017). For MNIST, Boundary Attack has
almost no effect on Neural ODEs. For CIFAR10, however, it seems that Neural
ODEs also fails in this gradient-free attack, but the performance of Neural
ODEs is still better than ResNet50 when $\epsilon$ is larger than 33.

Figure 3: The evaluation of ResNet and Neural ODEs under Boundary Attack. For
Boundary Attack, the number of queries is fixed at 1,000. The initial step
size and orthogonal step size are set to 0.1.

####  4.2.3 Comparing with adversarial training methods

As one can see, Neural ODEs can remain its robustness and outperform PGD,
TRADES, and YOPO without taking any adversarial training methods.

Model | Adversarial training method | Clean | PGD-20($\epsilon=8/255$)
---|---|---|---
PreAct-Res18 | PGD-10(Madry et al., 2017) | 84.82 | 41.61
PreAct-Res18 | TRADES-10(Zhang et al., 2019b) | 86.14 | 44.50
PreAct-Res18 | YOPO-5-3(Zhang et al., 2019a) | 83.99 | 44.72
Neural ODEs | - | 69.94 | 59.06
Table 2: The comparison of robustness between Neural ODEs with PreAct-Res18
trained by three types of adversarial training methods on CIFAR10.

##  5 Conclusion and discussion

In this paper, we highlighted and analyzed the natural robustness of Neural
ODEs. We proved there are two similar upper bounds for ResNet and Neural ODEs
under assumptions of continuity and step size $h\rightarrow 0$. We showed that
it is the step size $h=1$ causes ResNet and the other three deep neural
networks with skip connections fail on adversarial examples. Neural ODEs
define a continuous dynamic system with step size $h\rightarrow 0$. However,
in actual computations, Modern ODE solvers adaptively adjust the step size to
solve the continuous dynamic system and this brings us a weaker upper bound
for Neural ODEs. We experimentally showed that Neural ODEs is more robust on
adversarial examples compared with ResNet.

According to Theorem 3.1 and 3.2, the upper bound $c|\epsilon|$ and
$\widehat{c}|\epsilon|$ are related to Lipschitz constant and independent with
the step size $h$. Cisse et al. (2017) showed that the robustness of DNNs can
be improved by constraining the Lipschitz constant. In this paper, however, we
are more concerned with the step size $h$, because the upper bound cannot be
guaranteed if $h\rightarrow 0$ is not satisfied. We experimentally
demonstrated that the difference of ResNet with Neural ODEs in step size $h$
is sufficient to make a significant difference in robustness, even without
constraining the Lipschitz constant. Nevertheless, constraining the Lipschitz
constant is expected to make neural networks to be more robust, which we would
like to consider as our future work.

Due to the similarity between DNNs and numerical methods for solving ODEs, we
believe that we can learn from the numerical stability analysis to design more
robust deep neural networks.

## References

  * Akhtar et al. (2018) Naveed Akhtar, Jian Liu, and Ajmal Mian.  Defense against universal adversarial perturbations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3389–3398, 2018.
  * Bai et al. (2017) Wenjun Bai, Changqin Quan, and Zhiwei Luo.  Alleviating adversarial attacks via convolutional autoencoder.  In _2017 18th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD)_ , pp. 53–58. IEEE, 2017.
  * Brendel et al. (2017) Wieland Brendel, Jonas Rauber, and Matthias Bethge.  Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  _arXiv preprint arXiv:1712.04248_ , 2017.
  * Chen et al. (2018) Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  In _Advances in neural information processing systems_ , pp. 6571–6583, 2018.
  * Cisse et al. (2017) Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier.  Parseval networks: Improving robustness to adversarial examples.  _arXiv preprint arXiv:1704.08847_ , 2017.
  * Dormand & Prince (1980) John R Dormand and Peter J Prince.  A family of embedded runge-kutta formulae.  _Journal of computational and applied mathematics_ , 6(1):19–26, 1980.
  * Dupont et al. (2019) Emilien Dupont, Arnaud Doucet, and Yee Whye Teh.  Augmented neural odes.  In _Advances in Neural Information Processing Systems_ , pp. 3140–3150, 2019.
  * Dziugaite et al. (2016) Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy.  A study of the effect of jpg compression on adversarial images.  _arXiv preprint arXiv:1608.00853_ , 2016.
  * Fehlberg (1969) Erwin Fehlberg.  _Low-order classical Runge-Kutta formulas with stepsize control and their application to some heat transfer problems_ , volume 315.  National aeronautics and space administration, 1969.
  * Finlay et al. (2018) Chris Finlay, Adam M Oberman, and Bilal Abbasi.  Improved robustness to adversarial examples using lipschitz regularization of the loss.  2018\.
  * Gomez et al. (2017) Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse.  The reversible residual network: Backpropagation without storing activations.  In _Advances in neural information processing systems_ , pp. 2214–2224, 2017.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Haber & Ruthotto (2017) Eldad Haber and Lars Ruthotto.  Stable architectures for deep neural networks.  _Inverse Problems_ , 34(1):014004, 2017.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pp. 1097–1105, 2012.
  * Krotov & Hopfield (2018) Dmitry Krotov and John Hopfield.  Dense associative memory is robust to adversarial inputs.  _Neural computation_ , 30(12):3151–3167, 2018\.
  * Kutta (1901) Wilhelm Kutta.  Beitrag zur naherungsweisen integration totaler differentialgleichungen.  _Z. Math. Phys._ , 46:435–453, 1901.
  * Larsson et al. (2016) Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Fractalnet: Ultra-deep neural networks without residuals.  _arXiv preprint arXiv:1605.07648_ , 2016.
  * Lee et al. (2017) Hyeungill Lee, Sungyeob Han, and Jungwoo Lee.  Generative adversarial trainer: Defense to adversarial perturbations with gan.  _arXiv preprint arXiv:1705.03387_ , 2017.
  * Lu et al. (2018) Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong.  Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations.  In _International Conference on Machine Learning_ , pp. 3276–3285. PMLR, 2018.
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2574–2582, 2016.
  * Nayebi & Ganguli (2017) Aran Nayebi and Surya Ganguli.  Biologically inspired protection of deep networks from adversarial attacks.  _arXiv preprint arXiv:1703.09202_ , 2017.
  * Papernot et al. (2016) Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.  Distillation as a defense to adversarial perturbations against deep neural networks.  In _2016 IEEE Symposium on Security and Privacy (SP)_ , pp. 582–597. IEEE, 2016.
  * Ross & Doshi-Velez (2017) Andrew Slavin Ross and Finale Doshi-Velez.  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  _arXiv preprint arXiv:1711.09404_ , 2017.
  * Runge (1895) Carl Runge.  Über die numerische auflösung von differentialgleichungen.  _Mathematische Annalen_ , 46(2):167–178, 1895\.
  * Ruthotto & Haber (2019) Lars Ruthotto and Eldad Haber.  Deep neural networks motivated by partial differential equations.  _Journal of Mathematical Imaging and Vision_ , pp. 1–13, 2019.
  * Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.  Highway networks.  _arXiv preprint arXiv:1505.00387_ , 2015.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Tramèr et al. (2017) Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  _arXiv preprint arXiv:1705.07204_ , 2017.
  * Weinan (2017) E Weinan.  A proposal on machine learning via dynamical systems.  _Communications in Mathematics and Statistics_ , 5(1):1–11, 2017.
  * Xie et al. (2019) Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille.  Improving transferability of adversarial examples with input diversity.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2730–2739, 2019.
  * Xu et al. (2017) Weilin Xu, David Evans, and Yanjun Qi.  Feature squeezing: Detecting adversarial examples in deep neural networks.  _arXiv preprint arXiv:1704.01155_ , 2017.
  * Zantedeschi et al. (2017) Valentina Zantedeschi, Maria-Irina Nicolae, and Ambrish Rawat.  Efficient defenses against adversarial attacks.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , pp. 39–49, 2017.
  * Zhang et al. (2019a) Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle.  In _Advances in Neural Information Processing Systems_ , pp. 227–238, 2019a.
  * Zhang et al. (2019b) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan.  Theoretically principled trade-off between robustness and accuracy.  _arXiv preprint arXiv:1901.08573_ , 2019b.
  * Zhang et al. (2017) Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin.  Polynet: A pursuit of structural diversity in very deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 718–726, 2017.

##  Appendix A Training configuration of Neural ODEs

Table A1 summarizes our experiment settings for training Neural ODEs. We only
change some hyperparameters in Neural ODEs, without taking any help of
adversarial defense methods.

Training Configurations | MNIST | CIFAR10
---|---|---
training method | Adam | Adam
ODEslover | dopri5 | dopri5
convolution layers | 4 | 5
convolution channels | 128 | 256
start time $t_{0}$ | 0 | 0
stop time $t_{1}$ | 100 | 500
training epochs | 100 | 200
initial learning rate* | $10^{-3}$ | $10^{-4}$
mini-batches size | 32 | 32
Table A1: The training configurations of Neural ODEs on MNIST and CIFAR10.
*Every 50 epochs, the learning rate is reduced to half.

It is worth to mention that there are no common training configurations for
Neural ODEs on CIFAR10, Dupont et al. (2019) had reported that the accuracy of
Neural ODEs on CIFAR10 test sets is 53.7%$\pm$0.2%, our experiment result is
better than it.