# Ortho-ODE: Enhancing Robustness and of Neural ODEs against Adversarial
Attacks

Vishal Purohit

###### Abstract

Neural Ordinary Differential Equations (NODEs) proposed in the influential
work [5], probed the usage of numerical solvers to solve the differential
equation characterized by a Neural Network(NN), therefore initiating a new
paradigm of deep learning models with infinite depth. NODEs were designed to
tackle the irregular time series problem. However, NODEs have demonstrated
robustness against various noises and adversarial attacks. This paper is about
the natural robustness of NODEs and examines the cause behind such surprising
behavior. We show that by controlling the Lipschitz constant of the ODE
dynamics the robustness can be significantly improved. We derive our approach
from Grownwall’s inequality. Further, we draw parallels between contractivity
theory and Grownwall’s inequality. Experimentally we corroborate the enhanced
robustness on numerous datasets - MNIST, CIFAR-10, and CIFAR 100. We also
present the impact of adaptive and non-adaptive solvers on the robustness of
NODEs.

###### Contents

  1. 1 Introduction
  2. 2 Related Work
    1. 2.1 Neural-ODEs
    2. 2.2 Adversarial Attacks
    3. 2.3 Adversarial Defense
  3. 3 Methodology
    1. 3.1 Preliminaries on Neural ODE
    2. 3.2 Problem Formulation
    3. 3.3 The Grönwall–Bellman inequality
    4. 3.4 Connection between Grownwall’s Inequality and Contraction theory
    5. 3.5 Ortho-ODE
  4. 4 Experiments
    1. 4.1 Numerical Results
  5. 5 Limitations & Conclusion
  6. 6 Future Work

##  1 Introduction

Deep Learning (DL) has revolutionized and impacted diverse fields of science.
It has found successful applications in high-level vision tasks like - image
classification, object detection, and image segmentation, and low-level tasks
like image super-resolution, deburring, etc. Despite a plethora of
applications, deep learning algorithms suffer from fundamental problems that
limit their application to critical fields like medical imaging, security, and
surveillance. But [33] found that most of the existing state-of-the-art neural
networks are easily fooled by adversarial examples that are generated using
tiny perturbations to the input images. The Inputs corrupted with
imperceptible perturbations can easily fool many vanilla deep neural networks
(DNNs) into misclassifying them and degrading their performance. A new
subfield of deep learning, adversarial attacks [35], is dedicated to designing
such imperceptible perturbations to data and defenses for such attacks.
Recently, [36] [21] have applied Neural Ordinary Differential Equations
(NODEs) [5] to defend against adversarial attacks. Some works [36] explored
the natural robustness of NODEs against adversarial attacks, both, empirically
and theoretically. The work [36] made some interesting observations and
provided the reasoning behind such surprising properties of NODEs. NODEs were
introduced to tackle the irregular time series problem but their surprising
robustness against attacks has piqued the interest of a lot of researchers.
Though NODEs are robust against adversarial attacks, they still suffer from
poor performance on specific attacks, specifically gradient-free attacks.
Practically, it is impossible to defend against every adversarial attack out
in the wild. Meanwhile, a more important question to be asked is - why NODEs
are robust against some adversarial attacks, and how to improve their
robustness?

So far many techniques have been introduced to tackle adversarial attacks.
Probably one of the most successful techniques is adversarial training
introduced in works [23] [38]. In adversarial training, the adversarial
examples are simulated in each iteration of the model and used as a training
set in the next iteration of training. Using adversarial training is
computationally expensive since in every iteration we need to generate
adversarial examples and retrain the model on newly generated samples. In
contrast, NODEs offer robustness naturally without the need for adversarial
training which makes them attractive to computation-limited applications.

In this paper, our objective is to assess the effect of the Lipschitz constant
of dynamics of NODE on the robustness of the model against adversarial
attacks. To this end, we first propose to use orthogonal convolutional layers
[34] using Cayley transform to design the NN that signifies the dynamics of
non-linear dynamical system. Encouraging orthogonality in neural networks has
proven to yield several compelling benefits. Our work specifically uses two
such benefits - preserving gradient norms and enforcing low Lipschitz
constants. Controlling Lipschitz constants is nontrivial and has shown several
benefits [29], [13], [28] against perturbations in Convolutional Neural
Networks (CNNs). Different from CNNs, NODEs are infinite depth resnets [19].
Because of their infinite depth nature, we need to ensure that NODEs do not
suffer from degraded activations due to exploding and vanishing gradients
[27]. Orthogonal convolutional layers using the Cayley transform ensure stable
activation, preserving gradient norms and enforcing low Lipschitz constants.
We call our method as Ortho-ODE as in ODE with orthogonal convolutional
layers. Our method is backed theoretically by Grownwall’s inequality. Our
contributions are:

  * •

Our method proposes to use orthogonal convolutional layers to characterize the
NN representing dynamics of ODE. Thus, enabling us to upper bound the
Lipschitz constant of the dynamics making our model robust.

  * •

We theoretically justify that imposing orthogonality constraints on dynamics
ensures the representations of adversarial and pure samples remain close.
Therefore, increasing the classification accuracy of our method.

  * •

We test our method on multiple datasets and against many state-of-the-art
robust NODE methods. We draw parallels between our method and contractivity
theory to demonstrate that various theoretical pieces of evidence support our
method.

##  2 Related Work

###  2.1 Neural-ODEs

NODEs were first introduced in the work by Chen et al. [5] as continuous depth
formulation of ResNet architecture. Many notable architects can be interpreted
as different discretizations of the differential equations [1]. Many of the
subsequent works have followed up with an exploration of optimization issues
and the expressivity of NODE. For example, in the work [8] it was shown that
the expressivity of the NODEs is limited due to the topology-preserving nature
of NODEs. To overcome such issues [8] presented augmentedODE to learn more
complex functions. The work [36] was the first to evaluate the adversarial
robustness of NODEs theoretically as well as empirically.

Additionally, in [17] it was shown that injecting noise could be beneficial to
the stability of NODEs. Despite some appealing properties of NODEs, they are
computationally expensive. Hence a recent study [25] explored the efficient
implementation of the adjoint training method. The renewed interest in the
marriage of dynamic systems and deep learning has given rise to a plethora of
works combining the theory of dynamical systems with NODEs. In the original
formulation of NODE, there was no depth or input-dependent modification of the
dynamics. However, [24] suggested using neural ODE whose dynamics would depend
on the input.

###  2.2 Adversarial Attacks

Adversarial examples are seen as threats to neural networks, especially in
critical applications. Adversarial examples are fed to the neural network to
modify their predictions to the desired prediction. As one of the initial
applications of adversarial attacks dates back to work [6], [22] which
modified the spellings of the words to fool the spam filters. But, the first
adversarial attacks on computer vision models were introduced in work [33] and
[17]. These two works established a new field of deep learning focusing on the
design and defense of adversarial examples. It is puzzling to many researchers
as to why neural networks are sensitive to imperceptible perturbations
(adversarial attacks) in the image. The work [16] proposed a localize the
attack region to a small patch instead of adding noise to the whole image.
Many such attacks have been formulated for speech processing [3], [15].
Multiple works [10],[11] have proposed attacks designed specifically for
adaptive models similar to NODEs.

Since the introduction of adversarial examples, many works have been proposed
to refine the attacks and target various properties of the neural network.
Broadly, adversarial attacks could be classified as - black-box and white-box
attacks [2]. Black box attacks are those, where the attacker does not have
access to any knowledge about the model or its output. However, in white box
attacks, the attacker has access to the gradient information, outputs, or
model architecture. White box attacks are generally more effective because of
the design of targeted attacks designed for a specific model using the
available information. Two of the most famous attacks are FGSM [12] and PGD
[12], which is a white box attack with the goal of misclassification. PGD is a
gradient-based attack where the attacker has access to gradients of the model
during training. Since then many sophisticated attacks have been proposed.
Autoattack [12] is a suite of attacks carefully designed to do large-scale
evaluations of the robustness of NNs.

###  2.3 Adversarial Defense

The adversarial defense literature is equally rich with the most famous being
adversarial training [17], Bayesian adversarial training [1], and other
regularization-based methods [28]. Many variations of adversarial training
approaches also have been proposed [14]. Among various defense mechanisms,
[30] was the first to work to improve the robustness using control theory and
dynamic systems. Further, the use of Lyapunov-stable equilibrium for NODEs is
investigated in Stable Neural ODE [20].

##  3 Methodology

In this section, we first go over some of the preliminaries and problem
formulation for image classification using NODEs. We follow up with a detailed
description of methods and theorems supporting our hypothesis.

###  3.1 Preliminaries on Neural ODE

Under the neural ODE framework, we model the input and output as two states of
a continuous dynamical system by approximating the system’s dynamics with
trainable layers. The Neural ODEs are endowed with intrinsic invertibility,
yielding to a family of invertible models for solving inverse problems [9].
The following equations characterize the relation between input and output:

$\frac{dz(t)}{dt}=f_{\mathcal{W}}(z(t),t),z(0)=z_{in}$ (1)

where $z_{out}=z(T)$ and
$f_{\mathcal{W}}:\mathbb{R}^{n}\times[0,\infty)\rightarrow\mathbb{R}^{n}$
denotes the trainable layers that are parameterized by weights $\mathcal{W}$.
We assume that $f_{\mathcal{W}}$ is continuous in $t$ and globally Lipschitz
continuous in $z$. The input of neural ODE corresponds to the state at $t=0$
and output $z_{out}$ is associated with the state at some $T\in(0,\infty)$.
Given the input $z_{in}$, the output $z_{out}$ at time $t$ can be calculated
by solving ODE in Eq. 1 Therefore, the solution of neural ODE can be
represented as a $d-$dimensional function $\phi_{T}(.,.)$ i,e.

$z_{out}=z(T)=z(0)+\int_{0}^{T}f_{\mathcal{W}}(z(t),t)dt=\phi_{T}(z_{in},f_{\mathcal{W}})$
(2)

It is quite easy to see that NODEs are the continuous analog of residual
networks where the hidden layer of the resnet can be regarded as discrete-time
difference equations $z(t+1)=z{t}+f_{\mathcal{W}}(z(t),t)$.

For classification, task NODEs cannot directly be applied to images. We need
CNN layers before the NODE layer to extract the representation before passing
it to NODE. Additionally, we need a set of fully connected layers post NODEs
to classify the representation in various classes. The pre and post-CNN layers
be represented by $f_{pre}(.)$ and $f_{post}(.)$. NODEs have a property that
is formalized using the following proposition

######  Proposition 1.

(ODE integral curves do not intersect [18], [37]) Let $f$ be a function whose
derivative exists in every point, then $f$ is a continuous function. Let
$z_{1}(t)$ and $z_{2}(t)$ be two solutions of the ODE with different initial
conditions, i.e. $z_{1}(0)\neq z_{2}(0)$. $f_{\mathcal{W}}$ is continuous in
t, and globally Lipschitz is continuous in z.

Then, it holds that

$z_{1}(t)\neq z_{2}(t)$

for all

$t\in[0,\infty)$

###  3.2 Problem Formulation

Let $\mathcal{P}_{data}$ be the probability distribution of the data over
$\mathcal{X}\times\mathcal{Y}$, where $\mathcal{X}$ represent a set of input
data points and $\mathcal{Y}$ represents the corresponding labels. Let
$n^{th}$ pair of data be represented by pair $(x_{n},y_{n})$, where
$x_{n}\in\mathbb{R}^{m\times n}$ represents an image of $m\times n$, width and
height respectively and $y_{n}\in\mathbb{R}^{C}$. Here $C$ is the total number
of classes in the training dataset. Further, we assume that both training and
test data come from $\mathcal{P}_{data}$. We extract the features from input
$x_{n}$ using CNN layers and process the output of NODE using an MLP layer.
The whole processing pipeline is represented by the following functions -

During test time the adversarial samples are generated using PGD or FSGM be
represented by $x_{adv}$. Our goal is to correctly classify the adversarial
samples without adversarial training of NODE.

###  3.3 The Grönwall–Bellman inequality

A matrix $A\in\mathbb{R}^{m\times n}$ is said to be an orthogonal matrix if
$A^{T}A=I_{n}$, where $I_{n}$ is the identity matrix of $n\times n$ dimension.
Matrix $A$ is said to be semi-orthogonal if $A^{T}A=I_{n}$ or $AA^{T}=I_{n}$.
If $m\geq n$, then $A$ is norm preserving, and if $m\leq n$ then the mapping
is contractive. Alternatively, a matrix with all singular values is orthogonal
too. One of the consequences of orthogonality is 1-Lipschitzness. The
definition of Lipschitzness is as below

######  Definition 1.

(Lipschitness) A function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ is
L-Lipschitz w.r.t $l_{2}$ norm if and only if
$\frac{||f(x)-f(x^{{}^{\prime}})||_{2}}{||x-x^{{}^{\prime}}||_{2}}\leq L$
$\forall x,x^{{}^{\prime}}\in\mathbb{R}^{n}$. L is called lipschitz constant
of $f$

Figure 1: Architecture difference between Resnet, Vanilla ODE and Ortho-ODE

There are multiple works showcasing that bounding the Lipschitz constant of a
neural network has guaranteed robustness against small perturbations in the
input. Though the benefit of bounding the Lipschitz constant has been explored
for neural networks the impact of bounding the Lipschitz constant on the
dynamics of NODE is rarely investigated. According to Grownall’s inequality
[26], when stated informally - The difference between two terminal states of
NODE is bounded by the difference between the initial states times the
exponential of the Lipschitz constant of the dynamics. In neural ODE, dynamics
are represented by a neural network. Hence controlling the Lipschitz constant,
one can control how close two outputs are though initial conditions are quite
different. Formally the theorem is as below-

######  Theorem 1.

(Grönwall–Bellman inequality) Let $U\subset\mathbb{R}^{d}$ be an open set. Let
$f:U\times[0,T]\rightarrow\mathbb{R}^{d}$ be a continuous function and let
$z_{1}$ and $z_{2}$ satisfy the Initial Value Problem (IVP) problems -

$\frac{dz_{1}(t)}{dt}=f(z_{1}(t_{0}),t),z_{1}(t_{0})=x_{1}$ (3)
$\frac{dz_{2}(t)}{dt}=f(z_{2}(t_{0}),t),z_{1}(t_{0})=x_{2}$ (4)

Let $C$ be a constant such that $C\geq 0$ such that $\forall t\in[0,T]$,

$||f(z_{2}(t_{0}),t)-f(z_{1}(t_{0}),t)||\leq C||z_{2}(t)-z_{1}(t)||$ (5)

Consequently,

$||z_{2}(t)-z_{1}(t)||\leq||x_{2}-x_{1}||.e^{Ct}$ (6)

Where $C$ is the Lipschitz constant of the dynamics.

The key to improving the robustness of the NODEs is to control the difference
between the neighboring curves. From Theorem 1 we can bring two neighboring
curves close together by bounding the Lipschitz constant of the dynamics of
ODE. Directly bounding the Lipschitz constant is hard. Hence we resort to
Cayley transform to impose orthogonality constraints on the dynamics of ODE.
For our method to work, we make the following assumption -

######  Assumption 1.

(Representation closeness) Let $z_{pure}$ represent the input representation
of pure sample and $z_{adv}$ be the input representation of an adversarial
sample. Then we assume that $z_{adv}$ is in the $\epsilon$ neighborhood of
$z_{pure}$. i.e,

$||z_{pure}-z_{adv}||\leq\epsilon$ (7)

###  3.4 Connection between Grownwall’s Inequality and Contraction theory

Recently contraction theory has been employed in Neural Networks for multiple
purposes. For example, [14] explored the use of contractivity to improve the
well-posedness and robustness of implicit neural networks, and analysis of
Hopfield NNs [4]. In [7] author proposed Hamiltonian NODE which is contractive
by design to improve the robustness. Singh et al. [32] and Revay et al. [31]
utilize contraction theory to learn stabilizable nonlinear NN models from
available data. Contractivity is the property of a dynamical system and it
ensures that trajectories of the dynamical system converge to each other
asymptotically. Formally contractivity of a dynamic system is defined below -

######  Definition 2.

(Contractivity) The dynamics of an ODE is contractive with contraction rate
$\zeta>0$ if

$||\hat{z}_{t}-z_{t}||\leq e^{-\rho t}||\hat{z_{0}}-z_{0}||,\hskip
10.0pt\forall t\in[0,T]$ (8)

where $z_{0},\hat{z}_{0}\in\mathbb{R}^{n}$ are the initial conditions of IVP
and $\hat{z}_{t}$ and $z_{t}$ are its solutions.

Therefore we can say that a NODE is contractive, the Lipschitz constant
between the input and the output is smaller than 1. Making NODEs globally
contractive can significantly hamper expressive power. Our goal is to apply
local contractivity efficiently to harness the natural robustness provided by
locally contractive NODEs.

###  3.5 Ortho-ODE

A recent work by Torckman et al.[34] parameterized orthogonal convolutions
using Cayley transform in a scalable and efficient way. The key idea behind
the method in [34] is that multi-channel convolution in the Fourier domain
reduces to a batch of matrix-vector products, and making each of those
matrices orthogonal makes the convolution orthogonal. Since orthogonalization
directly controls the Lipschitz constant, we propose to model the layers of
the neural network describing the dynamics using orthogonal convolution layers
instead of normal convolution layers. A comparison between ResNets, Vanilla
ODE, and Ortho-ODE is shown in Figure.1. We briefly go over the Cayley
transform and how it can be used to impose orthogonal constraints on
convolution operation.

Consider convolutional layer with stride 1 with $c_{in}$ representing the
input channel and $c_{out}$ representing the output channels. Let the set of
weights of the convolutional layer mapping from input to output is of shape
$c_{out}\times c_{in}\times n\times n$, where $n\times n$ is the size of the
convolutional kernel. The convolutions are easier to be analyzed when they are
considered to be circular. The convolution is said to be circular when if the
kernel goes out of the bounds of input, it wraps around to the other side of
the input. We define $Conv(X)$ as the circular convolutional layer with weight
tensor $W\in\mathbb{R}^{c_{\text{out}}\times c_{\text{in}}\times n\times n}$
applied to an input tensor $X\in\mathbb{R}^{b\times c_{\text{in}}\times
h\times w}$. The resulting output tensor $Y=Conv(X)\in\mathbb{R}^{cout×n×n}$.
One can view the convolutional operation as doubly block-circulant matrix
$C\in\mathbb{R}^{c_{out}n^{2}c_{in}n^{2}}$. Similarly, we denote by
$Conv^{T}(X)$ the transpose of the same convolution.

The Cayley transform is a bĳection between skewsymmetric matrices $A$ and
orthogonal matrices $Q$ without −$1$ eigenvalues:

$Q=(I-A)(I+A)^{-1}$ (9)

A matrix is said to be skew-symmetric if $A=-A^{T}$. The Cayley transform of
such a skew-symmetric matrix is always orthogonal. Since convolutions are
linear transformations we can the Cayley transform to convolutions. As
described in [34] While it is possible to construct the matrix $C$
corresponding to a convolution $Conv$ and apply the Cayley transform to it,
this is highly inefficient in practice: Convolutions can be easily skew-
symmetrized by computing $Conv(X)-Conv^{T}(X)$, but finding their inverse is
challenging; instead, we manipulate convolutions in the Fourier domain, taking
advantage of the convolution theorem and the efficiency of the fast Fourier
transform.

Since we can construct multiple layers of CNN where each layer is orthogonal
and we use such layers to construct the neural network that represents the
dynamics of the NODE. As verified in [34] using the clayey transform it is
possible to maintain the orthogonalization constraint consistently.

##  4 Experiments

In this section, we first describe the datasets used to benchmark our
algorithm followed by details of existing ODE and Non-ODE based benchmarks
used for comparison. Further, we describe the training settings including
model description, metric and numerical results.

Datasets General datasets used for evaluating the robustness of NODEs include
MNIST, CIFAR-10, and CIFAR-100. MNIST is the easiest dataset among others.
CIFAR-10 and CIFAR-100 are difficult datasets used for the classification
task. For each dataset, we have 60,000 training samples and 10,000 test
samples. MNIST and CIFAR-10 have 10 classes but CIFAR-100 has 100 classes.

Benchmarks Our method with orthogonal layers is dubbed as Ortho-ODE. We
compare Ortho-ODE with parameter-wise equivalent ResNet-10, Vanilla-ODE which
uses normal convolutional layers, and TisODE [36]

Training Details All the methods are trained for 100 epochs with a learning
rate of 0.01 and no weight decay. We ensure the number of parameters in all
the architectures is almost similar. We augment each data with crop and
rotation augmentation. We evaluate all the models in the absence and presence
of adversarial samples. We use PGD and FGSM adversarial attacks to assess the
robustness of our method. Additionally, we use gaussian noise with different
standard deviations to assess the robustness against non-target attacks.

Metrics We present classification accuracy for each method on each dataset

###  4.1 Numerical Results

We evaluate our method against several benchmarks. We briefly describe the
benchmarks used for comparison. We evaluate all the methods in two training
configurations - one with adversarial attacks and another without adversarial
attacks. Additionally, we also evaluate the performance in the presence of
Gaussian Noise. ResNet10 is a normal CNN with residual connections and there
is a total of 10 layers, making its number of parameters almost equivalent to
parameters in other methods. As expected ResNet10 does not perform well when
it comes to adversarial attacks. On both FGSM and PGD-based attacks, ResNet10
struggles to give any good accuracy. All the results are tabulated in Table 1.
We find that ResNet10 outperforms all the methods under no attack
configuration.

We evaluate our method against Neural ODE or Vanilla ODE where the
convolutional layers used in neural network parameterize the dynamics are
normal convolutional layers. We ensure that all channels and the total number
of parameters remain almost the same as our method. The performance of Vanilla
ODE is far better than ResNet10 under all attacks. However, under no attack
configuration, ResNet10 dominates.

Further, TisODE uses the time invariance property to ensure that in the
solution of two slightly different initial conditions the final output from
NODE almost remains the same. In our, evaluation TisODE performs best in most
cases under different adversarial attacks. Compared to our method,
underperforms in some cases but there is no clear evidence suggesting that our
method completely outperforms TisODE.

Datasets | Methods | No Attacks | Gaussian | FGSM | PGD
---|---|---|---|---|---
$\sigma=100$ | FSGM - 5/255 | FSGM - 8/255 | PGD - 0.2 | PGD -0.3
MNSIT | ResNet10 | 99.15 | 98.75 | 28.20 | 16.07 | 32.67 | 0.0
| Vanilla ODE | 97.84 | 97.63 | 49.10 | 34.78 | 64.89 | 13.02
| TisODE | 99.13 | 98.9 | 50.23 | 36.98 | 67.47 | 13.7
| Ortho-ODE (Our Method) | 99.14 | 99.10 | 49.32 | 37.52 | 67.86 | 11.56
CIFAR-10 | ResNet10 | 91.12 | 90.56 | 38.10 | 17.05 | 30.45 | 1.2
| Vanilla ODE | 82.7 | 81.30 | 42.89 | 38.12 | 49.18 | 12.56
| TisODE | 85.30 | 84.12 | 44.23 | 37.46 | 50.34 | 13.1
| Ortho-ODE (Our Method) | 85.69 | 85.10 | 43.12 | 36.89 | 50.78 | 11.4
CIFAR-100 | ResNet10 | 68.57 | 66.89 | 18.57 | 14.54 | 17.13 | 0.0
| Vanilla ODE | 52.91 | 56.21 | 47.67 | 37.67 | 21.89 | 11.12
| TisODE | 53.62 | 55.71 | 48.12 | 36.41 | 23.72 | 12.34
| Ortho-ODE (Our Method) | 53.64 | 52.45 | 49.45 | 35.32 | 21.81 | 12.56

Table 1: Robustness Results of our method compared against existing approaches
on MNIST, CIFAR-10, and CIFAR-100

##  5 Limitations & Conclusion

Apart from poor performance under adversarial attacks, our method inherits the
disadvantages of the Cayley transform. Among all the methods shown in the
results table, our method is the slowest due to the use of the Cayley
transform which converts the signal to the Fourier domain using the Fast
Fourier transform. As normal convolution would not require a such step, hence
our method is the slowest.

Apart from slow training and inference, the accuracy of our method needs to
improve to be competitive with TisODE. Due to the orthogonality constraints on
the convolutional layers, we sacrifice the expressivity of the model to some
extent. The trade-off between expressivity and robustness is a common theme
shared across multiple algorithms.

In this work, we evaluated the use of Grownwall’s inequality to improve the
robustness of NODE. We constrained the Lipschitz constant of the neural
network representing the dynamics using orthogonal constraints. We choose to
use Cayley transform to constrain and impose the orthogonality requirement. We
evaluate our method on multiple datasets and compared it against various
benchmarks. Though our method does not outperform the benchmarks in all cases,
our method still works and sometimes outperforms TisODE.

##  6 Future Work

In the future, we would like to further explore the connection between the
bounding of the Lipschitz constant of NODE and the adversarial robustness. It
is important to figure out a faster method to impose the orthogonality
requirement in order to improve the training an inference speed of our method.

## References

  * [1] Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey and Michael P. Brenner  “Learning data-driven discretizations for partial differential equations”  Proceedings of the National Academy of Sciences, 2019, pp. 15344–15349
  * [2] Siddhant Bhambri, Sumanyu Muku, Avinash Tulasi and Arun Balaji Buduru  “A Study of Black Box Adversarial Attacks in Computer Vision”  In _CoRR_ abs/1912.01667, 2019
  * [3] Nicholas Carlini et al.  “Hidden Voice Commands”  In _Proceedings of the 25th USENIX Conference on Security Symposium_ , 2016, pp. 513–530
  * [4] Veronica Centorrino, Francesco Bullo and Giovanni Russo  “Contraction Analysis of Hopfield Neural Networks with Hebbian Learning”, 2022
  * [5] Ricky T.. Chen, Yulia Rubanova, Jesse Bettencourt and David K Duvenaud  “Neural Ordinary Differential Equations”  In _Advances in Neural Information Processing Systems_ 31 Curran Associates, Inc., 2018
  * [6] Nilesh Dalvi et al.  “Adversarial Classification”  In _Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ Association for Computing Machinery, 2004, pp. 99–108
  * [7] Thai P. Duong and Nikolay Atanasov  “Hamiltonian-based Neural ODE Networks on the SE(3) Manifold For Dynamics Learning and Control”  In _CoRR_ abs/2106.12782, 2021
  * [8] Emilien Dupont, Arnaud Doucet and Yee Whye Teh  “Augmented Neural ODEs”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019
  * [9] Will Grathwohl, Ricky T.. Chen, Jesse Bettencourt and David Duvenaud  “Scalable Reversible Generative Models with Free-form Continuous Dynamics”  In _International Conference on Learning Representations_ , 2019
  * [10] Mirazul Haque, Anki Chauhan, Cong Liu and Wei Yang  “ILFO: Adversarial Attack on Adaptive Neural Networks”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [11] Sanghyun Hong, Yigitcan Kaya, Ionut-Vlad Modoranu and Tudor Dumitras  “A Panda? No, It’s a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference”  In _CoRR_ abs/2010.02432, 2020
  * [12] Sandy H. Huang et al.  “Adversarial Attacks on Neural Network Policies”  In _CoRR_ abs/1702.02284, 2017
  * [13] Yujia Huang et al.  “Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds”  In _CoRR_ abs/2111.01395, 2021
  * [14] Saber Jafarpour, Alexander Davydov, Anton V. Proskurnikov and Francesco Bullo  “Robust Implicit Networks via Non-Euclidean Contractions”  In _CoRR_ abs/2106.03194, 2021
  * [15] Robin Jia and Percy Liang  “Adversarial Examples for Evaluating Reading Comprehension Systems”  In _CoRR_ abs/1707.07328, 2017
  * [16] Danny Karmon, Daniel Zoran and Yoav Goldberg  “LaVAN: Localized and Visible Adversarial Noise”  In _CoRR_ abs/1801.02608, 2018
  * [17] Alexey Kurakin, Ian J. Goodfellow and Samy Bengio  “Adversarial Machine Learning at Scale”  In _CoRR_ , 2016
  * [18] Norman Levinson and Earl A. Coddington  “Theory of Ordinary Differential Equations”  In _Theory of Ordinary Differential Equations_ , 1955
  * [19] Mufan Li, Mihai Nica and Dan Roy  “The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization”  In _Advances in Neural Information Processing Systems_ Curran Associates, Inc., 2021, pp. 7852–7864
  * [20] Xiyuan Li, Xin Zou and Weiwei Liu  “Defending Against Adversarial Attacks via Neural Dynamic System”  In _Advances in Neural Information Processing Systems_ , 2022
  * [21] Xuanqing Liu, Tesi Xiao, Uc Davis and Qin Cao  “How Does Noise Help Robustness? Explanation and Exploration under the Neural SDE Framework”  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 279–287
  * [22] Daniel Lowd and Christopher Meek  “Adversarial Learning”  In _Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining_ , 2005, pp. 641–647
  * [23] Aleksander Madry et al.  “Towards Deep Learning Models Resistant to Adversarial Attacks”  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_ OpenReview.net, 2018
  * [24] Stefano Massaroli et al.  “Dissecting Neural ODEs”  In _Advances in Neural Information Processing Systems_ 33 Curran Associates, Inc., 2020, pp. 3952–3963
  * [25] Takashi Matsubara, Yuto Miyatake and Takaharu Yaguchi  “Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory”  In _Advances in Neural Information Processing Systems_ , 2021
  * [26] D.. Mitrinović, J.. Pečarić and A.. Fink  “Gronwall Inequalities in Higher Dimensions”, 1991, pp. 401–432
  * [27] Razvan Pascanu, Tomás Mikolov and Yoshua Bengio  “Understanding the exploding gradient problem”  In _CoRR_ abs/1211.5063, 2012
  * [28] Patricia Pauli et al.  “Training Robust Neural Networks Using Lipschitz Bounds”  In _IEEE Control Systems Letters_ 6, 2022, pp. 121–126  DOI: 10.1109/LCSYS.2021.3050444
  * [29] Patricia Pauli, Anne Koch, Julian Berberich and Frank Allgöwer  “Training robust neural networks using Lipschitz bounds”  In _CoRR_ abs/2005.02929, 2020
  * [30] Arash Rahnama, André T. Nguyen and Edward Raff  “Robust Design of Deep Neural Networks against Adversarial Attacks based on Lyapunov Theory”  In _CoRR_ abs/1911.04636, 2019
  * [31] Max Revay, Ruigang Wang and Ian R. Manchester  “Recurrent Equilibrium Networks: Unconstrained Learning of Stable and Robust Dynamical Models”  In _CoRR_ abs/2104.05942, 2021
  * [32] Sumeet Singh et al.  “Learning stabilizable nonlinear dynamics with contraction-based regularization”  In _The International Journal of Robotics Research_ 40.10-11, 2021, pp. 1123–1150
  * [33] Christian Szegedy et al.  “Intriguing properties of neural networks”  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014
  * [34] Asher Trockman and J. Kolter  “Orthogonalizing Convolutional Layers with the Cayley Transform”  In _ArXiv_ abs/2104.07167, 2021
  * [35] Han Xu et al.  “Adversarial Attacks and Defenses in Images, Graphs and Text: A Review”  In _CoRR_ abs/1909.08072, 2019  arXiv:1909.08072
  * [36] Hanshu Yan, Jiawei Du, Vincent Yan Fu Tan and Jiashi Feng  “On Robustness of Neural Ordinary Differential Equations”  In _ArXiv_ abs/1910.05513, 2019
  * [37] Laurent Younes  “Shapes and Diffeomorphisms”  In _Shapes and Diffeomorphisms_ , 2010
  * [38] Dinghuai Zhang et al.  “You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle”  In _ArXiv_ abs/1905.00877, 2019

17