# Realization Theory Of Recurrent Neural ODEs Using Polynomial System
Embeddings

Martin Gonzalez  Institut de Recherche Technologique SystemX, Palaiseau,
France.  Thibault Defourneau 111The work presented in this paper was carried
out while T. Defourneau was employed by Centre de Recherche en Informatique,
Signal et Automatique de Lille. Trinov, 75001 Paris, France.  Hatem Hajri
Mihaly Petreczky  Centre de Recherche en Informatique, Signal et Automatique
de Lille, UMR CNRS 9189, France.

###### Abstract

In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long
Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into
the class of polynomial systems. This embedding preserves input-output
behavior and can suitably be extended to other neural DE architectures. We
then use realization theory of polynomial systems to provide necessary
conditions for an input-output map to be realizable by an ODE-LSTM and
sufficient conditions for minimality of such systems. These results represent
the first steps towards realization theory of recurrent neural ODE
architectures, which is is expected be useful for model reduction and learning
algorithm analysis of recurrent neural ODEs.

###### keywords:

Realization theory, Neural ODEs, Recurrent Neural Networks, Long Short-Term
Memory, System Identification.

††journal: Systems & Control Letters

## Introduction

Long Short-Term Memory networks (LSTMs) represent a generalization of
recurrent neural networks (RNNs) widely used in text analysis tasks such as
grammar correction and next word prediction. They were introduced [23], and
have been studied intensively ever since, due to their effectiveness for
learning long-term dependencies in comparison with RNN algorithms and its
variants (see [22] and [7] for details).

More recently, the introduction of neural ODEs [12] and other implicit network
architectures [4, 30, 47] has opened the door to new machine learning
paradigms tightly related to dynamical system modeling techniques. For
instance, neural ODEs can be seen as _continuous-time_ dynamical systems and
as infinitesimally connected ResNets while the latter can be seen as Euler
discretizations of neural ODEs. As such, they exploit a rich available theory
on both sides, offering memory efficiency whilst their recurrent analogs have
the ability of handling irregular data and are suitable for tackling
generative problems and time series (particularly in physics), becoming
relevant to both modern machine learning and traditional mathematical
modelling.

In this paper we make the first steps towards developing realization theory of
recurrent neural ODE architectures. We focus on the present article on neural
ODE analogs of RNNs and LSTMs and we aim at characterizing those input-output
maps which can be represented by these systems and understanding the minimal
size of such systems sufficient to be able to represent a given input-output
map.

The motivation for studying realization theory for neural ODEs is that
learning algorithms for such systems from data correspond to system
identification algorithms. Realization theory is central in system
identification as it can be viewed as an attempt to solve a system
identification problem through idealized qualitative analysis, where there is
infinite data and no modelling error. For linear systems, realization theory
[27, 32] allowed to address identifiability, canonical forms and gave rise to
subspace identification algorithms.

In order to make the discussion more precise, let us determine an ambient
class of dynamical systems containing ODE-RNNs, ODE-LSTMs222These will be
specified by a general class of two-scale ODE-based RNNs where multiple
continuous-time memory architectures arise. and polynomial systems as some of
its subclasses, and having all desired properties needed for our study. For a
subset $Z\subset{\mathbb{R}}^{k}$, $k\geq 1$, we denote:

  * 1.

$\mathcal{Z}$ (resp.
$\mathcal{Z}_{\operatorname{pc}},\mathcal{Z}_{\operatorname{ac}}$ ) the set of
continuous (resp. piecewise-continuous, resp. absolutely continuous )
functions from $[0;+\infty[$ to $Z$. Denote by
$\mathcal{Z}_{\operatorname{pco}}$ the set of all piecewise-constant functions
from $[0,+\infty[$ to $Z$ which are constant starting from a certain point,
i.e., $h\in\mathcal{Z}_{\operatorname{pco}}$, if $h$ is piecewise-constant and
there exists $T_{h}\geq 0$ such that the restriction of $h$ to
$[T_{h},+\infty[$ is constant.

  * 2.

$\mathcal{Z}^{f}$ the set of functions from $[0,T]$ to $Z$, for some $T>0$. We
let the reader combine this with the above notations.

The ambient class $\mathcal{F}$ of dynamical systems we will consider in this
paper is described by differential equations of the form

$\left\\{\begin{array}[]{lll}\dot{x}(t)=f\big{(}x(t),u(t)\big{)}\\\
y(t)=g\big{(}x(t)\big{)}\end{array}\right.\quad t\geq 0,$ (1)

with initial condition $x(0)=x_{0}\in{\mathbb{R}}^{n}$, where

  * 1.

$(x,u,y)\in\mathcal{X}_{ac}\times\mathcal{U}_{pco}\times\mathcal{S}_{pc}$
333Allowing $u\in\mathcal{U}_{pc}$ represents a relatively small technical
difficulty that we avoid here for clarity., where
$X={\mathbb{R}}^{n},S={\mathbb{R}}^{\mathrm{p}}$ denote respectively the state
and output spaces and
$U=\\{\alpha_{1},\ldots,\alpha_{K}\\}\subset{\mathbb{R}}^{m}$ is a finite
input space with cardinality $K$;

  * 2.

$f:X\times U\rightarrow X$ is analytic on its first argument and
$g:X\rightarrow S$ is analytic.

We will identify these systems with tuples of the form $\Sigma=(f,g,x_{0})$.
The triple $(m,n,\mathrm{p})$ will be called the format of $\Sigma$.

On the one hand, polynomial systems are a subclass of such systems and many
methods of computational algebra can be used to determine qualitative
properties of such systems such as observability, reachability and minimality.
On the other hand, one can think of ODE-RNNs and ODE-LSTMs as subclasses of
$\mathcal{F}$ which can be parameterized according to some class of learning
weight functions $\theta(t)$, which will be assumed to be constant for
simplicity. As such, under mild assumptions, we can associate polynomial
systems to large classes of ODE-RNNs and ODE-LSTMs and, by doing so, infer
such qualitative properties on these classes. More specifically,

  * 1.

We show that an i-o map can be realized by an ODE-RNN or an ODE-LSTM, only if
it can be realized by a polynomial system, i.e. a non-linear system defined by
vector fields and readout maps which are polynomials. We present an explicit
algorithmic construction of such a polynomial system.

  * 2.

We infer sufficient conditions for
minimality/observability/reachability/accessibility of ODE-RNNs and ODE-LSTMs
from the properties of their associated polynomial systems [37, 34, 5, 44].

  * 3.

We present a necessary condition for existence of a realization by ODE-RNNs
and ODE-LSTMs, using results from realization theory of polynomial systems.
This necessary condition is a generalization of the well-known rank condition
for Hankel matrices of linear systems.

Note that elements in ${\mathcal{F}}$ could be viewed as analytic systems for
which there is an existing realization theory [26, 24, 21]. However, as
analytical functions do not have a finite representation, this approach is not
computationally effective: there are no algorithms for checking minimality,
deciding equivalence of two systems neither transforming a system to a minimal
one. Nevertheless, seen as polynomial systems, computer algebra tools can be
used to address these issues [36]. In addition, since polynomial systems have
much more algebraic structures than analytic systems and the conditions for
minimality/observability/reachability studied here are less restrictive than
those which can be obtained by the analytic approach.

Related work:  To the best of our knowledge, the results of the paper are new.
Observability, controllability and minimality of ODE-RNNs were investigated in
[3, 40, 2], but no results on existence of a realization were provided, and
the results of [3, 40, 2] used certain assumptions on the weights of the ODE-
RNNs. In contrast to [3, 40, 2], in this paper we consider ODE-LSTMs and we
address the issue of existence of a realization by ODE-LSTM. Moreover, the
technique used in this paper is completely different from that of [3, 40, 2].
Rational embeddings and elements of realization theory of a subclass of ODE-
RNNs were considered in [15]. In comparison to [15], the main novelty is that
in this paper we consider both ODE-RNNs and ODE-LSTMs and that detailed proofs
and examples are provided. That is, the current paper extends the results of
[15].

##  1 Preliminaries

We denote ${\mathbb{R}}[X_{1},\dots,X_{n}]$ the algebra of real polynomials in
$n$ variables and denote ${\mathbb{R}}(X_{1},\ldots,X_{n})$ its quotient
field, whose elements are rational functions in $n$ variables. If $A$ is an
integral domain over ${\mathbb{R}}$ then the transcendence degree
$\mathrm{trdeg}A$ of $A$ over ${\mathbb{R}}$ is defined as the transcendence
degree over ${\mathbb{R}}$ of the field $F$ of fractions of $A$ and it equals
the greatest number of algebraically independent elements of $F$ over
${\mathbb{R}}$. Let $n,m$ be two integers and let
${\mathbb{R}}[X_{1},\ldots,X_{n};{\mathbb{R}}^{m}]$ be the set of tuples
$(P_{1},\ldots,P_{m})$ whose $m$ components are polynomials in $n$ variables.

Denote $\mathcal{C}^{\omega}({\mathbb{R}})$ the algebra of real analytic
functions over ${\mathbb{R}}$ and denote $\sigma^{(i)}$ the $i$-th derivative
of $\sigma\in\mathcal{C}^{\omega}({\mathbb{R}})$. We denote
$\mathcal{C}_{1}^{\omega}({\mathbb{R}})$ the subset of
$\mathcal{C}^{\omega}({\mathbb{R}})$ of those analytic functions satisfying

$\sigma^{(1)}=P(\sigma)\,,$ (2)

for some $P\in{\mathbb{R}}[X]$. In this paper we will consider only activation
functions of this sort. In particular, the hyperbolic tangent and the logistic
functions

$\forall x\in{\mathbb{R}}\,,\hskip
5.69054ptth(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\,,\hskip
5.69054ptS(x)=\frac{1}{1+e^{-x}}\,$

are both elements of $\mathcal{C}_{1}^{\omega}({\mathbb{R}})$ as they are the
unique solutions of the differential equations

$th^{\prime}(x)=1-th^{2}(x)\qquad S^{\prime}(x)=S(x)-S^{2}(x)$

with initial conditions $th(0)=0$ and $S(0)=\frac{1}{2}$.

For a map $\sigma:{\mathbb{R}}\rightarrow{\mathbb{R}}$ denote
$\overrightarrow{\sigma}:{\mathbb{R}}^{n}\rightarrow{\mathbb{R}}^{n}$ the map
defined by

$\overrightarrow{\sigma}:(x_{1},\ldots,x_{n})^{T}\mapsto(\sigma(x_{1}),\ldots,\sigma(x_{n}))^{T}$
(3)

Let $\odot$ be the Hadamard product $(A\odot B)_{ij}:=(A)_{ij}(B)_{ij}$ where
$A,B$ matrices of same dimension. In particular, for
$P,Q,R\in{\mathbb{R}}^{n}$, the expression $P\odot Q=R$ is equivalent to
$P_{i}Q_{i}=R_{i}$, for $i\in[n]:=\\{1,\ldots,n\\}$.

###  1.1 Polynomial systems

Define the subclass $\mathcal{F}_{0}$ of $\mathcal{F}$ consisting of those
systems described by

$\left\\{\begin{array}[]{ll}\dot{x}(t)=P_{u(t)}(x(t))\\\\[5.69054pt]
y(t)=g(x(t))\,,\end{array}\right.\quad t\geq 0,$ (4)

with initial condition $x(0)=x_{0}\in{\mathbb{R}}^{n}$, where
$P_{u(t)}\in{\mathbb{R}}[X_{1},\ldots,X_{n};{\mathbb{R}}^{n}]$ for $u(t)\in U$
and $g\in{\mathbb{R}}[X_{1},\ldots,X_{n};{\mathbb{R}}^{\mathrm{p}}]$. These
are polynomial systems and will be identified with tuples
$\mathscr{P}=(\\{P_{u(t)}\\}_{u(t)\in U},g,x_{0})$ and have at most one
solution
$(x,u,y)\in\mathcal{X}_{\operatorname{ac}}\times\mathcal{U}_{\operatorname{pco}}\times\mathcal{S}_{\operatorname{pc}}$
given an initial state $x_{0}$.

######  Definition 1.1.

We say that a polynomial system $\mathscr{P}\in\mathcal{F}_{0}$ is a
polynomial embedding of a system $\Sigma\in\mathcal{F}$ if for any solution
$(x,u,y)\in\mathcal{X}_{\operatorname{ac}}\times\mathcal{U}_{\operatorname{pco}}\times\mathcal{S}_{\operatorname{pc}}$
of $\Sigma$, there is a continuous injection $F$ such that $(F(x),u,y)\in
F(\mathcal{X}_{\operatorname{ac}})\times\mathcal{U}_{\operatorname{pco}}\times\mathcal{S}_{\operatorname{pc}}$
is a solution of $\mathscr{P}$. We will denote such systems by
$\mathscr{P}(\Sigma)$.

###  1.2 The algebra of input-output maps

In this section we introduce causal analytic i-o maps and verify that under
some assumptions their observation algebras are well-defined. For the latter,
we make use of some technical definitions (see [34, Definitions 4.2, 4.3]),
allowing us to define derivations and show that the ring of input-output maps
forms an integral domain structure.

To this end, we remark that any element $u$ of
$\mathcal{U}_{\operatorname{pco}}$ is completely determined by determined by
tuples $t_{1},\ldots,t_{l}\in[0,T],\alpha_{1},\ldots,\alpha_{l}\in U$ for some
$l\geq 1$ such that:

$u_{t_{1},\ldots,t_{l}}^{\alpha_{1},\ldots,\alpha_{l}}(t)=\left\\{\begin{array}[]{rl}\alpha_{i}&\mbox{
if }\;t\in[T_{i-1},T_{i}[,\hskip 2.84526pti\in[l]\\\ \alpha_{l}&\mbox{ if
}\;t\geq T_{l}\end{array}\right.$

for some interval decomposition

$T_{0}=0\,,\hskip 2.84526ptT_{i}=\sum_{j=1}^{i}t_{j},\hskip
2.84526pti\in[l]\quad T=T_{l}.$

Now, for
$p:\mathcal{U}_{\operatorname{pco}}\to\mathcal{S}_{\operatorname{pc}}$, each
component of $p(u):=(p_{1}(u),\ldots,p_{\operatorname{p}}(u))$ is of the form
$p_{k}(u_{t_{1},\ldots,t_{l}}^{\alpha_{1},\ldots,\alpha_{l}})$. This remark
will be used to define the class of causal and analytic input-output maps. In
turn, any input-output map realized by a system from $\mathcal{F}$ belongs to
this latter class.

######  Definition 1.2.

A map $p:\mathcal{U}_{pco}\rightarrow\mathcal{S}_{pc}$ is

  * 1.

_causal_ if, $\forall u,v\in\mathcal{U}_{pco},t\geqslant 0,$ we have:

$\left(u(s)=v(s),\forall
s\in[0,t]\right)\Rightarrow\left(p(u)(s)=p(v)(s),\forall s\in[0,t]\right).$

  * 2.

_analytic_ if $\forall
k\in[\mathrm{p}],\alpha_{1},\ldots,\alpha_{l}\in\mathcal{U},l>0$, the
following function is analytic

$\displaystyle\phi_{p,k,\alpha_{1},\ldots,\alpha_{l}}:([0,+\infty[)^{l}$
$\displaystyle\rightarrow$ $\displaystyle{\mathbb{R}}$
$\displaystyle(t_{1},\ldots,t_{l})$ $\displaystyle\mapsto$ $\displaystyle
p_{k}(u_{t_{1},\ldots,t_{l}}^{\alpha_{1},\ldots,\alpha_{l}})(T_{l}).$

Set $S^{0}={\mathbb{R}}$. Denote $\mathcal{A}(\mathcal{U}_{pco})$ the set of
causal analytic maps $p:\mathcal{U}_{pco}\rightarrow\mathcal{S}^{0}_{pc}$. It
is naturally a ${\mathbb{R}}$\- algebra.

For our purposes, we need to define a derivation operation on
$\mathcal{A}(\mathcal{U}_{pco})$. To this end, we will use the following
observation: for each $\alpha\in U,t>0$ and
$u\in\mathcal{U}_{\operatorname{pco}}$, we can construct an element
$u_{t,\alpha}\in\mathcal{U}_{\operatorname{pco}}$ by setting

$u_{t,\alpha}(\tau)=\left\\{\begin{array}[]{rl}u(\tau)&\tau\in[0,t[\\\
\alpha&\tau\geq t.\\\ \end{array}\right.$

Then for all $\alpha\in U$, we define the map

$D_{\alpha}:\mathcal{A}(\mathcal{U}_{pco})\rightarrow\mathcal{A}(\mathcal{U}_{pco})\quad\varphi\mapsto
D_{\alpha}(\varphi)$

given, for all $u\in\mathcal{U}_{pco},t\geq 0$ by,

$\big{(}D_{\alpha}\varphi(u)\big{)}(t)=\frac{d}{ds}\Big{(}\varphi\big{(}u_{t,\alpha})(t+s)\Big{)}_{|\,s=0}.$

The map $D_{\alpha}$ is a well-defined derivation.

Next, we will argue that $\mathcal{A}(\mathcal{U}_{pco})$ is an integral
domain. To this end notice that the set $\mathcal{U}^{f}_{\operatorname{pc}}$
of piecewise constant functions over finite intervals is closed by interval
truncation, concatenation and piecewise time dilatation. Hence,
$\mathcal{U}^{f}_{\operatorname{pc}}$ is a set of admissible inputs in the
sense of [34, Definition 4.1]. Consequently, we can use the definition of
analytic functions in the sense of [34, Definition 4.3]. Let us denote by
$\mathcal{A}(\mathcal{U}^{f}_{\operatorname{pc}},{\mathbb{R}})$ the set of
analytic functions $\mathcal{U}^{f}_{\operatorname{pc}}\to{\mathbb{R}}$ in the
sense of [34, Definition 4.3]. From [34, Theorem 4.4] it follows that the ring
$\mathcal{A}(\mathcal{U}^{f}_{\operatorname{pc}},{\mathbb{R}})$ is an integral
domain. Below we will present an $\mathbb{R}$-algebra isomorphism between
$\mathcal{A}(\mathcal{U}_{pco})$ and
$\mathcal{A}(\mathcal{U}^{f}_{\operatorname{pc}},{\mathbb{R}})$. The existence
of such an isomorphism then implies that $\mathcal{A}(\mathcal{U}_{pco})$ is
also an integral domain. In order to define this isomorphism, we observe that
for each $v:[0,T_{v}]\to U$ in $\mathcal{U}^{f}_{\operatorname{pc}}$ we can
construct a unique element $u_{v}\in\mathcal{U}_{\operatorname{pco}}$ by
setting

$u_{v}(t)=\left\\{\begin{array}[]{rl}v(t)&t\in[0,T_{v}[\\\ v(T_{v})&t\geq
T_{v}.\\\ \end{array}\right.$

Let us define the map

$\displaystyle\ell:\mathcal{A}(\mathcal{U}_{pco})$
$\displaystyle\longrightarrow$
$\displaystyle\mathcal{A}(\mathcal{U}^{f}_{\operatorname{pc}},{\mathbb{R}})$
$\displaystyle\Big{(}\varphi:\mathcal{U}_{pco}\to\mathcal{S}^{0}_{pc}\Big{)}$
$\displaystyle\longmapsto$
$\displaystyle\Big{(}\tilde{\varphi}:\mathcal{U}^{f}_{\operatorname{pc}}\to{\mathbb{R}}\Big{)}$

given, for each function $v:[0,T_{v}]\to U$ in
$\mathcal{U}^{f}_{\operatorname{pc}}$, by
$\widetilde{\varphi}(v)=\varphi(u_{v})(T_{v})$.

It then follows $\ell$ is an ${\mathbb{R}}$-algebra isomorphism and hennce
$\mathcal{A}(\mathcal{U}_{pco})$ is an integral domain.

The above discussion allow us to introduce the following.

######  Definition 1.3.

Let $p:\mathcal{U}_{pco}\rightarrow\mathcal{S}_{pc}$ be analytic and causal.
The _observation algebra_ $\mathcal{A}_{obs}(p)$ of $p$ is the smallest sub-
algebra $\mathcal{A}\big{(}\mathcal{U}_{pco}\big{)}$ containing each
$p_{k}\in\mathcal{A}_{obs}(p)$ and closed under $D_{\alpha}$, for all
$\alpha\in U$. The field of fractions $\mathcal{Q}_{obs}(p)$ of
$\mathcal{A}_{obs}(p)$ will be called observation field of $p$ and we denote
$trdeg\mathcal{A}_{obs}(p)$ the transcendence degree of $\mathcal{A}_{obs}(p)$
over ${\mathbb{R}}$.

######  Definition 1.4.

Let $\Sigma\in{\mathcal{F}}$ be a system with initial state $x_{0}$. It is
called a (piecewise constant) realisation of a map
$p:\mathcal{U}_{\operatorname{pco}}\rightarrow\mathcal{S}_{pc}$ if for all
$u\in\mathcal{U}_{\operatorname{pco}}$ the unique solution $(x,u,y)$ of
$\Sigma$ such that $x(0)=x_{0}$ satisfies $p(u)\equiv y$.

######  Remark 1.5.

If a system $\Sigma$ realizes an i-o map
$p:\mathcal{U}_{pco}\rightarrow\mathcal{S}_{pc}$, then the polynomial
embedding $\mathscr{P}(\Sigma)$, when it exists, also realizes $p$.

###  1.3 Minimality, reachability and observability of polynomial systems

Let $\Sigma\in{\mathcal{F}}$ be a system of format $(m,n,\mathrm{p})$ as in
the above subsection. The _dimension_ $dim(\Sigma)$ of $\Sigma$ is the
dimension of its state-space.

A polynomial system $\mathscr{P}$ realizing an i-o map $p$ is minimal if there
is no polynomial system $\mathscr{P}^{{}^{\prime}}$ realizing $p$ such that
$\dim(\mathscr{P}^{{}^{\prime}})<\dim(\mathscr{P})$. Define the set of
reachable states of a polynomial system $\mathscr{P}$ as:

$\mathrm{R}_{\mathscr{P}}(\upsilon_{0})=\\{\upsilon(t)\;|\;t\geqslant
0,(\upsilon,u,y)\mbox{ is a solution of
}\mathscr{P},\upsilon(0)=\upsilon_{0}\\}\,$

and recall from [6, Definition 4] that its _observation algebra_
$\mathcal{A}_{obs}(\mathscr{P})$ is the smallest sub-algebra of the ring
$R[X_{1},\ldots,X_{n}]$ which contains $h_{k}$, $k\in[p]$ and which is closed
under taking the formal Lie derivatives with respect to the formal vector
fields $f_{\alpha}=\sum_{i=1}^{n}P_{i,\alpha}\frac{\partial}{\partial X_{i}}$.
Its fraction field $\mathcal{Q}_{obs}(\mathscr{P})$ will be called its
observation field. Finally, $\mathscr{P}$ is minimal if
$\dim(\mathscr{P})=\mathrm{trdeg}\mathcal{A}_{obs}(p)$ (see [38, Lemma 1,
Theorem 4] for details). Notice that the other implication is true for
rational systems, but not for polynomial ones. A polynomial system
$\mathscr{P}$ is

  * 1.

_algebraically reachable_ , if there is no non-trivial polynomial which is
zero on $\mathrm{R}_{\mathscr{P}}(\upsilon_{0})$;

  * 2.

accessible, if $\mathrm{R}_{\mathscr{P}}(\upsilon_{0})$ contains an open
subset of ${\mathbb{R}}^{n}$;

  * 3.

_algebraically observable_ , if
$\mathcal{A}_{obs}(\mathscr{P})=R[X_{1},\ldots,X_{n}]$ (see [34]);

  * 4.

_semi-algebraically observable_ if
$\mathrm{trdeg}(\mathcal{A}_{obs}(\mathscr{P}))=n$ (see [35]);

  * 5.

_observable_ , if for every two distinct initial states
$\upsilon_{0},\upsilon_{0}^{{}^{\prime}}$ there exists solutions
$(\upsilon,u,y)$ and $(\upsilon^{{}^{\prime}},u,y^{{}^{\prime}})$ of
$\mathscr{P}$ such that $\upsilon(0)=\upsilon_{0}$,
$\upsilon^{{}^{\prime}}(0)=\upsilon_{0}^{{}^{\prime}}$, and $y\neq
y^{{}^{\prime}}$.

Accessibility implies algebraic reachability and algebraic observability
implies semi-algebraic observability, and semi-algebraic observability implies
observability. A polynomial system $\mathscr{P}$ is minimal if it is
algebraically reachable and algebraically observable (see [38, Theorem 4] and
[5] for details). Notice that the other implication is true for rational
systems but not for polynomial ones. Algebraic, rational and semi-algebraic
observability and algebraic reachability of polynomial systems can be checked
using methods of computational algebra [36].

Define the reachable set of a system $\Sigma$ in ${\mathcal{F}}$ of format
$(m,n,\mathrm{p})$ by

$\mathrm{R}_{\Sigma}(s_{0})=\\{s(t)\;|\;t\geqslant 0\,,(s,u,y)\mbox{ is a
solution of }\Sigma,s(0)=s_{0}\\}.$

We will say that $\Sigma$ is

  * 1.

_accessible_ , if $\mathrm{R}_{\Sigma}(s_{0})$ contains an open subset of
${\mathbb{R}}^{n}$;

  * 2.

_algebraically reachable_ if there is no non-trivial polynomial which is zero
on $R_{\Sigma}(s_{0})$;

  * 3.

_span-reachable_ , if the linear span of the elements
$\mathrm{R}_{\Sigma}(s_{0})$ is ${\mathbb{R}}^{n}$;

  * 4.

reachable if there exist no linear function which is zero on
$\mathrm{R}_{\Sigma}(s_{0})$;

  * 5.

_weakly observable_ if for every initial state $\hat{s}\in{\mathbb{R}}^{n}$
there is an open subset $V$ of ${\mathbb{R}}^{n}$ such that $\hat{s}\in V$ and
for every $\hat{s}\neq\overline{s}\in V$, there exist solution $(s,u,y)$ and
$(s^{\prime},u,y^{\prime})$ of $\Sigma$, with $s(0)=\hat{s}$ and
$s^{\prime}(0)=\overline{s}$, such that $y\neq y^{\prime}$;

  * 6.

_observable_ if for every initial state $\hat{s}\in{\mathbb{R}}^{n}$,
$V={\mathbb{R}}^{n}$ in the latter definition.

Accessibility implies algebraic reachability which in turn implies span-
reachability. Observability implies weak observability. Finally, if the system
$\Sigma$ realizes an i-o map $p$, is accessible and weakly observable, then it
is minimal dimensional among all the systems from $\mathcal{F}$ realizing $p$
(see [26, Theorem 1.12]).

##  2 Realization theory of dynamical neural networks

###  2.1 Recurrent neural nets and LSTM embeddings

Let us denote by $\mathcal{F}_{1}\subset\mathcal{F}$ the class of systems
described by differential equations

$\Sigma:\left\\{\begin{array}[]{ll}\dot{x}(t)=\overrightarrow{\sigma}\big{(}Ax(t)+Bu(t)\big{)}\\\\[5.69054pt]
y(t)=Cx(t)\end{array}\right.\quad t\geq 0,$ (5)

with initial condition $x(0)=x_{0}\in{\mathbb{R}}^{n}$ and where

  * 1.

$\sigma\in\mathcal{C}_{1}^{\omega}({\mathbb{R}})$ is Lipschitz continuous,

  * 2.

$A\in{\mathbb{R}}^{n\times n}$, $B\in{\mathbb{R}}^{n\times m}$ and
$C\in{\mathbb{R}}^{\mathrm{p}\times n}$ are matrices.

######  Definition 2.1.

An element of $\mathcal{F}_{1}$ will be called a recurrent neural ODE (ODE-
RNN). We will identify such systems with tuples $\Sigma=(A,B,C,\sigma,x_{0})$
and the triple $(m,n,\mathrm{p})$ will be its format.

Let us denote by $\mathcal{F}_{2}$ the subclass of ${\mathcal{F}}$ of systems
described by differential equations of the form

$\Sigma:\left\\{\begin{array}[]{llll}\dot{x}(t)=\mathfrak{U}^{0}x(t)+g^{2}(t)\odot
x(t)+g^{3}(t)\odot g^{1}(t)\\\\[5.69054pt] \dot{z}(t)=g^{4}(t)\\\\[5.69054pt]
y(t)=Cs(t)\end{array}\right.\quad t\geq 0,$ (6)

with initial condition
$s(0)=s_{0}=(x_{0}^{T},z_{0}^{T})^{T}\in{\mathbb{R}}^{2n}$, where

  * 1.

$\sigma=\\{\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4},\sigma_{5}\\}\subset
C_{1}^{\omega}({\mathbb{R}})$ are all Lipschitz continuous,

  * 2.

$\mathfrak{U}=\\{\mathfrak{U}^{0},\mathfrak{U}^{1},\mathfrak{U}^{2},\mathfrak{U}^{3},\mathfrak{U}^{4}\\}\subset{\mathbb{R}}^{n\times
n}$,

  * 3.

$\mathfrak{W}=\\{\mathfrak{W}^{1},\mathfrak{W}^{2},\mathfrak{W}^{3},\mathfrak{W}^{4}\\}\subset{\mathbb{R}}^{n\times
m}$, and $C\in{\mathbb{R}}^{\mathrm{p}\times 2n}$,

  * 4.

$\mathfrak{b}=\\{b^{1},b^{2},b^{3},b^{4}\\}\subset{\mathbb{R}}^{n}$,

  * 5.

$u(t)\in{\mathbb{R}}^{m}$, $s(t)=(x(t)^{T},z(t)^{T})^{T}\in{\mathbb{R}}^{2n}$,
$y(t)\in{\mathbb{R}}^{p}$,

  * 6.

$h(t)=z(t)\odot\overrightarrow{\sigma_{5}}(x(t))$,

  * 7.

$g^{i}(t):=\overrightarrow{\sigma_{i}}(\mathfrak{U}^{i}h(t)+\mathfrak{W}^{i}u(t)+b^{i})$,
for $i\in[4]$.

######  Definition 2.2.

An element of $\mathcal{F}_{2}$ will be called a long short-term neural ODE
(ODE-LSTM). We identify such systems with tuples
$\Sigma=(\mathfrak{U},\mathfrak{W},\mathfrak{b},C,\sigma,s_{0})$, set the
triple $(m,2n,\mathrm{p})$ to be its format and define the (ordered) set
$\sigma$ to be its activation.

We will restrict our attention to solutions of systems $\Sigma$ in
${\mathcal{F}}_{1}$ or in ${\mathcal{F}}_{2}$ of the form
$(x,u,y)\in\mathcal{X}_{\operatorname{ac}}\times\mathcal{U}_{\operatorname{pco}}\times\mathcal{S}_{\operatorname{pc}}$
and we recall that activations $\sigma$ are assumed to be in
$C_{1}^{\omega}({\mathbb{R}})$ so that global existence and uniqueness of
solutions is verified and determined by $u$ and for some initial value (see
[15] for details). Notice that the Euler discretization of such systems
correspond exactly to residual RNNs and residual LSTMs (see 4.1 for details).

######  Theorem 2.3.

All ODE-RNNs and ODE-LSTMs in ${\mathcal{F}}_{1}$ and ${\mathcal{F}}_{2}$ have
polynomial embeddings. Moreover, if an i-o map $p$ has a realization by a ODE-
LSTM, or a ODE-RNN then $p$ is causal, analytic and
$\mathrm{trdeg}~{}A_{obs}(p)<+\infty$.

The proof of this theorem will be done in Subsection 3.1.

We highlight the fact that our definitions of polynomial embeddings are
completely explicit and can be easily implemented algorithmically. Theorem 2.3
allows to infer qualitative properties on neural nets induced by properties of
their polynomial embeddings as we will show in the next section.

###  2.2 Qualitative properties of ODE-LSTMs

A system $\Sigma$ in ${\mathcal{F}}_{2}$ with given activation $\sigma$
realizing an i-o map $p$ is said to be _$\sigma$ -minimal_ there exists no
$\Sigma^{{}^{\prime}}$ in ${\mathcal{F}}_{2}$ with activation $\sigma$, such
that $\Sigma^{{}^{\prime}}$ is a realization of $p$ and
$\dim(\Sigma^{{}^{\prime}})<\dim(\Sigma)$.

######  Lemma 2.4.

Assume that an i-o map $p$ is realized by a system $\Sigma$ in
${\mathcal{F}}_{2}$ with given activation function $\sigma$. If one of the
conditions below holds, then $\Sigma$ is a $\sigma$-minimal realization of
$p$:

  1. 1.

$\mathscr{P}(\Sigma)$ is a minimal realization of $p$,

  2. 2.

$\mathrm{trdeg}\mathcal{A}_{obs}(p)=\operatorname{dim}(\Sigma)$,

  3. 3.

$\mathscr{P}(\Sigma)$ is semi-algebraically observable and algebraically
reachable,

  4. 4.

$\mathscr{P}(\Sigma)$ is algebraically observable and accessible.

###### Proof.

The second point comes from the first and from [38, Proposition 6]. The rest
of the proof is straightforward. ∎

######  Proposition 2.5.

Let $\Sigma$ be a system in ${\mathcal{F}}_{2}$.

  1. 1.

If $\mathscr{P}(\Sigma)$ is accessible, then $\Sigma$ is also accessible.

  2. 2.

If $\mathscr{P}(\Sigma)$ is algebraically reachable, then $\Sigma$ is span-
reachable. In particular, if $\mathscr{P}(\Sigma)$ is accessible, then
$\Sigma$ is span-reachable.

  3. 3.

If $\mathscr{P}(\Sigma)$ is observable, then $\Sigma$ is observable. In
particular, if $\mathscr{P}(\Sigma)$ is algebraically observable, then
$\Sigma$ is observable.

  4. 4.

If $\mathscr{P}(\Sigma)$ is semi-algebraically observable, then $\Sigma$ is
weakly observable.

The proof of the above proposition will be done in Subsection 3.2. Notice that
a similar result was established for ${\mathcal{F}}_{1}$ in [15] with the help
of an auxiliary polynomial embedding.

Note that the variables involved in the polynomial embedding
$\mathscr{P}(\Sigma)$ can be naturally reordered, or simply reduced, following
the expressions of the activation functions $\sigma$ of $\Sigma$.

Accessibility and algebraic/semi-algebraic observability conditions for
rational/ polynomial systems can be checked by using methods of computer
algebra [36]. In contrast, for checking accessibility and (weak) observability
of an ODE-LSTM the only systematic tools are the rank conditions [21, Theorems
2.2, 2.5, 3.1, 3.5] or [24, Corallaries 2.2.5,2.3.5], which are not
computational effective manner for analytic $\sigma$. Notice that minimality
of $\mathscr{P}(\Sigma)$ is a much weaker condition than accessibility and
weak observability of $\Sigma$. This suggests that using realization theory of
polynomial systems is likely to yield more useful results for ODE-LSTMs than
using realization theory of general analytic systems.

##  3 Proofs

###  3.1 Proof of Theorem 2.3

Let us prove that all ODE-RNN have polynomial embeddings. Let
$\Sigma=(A,B,C,\sigma,x_{0})$ be an ODE-RNN with format $(m,n,\mathrm{p})$.
Denote $L=Kn+n$ and consider the bijection

$\displaystyle\phi:[K]\times[n]$ $\displaystyle\longrightarrow$
$\displaystyle[Kn]$ $\displaystyle(r,j)$ $\displaystyle\longmapsto$
$\displaystyle\phi(r,j):=r+K(j-1)$

For $L$ formal symbols $X_{\gamma}$, let us write $X_{\phi(j,r)}$ for the
unique index $\gamma\in[Kn]$ such that $\phi(j,r)=\gamma$.

Then one can construct an associated polynomial system
$\mathscr{P}(\Sigma)=(\\{P_{\alpha}\\}_{\alpha},h,\upsilon_{0})$ with

  * 1.

$P_{\alpha}\in{\mathbb{R}}[X_{1},\ldots,X_{L};{\mathbb{R}}^{Kn}]$

  * 2.

$h\in{\mathbb{R}}[X_{1},\ldots,X_{L};{\mathbb{R}}^{\mathrm{p}}]$

  * 3.

$\upsilon_{0}\in{\mathbb{R}}^{L}$.

as follows:

$\displaystyle
P_{\phi(j,r),u(t)}=U(X_{j,r})\left(\sum_{k=1}^{n}a_{j,k}U_{0}(X_{k,u(t)})\right)$
$\displaystyle P_{Kn+j,r}=U_{0}(X_{j,r}),$ $\displaystyle
h_{k}=\sum_{j=1}^{n}c_{k,j}X_{j+Kn}$
$\displaystyle(\upsilon_{0})_{\phi(j,r)}=\sigma\big{(}e_{j}^{T}(Ax_{0}+B\alpha_{r})\big{)},$
$\displaystyle(\upsilon_{0})_{Kn+j}=e_{j}^{T}x_{0}.$

where $k\in[\mathrm{p}],j\in[n],r\in[K]$ and $u(t)$ ranges the $K$\-
components of $\\{P_{\alpha}\\}_{\alpha}$. By [15, Lemma 2], if $(x,u,y)$ is a
solution of a ODE-RNN $\Sigma$, then $(F(x),u,y)$ is a solution of
$\mathscr{P}(\Sigma)$ where $F:\mathbb{R}^{n}\rightarrow\mathbb{R}^{L}$ is
given by $F(x)=(z_{1},\ldots,z_{nK},x^{T})^{T}$, where
$z_{\phi(j,\alpha)}=\sigma(e_{j}^{T}(Ax+B\alpha))$.

Let us now prove that all ODE-LSTMs have polynomial embeddings. Let
$\Sigma=(\mathfrak{U},\mathfrak{W},\mathfrak{b},C,\sigma,s_{0})$ be a ODE-LSTM
with format $(m,2n,\mathrm{p})$, denote $U=\\{\alpha_{1},\ldots,\alpha_{K}\\}$
its input space and $s(t)=(x(t)^{T},z(t)^{T})^{T}$ its state trajectory.
Consider the ordering

$\displaystyle\phi:[4]\times[n]\times[K]$ $\displaystyle\longrightarrow$
$\displaystyle[4nK],$ $\displaystyle(l,j,r)$ $\displaystyle\longmapsto$
$\displaystyle\phi(l,j,r):=j+n(l-1)+4n(r-1).$

Recall the discussion after (6) the function
$h:[0,+\infty[\rightarrow{\mathbb{R}}^{n}$, i.e.,
$h(t)_{j}=z_{j}(t)\sigma_{5}(x_{j}(t))$, $j=1,2,\ldots,n$. In particular,
there exists a function $\chi:\mathbb{R}^{2n}\rightarrow\mathbb{R}^{n}$,
$\chi(s(t))=h(t)$. Clearly, we can defined a function
$R^{l}:U\times{\mathbb{R}}^{2n}\rightarrow{\mathbb{R}}^{n}$ such that for
$\alpha\in U$, $R(\alpha,s)$ is polynomial in $s$ and

$R^{l}(\alpha,s(t))=\mathfrak{U}^{l}\underbrace{\chi(s(t))}_{h(t)}+\mathfrak{W}^{l}\alpha+b^{l},\quad
l\in[4].$

By writing
$\zeta_{\phi(l,j,r)}(s(t))=\sigma_{l}(e^{T}_{j}(R^{l}(\alpha_{r},s(t))))$ we
can define a map $F:{\mathbb{R}}^{2n}\rightarrow{\mathbb{R}}^{4nK+3n}$ such
that

$F(s(t))=(\zeta_{1}(s(t)),\ldots,\zeta_{4nK}(s(t)),\overrightarrow{\sigma_{5}}(x(t)),s(t)).$

Define $\upsilon(t)=F(s(t))$. Its coordinates are explicitly written as

${\left\\{\begin{array}[]{ll}\upsilon_{\phi(l,j,r)}(t)=\sigma_{l}\big{(}e_{j}^{T}\big{(}R^{l}(\alpha_{r},s(t))\big{)}\big{)}\,,\\\\[5.69054pt]
\upsilon_{4nK+j}(t)=\sigma_{5}(x_{j}(t))\,,\quad j\in[n]\\\\[5.69054pt]
\upsilon_{4nK+n+k}(t)=s_{k}(t)\quad k\in[2n]\end{array}\right.}$ (7)

Let us ease the notation and write

  * 1.

$\upsilon_{\phi(l,j,r)}(t)=\upsilon_{(l-1)n+j,r}(t)$,

  * 2.

$\upsilon_{\alpha_{r}}(t)=(\upsilon_{1,r}(t),\ldots,\upsilon_{4n,r}(t))$,

  * 3.

$\upsilon_{4nK+*}(t)=(\upsilon_{4nK+1}(t),\ldots,\upsilon_{4nK+n}(t))$,

  * 4.

$\upsilon(t)=(\upsilon_{\alpha_{1}}(t),\ldots,\upsilon_{\alpha_{K}}(t),\upsilon_{4nK+*}(t),x(t)^{T},z(t)^{T})^{T}$

  * 5.

$x_{j}(t)=\upsilon_{4nK+n+j}(t)$ and $z_{j}(t)=\upsilon_{4nK+2n+j}(t)$,

Set $\mathfrak{U}^{l}=(\mathfrak{U}^{l}_{i,j})_{i,j=1}^{n}$, $0\leq l\leq 4$,
and $\sigma_{k}^{(1)}=P_{k}(\sigma_{k}),k\in[5]$. For each $r=1,2,\ldots,K$,
define the polynomials $Q_{j,r}$, $Q_{5n+j,r}$, $Q_{ln+j,r}$, $j=1,\ldots,n$,
$l\in[4]$, as follows:

  * 1.

$Q_{j,r}(\upsilon(t))=\sum_{i=1}^{n}\mathfrak{U}^{0}_{j,i}x_{i}(t)+\upsilon_{n+j,r}(t)x_{j}(t)+\upsilon_{2n+j,r}(t)\upsilon_{j,r}(t)$

  * 2.

$Q_{5n+j,r}(\upsilon(t))=Q_{j,r}(\upsilon(t))P_{5}(\upsilon_{4nK+j}(t))$

  * 3.

for $l\in[4]$,

$\begin{split}&Q_{ln+j,r}(\upsilon(t))=P_{l}(\upsilon_{\phi(l,j,r)}(t))\times\\\
&\sum_{i=1}^{n}\Big{(}\mathfrak{U}_{j,i}^{l}\big{(}\upsilon_{\phi(4,i,r)}(t)\upsilon_{4nK+i}(t)+z_{j}(t)Q_{i,r}(\upsilon(t))P_{5}(\upsilon_{4nK+i}(t)\big{)}\Big{)}\end{split}$

We set $\upsilon(t)$ to be the state variables of the polynomial system
$\mathscr{P}(\Sigma)=(\\{Q_{u}\\}_{u},C,\upsilon_{0})$ defined,for
$l\in[4],j\in[n]$, by

$\displaystyle\dot{x}_{j}(t)=Q_{j,u(t)}(\upsilon(t))\,,~{}x_{j}(0)=e_{j}^{T}x_{0},$
$\displaystyle\dot{z}_{j}(t)=\upsilon_{3n+j,u(t)}(t)\,,~{}z_{j}(0)=e_{j}^{T}z_{0}$
$\displaystyle\dot{\upsilon}_{(l-1)n+j,u(t)}(t)=Q_{ln+j,u(t)}(\upsilon(t))\,,$
$\displaystyle\dot{\upsilon}_{4nK+j}(t)=Q_{5n+j,u(t)}(\upsilon(t))$
$\displaystyle\upsilon_{\phi(l,j,r)}(0)=\sigma_{l}\big{(}e_{j}^{T}(R^{l}(\alpha_{r},s(0)))\big{)}\,,\hskip
5.69054pt$ $\displaystyle\upsilon_{4nK+j}(0)=\sigma_{5}(e_{j}^{T}x_{0})$
$\displaystyle h(0)=z_{0}\odot\overrightarrow{\sigma_{5}}(x_{0})\,,$
$\displaystyle y(t)=Cs(t)$

with initial state $\upsilon_{0}=\upsilon(0)$.

######  Lemma 3.1.

Let $\upsilon(t)=F(s(t))$. If $(s,u,y)$ is a solution of a LSTM $\Sigma$, then
$(\upsilon,u,y)$ is a solution of $\mathscr{P}(\Sigma)$.

###### Proof.

Let $(s,u,y)$ be a solution of $\Sigma$. To show that $(\upsilon,u,y)$ is a
solution of $\mathscr{P}(\Sigma)$, it suffices to prove that $\upsilon$
satisfies the differential equation (3.1). For $u(t)\in U$, we calculate the
first time-derivative of $x_{j}(t)$. Following (6), we know that we have

$\displaystyle\dot{x}_{j}(t)$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}U^{0}_{j,i}x_{i}(t)+g^{2}_{j}(t)x_{j}(t)$
$\displaystyle+g^{2}_{j}(t)\sigma_{1}\big{(}e_{j}^{T}(R^{1}(u(t),t))\big{)}$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n}U^{0}_{j,i}x_{i}(t)+\sigma_{2}\big{(}e_{j}^{T}(R^{2}(u(t),t))\big{)}x_{j}(t)$
$\displaystyle+\sigma_{3}\big{(}e_{j}^{T}(R^{3}(u(t),t))\big{)}\,\sigma_{1}\big{(}e_{j}^{T}(R^{1}(u(t),t))\big{)}$
$\displaystyle=$ $\displaystyle Q_{j,u(t)}(\upsilon(t))\,,$

as desired. Now notice that $\dot{z}_{j}(t)=\upsilon_{\phi(4,j,r)}(t)$, where
$r,t$ are such that $u(t)=\alpha_{r}\in U$. Then we get

$\displaystyle\dot{\upsilon}_{4nK+j}(t)$ $\displaystyle=$
$\displaystyle\frac{d}{dt}\big{(}\sigma_{5}(x_{j}(t))\big{)}$ $\displaystyle=$
$\displaystyle\dot{x}_{j}(t)P_{5}\big{(}\sigma_{5}(x_{j}(t))\big{)}$
$\displaystyle=$ $\displaystyle
Q_{j,u(t)}(\upsilon(t))P_{5}(\upsilon_{4nK+j}(t))\,.$

Now take $l\in[4]$. We obtain

$\displaystyle\dot{\upsilon}_{\phi(l,j,r)}(t)=\frac{d}{dt}\Big{(}\sigma_{l}\big{(}e_{j}^{T}(R^{l}(\alpha_{r},s(t))\big{)}\Big{)}$
$\displaystyle=$ $\displaystyle
P_{l}(\upsilon_{\phi(l,j,r)}(t))\Big{(}\sum_{i=1}^{n}\mathfrak{U}_{j,i}^{l}\dot{h}(t)\Big{)}$
$\displaystyle=$ $\displaystyle P_{l}(\upsilon_{\phi(l,j,r)}(t))$
$\displaystyle\Big{(}\sum_{i=1}^{n}\mathfrak{U}_{j,i}^{l}\big{(}\dot{z}_{i}(t)\sigma_{5}(x_{i}(t))+z_{i}(t)\dot{x_{i}}(t)P_{5}(\sigma_{5}(x_{i}(t)))\big{)}\Big{)}$
$\displaystyle=$ $\displaystyle
P_{l}(\upsilon_{(\phi(l,j,r)}(t))\Big{(}\sum_{i=1}^{n}\mathfrak{U}_{j,i}^{l}\big{(}\upsilon_{3n+i,u(t)}(t)\upsilon_{4nK+i}(t)$
$\displaystyle\qquad\qquad\qquad\quad+z_{i}(t)Q_{i,u(t)}(\upsilon(t))P_{5}(\upsilon_{4nK+i}(t))\big{)}\Big{)}$
$\displaystyle=$ $\displaystyle Q_{ln+j,u(t)}(\upsilon(t))\,.$

This completes the proof of Lemma 3.1. ∎

We observe that the map $F$ is a smooth map, in particular it is a continuous
map and Lemma 3.1 showed that, for all $u\in\mathcal{U}_{pco}$, if $(s,u,y)$
is a solution of $\Sigma$, then $(\upsilon,u,y)$ is a solution of
$\mathscr{P}(\Sigma)$, with

$\upsilon(t)=F(s(t))\,,\quad\forall t\geqslant 0,$ (8)

where $s(t)=(x(t)^{T},z(t)^{T})^{T}\in{\mathbb{R}}^{2n}$.

Finally, if an i-o map
$p:\mathcal{U}_{pco}\rightarrow\mathcal{S}_{\operatorname{pc}}$ is realized by
a system $\Sigma$ in ${\mathcal{F}}_{1}$ or ${\mathcal{F}}_{2}$, then the
polynomial embedding $\mathscr{P}(\Sigma)$ also realizes $p$ by [15, Lemma 2]
and Lemma 3.1. Thus $p$ is causal and analytic and
$\operatorname{trdeg}\mathcal{A}_{obs}(p)<\infty$ (by [5, Theorem 3], [37,
Theorem 5.16]).

This concludes the proof of Theorem 2.3.

###  3.2 Proof of Proposition 2.5

First, we prove that, if $\mathscr{P}(\Sigma)$ is accessible, then $\Sigma$ is
also accessible. By definition of the map $F$ from (7), we obtain

$F\big{(}R_{\Sigma}(s_{0})\big{)}=R_{\mathscr{P}(\Sigma)}(\upsilon_{0})\,,$
(9)

where $R_{\Sigma}(s_{0})$ and $R_{\mathscr{P}(\Sigma)}(\upsilon_{0})$ are
respectively the reachable set of $\Sigma$ and of $\mathscr{P}(\Sigma)$, and
$\upsilon_{0}=\upsilon(0)$. Now suppose that the polynomial system
$\mathscr{P}(\Sigma)$ is accessible, i.e. there exists a non-empty open set
$O$ included in $R_{\mathscr{P}(\Sigma)}(\upsilon_{0})$. Thus $F^{-1}(O)$ is a
non-empty open set (because $F$ is a continuous map) included in
$R_{\Sigma}(s_{0})$, so that $\Sigma$ is accessible.

Next, we prove that, if $\mathscr{P}(\Sigma)$ is algebraically reachable, then
$\Sigma$ is span-reachable. Suppose that $\mathscr{P}(\Sigma)$ is
algebraically reachable, i.e. there is no non-trivial polynomial which
vanishes on the reachable set $R_{\mathscr{P}(\Sigma)}(\upsilon_{0})$. Take
$u\in\mathcal{U}_{pco}$ such that $(s,u,y)$ a solution of $\Sigma$, with
$s(t)=(x(t)^{T},z(t)^{T})^{T}\in{\mathbb{R}}^{2n}$, for $t\geqslant 0$.
Consider $(\upsilon,u,y)$ the solution of $\mathscr{P}(\Sigma)$ obtained by
Lemma 3.1. Assume that $\Sigma$ is not span-reachable, i.e. there exist reals
$\lambda_{1},\ldots,\lambda_{2n}$ such that

$\displaystyle\sum_{j=1}^{n}\lambda_{j}x_{j}(t)+\sum_{j=1}^{n}\lambda_{n+j}z_{j}(t)=0\,.$

Then taking the first derivative of the above equation gives

$\displaystyle\sum_{j=1}^{n}\lambda_{j}Q_{j,u(t)}(\upsilon(t))+\sum_{j=1}^{n}\lambda_{n+j}\upsilon_{3n+j,u(t)}(t)=0\,,$

which is a contradiction, because there exists at least one non-trivial
polynomial (given by the above equation) vanishing on the reachable set
$R_{\mathscr{P}(\Sigma)}(\upsilon_{0})$.

Next, we prove that, if $\mathscr{P}(\Sigma)$ is observable, the $\Sigma$ is
also observable. Take $s_{0},s_{0}^{\prime}\in{\mathbb{R}}^{2n}$ two initial
states of $\Sigma$ such that $s_{0}\neq s_{0}^{\prime}$. Thus we have
$\upsilon_{0}=F(s_{0})\neq F(s_{0}^{\prime})=\upsilon_{0}^{\prime}$, because
the map $F$, defined in (7), is injective. As $\mathscr{P}(\Sigma)$ is
observable, there exist solutions $(\upsilon,u,y)$ and
$(\upsilon^{\prime},u,y^{\prime})$ of $\mathscr{P}(\Sigma)$ such that
$\upsilon(0)=\upsilon_{0}$ and $\upsilon^{\prime}(0)=\upsilon_{0}^{\prime}$,
and $y\neq y^{\prime}$. By Lemma 3.1 and by (9), there exist solutions
$(s,u,y)$ and $(s^{\prime},u,y^{\prime})$ of $\Sigma$ with $s(0)=s_{0}$ and
$s^{\prime}(0)=s_{0}^{\prime}$, satisfying

$\forall t\geqslant 0\,,\quad F(s(t))=\upsilon(t)\,,\quad\mbox{and}\quad
F(s^{\prime}(t))=\upsilon^{\prime}(t)\,,$ (10)

and $y\neq y^{\prime}$. Thus $\Sigma$ is observable, as desired. Now if
$\mathscr{P}(\Sigma)$ is algebraically reachable, then it is observable, by
[5, Proposition 3]. Then $\Sigma$ is observable, by the above arguments.

Finally, we prove that, if $\mathscr{P}(\Sigma)$ is semi-algebraically
observable, then $\Sigma$ is weakly observable. By [35, Proposition 4.20,
Corollary 4.22], the polynomial system $\mathscr{P}(\Sigma)$ is weakly
observable. Let $s_{0}\in{\mathbb{R}}^{2n}$ be an initial state of $\Sigma$,
and let $\upsilon_{0}=F(s_{0})$. As $\mathscr{P}(\Sigma)$ is weakly
observable, there exist a open set $V$ with $\upsilon_{0}\in V$ such that, for
all $\upsilon_{0}^{\prime}\neq\upsilon_{0}\in V$, there exist solutions
$(\upsilon,u,y)$ and $(\upsilon^{\prime},u,y^{\prime})$ satisfying
$\upsilon(0)=\upsilon_{0}$, $\upsilon^{\prime}(0)=\upsilon_{0}^{\prime}$, and
$y\neq y^{\prime}$. We set $U=F^{-1}(V)$ which is an open set (because $F$ is
a continuous map) such that $s_{0}\in U$. Take $s_{0}^{\prime}\neq s_{0}\in
U$, and set $\upsilon_{0}^{\prime}=F(s_{0}^{\prime})\in V$. By injectivity of
$F$, $\upsilon_{0}^{\prime}\neq\upsilon_{0}\in V$. Thus we can find solutions
$(\upsilon,u,y)$ and $(\upsilon^{\prime},u,y^{\prime})$ of
$\mathscr{P}(\Sigma)$ as above. Then there exist solutions $(s,u,y)$ and
$(s^{\prime},u,y^{\prime})$ of $\Sigma$ such that (10) holds. We know that
$y\neq y^{\prime}$. Thus $\Sigma$ is weakly observable.

##  4 Examples

Linear systems are a particular case of ODE-LSTMs, by taking $\sigma_{1}$ the
identity map, $\sigma_{2}=\sigma_{4}=0$ (the constant functions equal to $0$),
$\sigma_{3}=1$ (the constant function equal to $1$), $b^{1}\in\mathbb{R}^{n}$
the trivial vector and $C\in\mathbb{R}^{p\times 2n}$ a matrix of the form
$(\widetilde{C},0)$ with $\widetilde{C}\in\mathbb{R}^{p\times n}$. Also ODE-
RNNs are particular cases of ODE-LSTMs by taking $\sigma_{2},=\sigma_{4}=0$,
$\sigma_{3}=1$, $\sigma_{1}$ to be any non-constant continuous globally
Lipschitz function, and taking $b_{1}$ to be trivial.

######  Remark 4.1.

For a suitable choice of $C$ and $U^{0}$, and taking
$\sigma_{3}=\sigma_{4}=\sigma_{2}$, the Euler discretization of an ODE-LSTM in
${\mathcal{F}}_{2}$ is given by

$\Sigma_{discretized}:\left\\{\begin{array}[]{lllll}x(k+1)=x(k)+f(k)\odot
x(k)+i(k)\odot g^{1}(k)\\\\[5.69054pt]
f(k)=\overrightarrow{\sigma_{2}}\big{(}\mathfrak{U}^{2}h(k)+\mathfrak{W}^{2}u(k)+b^{2}\big{)}\\\\[5.69054pt]
i(k)=\overrightarrow{\sigma_{3}}\big{(}\mathfrak{U}^{3}h(k)+\mathfrak{W}^{3}u(k)+b^{3}\big{)}\\\\[5.69054pt]
z(k+1)=z(k)+\overrightarrow{\sigma_{2}}\big{(}\mathfrak{U}^{4}h(k)+\mathfrak{W}^{4}u(k)+b^{4})\\\\[5.69054pt]
h(k)=z(k)\odot\overrightarrow{\sigma_{5}}(x(k))\\\\[5.69054pt]
x(0)=x_{0}\,,\;z_{0}=z(0)\\\\[5.69054pt] y(k)=z(k).\end{array}\right.$

This is closely related to LSTM networks defined in [18], where, at the $k$th
step, $x(k),f(k),i(k)$ are usually called respectively the cell, the forget
gate and the input gate and $u(k),z(k)$ respectively the input and the output.
The presence of a skip connection makes this actually a residual LSTM which,
being the discretization of an ODE, enjoys of (gradient) stable dynamics,
contrary to vanilla LSTMs. In addition, our construction should be readily
applicable to LEM networks defined in [43] which are also presented as
discretized two-gated recurrent neural ODEs.

######  Example 4.2.

Let us exhibit an ODE-LSTM whose polynomial embedding is minimal. Set
$\mathcal{U}=\\{u\\}\subset{\mathbb{R}}$ and consider:

$\Sigma~{}:~{}\left\\{\begin{array}[]{lllll}\dot{x}(t)=\sigma(x(t)z(t)+u)x(t)\\\
\dot{z}(t)=0\\\ x(0)=0\,,z(0)=a\,,~{}\mbox{with}~{}a\neq 0\\\
y(t)=x(t),\end{array}\right.$ (11)

where $\sigma$ is the sigmoid function. Here
$\sigma_{1}=\sigma_{3}=\sigma_{4}=0$, $\sigma_{5}$ is the identity, $U^{2}=1$,
$W^{2}=1$, $b^{2}$ is the zero vector and $C=(1,0)\in\mathbb{R}^{1\times 2}$.
We can rewrite (11) as:

$\Sigma~{}:~{}\left\\{\begin{array}[]{lllll}\dot{x}(t)=\sigma(ax(t)+u)x(t),\\\
x(0)=0,\\\ y(t)=x(t).\end{array}\right.\,$ (12)

Then, as $m(\sigma)=2$, $\mathscr{P}(\Sigma)$ is given by

$\mathscr{P}(\Sigma)~{}:~{}\left\\{\begin{array}[]{lllll}\dot{\upsilon}_{1,u}=a\upsilon_{2}\upsilon_{1,u}(1-\upsilon_{1,u}),\\\
\dot{\upsilon_{2}}=\upsilon_{1,u}\upsilon_{2},\\\
\upsilon_{1,u}(0)=\sigma(u)\,,\,\upsilon_{2}(0)=0\,,\\\
y(t)=h(\upsilon(t))=\upsilon_{2}(t).\end{array}\right.\,$

where $\upsilon(t)=(\upsilon_{1,u}(t),\upsilon_{2}(t))^{T}$
$\upsilon_{1,u}(t)=\sigma(ax(t)+u)$ and $\upsilon_{2}(t)=x(t)$, for
$t\geqslant 0$. It is clear that $\upsilon_{1,u}$ and $\upsilon_{2}$ belong to
the observation field $\mathcal{Q}_{obs}(\mathscr{P}(\Sigma))$.

Now we prove that $dim(\mathscr{P}(\Sigma))=2$. Observe that

$\forall t\geqslant
0\,,\quad\frac{\dot{\upsilon}_{1,u}(t)}{1-\upsilon_{1,u}(t)}=a\dot{\upsilon_{2}}(t),$

and notice that $\upsilon_{1,u}(t)<1$, because the sigmoid function takes
value in $]0;1[$. By taking the primitives of both sides of the above
equation, there is $c\in\mathbb{R}$ such that

$\forall t\geqslant 0\,,\quad ln(1-\upsilon_{1,u}(t))=a\upsilon_{2}(t)+c\,,$

which implies that

$\forall t\geqslant
0\,,\quad\upsilon_{1,u}(t)=1-Ke^{a\upsilon_{2}(t)}\,,~{}\mbox{with}~{}K=e^{c}\,.$

Thus, there is no non-trivial polynomial which vanishes on the set of
reachable states of $\mathscr{P}(\Sigma)$, i.e. $\mathscr{P}(\Sigma)$ is
algebraically reachable. We conclude that
$dim(\mathscr{P}(\Sigma))=2=m(\sigma)$ which implies that $\Sigma$ is minimal.

In what follows we set
$\mathcal{U}=\\{\alpha_{1},\alpha_{2}\\}\subset\mathbb{R}$ with
$\alpha_{1}\neq\alpha_{2}$.

######  Example 4.3.

Let us exhibit a reduction method of polynomial ODE-LSTM embeddings. Consider:

$\Sigma~{}:~{}\left\\{\begin{array}[]{lllll}\dot{x}(t)=\sigma(h(t)+u(t))x(t)+\sigma(h(t)+u(t)),\\\
\dot{z}(t)=\sigma(h(t)+u(t)),\\\ h(t)=x(t)z(t),\\\ x(0)=0\,,z(0)=0,\\\
y(t)=z(t),\end{array}\right.\,$

where we take $\sigma_{1}=\sigma_{2}=\sigma_{4}=\sigma$ to be the sigmoid
function, $\sigma_{3}=1$ and $\sigma_{5}$ the identity map. Here
$n(\sigma)=2$, $k(\sigma)=1$ so that $m(\sigma)=4$. For $k=1,2$ and
$t\geqslant 0$, using Lemma 3.1, let us set
$\upsilon_{1,\alpha_{k}}(t)=\sigma_{1}(x(t)z(t)+\alpha_{k})=\sigma(x(t)z(t)+\alpha_{k})=\upsilon_{2,\alpha_{k}}(t)=\upsilon_{4,\alpha_{k}}(t)$,
$\upsilon_{3,\alpha_{k}}(t)=1$, and $\upsilon_{5}(t)=x(t)$. Thus,
$\mathscr{P}(\Sigma)$ is simply given by

$\left\\{\begin{array}[]{lllll}\dot{\upsilon}_{1,\alpha_{1}}=(\upsilon_{1,u(t)}\upsilon_{2}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{2})\upsilon_{1,\alpha_{1}}(1-\upsilon_{1,\alpha_{1}})\,,\\\
\dot{\upsilon}_{1,\alpha_{2}}=(\upsilon_{1,u(t)}\upsilon_{2}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{2})\upsilon_{1,\alpha_{2}}(1-\upsilon_{1,\alpha_{2}})\,,\\\
\dot{\upsilon}_{2}=\upsilon_{1,u(t)}\upsilon_{2}+\upsilon_{1,u(t)}\,,\\\
\dot{\upsilon}_{3}=\upsilon_{1,u(t)}\,,\\\
\upsilon_{1,\alpha_{k}}(0)=\sigma(\alpha_{k}),\,\upsilon_{2}(0)=\upsilon_{3}(0)=0\,,~{}\mbox{for}~{}k=1,2\,,\\\
y(t)=\upsilon_{3}(t),\end{array}\right.\,$

where we set $x(t)=\upsilon_{2}(t)$ and $z(t)=\upsilon_{3}(t)$.

######  Example 4.4.

Let us now exhibit an accessible ODE-LSTM whose polynomial embedding is not
accessible:

$\Sigma~{}:~{}\left\\{\begin{array}[]{lllll}\dot{x}(t)=\sigma(h(t)+u(t))^{2}\,,\\\
\dot{z}(t)=\sigma(h(t)+u(t))\,,\\\ h(t)=x(t)z(t)\,,\\\
s_{0}=(x(0),z(0))=(0,0)^{T}\,,\\\ y(t)=z(t).\end{array}\right.\,$ (13)

In this case, $n(\sigma)=2$. We first prove that $\Sigma$ is accessible. Let
$f_{\alpha_{1}},f_{\alpha_{2}}:\mathbb{R}^{2}\rightarrow\mathbb{R}^{2}$ be
vector fields generated by $\Sigma$. We denote by
$\mathcal{L}_{\Sigma}(s_{0})$ the smallest Lie algebra containing
$f_{\alpha_{1}},f_{\alpha_{2}}$ and closed by Lie brackets. We then have

$\left\\{\begin{array}[]{ll}\frac{1}{\sigma(\alpha_{2})}f_{\alpha_{2}}(s_{0})-\frac{1}{\sigma(\alpha_{1})}f_{\alpha_{1}}(s_{0})=(\sigma(\alpha_{2})-\sigma(\alpha_{1}))\;(1,0)^{T}\in\mathbb{R}^{2}\,,\\\\[5.69054pt]
\frac{1}{\sigma(\alpha_{2})^{2}}f_{\alpha_{2}}(s_{0})-\frac{1}{\sigma(\alpha_{1})^{2}}f_{\alpha_{1}}(s_{0})=\frac{\sigma(\alpha_{1})-\sigma(\alpha_{2})}{\sigma(\alpha_{1})\sigma(\alpha_{2})}\;(0,1)^{T}\in\mathbb{R}^{2}.\end{array}\right.$

As $\sigma(\alpha_{1}),\sigma(\alpha_{2})>0$ and
$\sigma(\alpha_{1})\neq\sigma(\alpha_{2})$ because $\alpha_{1}\neq\alpha_{2}$
and $\sigma$ is bijective and takes values in $]0;1[$, then
$dim\mathcal{L}_{\Sigma}(s_{0})=2=n(\sigma)$. By [25, Theorem 3.10], $\Sigma$
is accessible. Now $\mathscr{P}(\Sigma)$ is given by

$\left\\{\begin{array}[]{llllll}\dot{\upsilon}_{1,\alpha_{1}}=(\upsilon_{1,u(t)}^{2}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{2})\upsilon_{1,\alpha_{1}}(1-\upsilon_{1,\alpha_{1}})\,,\\\
\dot{\upsilon}_{1,\alpha_{2}}=(\upsilon_{1,u(t)}^{2}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{2})\upsilon_{1,\alpha_{2}}(1-\upsilon_{1,\alpha_{2}})\,,\\\
\dot{\upsilon_{2}}=\upsilon_{1,u(t)}^{2}\,,\\\
\dot{\upsilon_{3}}=\upsilon_{1,u(t)}\,,\\\
\upsilon_{1,\alpha_{k}}(0)=\sigma(\alpha_{k}),\,\upsilon_{2}(0)=\upsilon_{3}(0)=0\,,~{}\mbox{for}~{}k=1,2\,,\\\
y(t)=\upsilon_{3}(t),\end{array}\right.\,$

where, for $t\geqslant 0,\alpha\in\mathcal{U}$, we set
$\upsilon_{1,\alpha}(t)=\sigma(x(t)z(t)+\alpha)$, $\upsilon_{2}(t)=x(t)$ and
$\upsilon_{3}(t)=z(t)$. We denote
$\upsilon_{0}=(\sigma(\alpha_{1}),\sigma(\alpha_{2}),0,0)^{T}\in\mathbb{R}^{4}$
the initial state of $\mathscr{P}(\Sigma)$ and let
$g_{\alpha_{1}},g_{\alpha_{2}}:\mathbb{R}^{4}\rightarrow\mathbb{R}^{4}$ be
vector fields generated by the polynomial system $\mathscr{P}(\Sigma)$. We
denote $\mathcal{L}_{\mathscr{P}(\Sigma)}(\upsilon_{0})$ the smallest Lie
algebra containing $g_{\alpha_{1}},g_{\alpha_{2}}$ and closed by Lie brackets.
It is easy to prove that
$(g_{\alpha_{1}}(\upsilon_{0}),g_{\alpha_{2}}(\upsilon_{0}))$ is linearly
independent, so that $2\leqslant
dim\mathcal{L}_{\mathscr{P}(\Sigma)}(\upsilon_{0})$.

######  Example 4.5.

Let us exhibit an ODE-LSTM $\Sigma$ which, seen as an analytic system, is both
accessible and weakly observable and thus is minimal:

$\Sigma~{}:~{}\left\\{\begin{array}[]{lllll}\dot{x}(t)=\sigma(h(t)+u(t))x(t)+\sigma(h(t)+u(t))^{2},\\\
\dot{z}(t)=\sigma(h(t)+u(t)),\\\ h(t)=x(t)z(t),\\\ x(0)=0\,,z(0)=0,\\\
y(t)=z(t),\end{array}\right.\,$ (14)

where $\sigma_{1}=\sigma_{2}=\sigma_{3}=\sigma_{4}=\sigma$ is the sigmoid
function and $\sigma_{5}$ the identity map. Here $n(\sigma)=2$ and
$\mathscr{P}(\Sigma)$ is given by:

$\left\\{\begin{array}[]{llllll}\dot{\upsilon}_{1,\alpha_{1}}=(\upsilon_{1,u(t)}\upsilon_{2}\upsilon_{3}+\upsilon_{1,u(t)}^{2}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{2})\upsilon_{1,\alpha_{1}}(1-\upsilon_{1,\alpha_{1}})\,,\\\
\dot{\upsilon}_{1,\alpha_{2}}=(\upsilon_{1,u(t)}\upsilon_{2}\upsilon_{3}+\upsilon_{1,u(t)}^{2}\upsilon_{3}+\upsilon_{1,u(t)}\upsilon_{2})\upsilon_{1,\alpha_{2}}(1-\upsilon_{1,\alpha_{2}})\,,\\\
\dot{\upsilon_{2}}=\upsilon_{1,u(t)}\upsilon_{2}+\upsilon_{1,u(t)}^{2}\,,\\\
\dot{\upsilon_{3}}=\upsilon_{1,u(t)}\,,\\\
\upsilon_{1,\alpha_{1}}(0)=\sigma(\alpha_{1})\,,\upsilon_{1,\alpha_{2}}(0)=\sigma(\alpha_{2})\,,\upsilon_{2}(0)=\upsilon_{3}(0)=0\,,\\\
y(t)=\upsilon_{3}(t).\end{array}\right.\,$

Now, for $t\geqslant 0$, set
$\upsilon_{1,\alpha_{k}}(t)=\sigma(x(t)z(t)+\alpha_{k})$, where $k=1,2$, and
set $\upsilon_{2}(t)=x(t)$ and $\upsilon_{3}(t)=z(t)$.
Notice that equations (13) hold in this case, so
$dim\mathcal{L}_{\Sigma}(s_{0})=2=n(\sigma)$. Then, by [25, Theorem 3.10], we
conclude that $\Sigma$ is accessible (i.e. its reachable set from
$s_{0}=(0,0)^{T}$ contains a non-empty open set).

Now let’s prove that $\Sigma$ is weakly observable by proving that
$\mathscr{P}(\Sigma)$ is semi-algebraically observable. Denote
$g_{\alpha_{1}},g_{\alpha_{2}}:\mathbb{R}^{4}\rightarrow\mathbb{R}^{4}$ the
vector fields generated by $\mathscr{P}(\Sigma)$, and $L_{g_{\alpha}}$ the Lie
derivative operator along $g_{\alpha}$, for $\alpha\in\mathcal{U}$. The output
map of $\mathscr{P}(\Sigma)$ is $h=\upsilon_{3}$. It is then clear that
$\upsilon_{3}$ belongs to the observation algebra of $\mathscr{P}(\Sigma)$.
Moreover we have

$\displaystyle L_{g_{\alpha_{1}}}h=\upsilon_{1,\alpha_{1}}\,,\quad
L_{g_{\alpha_{2}}}h=\upsilon_{1,\alpha_{2}}\,,$

which shows that $\upsilon_{1,\alpha_{1}},\upsilon_{1,\alpha_{2}}$ also belong
to the observation algebra $\mathcal{A}_{obs}(\mathscr{P}(\Sigma))$ of
$\mathscr{P}(\Sigma)$ as the latter algebra is closed under Lie derivatives
along $g_{\alpha_{1}},g_{\alpha_{2}}$. Thus, in
$\mathcal{Q}_{obs}(\mathscr{P}(\Sigma))$ we have

$\upsilon_{2}=\frac{\frac{L_{g_{\alpha_{2}}}L_{g_{\alpha_{1}}}h}{L_{g_{\alpha_{1}}h(1-L_{g_{\alpha_{1}}})}}-h(L_{g_{\alpha_{2}}}h)^{2}}{h(L_{g_{\alpha_{2}}}h)+L_{g_{\alpha_{2}}}h}\,.$

This proves that $\mathscr{P}(\Sigma)$ is semi-algebraically observable. Thus
$\Sigma$ is weakly observable by Lemma 2.5. As $\Sigma$ is seen as an analytic
system and is accessible and weakly observable, it is then minimal.

##  5 Conclusions and perspectives

We have shown that i-o maps realized by large classes of recurrent neural ODEs
(namely ODE-RNNs and ODE-LSTMs) can be represented by polynomial systems, and
we used this fact to derive necessary and sufficient conditions for the
existence of realizations by such systems and their minimality. Future
research will be directed towards improving these results to derive a complete
realization theory for ODE-LSTMs and apply them to formulating theoretical
guarantees for learning ODE-LSTMs.

## References

  * [1] F. Albertini, P. Dai Pra, Forward accessibility for recurrent neural networks, IEEE Transactions on Automatic Control, 1995, pages 1962-1968, volume 40.
  * [2] F. Albertini, E.D. Sontag, For Neural Networks, Function Determines Form, Neural Networks 6(1993): 975-990.
  * [3] F. Albertini and E.D. Sontag, State observability in recurrent neural networks, Systems f3 Control Letters 22(1994): 235-244.
  * [4] S. Bai, J. Z. Kolter, V. Koltun, Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
  * [5] Z. Bartoszewicz, Minimal polynomial realizations, Mathematics of control, signals, and systems, 1988, pages 227-237, volume 1.
  * [6] Z. Bartoszewicz, Rational systems and observation fields, Systems & Control Letters 9, 1987, pages 379-386.
  * [7] Y. Bengio, P. Simard, and P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Transactions on neural networks, 5(2), pages 157-166, 1994.
  * [8] S. P. Bhat and D. S. Bernstein, Geometric homogeneity with applications to finite time stability, Mathematics of Control, Signals and Systems, vol. 17, pp. 101 – 127, 2005.
  * [9] A. Fermanian, P. Marion, J.-P. Vert and G. Biau, Framing RNN as a kernel method: A neural ODE approach Advances in Neural Information Processing Systems 34, 2021.
  * [10] J. Bochnak, M. Coste, and M.-F. Roy, Real algebraic geometry, Springer-Verlag, Berlin Heidelberg, 1998.
  * [11] A. Caterini and D. E. Chang, Deep Neural Networks in a Mathematical Framework, Springer, SpringerBriefs in Computer Science, 2018.
  * [12] R.T. Chen, Y. Rubanova, J. Bettencourt and D. Duvenaud, Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  * [13] D. Cox, J. Little, and D. O’Shea, Ideals, varieties, and algorithms: An introduction to computational algebraic geometry and commutative algebra, Springer, third edition, 2007.
  * [14] P. D’Alessandro, A. Isidori and A. Ruberti, Realization and structure theory of bilinear dynamical systems. SIAM J. Control and Optimization, 517–535, 1974.
  * [15] T. Defourneau, M. Petreczky, Realization theory of recurrent neural networks and rational systems, IEEE 58th Conference on Decision and Control (CDC), 2019.
  * [16] F.A. Gers, N. Schraudolph, and J. Schmidhuber, Learning precise timing with LSTM recurrent networks, Journal of Machine Learning Research, 3, pages 115-143, 2002.
  * [17] F.A. Gers, and J. Schmidhuber, LSTM recurrent networks learn simple context free and context sensitive languages, IEEE Transactions on Neural Networks, 12(6), pages 1333-1340, 2001.
  * [18] F.A. Gers, and J. Schmidhuber, Recurrent nets that time and count, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, volume 3, 2000.
  * [19] J. Hanson, M. Raginsky, E. Sontag Learning Recurrent Neural Net Models of Nonlinear Systems, Proceedings of the 3rd Conference on Learning for Dynamics and Control, PMLR 144, 2021.
  * [20] K. He, X.Zhang, S. Ren and J. Sun, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [21] R. Hermann, and A. J. Krener, Nonlinear controllability and observability, IEEE Transactions on automatic control, 1977, pages 728-740, volume 22.
  * [22] S. Hochreiter, Untersuchungen zu dynamischen neuronalen Netzen, Institut für Informatik Technische Universität München, Germany, Ph.D thesis under the direction of W. Brauer, 1991.
  * [23] S. Hochreiter, and J. Schmidhuber, Long short term memory, Neural Computation, 9(8), pages 1735-1780, 1997.
  * [24] A. Isidori, Nonlinear control systems, Springer, third edition, 2013.
  * [25] B. Jakubczyk, Introduction to geometric nonlinear control ; controllability and Lie bracket, Lectures given at the Summer School on Mathematical Control Theory, Trieste 3-28 Septembre 2001, International Atomic Energy Agency (IAEA), Volumn 38, 2002.
  * [26] B. Jakubczyk, Realization theory for nonlinear systems: three approaches. In M. Fliess and M. Hazenwinkel, editors, Algebraic and Geometric Methods in Nonlinear Control Theory, pages 3–31. D. Reidel Publishing Company, Dordrecht, 1986.
  * [27] T. Kailath, Linear Systems, Prentice-Hall, New Jersey, 1979, ISBN 978-0-13-536961-6.
  * [28] R.E. Kalman, Mathematical description of linear dynamical systems. SIAM J. Control and Optimization, 1(2): 152–159, 1963.
  * [29] R.E. Kalman, On minimal partial realization of a linear input-output map, Topics in Mathematical Systems Theory, 1969.
  * [30] P. Kidger, J. Morrill, J. Foster and T. Lyons, Neural Controlled Differential Equations for Irregular Time Series, Advances in Neural Information Processing Systems, 33, 2020.
  * [31] E. Kunz, Introduction to commutative algebra and algebraic geometry, Birkhäuser, Boston,1985.
  * [32] A. Lindquist and G. Picci, Linear Stochastic Systems, Series in Contemporary Mathematics. Springer-Verlag Berlin Heidelberg, 2015, vol. 1.
  * [33] L. Ljung, System identification: theory for the user (second edition), Precice-Hall, Englewood Cliffs, NJ, 1999.
  * [34] J. Němcová, Rational Systems in Control and System Theory, Centrum Wiskunde & Informatica (CWI), Amsterdam, Ph.D. thesis under the direction of Jan H. van Schuppen, 2009.
  * [35] J. Němcová, M. Petreczky and J.H. van Schuppen, Realization theory of Nash systems,  SIAM J. Control & Optimization, 2013, pages 3386-3414, volume 51.
  * [36] J. Němcová, M. Petreczky, J. H. van Schuppen, Observability reduction algorithm for rational systems, IEEE Conference on Decision and Control (CDC), 2016, pages 5738-5743.
  * [37] J. Němcová, and J.H. van Schuppen, Realization theory for rational systems: The existence of rational realizations, SIAM J. Control Optim., 2009, pages 2840-2856, volume 48.
  * [38] J. Němcová, and J.H. van Schuppen, Realization theory for rational systems: Minimal rational realizations, Acta Applicandae Mathematica, 2010, pages 605-626, volume 110.
  * [39] I. Sustkever, Training Recurrent Neural Networks, PhD thesis, University of Toronto, 2013.
  * [40] Y. Qiao, E.D. Sontag, Further results on controllability of recurrent neural networks, Systems & Control Letters 36, 1999, pages 121-129.
  * [41] L. Rosier, Homogeneous Lyapunov function for homogeneous continuous vector field, Systems & Control Letters, vol. 19, pp. 467 - 473, 1992.
  * [42] Y. Rubanova, R.T.Q. Chen, D.K. Duvenaud, Latent ordinary differential equations for irregularly-sampled time series Advances in neural information processing systems 32, 2019.
  * [43] T. K. Rusch, S. Mishra, N. B. Erichson and M. W. Mahoney, Long Expressive Memory for Sequence Modeling, International Conference on Learning Representations, 2022.
  * [44] Y. Wang and E.D. Sontag, Algebraic differential equations and rational control systems, SIAM J. Control Optim., 30(5):1126–1149, 1992.
  * [45] E.D. Sontag, Y. Wang, and A. Megretski, Input classes for identification of bilinear systems, IEEE Transactions Autom. Control, 2009, pages 195–207, volume 54.
  * [46] J.M. Van den Hof, System theory and system identification of compartmental systems, PhD thesis, University of Groningen, 1996.
  * [47] B. Tzen, M. Raginsky, Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883, 2019.
  * [48] O. Zariski and P. Samuel, Commutative algebra I, II., Springer, 1958.