# Geometric deep learning approach to knot theory

Lennart Jaretzki
<EMAIL_ADDRESS>
Leipzig Germany

###### Abstract

In this paper, we introduce a novel way to use geometric deep learning for
knot data by constructing a functor that takes knots to graphs and using graph
neural networks. We will attempt to predict several knot invariants with this
approach. This approach demonstrates high generalization capabilities.

##  1 Introduction

Recent papers have researched the application of machine learning and neural
networks to knot theory. One common approach to learning knots is to use their
braid representation as input instead. This has the downside that extra
crossings have to be added. Also, it makes the data less dense, and the neural
network has to abstract the underlying knot. The neural networks used are
often dense models that operate on a fixed-sized input, sequence-to-sequence
models, transformers, and other natural language processing approaches [1, 9,
10]. There was some research conducted on using a more geometric approach by
using rectangular diagrams, but they used LSTMs instead of a geometric deep
learning approach, and in the data representation, arbitrary choices had to be
made [3]. It seems much more intuitive to use methods from geometric deep
learning to learn knot data [13]. We address these issues by proposing a
method to use graph neural networks to learn knot data. For this purpose, we
construct a functor that takes knots to graphs, which are learned by graph
neural networks. We will attempt to predict both geometric and combinatorial
invariants. We will conduct experiments using the graph transformer operator.
The code for this project can be found on github.

##  2 Graph construction

Using graph neural networks for knot learning requires a way to represent a
given knot as a graph. One method to do this would be by using black graphs,
but they suffer from the problem that the graph is often a self-connected
graph, i.e. it contains loops. Another problem is that a large number of black
graphs are only pseudographs. This makes it hard to apply graph neural
networks to them. Our proposed functor is similar and takes knots to graphs by
taking the faces of the graph to the vertices of the graph and taking edges
between crossings to perpendicular graph edges. This induces taking the
crossings of the knot to the faces of the resulting graph. By definition, the
resulting graph is planar. This brings up the question if there is a unique
way to reconstruct the knot from a given knot. Such that a unique
reconstruction is possible, the edges of the resulting graph need to have two
edge attributes. The first attribute is a measure of distance that is
necessary to preserve the general structure of the knot, and the second edge
attribute is a boolean that indicates if a given edge between crossings is
altering between ”over” and ”under”. Since only the alteration is given, it is
not possible to differentiate chirality and handedness. Note that this
procedure generalizes to links. It is probably also possible to generalize the
procedure to virtual knots.

Figure 1: The described Functor at the example of the $6\\_3$ with the
crossing in the middle flipped.

###  2.1 Rigorous definition

Given a knot, first all edges of the knot diagram need to be enumerated, i. e.
labeled with increasing numbers (increasing while we go along the paths that
connect two crossings). Next, the alteration of each edge needs to be
determined. Now that we have gathered this information, we can transform our
knot into a graph.

$V:=\\{F|\partial F=\bigcup\limits_{i\in A}e_{i}\text{ with }\forall i\in
A:e_{i}\text{ is edge in the knot diagram},F\text{ is connected region}\\}\,.$
(1)

$E:=\\{(F_{0},F_{1})|F_{0},F_{1}\in V,F_{0}\cap F_{1}\neq\emptyset\\}\,.$ (2)

Note that this has a similar problem to black graphs because two faces may
have a common edge in two or more distinct positions. This is relatively rare
and did not happen once in the dataset. It does occur when the knot gets
twisted or two knots get added.

$G=(V,E).$ (3)

Let $e\in E$:

$l(e):=\text{label of the edge}$ (4)

Note that by definition, if and only if $a,b\in E$, and $a$ and $b$ are on the
opposite side of the same crossing, then:

$l(a)\equiv l(b)\pm 1\mod{2\cdot\text{\\#Crossings(K)}}$ (5)

###  2.2 Restore knot

To restore the knot, it is necessary to find a planar embedding of the knot
graph.

$p:V\to\mathbb{R}^{2}$ (6)

Such that $p$ is planar, i. e. there are no two graph edges that cross each
other. To be able to reconstruct $K$ uniquely, we need the additional
condition that, given an arbitrary graph face, this face is composed of
exactly four edges. It is also necessary that opposing edges fulfill “Eq.
(5)”. To reconstruct the knot diagram, every face gets mapped to a crossing.
The outgoing edges of the constructed knot crossing get connected through the
graph edges they are affiliated with, such that they connect with the
crossings created by the adjoint graph faces. The last step is to assign an
arbitrary crossing where the in-going path goes ”under” After that, you can
derive the state of the other crossings via the second edge attribute.

######  Theorem 2.1.

Two embeddings $p_{0}$, $p_{1}$ of $G$ restore ambient isotopic knots.

###### Proof.

Let $v_{0}\in V$ be an arbitrary vertex. Next, $p_{0}(G)$ and $p_{1}(G)$ need
to be embedded in $\mathbb{R}^{3}$. Define $w_{p}(v):=(p(v),d(v_{0},v))$ where
$d$ is a distance measure (the number of edges of the shortest path connecting
$v_{0}$ and $v$). We can now transform $w_{p_{0}}(G)$ to $w_{p_{1}}(G)$ by
shifting the vertecies. Define $u_{t}(w_{p}(g)):=(1-t)\cdot
w_{p_{0}}(g)+t\cdot w_{p_{1}}(g)$ with $0\leq t\leq 1$. We can find a
corresponding knot embedded in $\mathbb{R}^{3}$ for each $u_{t}$ by embedding
the edges of $G$ as straight lines and then embedding the crossings of the
knot in the center of the faces that fulfill “Eq. (5)” while letting the knot
edges intersect with the midpoint of the embedded graph edges. This procedure
is continuous along $t$ 111You may have to find a homeomorphism of the
embedding such that injectivity is preserved throughout the entire process..
Since we can also lift the two graph embeddings continuously. Applying
$w_{p_{1}}^{-1}\circ u_{1}\circ w_{p_{0}}$ induces ambient isotopy 222Note
that the face may be flipped when applying $u$ but since we only have
information about the crossings in relation to each other, this makes no
difference.. ∎

##  3 Dataset

The foundation of the dataset consists of 2977 knots from the Knotinfo
Database [2]. These consist of all knots up to 12 crossings.

###  3.1 Data augmentation

Since the purpose is to predict invariance, we can easily extend the dataset
by applying random Reidemeister moves to the given knots. The exact algorithm
used for shuffling the knots is described in A. This returns a new randomly
shuffled knot, which is ambient isotopic to the original knot and is non-
simplifiable, i. e. can’t be solved by just undoing the first and second
reidemeister moves. For every knot, there are 89 shuffled versions of the
knot, such that the entire dataset of shuffled knots consists of 264,953
knots. The mean number of crossings of these knots is 34.56, with a standard
derivation of 11.38. Also, a dataset with 2977 knots that have at least 70
crossings was generated. Later, this dataset will be used to test how well the
neural networks can generalize to bigger knots. The mean number of knot
crossings in this dataset is 91.3, with a standard deviation of 14.91.

###  3.2 Validation datasets

One validation dataset is generated by a test train split, with $20\%$ of the
data being used for validation. We will also attempt to measure the
generalization capabilities by training only on knots that have 11 or fewer
crossings when solved and testing on knots that have 12 crossings when solved.
As well as testing on the knots that have at least 70 crossings.

###  3.3 Data preperation

We need to address two issues before we can use our data as input for neural
networks. First of all, in the distance measure, there was an arbitrary edge
chosen to enumerate the edges. To work around the problem, we instead
calculate the distance between the adjacent knot edges. Because neural
networks work better with smooth values that lie in a predefined interval, we
also apply an activation function to the distances.

Figure 2: Two crossings connected by one edge. The labels of the enumerated
edges are $a,b,c,d$. T.

The edge attribute that can be used by the neural network is given by

$1-2^{-\frac{d}{d_{s}}}.$ (7)

with $d\equiv\frac{\lvert(a+b-(c+d))\lvert}{2}\mod{\\#Crossings(K)}$ being the
distance, i. e. the minimal number of edges you need to walk to get from one
to the other. If $d$ is higher than $d_{s}$ the edge attribute will be higher
than $\frac{1}{2}$ and gets squished, while values lower than are more
distinguishable and can therefore be processed easier. This may be
advantageous because edges that are closer together are more intertwined and
are easier to process. $d_{s}=15$ was arbitrarily chosen. Also, to make the
edge attribute altering usable for the network, $altering$ and $notaltering$
get mapped to $1$ and $-1$ respectively.

##  4 Experiments

###  4.1 Models

Since the information about the knot is given by the edge attributes, it is
necessary to choose a graph neural network architecture that can process edge
attributes. One popular approach is graph convolution networks, which work by
aggregating information from neighboring nodes and edges in a graph using a
learned convolution operation to produce a new representation for each node in
the graph [8]. The transformer is a powerful architecture that utilizes
attention mechanisms and has been adeptly used in the graph transformer [4, 7,
15]. We observed that the results were best when using the graph transformer.
We apply the layer, which we batch normalize, and apply the $tanh$ activation
function on this operation, which is then stacked four times. Since we want to
make graph-level predictions, we then use global pooling, which means that we
calculate the $mean$, $max$, and $min$ of all node features of the graph.
These are then concatenated with 10 additional features of the given knot
333Similar to [1] we use the following features: alternating, fibered,
positive braid closure, small or large, crossing number, signature, arc index,
determinant, and Rasmussen invariant . They are then again normalized, which
increases results when generalizing to larger knots. These are the same as in
the given tensor, which is then the input for a 4-layer multi-layer perceptron
with 25 hidden neurons per layer. And one neuron that is predicting the
feature. Then the mean absolute error will be calculated, and the model
parameters will be updated using backpropergation and the Adam optimizer.

###  4.2 Training

The model will be fitted to the data over multiple epochs with a learning rate
of $0.001$. Comparison of different graph neural networks fitting different
knot invariants. The models were trained, and implemented using pytorch, and
pytorch geometric [5].

###  4.3 Results

The models have been trained over 200 epoches, and the one with the highest
accuracy has been picked. In the case of volume, the mean distance to the
correct result is displayed.

Feature | used Additional | Validation Dataset | Large Knot Dataset |  $\leq$ 11 Results 444When the model is trained only on knots that have crossings that are less than or equal to 11 when simplified and then tested on knots that have 12 crossings when simplified.
---|---|---|---|---
Q-Positivity | Yes | 92.38% | 91.57% | 91.13%
Rasmussen-S | Yes | 95.63 | 95.36% | -
Volume | Yes | 0.5498 | 0.7905 | 0.8485
Volume | No | 2.0442 | 2.6864 | -
Ozsvath-Szabo $\tau$ | Yes | 99.78% | 98.79% | 99.92%

We can observe that even though the accuracy is not that high, the networks
demonstrate high generalization both to larger knots and to knots that have
more crossings when simplified. Also note that it works better on
geometric/hyperbolic than on numerical/combinatorical [11, 14].

##  5 Similar work

While conducting this research, a similar approach has been published that
utilizes geometric deep learning to predict if two given 3-manifolds are
homeomorphic [16]. This approach uses plumbing graphs as input. They also use
techniques from reinforcement learning and graph attention networks.

##  Appendix A Shuffle algorithm

Given a knot and a complexity $c$ the here used shuffle algorithm starts by
applying $2\cdot c$ times random type 1 and type 2 reidemeister moves with
probabilities of $20\%$ and $80\%$ respectively. After this, it applies $5$
random type 1 and 2 reidemeister moves with the same probabilities, after
which $\lfloor\frac{c}{20}\rfloor$ random type 3 reidemeister moves are
applied. This process is repeated $c$ times. After this, all excess type 1 and
2 reidemeister moves are reversed. Increasing $c$ increases the probability of
getting bigger and more complex knots. This algorithm was implemented using
[6].

## References

  * [1] M. C. Hughes, A Neural Network Approach to Predicting and Computing Knot Invariants, J. Knot Theory Ramifications 29 (2020) 2050005.
  * [2] C. Livingston and A. H. Moore, KnotInfo: Table of Knot Invariants, knotinfo.math.indiana.edu (April 24, 2023).
  * [3] Kauffman, L. & Russkikh, N. & Taimanov, I.. (2022). Rectangular knot diagrams classification with deep learning. Journal of Knot Theory and Its Ramifications. 31. 10.1142/S0218216522500675.
  * [4] Shi, Yunsheng & Zhengjie, Huang & Feng, Shikun & Zhong, Hui & Wang, Wenjing & Sun, Yu. (2021). Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification. 1548-1554. 10.24963/ijcai.2021/214.
  * [5] Fey, Matthias & Lenssen, Jan. (2019). Fast Graph Representation Learning with PyTorch Geometric.
  * [6] M. Culler, N. M. Dunfield, M. Goerner, and J. R. Weeks, SnapPy, a computer program for studying the geometry and topology of 3-manifolds, http://snappy.computop.org
  * [7] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph Attention Networks. International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=rJXMpikCZ
  * [8] Corso, Gabriele & Cavalleri, Luca & Beaini, Dominique & Lio, Pietro & Veličković, Petar. (2020). Principal Neighbourhood Aggregation for Graph Nets.
  * [9] Lisitsa, Alexei & Salles, Mateo & Vernitski, Alexei. (2022). An application of neural networks to a problem in knot theory and group theory (untangling braids). 10.48550/arXiv.2206.05373.
  * [10] Halverson, James & Gukov, Sergei & Sulkowski, Piotr & Ruehle, Fabian. (2021). Learning to Unknot. Machine Learning: Science and Technology. 2. 10.1088/2632-2153/abe91f.
  * [11] Jejjala, Vishnu & Kar, Arjun & Parrikar, Onkar. (2019). Deep learning the hyperbolic volume of a knot. Physics Letters B. 799. 135033. 10.1016/j.physletb.2019.135033.
  * [12] Craven, Jessica & Hughes, Mark & Jejjala, Vishnu & Kar, Arjun. (2023). Learning knot invariants across dimensions. SciPost Physics. 14. 10.21468/SciPostPhys.14.2.021.
  * [13] Bronstein, Michael & Bruna, Joan & Cohen, Taco & Veličković, Petar. (2021). Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.
  * [14] Grünbaum, Daniel. (2022). Narrowing the Gap between Combinatorial and Hyperbolic Knot Invariants via Deep Learning.
  * [15] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L. & Polosukhin, I. Attention Is All You Need. (2017)
  * [16] Putrov, Pavel & Ri, Song. (2023). Graph Neural Networks and 3-Dimensional Topology.


# Fourier Neural Networks: A Comparative Study

Abylay Zhumekenov, Malika Uteuliyeva, Rustem Takhanov, Zhenisbek Assylbekov,
Alejandro J. Castro
Department of Mathematics, Nazarbayev University &Olzhas Kabdolov
BTS Digital

###### Abstract

We review neural network architectures which were motivated by Fourier series
and integrals and which are referred to as Fourier neural networks. These
networks are empirically evaluated in synthetic and real-world tasks. Neither
of them outperforms the standard neural network with sigmoid activation
function in the real-world tasks. All neural networks, both Fourier and the
standard one, empirically demonstrate lower approximation error than the
truncated Fourier series when it comes to approximation of a known function of
multiple variables.

##  1 Introduction

Over the past few years, neural networks have re-emerged as powerful machine-
learning models, yielding state-of-the-art results in fields such as computer
vision, speech recognition, and natural language processing. In this work we
explore several neural network architectures, the authors of which were
inspired by Fourier series and integrals. Such architectures will be
collectively referred to as Fourier Neural Networks (FNNs). First FNNs were
proposed in 80s and 90s, but they are not widely used nowadays. Is there any
reasonable explanation for this, or were they simply not given enough
attention? To answer this question we perform empirical evaluation of the
FNNs, found in the existing literature, on synthetic and real-world datasets.
We are mainly interested in the following hypotheses: Is any of the FNNs
superior to others? Does any FNN outperform conventional feedforward neural
network with the logistic sigmoid activation? Our experiments show that the
FNN of Gallant and White (1988) outperforms all other FNNs, and that all FNNs
are not better than the standard feedforward neural architecture with sigmoid
activation function except the case of modeling synthetic data.

##  2 Preliminaries

Notation. We let $\mathbb{Z}$ and $\mathbb{R}$ denote the integer and real
numbers, respectively. Bold-faced letters ($\mathbf{x}$, $\mathbf{y}$) denote
vectors in $d$-dimensional Euclidean space $\mathbb{R}^{d}$, and plain-faced
letters ($x$, $f$) denote either scalars or functions.
$\langle\cdot,\cdot\rangle$ denotes inner product:
$\langle\mathbf{x},\mathbf{y}\rangle:=\sum_{j=1}^{d}x_{j}y_{j}$; and
$\|\cdot\|$, $\|\cdot\|_{2}$ denote the Euclidean norm:
$\|\mathbf{x}\|:=\|\mathbf{x}\|_{2}:=\sqrt{\langle\mathbf{x},\mathbf{x}\rangle}$.

Feedforward Neural Networks. Following a standard convention, we define a
feedforward neural network with one hidden layer of size $n$ on inputs in
$\mathbb{R}^{d}$ as

$\mathbf{x}\mapsto
v_{0}+\sum_{k=1}^{n}v_{k}\sigma(\langle\mathbf{x},\mathbf{w}_{k}\rangle+b_{k})$
(1)

where $\sigma(\cdot)$ is the activation function, and
$v_{k},b_{k}\in\mathbb{R}$, $\mathbf{w}_{k}\in\mathbb{R}^{d}$, $k=1,\ldots,n$,
are parameters of the network. The universal approximation theorem (Hornik et
al. (1989); Cybenko (1989)) states that a feedforward network (1) with any
“squashing” activation function $\sigma(\cdot)$, such as the logistic sigmoid
function, can approximate any Borel measurable function $f(\mathbf{x})$ with
any desired non-zero amount of error, provided that the network is given
enough hidden layer size $n$. Universal approximation theorems have also been
proved for a wider class of activation functions, which includes the now
commonly used $\mathrm{ReLU}$ (Leshno et al. (1993)). The neural network (1)
with logistic sigmoid activation $\sigma(x):=1/(1+e^{-x})$ is referred to as
standard or vanilla feedforward neural network.

Fourier Series. Let $f(\mathbf{x})$ be a function integrable in the
$d$-dimensional cube $[-\pi,\pi]^{d}$. The Fourier series of the function
$f(\mathbf{x})$ is the series

$\sum_{\mathbf{k}\in\mathbb{Z}^{d}}\hat{f}_{\mathbf{k}}e^{i\langle\mathbf{x},\mathbf{k}\rangle},$
(2)

where the numbers $\hat{f}_{\mathbf{k}}$, called Fourier coefficients, are
defined by

$\hat{f}_{\mathbf{k}}:=(2\pi)^{-d}\underset{[-\pi,\pi]^{d}}{\int}f(\mathbf{y})e^{-i\langle\mathbf{y},\mathbf{k}\rangle}d\mathbf{y},$

Conceptually, the feedforward neural network with one hidden layer (1) and the
partial sum of the Fourier series (2) are similar in a sense that both are
linear combinations of non-linear transformations of the input $\mathbf{x}$.
The major differences between them are as follows:

  * •

The Fourier series has a direct access to the function $f(\mathbf{x})$ being
approximated, whereas the neural network does not have it — instead it is
usually given a training set of pairs
$\\{\mathbf{x}_{i},f(\mathbf{x}_{i})+\epsilon_{i}\\}$, where $\epsilon_{i}$ is
a noise (error).

  * •

The coefficients and linear transformations of the input in the Fourier series
are fixed, but they are trainable in the neural network and are subject to
estimation based on the training set
$\\{\mathbf{x}_{i},f(\mathbf{x}_{i})+\epsilon_{i}\\}$.

There exists a variety of results on convergence of different types of partial
sums (rectangular, square, spherical) of the multiple Fourier series (2) to
$f(\mathbf{x})$ in various senses (uniform, mean, almost everywhere). We refer
the reader to the works of Alimov et al. (1976) and Alimov et al. (1977) for a
survey of such results. It seems that the existence of such convergence
guarantees has motivated several authors to design the activation functions
for (1) in such a way that the resulting neural networks mimic the behavior of
the Fourier series (2). In the next section we give a brief overview of such
networks.

##  3 Fourier Neural Networks

FNN of Gallant and White (1988): The earliest attempt on making a neural
network resemble the Fourier series is due to Gallant and White (1988) who
have suggested the “cosine squasher”

$\sigma_{\rm GW}(x):=\begin{cases}0,&x\in(-\infty,-\frac{\pi}{2}),\\\
\frac{1}{2}\left(\cos\left(x+\frac{3\pi}{2}\right)+1\right),&x\in[-\frac{\pi}{2},\frac{\pi}{2}],\\\
1,&x\in(\frac{\pi}{2},+\infty),\end{cases}$ (3)

as an activation function in the feedforward network (1). Moreover, they show
that when additionally the connections $\mathbf{w}_{i}$, $b_{i}$ from input to
hidden layer are hardwired in a special way, the obtained feedforward network
yields a Fourier series approximation to a given function $f(\mathbf{x})$.
Thus, such networks possess all the approximation properties of Fourier series
representations. In particular, approximation to any desired accuracy of any
square integrable function can be achieved by such a network, using
sufficiently many hidden units. McCaffrey and Gallant (1994) showed that the
squared approximation error for sufficiently smooth functions is of order
$O(n^{-1})$, where $n$ is the network’s hidden layer size. We notice here that
Barron (1993) has established the same order of the approximation error for
the feedforward networks with any sigmoidal activation111A bounded measurable
function $\phi(x)$ on the real line is called sigmoidal if $\phi(x)\to 1$ as
$x\to+\infty$ and $\phi(x)\to 0$ as $x\to-\infty$. and when the function being
approximated $f(\mathbf{x})$ has a bound on the first moment of the magnitude
distribution of the Fourier transform. FNN of Gallant and White (1988) is
denoted as $f_{\rm GW}$.

FNN of Silvescu (1999): Another attempt to mimic the behavior of the Fourier
series by a neural network was done by Silvescu (1999), who introduced the
following FNN:

$f_{\rm S}:\mathbf{x}\mapsto v_{0}+\sum_{k=1}^{n}v_{k}\sigma_{\rm
S}(\mathbf{x};\boldsymbol{\omega}_{k},\boldsymbol{\phi}_{k}),$ (4)

with

$\sigma_{\rm
S}(\mathbf{x};\boldsymbol{\omega}_{k},\boldsymbol{\phi}_{k}):=\prod_{j=1}^{d}\cos(\omega_{kj}x_{j}+\phi_{kj}),$
(5)

where $\boldsymbol{\omega}_{k},\boldsymbol{\phi}_{k}\in\mathbb{R}^{d}$,
$v_{k}\in\mathbb{R}$ are trainable parameters. As we can see, Silvescu’s FNN
(4) does not follow the framework of the standard feedforward neural networks
(1), and moreover its activation function is not sigmoidal. Figure 1 depicts
the difference between (1) and (4) for the case when $d=3$ and $n=2$.

Figure 1: Standard feedforward NN (top) vs Silvescu’s Fourier NN (bottom). In
the standard NN non-linearity is applied on top of the linear transformation
of the whole input, whereas in the Silvescu’s network non-linearity is applied
separately to each component of the input vector.

Because of this difference, the result of Barron (1993) is not applicable to
Silvescu’s FNN. However, we conjecture that the same convergence rate is valid
for the Silvescu’s FNN. The proof (or disproof) of this conjecture is deferred
to our future work.

FNN of Liu (2013): More recently, several authors suggested the following
architecture

$f_{\rm L}:\mathbf{x}\mapsto
v_{0}+\sum_{k=1}^{n}v_{k}\cos(\langle\mathbf{w}_{k},\mathbf{x}\rangle+b_{k})+u_{k}\sin(\langle\mathbf{p}_{k},\mathbf{x}\rangle+q_{k}),$
(6)

where $\mathbf{w}_{k},\mathbf{p}_{k}\in\mathbb{R}^{d}$,
$b_{k},q_{k}\in\mathbb{R}$ are either hardwired or trainable, and
$v_{k},u_{k}\in\mathbb{R}$ are trainable parameters. Tan (2006) explored
aircraft engine fault diagnostics using (6), Zuo and Cai (2005), Zuo and Cai
(2008), Zuo et al. (2009) used it for the control of a class of uncertain
nonlinear systems. The above-mentioned authors did not provide rigorous
mathematical analysis of this architecture, instead they used it as an ad-hoc
solution in their engineering tasks. Although this FNN fits into the general
feedforward framework (1), its activations are not sigmoidal, and thus the
result of Barron (1993) is not applicable here as well. Liu (2013) empirically
evaluated (6) on various datasets and showed that in certain cases it
converges faster than the feedforward network with sigmoid activation and has
equally good predicting accuracy and generalization ability. Also, only in the
work of Liu (2013) all the weights in (6) are allowed to be trainable, hence
we refer to this architecture as $f_{\rm L}$.

##  4 Empirical Evaluation

In this section we will perform empirical evaluation of the Fourier neural
networks $f_{\rm GW}$, $f_{\rm S}$, $f_{\rm L}$ from Section 3 against vanilla
feedforward network (1) with sigmoid activation222I.e. we put
$\sigma(x):=\frac{1}{1+\exp(-x)}$ in (1). on synthetic and real-world
datasets. By “synthetic datasets” we mean datasets generated from a known
function. In this case we can also compare the performance of Fourier neural
networks to the approximation error given by the partial Fourier series.

###  4.1 Synthetic tasks

We try to approximate a function of one variable $x\mapsto|x|$,
$x\in[-\pi,\pi]$, and a function of $d=100$ variables:
$\mathbf{x}\mapsto\mathbb{I}[\|\mathbf{x}\|\leq 1]$,
$\mathbf{x}\in\\{\mathbf{x}\in\mathbb{R}^{100}:\,\|\mathbf{x}\|\leq 2\\}$,
where $\mathbb{I}[\cdot]$ is the indicator function.333This means that
$\mathbb{I}[\|\mathbf{x}\|\leq 1]=1$ if $\|\mathbf{x}\|\leq 1$, and $0$
otherwise. In both cases we sampled $5\cdot 10^{5}$ data instances uniformly
from the domains of the functions. To each instance, we associated a target
value according to the target function $|x|$ or $\mathbb{I}[\|\mathbf{x}\|\leq
1]$. Another $10\cdot 10^{4}$ examples were generated in a similar manner, of
which $5\cdot 10^{4}$ examples were used as a validation set, and $5\cdot
10^{4}$ examples were used as a test set. We trained 32 networks on these
datasets: for each of the above-mentioned models (vanilla feedforward network,
$f_{\rm GW}$, $f_{\rm S}$, $f_{\rm L}$) we varied the hidden layer size from
100 to 800 with the step 100. Training was performed with Adam optimizer
(Kingma and Ba (2015)). We used the squared loss
$l(y,\hat{y}):=(y-\hat{y})^{2}$ and batches of size 100. For each model, a
learning rate was tuned separately on the validation set. The results are
presented in Figure 2.

Figure 2: Results of approximating $|x|$ (left) and
$\mathbb{I}[\|\mathbf{x}\|\leq 1]$ (right) by Fourier neural networks and
Fourier series. MSE stands for the mean squared error,
$\frac{1}{T}\sum_{i=1}^{T}(y_{i}-\hat{y}_{i})^{2}$. Dashed curves were
obtained by regressing $\log(\mathrm{MSE})$ of $f_{\rm GW}$ on $\log n$.
∗Evaluation of $f_{\rm S}$ for the indicator function data is in progress.

Vanilla feedforward network (1) obtains lowest mean squared error (MSE) for
$|x|$, whereas the FNN of Gallant and White (1988) outperforms all other
models for $\mathbb{I}[\|\mathbf{x}\|\leq 1]$. According to the regression
fits (dashed curves in Fig. 2), the function $x\mapsto|x|$ is approximated by
the neural networks with error $O(n^{-0.48})$, and this is much worse than the
approximation error given by the partial sums of the Fourier series of
$f(x)=|x|$, which, according to Lemma 1 below, is of order $O(n^{-3})$. For
the function $\mathbf{x}\mapsto\mathbb{I}[\|\mathbf{x}\|\leq 1]$ results are
to other way around: the approximation error by the neural networks is of
order $O(n^{-0.33})$, while it is of the order $O(n^{-1/100})$ by the
truncated Fourier series (see Lemma 2 below). We keep in mind that the
theoretical result of Barron (1993) states that for any function from a
certain class444Functions with bounded first moment of the magnitude
distribution of the Fourier transform, which we refer to as Barron functions
in agreement with Lee et al. (2017). (to which the indicator function does
belong) a feedforward neural network with one hidden layer of size $n$ will be
able to approximate this function with a squared error of order $O(n^{-1})$.
We are not guaranteed, however, that the training algorithm will be able to
learn that function. Even if the neural network is able to represent the
function, learning can fail, since the optimization algorithm used for
training may not be able to find the value of the parameters that corresponds
to the desired function. We attribute the mismatch, $O(n^{-1/3})$ instead of
$O(n^{-1})$, between the orders of approximation errors to the suboptimal
estimation of the parameters of the networks by the Adam optimizer. However,
in general we have experimentally confirmed Barron’s claim that neural
networks with $n$ hidden units can approximate functions with much smaller
error than series expansions with $n$ terms.

We also notice here that directly comparing neural networks with truncated
Fourier series is somewhat unfair, as these are two different categories of
approximation: Fourier series serve as some theoretical reference, which is
possible only when we have access to the function being approximated.

######  Lemma 1.

For the $2\pi$-periodic function $f(x):=|x|$, $x\in[-\pi,\pi]$, let $S_{n}(x)$
be the $n^{\text{th}}$ partial sum of its Fourier series. Then, for some
constant $C$,

$\|f-S_{n}\|_{2}^{2}\leq\frac{C}{n^{3}}.$ (7)

###### Proof.

The Fourier series expansion of $f$ is given by

$f(x)=\frac{\pi}{2}+\sum_{k=1}^{\infty}a_{k}\cos(2k-1)x,\qquad
a_{k}:=-\frac{4}{\pi}\frac{1}{(2k-1)^{2}},$ (8)

(see Example 1, p. 23, from Folland (1992)), and therefore by Parseval’s
Theorem,

$\displaystyle\|f-S_{n}\|_{2}^{2}$
$\displaystyle:=\int_{-\pi}^{\pi}(f(x)-S_{n}(x))^{2}dx=\pi\sum_{k=n+1}^{\infty}{a_{k}^{2}}$
$\displaystyle\leq\pi\sum_{k=n+1}^{\infty}\Big{(}-\frac{4}{\pi}\frac{1}{(2k-1)^{2}}\Big{)}^{2}=\frac{16}{\pi}\sum_{k=n+1}^{\infty}\frac{1}{(2k-1)^{4}}.$
(9)

Since $(2k-1)^{-4}$ is a monotonically decreasing sequence, we have

$\int_{n+1}^{\infty}\frac{du}{(2u-1)^{4}}\leq\sum_{k=n+1}^{\infty}\frac{1}{(2k-1)^{4}}\leq\int_{n}^{\infty}\frac{du}{(2u-1)^{4}},$

that is,

$\frac{1}{6(2n+1)^{3}}\leq\sum_{k=n+1}^{\infty}\frac{1}{(2k-1)^{4}}\leq\frac{1}{6(2n-1)^{3}}.$
(10)

Combining (9) and (10) we obtain (7). ∎

######  Lemma 2.

Let $\mathbf{x}\in[-\pi,\pi]^{d}$ and $f(\mathbf{x})$ be the indicator
function of the unit ball in $\mathbb{R}^{d}$, that is,
$f(\mathbf{x}):=\mathbb{I}[\mathbf{x}\leq 1]$. Let $S_{R}(\mathbf{x})$ be the
truncated Fourier Series of $f(\mathbf{x})$, where $R\geq 1$ is the radius of
the partial spherical summation and $n$ is the number of terms in the partial
sum. Then, for some dimensional dependent constant $C_{d}$, the following
holds

$\|f-S_{R}\|^{2}_{2}\leq\frac{C_{d}}{n^{1/d}}.$ (11)

###### Proof.

For $\mathbf{x}\in\mathbb{R}^{d}$, denote
$\|\mathbf{x}\|_{1}:=|x_{1}|+\ldots+|x_{d}|$, and
$\|\mathbf{x}\|_{\infty}:=\max_{1\leq i\leq d}|x_{i}|$. It is known that

$\|\mathbf{x}\|_{2}\leq\|\mathbf{x}\|_{1}\leq\sqrt{d}\|\mathbf{x}\|_{2},\quad\|\mathbf{x}\|_{\infty}\leq\|\mathbf{x}\|_{1}\leq
d\|\mathbf{x}\|_{\infty},$ (12)

which in particular implies

$\|\mathbf{x}\|_{\infty}\geq\frac{1}{d}\|\mathbf{x}\|_{1}\geq\frac{1}{d}\|\mathbf{x}\|_{2}.$
(13)

From (12) and (13) it follows that

$\\{\mathbf{k}\in\mathbb{Z}^{d}:\,\|\mathbf{k}\|_{2}>R\\}\subset\\{\mathbf{k}\in\mathbb{Z}^{d}:\,\|\mathbf{k}\|_{\infty}>R/d\\},$
(14)

and, therefore,

$\sum_{\|\mathbf{k}\|_{2}>R}\frac{1}{\|\mathbf{k}\|^{d+1}_{2}}\leq\sum_{\|\mathbf{k}\|_{\infty}>R/d}\frac{1}{\|\mathbf{k}\|_{2}^{d+1}}\lesssim\sum_{\|\mathbf{k}\|_{\infty}>R/d}\frac{1}{\|\mathbf{k}\|_{1}^{d+1}}.$
(15)

Here “$A\lesssim B$” means that “$A\leq C_{d}B$”, for some dimensional
dependent constant $C_{d}$. Analogously, we write “$A\sim B$” if “$A\lesssim
B$” and “$B\lesssim A$”. Denoting $\tilde{R}:=R/d$, we obtain the following
decomposition

$\\{\|\mathbf{k}\|_{\infty}>\tilde{R}\\}\\\ =\bigcup_{1\leq j\leq
d}\bigcup_{1\leq i_{1}\neq\dots\neq i_{d}\leq
d}\Big{\\{}|k_{i_{\alpha}}|>\tilde{R},\,1\leq\alpha\leq j;\\\
|k_{i_{\alpha}}|\leq\tilde{R},\,j<\alpha\leq d\Big{\\}}.$

Thus the latter sum in (15) can be estimated as follows,

$\displaystyle\sum_{\|\mathbf{k}\|_{\infty}>\tilde{R}}\frac{1}{\|\mathbf{k}\|_{1}^{d+1}}=\sum_{\|\mathbf{k}\|_{\infty}>\tilde{R}}\frac{1}{(\|\mathbf{k}\|_{1}^{(d+1)/j})^{j}}$
$\displaystyle\leq\sum_{j=1}^{d}\sum_{1\leq i_{1}\neq\dots\neq i_{d}\leq
d}\sum_{|k_{i_{1}}|>\tilde{R}}\cdots\sum_{|k_{i_{j}}|>\tilde{R}}\sum_{|k_{i_{j+1}}|\leq\tilde{R}}\cdots\sum_{|k_{i_{d}}|\leq\tilde{R}}$
$\displaystyle\qquad\times\frac{1}{|k_{i_{1}}|^{(d+1)/j}\cdots|k_{i_{j}}|^{(d+1)/j}}$
$\displaystyle\lesssim\sum_{j=1}^{d}\tilde{R}^{d-j}\Big{(}\sum_{|\ell|>\tilde{R}}\frac{1}{|\ell|^{(d+1)/j}}\Big{)}^{j}\sim\sum_{j=1}^{d}\tilde{R}^{d-j}\Big{(}\int_{\tilde{R}}^{\infty}\frac{du}{u^{(d+1)/j}}\Big{)}^{j}$
$\displaystyle\sim\sum_{j=1}^{d}\frac{\tilde{R}^{d-j}}{\tilde{R}^{d+1-j}}\sim\frac{1}{\tilde{R}}\sim\frac{1}{R}.$
(16)

Combining (15) and (16) we get

$\sum_{\|\mathbf{k}\|_{2}>R}\frac{1}{\|\mathbf{k}\|^{d+1}_{2}}\lesssim\frac{1}{R}.$
(17)

Let $\mathrm{SE}$ denote the squared error in the left-hand side of (11).
Then, Parseval’s Theorem allows us to write

$\displaystyle\mathrm{SE}$
$\displaystyle:=\int_{[-\pi,\pi]^{d}}|f(\mathbf{x})-S_{R}(\mathbf{x})|^{2}\,d\mathbf{x}=\int_{[-\pi,\pi]^{d}}\Big{|}\sum_{\|\mathbf{k}\|_{2}>R}\hat{f}_{\mathbf{k}}e^{i\mathbf{k}\cdot\mathbf{x}}\Big{|}^{2}d\mathbf{x}$
$\displaystyle=(2\pi)^{d}\sum_{\|\mathbf{k}\|_{2}>R}|\hat{f}_{\mathbf{k}}|^{2}.$

Using the estimates of the Fourier coefficients for the indicator function of
a ball (Pinsky et al. (1993), p. 120), and denoting
$\alpha:=\|\mathbf{k}\|_{2}-(d-1)\pi/4$, we get

$\displaystyle\mathrm{SE}$
$\displaystyle=(2\pi)^{d}\sum_{\|\mathbf{k}\|_{2}>R}\Big{[}\frac{C_{d}}{\|\mathbf{k}\|_{2}^{(d+1)/2}}\Big{\\{}\sin\alpha+O\Big{(}\frac{1}{\sqrt{\|\mathbf{k}\|_{2}}}\Big{)}\Big{\\}}\Big{]}^{2}$
$\displaystyle\sim\sum_{\|\mathbf{k}\|_{2}>R}\frac{1}{\|\mathbf{k}\|_{2}^{d+1}}\Big{[}\sin^{2}\alpha+2\sin\alpha\,O\Big{(}\frac{1}{\sqrt{\|\mathbf{k}\|_{2}}}\Big{)}+O\Big{(}\frac{1}{\|\mathbf{k}\|_{2}}\Big{)}\Big{]}$
$\displaystyle\lesssim\sum_{\|\mathbf{k}\|_{2}>R}\frac{1}{\|\mathbf{k}\|_{2}^{d+1}}.$
(18)

From (17) and (18) it follows that

$\mathrm{SE}\lesssim\frac{1}{R}.$ (19)

The number of terms $n$ in the spherical partial sum $S_{R}(\mathbf{x})$ is
equal to the number of integer points in the $d$-ball of radius $R$, which is,
according to Götze (2004), approximated by the volume of such ball up to an
error $O(R^{d-2})$, i.e.

$n\sim R^{d}.$

Combining this with (19), we get $\mathrm{SE}\lesssim n^{-1/d}$. ∎

###  4.2 Image recognition

We performed evaluation of the FNNs in the image recognition task using the
MNIST dataset (LeCun et al. (1998)), which is commonly used for training
various image processing systems. It consists of handwritten digit images,
$28\times 28$ pixels in size, organized into 10 classes (0 to 9) with 60,000
training and 10,000 test samples. Portion of training samples was used as
validation data. Images were represented as vectors in $\mathbb{R}^{784}$,
hidden layer size was fixed at 64 for all networks, and classification was
done based on the softmax normalization. Training was performed with Adam
optimizer (Kingma and Ba (2015)). We used the cross-entropy loss and batches
of size 100. Learning rate was tuned separately for each model on the
validation data. Table 2 compares classification accuracy obtained by the
models.

Model | Accuracy | Learning rate
---|---|---
Vanilla feedforward NN | 0.9648 | 0.0096
FNN of Gallant and White (1988) | 0.9695 | 0.0045
FNN of Silvescu (1999) | 0.9659 | 0.0134
FNN of Liu (2013) | 0.9638 | 0.0034
Table 1: Evaluation of the networks on MNIST data.

As we can see, all the networks demonstrate similar performance in this task.
In fact, the differences between accuracy results are not significant across
the models, Pearson’s Chi-square test of independence $\chi^{2}_{3}=5.6449$,
$p$-value $>0.1$.

###  4.3 Language modeling

A statistical language model (LM) is a model which assigns a probability to a
sequence of words. Below we specify one type of such models based on the
structurally constrained recurrent network (SCRN) of Mikolov et al. (2015).

Let $\mathcal{W}$ be a finite vocabulary of words. We assume that words have
already been converted into indices. Let
$\mathbf{E}\in\mathbb{R}^{|\mathcal{W}|\times d_{\mathcal{W}}}$ be an input
embedding matrix for words — i.e., it is a matrix in which the $w$th row
(denoted as $\mathbf{w}$) corresponds to an embedding of the word
$w\in\mathcal{W}$. Based on word embeddings
$\mathbf{w_{1:k}}=\mathbf{w_{1}},\ldots,\mathbf{w_{k}}$ for a sequence of
words $w_{1:k}$, the SCRN model produces two sequences of states,
$\mathbf{s_{1:k}}$ and $\mathbf{h_{1:k}}$, according to the
equations555Vectors are assumed to be row vectors, which are right multiplied
by matrices ($\mathbf{x}\mathbf{W}+\mathbf{b}$). This choice is somewhat non-
standard but it maps better to the way networks are implemented in code using
matrix libraries such as TensorFlow.

$\displaystyle\mathbf{s_{t}}=(1-\alpha)\mathbf{w_{t}}\mathbf{B}+\alpha\mathbf{s_{t-1}},$
(20)
$\displaystyle\mathbf{h_{t}}=\sigma(\mathbf{w_{t}}\mathbf{A}+\mathbf{s_{t}}\mathbf{P}+\mathbf{h_{t-1}}\mathbf{R}),$
(21)

where $\mathbf{B}\in\mathbb{R}^{|\mathcal{W}|\times d_{s}}$,
$\mathbf{A}\in\mathbb{R}^{|\mathcal{W}|\times d_{h}}$,
$\mathbf{P}\in\mathbb{R}^{d_{s}\times d_{h}}$,
$\mathbf{R}\in\mathbb{R}^{d_{h}\times d_{h}}$, $d_{s}$ and $d_{h}$ are
dimensions of $\mathbf{s_{t}}$ and $\mathbf{h_{t}}$, $\sigma(\cdot)$ is the
logistic sigmoid function. The last couple of states
$(\mathbf{s_{k}},\mathbf{h_{k}})$ is assumed to contain information on the
whole sequence $w_{1:k}$ and is further used for predicting the next word
$w_{k+1}$ of a sequence according to the probability distribution

$\Pr(w_{k+1}|w_{1:k})=\operatorname{softmax}(\mathbf{s_{k}}\mathbf{U}+\mathbf{h_{k}}\mathbf{V}),$
(22)

where $\mathbf{U}\in\mathbb{R}^{d_{s}\times|\mathcal{W}|}$ and
$\mathbf{V}\in\mathbb{R}^{d_{h}\times|\mathcal{W}|}$ are output embedding
matrices. For the sake of simplicity we omit bias terms in (21) and (22).
Being conceptually much simpler, the SCRN architecture demonstrates
performance comparable to the widely used LSTM model in language modeling task
(Kabdolov et al. (2018)), and this is why we chose it for our experiments.

We train and evaluate the SCRN model for $(d_{h},d_{s})$ $\in\\{(40,10)$,
$(90,10)$, $(100,40)$, $(300,40)\\}$ on the PTB (Marcus et al. (1993)) data
set, for which the standard training (0-20), validation (21-22), and test
(23-24) splits along with pre-processing per Mikolov et al. (2010) is
utilized. We replace $\sigma$ in (21) with $\sigma_{\rm GW}$, $\sigma_{\rm
S}$, and $\sigma_{\rm L}$ defined in Section 3, and we refer to such
modification as Fourier layers. The choice of hyperparameters is guided by the
work of Kabdolov et al. (2018), except that for the Fourier layers we
additionally tune the learning rate, its decay schedule and the initialization
scale over the validation split. To evaluate the performance of the language
models we use perplexity (PPL) over the test set. The results are provided in
Table 2.

Activation | (40, 10) | (90, 10) | (100, 40) | (300, 40)
---|---|---|---|---
$\sigma$ | 128.0 | 118.6 | 118.7 | 120.6
$\sigma_{\rm GW}$ | 132.8 | 119.6 | 120.1 | 127.9
$\sigma_{\rm S}$ | 144.4 | 133.4 | 127.7 | 125.9
$\sigma_{\rm L}$ | 165.7 | 139.3 | 147.5 | 156.8
Table 2: Evaluation of the SCRN language models on the PTB data. Columns 2–5
correspond to different configurations of the hidden ($d_{h}$) and context
($d_{s}$) states sizes.

As one can see, the conventional sigmoid activation outperforms all Fourier
activations, and, as in the case of synthetic data, the Fourier layer of
Gallant and White (1988) is better than other Fourier layers for most of the
architectures.

##  5 Discussion

The FNNs of Silvescu (1999) (4) and of Liu (2013) (6) have non-sigmoidal
activations, which makes their optimization more difficult. Although the
activation function of $f_{\mathrm{GW}}(\cdot)$ is sigmoidal, it still
underperforms the standard feedforward neural network in almost all cases. We
hypothesize that this is because $\sigma_{\mathrm{GW}}$ is constant outside
$\left[-\frac{\pi}{2},\frac{\pi}{2}\right]$, while $\sigma(x)=1/(1+e^{-x})$ is
never constant. This means that $\forall x_{1},\,x_{2}\in(\pi/2,\infty)$:
$\sigma_{\mathrm{GW}}(x_{1})=\sigma_{\mathrm{GW}}(x_{2})$, i.e. the activation
of Gallant and White (1988) (3) does not distinguish between any values to the
right from $\pi/2$ (and to the left from $-\pi/2$). The standard sigmoid
activation $\sigma(\cdot)$, on the other hand, can theoretically666In
practice, when implemented on a computer $\sigma(\cdot)$ will also be
“constant” outside an interval $[-a,a]$, where $a$ depends on the type of
precision used for computations. distinguish between any pair
$x_{1},\,x_{2}\in\mathbb{R}$: $x_{1}\neq x_{2}$. To see whether the constant
behavior of $\sigma_{\mathrm{GW}}$ indeed causes problems, we look at the pre-
activated values $x\cdot w+b$ for $x$ from the validation split in the
synthetic task of approximating $x\mapsto|x|$, $x\in[-\pi,\pi]$. The histogram
of these pre-activated values for the $f_{\mathrm{GW}}$ with hidden layer size
$n=100$ is given in Figure 3.

Figure 3: Histogram of pre-activated values ($x\cdot w+b$) in the FNN of
Gallant and White (1988). Frequencies are at log-scale.

It turns out that $\approx 8\%$ of pre-activated values are outside of
$[-\pi/2,\pi/2]$, and this information is lost when filtered through
$\sigma_{\mathrm{GW}}$.

##  6 Conclusion and Future Work

All Fourier neural networks are not better than the standard neural network
with sigmoid activation except when it comes to modeling synthetic data. The
architecture of Gallant and White (1988) is the best among Fourier neural
networks. When the function being approximated is known and depends on
multiple variables, the neural networks with just one hidden layer may provide
much better approximation compared to truncated Fourier series.

In this paper we focused on neural architectures with one hidden layer. It is
interesting to compare Fourier neural networks in a multilayer setup. We defer
such study to our future work which will also include experiments with a
larger variety of functions, as well as mathematical analysis of the
approximation of Barron functions by Silvescu’s and Liu’s Fourier neural
networks.

## Compliance with ethical standards

Conflict of Interest. The authors declare that they have no conflict of
interest.

## References

  * Alimov et al. [1976] Sh A Alimov, Vladimir Aleksandrovich Il’in, and Evgenii Mikhailovich Nikishin.  Convergence problems of multiple trigonometric series and spectral decompositions. i.  _Russian Mathematical Surveys_ , 31(6):29, 1976\.
  * Alimov et al. [1977] Sh A Alimov, Vladimir Aleksandrovich Il’in, and Evgenii Mikhailovich Nikishin.  Problems of convergence of multiple trigonometric series and spectral decompositions. ii.  _Russian Mathematical Surveys_ , 32(1):115–139, 1977.
  * Barron [1993] Andrew R Barron.  Universal approximation bounds for superpositions of a sigmoidal function.  _IEEE Transactions on Information theory_ , 39(3):930–945, 1993.
  * Cybenko [1989] George Cybenko.  Approximation by superpositions of a sigmoidal function.  _Mathematics of control, signals and systems_ , 2(4):303–314, 1989.
  * Folland [1992] Gerald B Folland.  _Fourier analysis and its applications_ , volume 4.  American Mathematical Soc., 1992.
  * Gallant and White [1988] A Ronald Gallant and Halbert White.  There exists a neural network that does not make avoidable mistakes.  In _Proceedings of the Second Annual IEEE Conference on Neural Networks, San Diego, CA, I_ , 1988.
  * Götze [2004] Friedrich Götze.  Lattice point problems and values of quadratic forms.  _Inventiones mathematicae_ , 157(1):195–226, 2004\.
  * Hornik et al. [1989] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  _Neural networks_ , 2(5):359–366, 1989.
  * Kabdolov et al. [2018] Olzhas Kabdolov, Zhenisbek Assylbekov, and Rustem Takhanov.  Reproducing and regularizing the scrn model.  In _Proc. of COLING_ , 2018.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _Proceedings of ICLR_ , 2015.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee et al. [2017] Holden Lee, Rong Ge, Tengyu Ma, Andrej Risteski, and Sanjeev Arora.  On the ability of neural nets to express distributions.  In _Conference on Learning Theory_ , pages 1271–1296, 2017.
  * Leshno et al. [1993] Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken.  Multilayer feedforward networks with a nonpolynomial activation function can approximate any function.  _Neural networks_ , 6(6):861–867, 1993.
  * Liu [2013] Shuang Liu.  Fourier neural network for machine learning.  In _Machine Learning and Cybernetics (ICMLC), 2013 International Conference on_ , volume 1, pages 285–290. IEEE, 2013.
  * Marcus et al. [1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini.  Building a large annotated corpus of english: The penn treebank.  _Computational linguistics_ , 19(2):313–330, 1993\.
  * McCaffrey and Gallant [1994] Daniel F McCaffrey and A Ronald Gallant.  Convergence rates for single hidden layer feedforward networks.  _Neural Networks_ , 7(1):147–158, 1994.
  * Mikolov et al. [2010] Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur.  Recurrent neural network based language model.  In _Proc. of INTERSPEECH_ , 2010.
  * Mikolov et al. [2015] Tomas Mikolov, Armand Joulin, Sumit Chopra, Michael Mathieu, and Marc’Aurelio Ranzato.  Learning longer memory in recurrent neural networks.  In _Proc. of ICLR Workshop Track_ , 2015.
  * Pinsky et al. [1993] Mark A. Pinsky, Nancy K. Stanton, and Peter. E Trapa.  Fourier series of radial functions in several variables.  _Journal of Functional Analysis_ , 116(1):111–132, 1993.
  * Silvescu [1999] Adrian Silvescu.  Fourier neural networks.  In _Neural Networks, 1999. IJCNN’99. International Joint Conference on_ , volume 1, pages 488–491. IEEE, 1999.
  * Tan [2006] HS Tan.  Fourier neural networks and generalized single hidden layer networks in aircraft engine fault diagnostics.  _Journal of engineering for gas turbines and power_ , 128(4):773–782, 2006.
  * Zuo and Cai [2005] Wei Zuo and Lilong Cai.  Tracking control of nonlinear systems using fourier neural network.  In _Advanced Intelligent Mechatronics. Proceedings, 2005 IEEE/ASME International Conference on_ , pages 670–675. IEEE, 2005.
  * Zuo and Cai [2008] Wei Zuo and Lilong Cai.  Adaptive-fourier-neural-network-based control for a class of uncertain nonlinear systems.  _IEEE Transactions on Neural Networks_ , 19(10):1689–1701, 2008.
  * Zuo et al. [2009] Wei Zuo, Yang Zhu, and Lilong Cai.  Fourier-neural-network-based learning control for a class of nonlinear systems with flexible components.  _IEEE transactions on neural networks_ , 20(1):139–151, 2009.