marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

Submission and Formatting Instructions for
International Conference on Machine Learning (ICML 2020)

Anonymous Authors1

††footnotetext:  1Anonymous Institution, Anonymous City, Anonymous Region,
Anonymous Country. Correspondence to: Anonymous Author
<EMAIL_ADDRESS>
Preliminary work. Under review by the International Conference on Machine
Learning (ICML). Do not distribute.

###### Abstract

This document provides a basic paper template and submission guidelines.
Abstracts must be a single paragraph, ideally between 4–6 sentences long.
Gross violations will trigger corrections at the camera-ready phase.

##  1 Electronic Submission

Submission to ICML 2020 will be entirely electronic, via a web site (not
email). Information about the submission process and LaTeX templates are
available on the conference web site at:

http://icml.cc/

The guidelines below will be enforced for initial submissions and camera-ready
copies. Here is a brief summary:

  * •

Submissions must be in PDF.

  * •

Submitted papers can be up to eight pages long, not including references, plus
unlimited space for references. Accepted papers can be up to nine pages long,
not including references, to allow authors to address reviewer comments. Any
paper exceeding this length will automatically be rejected.

  * •

Do not include author information or acknowledgements in your initial
submission.

  * •

Your paper should be in 10 point Times font.

  * •

Make sure your PDF file only uses Type-1 fonts.

  * •

Place figure captions _under_ the figure (and omit titles from inside the
graphic file itself). Place table captions _over_ the table.

  * •

References must include page numbers whenever possible and be as complete as
possible. Place multiple citations in chronological order.

  * •

Do not alter the style template; in particular, do not compress the paper
format by reducing the vertical spaces.

  * •

Keep your abstract brief and self-contained, one paragraph and roughly 4–6
sentences. Gross violations will require correction at the camera-ready phase.
The title should have content words capitalized.

###  1.1 Submitting Papers

Paper Deadline: The deadline for paper submission that is advertised on the
conference website is strict. If your full, anonymized, submission does not
reach us on time, it will not be considered for publication.

Anonymous Submission: ICML uses double-blind review: no identifying author
information may appear on the title page or in the paper itself. Section 2.3
gives further details.

Simultaneous Submission: ICML will not accept any paper which, at the time of
submission, is under review for another conference or has already been
published. This policy also applies to papers that overlap substantially in
technical content with conference papers under review or previously published.
ICML submissions must not be submitted to other conferences during ICML’s
review period. Authors may submit to ICML substantially different versions of
journal papers that are currently under review by the journal, but not yet
accepted at the time of submission. Informal publications, such as technical
reports or papers in workshop proceedings which do not appear in print, do not
fall under these restrictions.

Authors must provide their manuscripts in PDF format. Furthermore, please make
sure that files contain only embedded Type-1 fonts (e.g., using the program
pdffonts in linux or using File/DocumentProperties/Fonts in Acrobat). Other
fonts (like Type-3) might come from graphics files imported into the document.

Authors using Word must convert their document to PDF. Most of the latest
versions of Word have the facility to do this automatically. Submissions will
not be accepted in Word format or any format other than PDF. Really. We’re not
joking. Don’t send Word.

Those who use LaTeX should avoid including Type-3 fonts. Those using latex and
dvips may need the following two commands:

    dvips -Ppdf -tletter -G0 -o paper.ps paper.dvi
    ps2pdf paper.ps

It is a zero following the “-G”, which tells dvips to use the config.pdf file.
Newer TeX distributions don’t always need this option.

Using pdflatex rather than latex, often gives better results. This program
avoids the Type-3 font problem, and supports more advanced features in the
microtype package.

Graphics files should be a reasonable size, and included from an appropriate
format. Use vector formats (.eps/.pdf) for plots, lossless bitmap formats
(.png) for raster graphics with sharp lines, and jpeg for photo-like images.

The style file uses the hyperref package to make clickable links in documents.
If this causes problems for you, add nohyperref as one of the options to the
icml2020 usepackage statement.

###  1.2 Submitting Final Camera-Ready Copy

The final versions of papers accepted for publication should follow the same
format and naming convention as initial submissions, except that author
information (names and affiliations) should be given. See Section 2.3.2 for
formatting instructions.

The footnote, “Preliminary work. Under review by the International Conference
on Machine Learning (ICML). Do not distribute.” must be modified to
“Proceedings of the $\mathit{37}^{th}$ International Conference on Machine
Learning, Vienna, Austria, PMLR 119, 2020. Copyright 2020 by the author(s).”

For those using the LaTeX style file, this change (and others) is handled
automatically by simply changing $\mathtt{\backslash
usepackage\\{icml2020\\}}$ to

$\mathtt{\backslash usepackage[accepted]\\{icml2020\\}}$

Authors using Word must edit the footnote on the first page of the document
themselves.

Camera-ready copies should have the title of the paper as running head on each
page except the first one. The running title consists of a single line
centered above a horizontal rule which is $1$ point thick. The running head
should be centered, bold and in $9$ point type. The rule should be $10$ points
above the main text. For those using the LaTeX style file, the original title
is automatically set as running head using the fancyhdr package which is
included in the ICML 2020 style file package. In case that the original title
exceeds the size restrictions, a shorter form can be supplied by using

`\icmltitlerunning{...}`

just before $\mathtt{\backslash begin\\{document\\}}$. Authors using Word must
edit the header of the document themselves.

##  2 Format of the Paper

All submissions must follow the specified format.

###  2.1 Dimensions

The text of the paper should be formatted in two columns, with an overall
width of 6.75 inches, height of 9.0 inches, and 0.25 inches between the
columns. The left margin should be 0.75 inches and the top margin 1.0 inch
(2.54 cm). The right and bottom margins will depend on whether you print on US
letter or A4 paper, but all final versions must be produced for US letter
size.

The paper body should be set in 10 point type with a vertical spacing of 11
points. Please use Times typeface throughout the text.

###  2.2 Title

The paper title should be set in 14 point bold type and centered between two
horizontal rules that are 1 point thick, with 1.0 inch between the top rule
and the top edge of the page. Capitalize the first letter of content words and
put the rest of the title in lower case.

###  2.3 Author Information for Submission

ICML uses double-blind review, so author information must not appear. If you
are using LaTeX and the icml2020.sty file, use `\icmlauthor{...}` to specify
authors and `\icmlaffiliation{...}` to specify affiliations. (Read the TeX
code used to produce this document for an example usage.) The author
information will not be printed unless accepted is passed as an argument to
the style file. Submissions that include the author information will not be
reviewed.

####  2.3.1 Self-Citations

If you are citing published papers for which you are an author, refer to
yourself in the third person. In particular, do not use phrases that reveal
your identity (e.g., “in previous work langley00, we have shown …”).

Do not anonymize citations in the reference section. The only exception are
manuscripts that are not yet published (e.g., under submission). If you choose
to refer to such unpublished manuscripts anonymous, anonymized copies have to
be submitted as Supplementary Material via CMT. However, keep in mind that an
ICML paper should be self contained and should contain sufficient detail for
the reviewers to evaluate the work. In particular, reviewers are not required
to look at the Supplementary Material when writing their review.

####  2.3.2 Camera-Ready Author Information

If a paper is accepted, a final camera-ready copy must be prepared. For
camera-ready papers, author information should start 0.3 inches below the
bottom rule surrounding the title. The authors’ names should appear in 10
point bold type, in a row, separated by white space, and centered. Author
names should not be broken across lines. Unbolded superscripted numbers,
starting 1, should be used to refer to affiliations.

Affiliations should be numbered in the order of appearance. A single footnote
block of text should be used to list all the affiliations. (Academic
affiliations should list Department, University, City, State/Region, Country.
Similarly for industrial affiliations.)

Each distinct affiliations should be listed once. If an author has multiple
affiliations, multiple superscripts should be placed after the name, separated
by thin spaces. If the authors would like to highlight equal contribution by
multiple first authors, those authors should have an asterisk placed after
their name in superscript, and the term “*Equal contribution” should be placed
in the footnote block ahead of the list of affiliations. A list of
corresponding authors and their emails (in the format Full Name
<EMAIL_ADDRESS>can follow the list of affiliations. Ideally only one or
two names should be listed.

A sample file with author names is included in the ICML2020 style file
package. Turn on the [accepted] option to the stylefile to see the names
rendered. All of the guidelines above are implemented by the LaTeX style file.

###  2.4 Abstract

The paper abstract should begin in the left column, 0.4 inches below the final
address. The heading ‘Abstract’ should be centered, bold, and in 11 point
type. The abstract body should use 10 point type, with a vertical spacing of
11 points, and should be indented 0.25 inches more than normal on left-hand
and right-hand margins. Insert 0.4 inches of blank space after the body. Keep
your abstract brief and self-contained, limiting it to one paragraph and
roughly 4–6 sentences. Gross violations will require correction at the camera-
ready phase.

###  2.5 Partitioning the Text

You should organize your paper into sections and paragraphs to help readers
place a structure on the material and understand its contributions.

####  2.5.1 Sections and Subsections

Section headings should be numbered, flush left, and set in 11 pt bold type
with the content words capitalized. Leave 0.25 inches of space before the
heading and 0.15 inches after the heading.

Similarly, subsection headings should be numbered, flush left, and set in 10
pt bold type with the content words capitalized. Leave 0.2 inches of space
before the heading and 0.13 inches afterward.

Finally, subsubsection headings should be numbered, flush left, and set in 10
pt small caps with the content words capitalized. Leave 0.18 inches of space
before the heading and 0.1 inches after the heading.

Please use no more than three levels of headings.

####  2.5.2 Paragraphs and Footnotes

Within each section or subsection, you should further partition the paper into
paragraphs. Do not indent the first line of a given paragraph, but insert a
blank line between succeeding ones.

You can use footnotes111Footnotes should be complete sentences. to provide
readers with additional information about a topic without interrupting the
flow of the paper. Indicate footnotes with a number in the text where the
point is most relevant. Place the footnote in 9 point type at the bottom of
the column in which it appears. Precede the first footnote in a column with a
horizontal rule of 0.8 inches.222Multiple footnotes can appear in each column,
in the same order as they appear in the text, but spread them across columns
and pages if possible.

Figure 1: Historical locations and number of accepted papers for International
Machine Learning Conferences (ICML 1993 – ICML 2008) and International
Workshops on Machine Learning (ML 1988 – ML 1992). At the time this figure was
produced, the number of accepted papers for ICML 2008 was unknown and instead
estimated.

###  2.6 Figures

You may want to include figures in the paper to illustrate your approach and
results. Such artwork should be centered, legible, and separated from the
text. Lines should be dark and at least 0.5 points thick for purposes of
reproduction, and text should not appear on a gray background.

Label all distinct components of each figure. If the figure takes the form of
a graph, then give a name for each axis and include a legend that briefly
describes each curve. Do not include a title inside the figure; instead, the
caption should serve this function.

Number figures sequentially, placing the figure number and caption _after_ the
graphics, with at least 0.1 inches of space before the caption and 0.1 inches
after it, as in Figure 1. The figure caption should be set in 9 point type and
centered unless it runs two or more lines, in which case it should be flush
left. You may float figures to the top or bottom of a column, and you may set
wide figures across both columns (use the environment figure* in LaTeX).
Always place two-column figures at the top or bottom of the page.

###  2.7 Algorithms

If you are using LaTeX, please use the “algorithm” and “algorithmic”
environments to format pseudocode. These require the corresponding stylefiles,
algorithm.sty and algorithmic.sty, which are supplied with this package.
Algorithm 1 shows an example.

Algorithm 1 Bubble Sort

Input: data $x_{i}$, size $m$

repeat

Initialize $noChange=true$.

for $i=1$ to $m-1$ do

if $x_{i}>x_{i+1}$ then

Swap $x_{i}$ and $x_{i+1}$

$noChange=false$

end if

end for

until $noChange$ is $true$

###  2.8 Tables

You may also want to include tables that summarize material. Like figures,
these should be centered, legible, and numbered consecutively. However, place
the title _above_ the table with at least 0.1 inches of space before the title
and the same after it, as in Table 1. The table title should be set in 9 point
type and centered unless it runs two or more lines, in which case it should be
flush left.

Table 1: Classification accuracies for naive Bayes and flexible Bayes on
various data sets.
Data set | Naive | Flexible | Better?
---|---|---|---
Breast |  95.9$\pm$ 0.2 |  96.7$\pm$ 0.2 | $\surd$
Cleveland |  83.3$\pm$ 0.6 |  80.0$\pm$ 0.6 | $\times$
Glass2 |  61.9$\pm$ 1.4 |  83.8$\pm$ 0.7 | $\surd$
Credit |  74.8$\pm$ 0.5 |  78.3$\pm$ 0.6 |
Horse |  73.3$\pm$ 0.9 |  69.7$\pm$ 1.0 | $\times$
Meta |  67.1$\pm$ 0.6 |  76.5$\pm$ 0.5 | $\surd$
Pima |  75.1$\pm$ 0.6 |  73.9$\pm$ 0.5 |
Vehicle |  44.9$\pm$ 0.6 |  61.5$\pm$ 0.4 | $\surd$

Tables contain textual material, whereas figures contain graphical material.
Specify the contents of each row and column in the table’s topmost row. Again,
you may float tables to a column’s top or bottom, and set wide tables across
both columns. Place two-column tables at the top or bottom of the page.

###  2.9 Citations and References

Please use APA reference format regardless of your formatter or word
processor. If you rely on the LaTeX bibliographic facility, use natbib.sty and
icml2020.bst included in the style-file package to obtain this format.

Citations within the text should include the authors’ last names and year. If
the authors’ names are included in the sentence, place only the year in
parentheses, for example when referencing Arthur Samuel’s pioneering work
(Samuel59). Otherwise place the entire reference in parentheses with the
authors and year separated by a comma Samuel59. List multiple references
separated by semicolons kearns89; Samuel59; mitchell80. Use the ‘et al.’
construct only for citations with three or more authors or after listing all
authors to a publication in an earlier reference MachineLearningI.

Authors should cite their own work in the third person in the initial version
of their paper submitted for blind review. Please refer to Section 2.3 for
detailed instructions on how to cite your own papers.

Use an unnumbered first-level section heading for the references, and use a
hanging indent style, with the first line of the reference flush against the
left margin and subsequent lines indented by 10 points. The references at the
end of this document give examples for journal articles Samuel59, conference
publications langley00, book chapters Newell81, books DudaHart2nd, edited
volumes MachineLearningI, technical reports mitchell80, and dissertations
kearns89.

Alphabetize references by the surnames of the first authors, with single
author entries preceding multiple author entries. Order references for the
same authors by year of publication, with the earliest first. Make sure that
each reference includes all relevant information (e.g., page numbers).

Please put some effort into making references complete, presentable, and
consistent. If using bibtex, please protect capital letters of names and
abbreviations in titles, for example, use {B}ayesian or {L}ipschitz in your
.bib file.

## Software and Data

If a paper is accepted, we strongly encourage the publication of software and
data with the camera-ready version of the paper whenever appropriate. This can
be done by including a URL in the camera-ready copy. However, do not include
URLs that reveal your institution or identity in your submission for review.
Instead, provide an anonymous URL or upload the material as “Supplementary
Material” into the CMT reviewing system. Note that reviewers are not required
to look at this material when writing their review.

## Acknowledgements

Do not include acknowledgements in the initial version of the paper submitted
for blind review.

If a paper is accepted, the final camera-ready version can (and probably
should) include acknowledgements. In this case, please place such
acknowledgements in an unnumbered section at the end of the paper. Typically,
this will include thanks to reviewers who gave useful comments, to colleagues
who contributed to the ideas, and to funding agencies and corporate sponsors
that provided financial support.

##  Appendix A Do _not_ have an appendix here

_Do not put content after the references._ Put anything that you might
normally include after the references in a separate supplementary file.

We recommend that you build supplementary material in a separate document. If
you must create one PDF and cut it up, please be careful to use a tool that
doesn’t alter the margins, and that doesn’t aggressively rewrite the PDF file.
pdftk usually works fine.

Please do not use Apple’s preview to cut off supplementary material. In
previous years it has altered margins, and created headaches at the camera-
ready stage.


# Recurrent Rational Networks

Quentin Delfosse  Patrick Schramowski  Alejandro Molina  Kristian Kersting

###### Abstract

Latest insights from biology show that intelligence does not only emerge from
the connections between the neurons, but that individual neurons shoulder more
computational responsibility. Current Neural Network architecture design and
search are biased on fixed activation functions. Using more advanced learnable
activation functions provide Neural Networks with higher learning capacity.
However, general guidance for building such networks is still missing. In this
work, we first explain why rationals offer an optimal choice for activation
functions. We then show that they are closed under residual connections, and
inspired by recurrence for residual networks we derive a self-regularized
version of Rationals: Recurrent Rationals. We demonstrate that (Recurrent)
Rational Networks lead to high performance improvements on Image
Classification and Deep Reinforcement Learning.

Machine Learning, ICML

##  1 Introduction

Neural Networks’ efficiency in approximating any function has made them the
most used approximation function for many machine learning tasks. They consist
of successive layers, that topologically reshape the input space, transform
and extract new features, that are then separately activated. Neuro-Scientists
first explained that the brainpower resides in the combinations happening
through trillions of connections. However, research has progressively shown
that individual neurons are actually more capable than first thought, and
latest results have shown that dendritic compartments can compute complex
functions, (_e.g._ XOR), previously categorized as unsolvable by single-neuron
systems (Gidon et al., 2020).

In current neural network design, most of the research is focusing on well-
performing architectural patterns (Liu et al., 2018; Xie et al., 2019), while
keeping a fixed, predetermined activation function. Nevertheless, many
different functions have been adopted across different domains (_e.g._ Leaky
ReLU in YOLO (Redmon et al., 2016), GELU in GTP-3 (Brown et al., 2020), Tanh
in PPO (Schulman et al., 2017)). This indicates that the performance of neural
networks jointly depends on the task, the architecture, hyper-parameters, the
dataset (or environment), as well as the activation functions.

Figure 1: Recurrent Rational Network in its folded and unfolded forms. The
input passed along the layers and recurrently in the rational function of the
network.

To reduce the bias introduced by a fixed activation function and achieve
higher expressive power, one can use learnable activation functions. While
Neural Architecture Search uses Reinforcement Learning or evolutionary
algorithms on a preset family of activation functions to find performing
patterns (Zoph & Le, 2017; Liu et al., 2018), one can directly make use of
gradient descent to optimize parametric activation functions. In this way, one
can learn a linear combination of an arbitrary family of activation functions
(Manessi & Rozza, 2018), or use polynomial activation functions, where the
polynomials’ coefficients are considered as weights to be optimized (Goyal et
al., 2019).

A finer approach consists in learning a rational function (i.e. a ratio of
polynomials, see Eq 1). Rational functions and polynomials can converge to any
continuous function, but rational functions are better approximants than
polynomials in terms of convergence (Telgarsky, 2017). Rational activation
functions were first introduced as Padé Activation Units (Molina et al.,
2020).

$\operatorname{R}(x)=\frac{\operatorname{P}(x)}{\operatorname{Q}(x)}=\frac{\sum_{j=0}^{m}a_{j}x^{j}}{1+\sum_{k=1}^{n}b_{k}x^{k}}$
(1)

Molina _et al._ (2020) have shown that Rational Units outperform other
activation functions on several supervised learning tasks. However, neural
networks embedding rationals tend to overfit and ways to regularize the
rationals are required.

In this paper, we uncover important properties of rationals that explain their
performance. We show that they can learn various features, carried by
different handcrafted activation functions (Clevert et al., 2016; Elfwing et
al., 2018; Ramachandran et al., 2017; Misra, 2020; Georgescu et al., 2020). In
addition, we demonstrate that, if properly constructed, they can express
similar behavior to residual blocks, as they are closed under residual
connection. Therefore, we use rational functions to replace residual blocks in
a residual neural network and show that the performance is maintained or even
increased, while the number of parameters decreases. Furthermore, inspired by
weight sharing for Residual Networks, we introduce Recurrent Rational
Networks, whose recurrence acts as a natural regularizer. We evaluate Rational
and Recurrent Rational both on supervised learning tasks and Reinforcement
Learning. Our empirical evidence111$20.000$ GPU hours, carried on DGX-2
Machine with Nvidia Tesla V100with 32GB. demonstrates that (recurrent)
rational functions dominate domain. We used five different seeds for networks
and (potentially environment) initialization. For all the details and hyper
parameters of our experiments, refer to the appendix.

Our source code222github.com/ml-research/rational_sl
and github.com/ml-research/rational_rl to reproduce our experimental
evaluation and our Rational Network library333github.com/ml-
research/rational_activations are publicly available.

Contributions. We make the following contributions:

  * •

We investigate the performance gains and analyze important properties of
Rational Units, _e.g._ the natural embedding of residual connections.

  * •

Moreover, we introduce Recurrent Rational Networks and showcase their
advantages over Rational Networks on supervised image classification tasks.

  * •

We demonstrate that Rational Networks bring massive improvements in
Reinforcement Learning and even further increase the performances with our
introduced Recurrent Rational Networks.

  * •

We provide a Rational Activation Functions library to create (Recurrent)
Rational Networks compatible with most popular machine learning frameworks
PyTorch, Keras, Tensorflow and MXNet.

We proceed as follows. We start by describing the role of activation in Neural
Networks. Then, we analyze the properties of rational activation functions,
notably showing that they can incorporate residual blocks. Inspired by
recurrence for residual networks, we introduce Recurrent Rational Networks. We
demonstrate that these self-regularized networks improve generalization
performances on supervised learning classification. We then evaluate on
Reinforcement Learning tasks, and show Rational Networks outperforms every
non-learnable baselines and our Recurrent version dominate. Before concluding,
we discuss best practices and outlooks for these emerging Rational Network
types.

##  2 On the Role of Activation Functions

Let us start by investigating the role of activation functions in (deep)
neural networks.

Activation Functions Perform Internal Space Transformations. Contrary to the
rest of the layer, activation functions separately act on every dimension of
the input, _i.e._ they perform elementwise operations. They can be compared to
the brains’ dendrites that perform scalar transformations on the signal they
propagate, further merged into the neuron body (Li et al., 2020). The
transformation operated by the activation function cannot extract new feature
spaces by recombining the inputs, as convolutions do, but only stretch,
compress and cut the space. We refer to these transformations as internal
transformations, and to the ones performed by layers (without their activation
functions) as external ones. Different types of internal transformations
exist: Naitzat _et al._ (2020) showed that ReLU and its variants not only help
to tackle the vanishing gradient problem but allow non-homeomorphic
topological transformations. In contrast, Sigmoid or Tanh can only perform
homeomorphic ones. In other terms, ReLU can fold its input space, sending the
whole negative to one point, whereas Sigmoid and Tanh continuously squish it.

Diversity Inside the Network. Dendrites do not share identical behavior inside
the biological brain (Tavosanis, 2012). However, most common neural network
architectures use a fixed function across the network during learning. Some
architectures use a cell that contains different activation functions, as
LSTMs (Greff et al., 2015), or in Networks discovered by Neural Architecture
Search (Liu et al., 2018). Providing each layer with the ability to refine the
activation function for its specific need leads to higher modeling capacity.
In Rational Networks, we thus place different rational functions at each
layer, as done in (Molina et al., 2020).

Rationals Embed Desired Properties. Researchers have often searched for a
golden activation function. A combination of exhaustive and reinforcement
learning-based search led to SiLU/Swish (Ramachandran et al., 2017), later
refined with Mish by Misra _et al._ (2020), who showed common properties of
well-performing functions such as bounded negative domains and non-monotonic
shapes. This properties are learned by some of our trained Rationals (cf. Fig.
2 right). Inspired by the biological results described above, ADA,
particularly suited for performing XOR operation, was recently introduced
(Georgescu et al., 2020). This complex operation is feasible because of the
spike (high non-monocity) of ADA. We, here again, observe this property
learned by a trained Rational Unit. (cf. Fig. 2 left).

Figure 2: Rational Units (Rat.) discover biologically relevant operations
automatically. Rational Unit from trained DQN Agents on Time Pilot compared to
ADA (Georgescu et al., 2020), which like biological neurons can perform XOR,
and Recurrent one on Tuthankham to dSiLU, with bounded negative domains and
non-monotonic shape.

##  3 Rationals to Replace Residual Blocks

Veit _et al._ (2016) have exhibited Residual Networks (ResNet) ability to
tackle the Vanishing Gradient Problem. However, they were first introduced
following the idea that, for very deep networks, it is easier to start
learning from an identity layer and deviate from it, rather than trying to
converge to it if necessary (He et al., 2016). ResNets propagate an input
$\mathbf{x}$ through two paths: a transforming block of layers that preserve
dimensionality ($F$) and a residual connection (identity). The output
$\mathbf{y}$ of such a block satisfies Eq 2.

$\mathbf{y}(\mathbf{x})=F(\mathbf{x})+\mathbf{x}$ (2)

Rational functions can also follow this precept. They can be initialized as
the identity function, by having $a_{j}=\mathds{1}_{j=1}$ and $b_{k}=0$ for
all $j,k$ in Eq 1.

Rationals are Closed Under Residual Connections. Moreover, adding a residual
connection to a Rational Unit $\operatorname{R}(x)$ gives:

$\displaystyle\operatorname{R}(x)+x$
$\displaystyle=\frac{\sum\nolimits_{j=0}^{m}a_{j}x^{j}}{\sum\nolimits_{k=0}^{n}b_{k}x^{k}}+x$
$\displaystyle=\frac{\sum\nolimits_{j=0}^{m}a_{j}x^{j}+\sum\nolimits_{k=0}^{n}b_{k}x^{k+1}}{\sum\nolimits_{k=0}^{n}b_{k}x^{k}}$
(3)
$\displaystyle=\frac{\sum\nolimits_{j=0}^{M}c_{j}x^{j}}{\sum\nolimits_{k=0}^{n}b_{k}x^{k}}=\operatorname{R^{{}^{\prime}}}(x)\
,$

where:

$\displaystyle M=\max(m,n+1)\;,$ $\displaystyle\forall
j\in\\{0,\dots,M\\},c_{j}=a_{j}+b_{j-1}\;,$ $\displaystyle\forall
j\notin\\{0,\dots,m\\},a_{j}=0\;,\quad\text{and}$ $\displaystyle\forall
k\notin\\{0,\dots,n\\},b_{k}=0\ .$

This shows that rationals (with $m>n$) embed residual connections, and as
rationals are also closed under compositions, successive (residual) rationals
can be embedded into a single rational function. Hence, in order to provide
Rational Units with residual connections, one should select a numerator with a
higher order than the denominator.

ResNets Need Internal Space Transformations. However, ResNet blocks differ
from rationals as they act on multi-dimensional input space. Whereas the
rational acts on every dimension separately, they perform internal
transformations.

For very deep ResNet, Greff _et al._ (2017) showed that Residual blocks are
unrolled iterative estimation that refines upon their input representation.
They show that feature recombination does not occur inside the blocks, which
thus perform only internal transformations. These transitions to new levels of
representations (external transformations) occur during dimensionality change.
This view explains why lesioning and shuffling do not affect ResNet
performances significantly, and is also compatible with the findings obtained
by Molina _et al._ (2020), in pruned ResNets. We hypothesize that the Residual
Networks mentioned in these papers could benefit more from internal
transformations than from the external ones that these blocks can provide.

To test this hypothesis, we conduct lesioning experiments on a pretrained
ResNet101, on which we either remove the residual block (as in (Veit et al.,
2016)), or we replace it with a Rational units. We finetune the surrounding
blocks and in our case the identity-initialized rational. Using Rational Units
lead to performance improvements, even above the original model (_cf._ Tab.
1). We are here mainly interested in the train loss, as it shows how
effectively the model can learn. Nevertheless, the test accuracy is also
provided.

Eval | Lesion | L2.B3 | L3.B13 | L3.B19 | L4.B2
---|---|---|---|---|---
train | Std. (Veit et al.) | 100.9 | 120.2 | 90.5 | 58.9
Rat. (our) | 101.1 | 120.3 | 104.0 | 91.1
test | Std. (Veit et al.) | 93.1 | 102.0 | 97.1 | 81.7
Rat. (our) | 90.5 | 102.6 | 97.6 | 85.3
% dropped params | 0.63 | 2.51 | 2.51 | 10.0
Table 1: Rational functions improve lesioning. Recovery percentage for
retrained networks after lesioning (Veit et al., 2016) of a ResNet layer’s (L)
block (B). Residual blocks were lesioned, _i.e._ replaced, with a Identity
(Std.) or a Rational (Rat.) from a pretrained ResNet101 (44M parameters).
Then, the surrounding blocks (and implanted Rational Activation) are retrained
for 15 epochs. The best results are highlighted bold.

These findings empirically show that Residual Networks (ResNets) with rational
activation functions are less affected when compressing the neural network,
i.e. that internal transformations performed by rationals compensate for the
dropped neural connections.

Nonetheless, retraining with rationals on pretrained network, whose
architecture and weights have been optimized along with identity blocks might
have led the architecture to a local optimum. Using rationals from the
beginning might lead to a different optimum, and would require more
investigations.

Additionally, Lu _et al._ (2018) have bridged the gap between Numerical
Differential Equations and Residual Networks. Binding together Rational Units
with the latter thus implicitly connect rational functions to Linear Sytems.

Network | LReLU | RN | RRN
---|---|---|---
VGG8 | Train | $98.7\pm.0$ | $\bf 99.5\pm.0\bullet$ | $99.2\pm.0\bullet$
Test | $61.7\pm.0$ | $61.3\pm.3$ | $\bf 64.0\pm.1\bullet$
VGG11 | Train | $99.7\pm.0$ | $\bf 99.8\pm.0\bullet$ | $\bf 99.8\pm.0\bullet$
Test | $56.6\pm.4$ | $57.7\pm.3\bullet$ | $\bf 60.0\pm.2\bullet$
Table 2: Recurrence helps the network to regularize and thus improves the
performances on the test set. Train and test accuracies (with std. dev.) for
different architectures trained with Leaky ReLU (baseline), Rational and
Recurrent Rational Networks. The best results are highlighted bold and results
outperforming the LReLU baseline with $\bullet$ markers.

##  4 Recurrent Rational Networks

In the previous section, we bridged the gap between rationals and Residual
Networks. Continuing along these lines, Greff _et al._ (2017) further indicate
that sharing the weights in blocks can improve learning performances, as shown
for Highway (Lu & Renals, 2016) and Residual (Liao & Poggio, 2016) Networks.

Recurrent Rational Networks. Liao _et al._ (2016) link Residual Learning,
Recurrent Neural Networks and Visual Cortex. They first show that very deep
ResNets approximate shallow Recurrent Neural Networks with weight sharing
among the layer. Inspired by this findings, we introduce Recurrent Rational
Networks (RRN), a novel type a Recurrent Network where weights are shared
between rationals. In this type of Networks, the input is propagated through
different layers, but always passed in the same rational activation function.

Moreover, Molina _et al._ (2020) showed that rational tend to overfit and
suggested to further investigate regularization for Rationals. We advocate
that recurrence can act as a natural regularizer for rationals. We test this
hypothesis by comparing Rational Networks and Recurrent Rational Networks to a
baseline in Tab. 2 and later on Reinforcement Learning Tasks (_cf._ Section
4). Our empirical results also indicate that recurrence helps regularize on
CIFAR100.

Localized Recurrence. Liao _et al._ (2016) also investigate where weight
sharing could be the most beneficial for the network. They show that for
visual tasks, recurrent computations are more beneficial in early visual areas
than at middle and later stages, and as explained in their conclusion, these
findings later found psychophysics support (Eberhardt et al., 2016). We thus
evaluate networks different recurrence localizations, and report the results
in Tab. 3 (and Tab. LABEL:tab:cifar100_vgg_mixed in the appendix). Our
findings partially confirm these hypotheses, as the best performing networks
are the one sharing weights in the two early and the two last layers.
Localized recurrence is thus a good balance between modeling power of the
network and regularization.

Location | All | Early | Middle | Late
---|---|---|---|---
10 | Train | 74.3$\pm$.4 | 76.6$\pm$.6 | 75.1$\pm$.5 | 75.9$\pm$1.
Test | 74.9$\pm$.6 | 76.7$\pm$.5 | 74.9$\pm$.4 | 75.7$\pm$.0
100 | Train | 40.7$\pm$1. | 43.5$\pm$.8 | 43.7$\pm$1. | 44.4$\pm$.0
Test | 40.4$\pm$.3 | 42.5$\pm$.4 | 42.3$\pm$.0 | 42.1$\pm$.9
Table 3: Recurrence in early stages leads to the best performances. Mean
accuracies with std. dev. are provided. The evaluation is conducted on a
simple Convolutional Neural Network on CIFAR10 (10) and CIFAR100 (100) for
Recurrent Rational Networks and networks where recurrence is shared for two
successive layers. For local recurrent networks, best train accuracy (for 10)
and test accuracies are obtained when weights are shared at the first two
layers, best train accuracy (for 100) for recurrence in the last layers. The
best results are highlighted bold.

For bigger networks, testing every possible localized recurrence combination
is resource intensive, as the number of network architectures to test grows
exponentially ($2^{n-1}$ different networks for $n$ activation functions).
Therefore, having a way to automatically detect which layer need weight
sharing would considerably reduce computations. In Fig. 3, we plot the shapes
of the learned activation functions and note that the first two functions have
similar profiles as well as the last two functions. However, we here want to
quantify objectively the similarity, or the distance, between profiles of
functions.

Figure 3: Learned rational functions confirm biological intuition that weight
sharing in early layers is beneficial. Shown are the profiles of rational
functions learned at each layer (increasing layers from left to right) of a
simple convolutional network on CIFAR10. As one can see, similar profiles are
learned in the early resp. late layers. The input distributions, as well as
the neural distance between each function are reported on the graph, and
confirms the intuition.

Neurally equivalent activation functions. We first concentrate on equivalence
between neural networks, to derive properties of a suited metric. Research on
neural network equivalence is usually interested in changing the shape of the
neural network, especially reducing its amount of parameters (Al-Rawi & Al-
Rawi, 2010; Kumar et al., 2019). Two networks are defined to be equivalent if
they produce the same output for any given input. In this work, we focus on
equivalence between neural networks that differ only in their activation
functions. The neural networks we consider are equipped with linear layers
such as convolutional or fully connected ones. To provide some intuition, we
first point out that such neural networks, at any given layer, dividing the
activation function by a constant and multiplying the weights of the next
layer by this constant produces an equivalent neural network. Any learned
activation function is thus scale independent (for vertical scaling). Such
transformations can be applied to the bias with addition on the activation
function. Formally, we consider $f_{1}$ and $f_{2}$ two functions such that:

$\forall x,\ f_{1}(x)=a.f_{2}(c.x+d)+b\>,$

we derive:

$\displaystyle\mathbf{W}_{i}f_{1}(\mathbf{W}_{i-1}x+\mathbf{B}_{i-1})+\mathbf{B}_{i}$
$\displaystyle=\ $ $\displaystyle
a.\mathbf{W}_{i}f_{2}(c.\mathbf{W}_{i-1}x+(c.\mathbf{B}_{i-1}+\textbf{d}))+(\mathbf{B}_{i}+\textbf{b})$
$\displaystyle=\ $
$\displaystyle\mathbf{W}^{\prime}_{i}f_{1}(\mathbf{W}^{\prime}_{i-1}x+\mathbf{B}^{\prime}_{i-1})+\mathbf{B}^{\prime}_{i}\
,$

Thus, replacing $f_{1}$ by $f_{2}$ at a given layer of a neural network and
changing the weights and bias of this layer or the next one produces an
equivalent neural network.

Equivalence between activation is scale and shift independent. Considering
this, we introduce a new metric between two functions $f_{1}$ and $f_{2}$ in
Equation 4, that is shift and scale —both vertical and horizontal—
independent.

$\operatorname{ND}(f_{1},f_{2})=\displaystyle\min_{a,b,c,d}\int|f_{1}(x)-(af_{2}(c.x+d)+b)|\
dx$ (4)

With this definition, $\operatorname{ND}$ is actually a quasi-pseudo-metric,
as it does not respect the symmetry and the positive definiteness. We can
easily turn it into a pseudo-metric by taking the minimum between
$\operatorname{ND}(f_{1},f_{2}$) and $\operatorname{ND}(f_{2},f_{1})$ as the
new metric, thus solving the symmetry. However, the positive definite property
(i.e. $\operatorname{ND}(f_{1},f_{2})=0\Rightarrow f_{1}=f_{2}$) is not
verified on purpose, as this is used as neural equivalence property. We
provide the distance measured between the rational functions learned at each
layer in the Fig. 3. The two smallest distances correspond to the pairs of
functions with similar profiles, and confirm that merging those functions is
an optimal choice, as it is a good balance between regularization and modeling
capacity.

##  5 Boosting DQN using Recurrent RNs

Figure 4: Rational and Recurrent Rational Networks lead to superhuman
performance. DQN agents equipped with the Leaky ReLU baseline (LReLU),
Rational Networks (RN) and Recurrent Rational Networks (RRN). The asterisk (*)
is highlighting games for which RN/RRN performs better than human baseline,
whereas baseline does not ($67\%$).

As mentioned in the introduction, different activation functions are often
used for different tasks. Earlier work has shown that Rationals are suited for
supervised learning classification (Molina et al., 2020) and image generation
(Boullé et al., 2020). In this section, we would like to extend the use of
Rational Networks to Deep Reinforcement Learning.

Deep Reinforcement Learning requires additional constraints on the network, as
the policy influences the input and output distributions, and it has often be
shown that unstable policies could lead to a drop in performance (Schulman et
al., 2015, 2017). Therefore, plugging rational networks in a Reinforcement
Learning agent might lead to instabilities. In this section, we demonstrate
that rational networks indeed enhance the performance of the original DQN
agent. We also evaluate the self-regularized Recurrent Rational version, that
helps to prevent potential instabilities introduced by the higher modeling
capacity of Rational Networks. Here the scores are normalized according to Eq
5.

$\text{score}_{\text{normalized }}=100\times\frac{\text{ score }_{\text{agent
}}-\text{ score }_{\text{random }}}{\text{ score }_{\text{baseline }}-\text{
score }_{\text{random }}}.$ (5)

Game | Learnable | Fixed
---|---|---
RRN | RN | SiLU | d+SiLU
Asterix | $9059\bullet$ | $\bf 13018$ | 27.9 | 116
Battlezone | $\bf 679$ | $615\bullet$ | 200 | 99
Breakout | $\bf 217$ | $203\bullet$ | 16.8 | 2.1
Enduro | $791\bullet$ | $\bf 862$ | 22 | 2.3
Jamesbond | $\bf 3622$ | $3572\bullet$ | 70.5 | 61.2
Kangaroo | $\bf 1636$ | 911 | $1085\bullet$ | 118
Pong | $\bf 106\bullet$ | $\bf 106\bullet$ | 95.1 | 103
Qbert | $210\bullet$ | $\bf 216$ | 25.7 | 5
Seaquest | $\bf 3239$ | $2865\bullet$ | 646 | 32.4
Skiing | $151\bullet$ | $\bf 152$ | 82 | 106
SpaceInv. | $\bf 280$ | $125\bullet$ | 97.6 | 95.3
Tennis | $\bf 2967$ | $2960\bullet$ | 300 | 900
Timepilot | $1699\bullet$ | $\bf 2290$ | 129 | 122.4
Tutankham | $\bf 452050$ | $439525\bullet$ | 195950 | 9750
VideoPinball | $206\bullet$ | $\bf 368$ | 12.7 | -0.9
$\\#>$ LReLU | $\bf 15/15$ | $\bf 15/15$ | $6/15$ | $7/15$
Table 4: Rational Networks leading to dramatic performance improvement. Normalized mean scores (_cf._ Eq 5 in percentage with LReLU as baseline), averaged over $5$ reruns, of DQN agents equipped with Recurrent Rational and Rational Networks, and with SiLU and SiLU + dSiLU. The best results are highlighted bold and runner-up with $\bullet$ markers. The last row summarizes the number of improvements over the Leaky ReLU baseline. Game | DQN | DDQN
---|---|---
LReLU | RRN | RN | LReLU
Asterix | $206$ | $12621\bullet$ | $\bf 18109$ | 3723
Battlezone | $4464$ | $\bf 25749$ | $23403\bullet$ | 22775
Breakout | $155$ | $\bf 336$ | $316\bullet$ | 79
Enduro | $121$ | $957\bullet$ | $\bf 1043$ | 353
Jamesbond | $38$ | $\bf 1137$ | $1122\bullet$ | 45
Kangaroo | $335$ | $\bf 5266$ | $2941\bullet$ | 484
Pong | $16$ | $\bf 18.13$ | $18.04\bullet$ | 12
Qbert | $6715$ | $14080\bullet$ | $\bf 14436$ | 8954
Seaquest | $250$ | $\bf 7460$ | $6603\bullet$ | 898
Skiing | $-27365$ | $-23582\bullet$ | $\bf-23487$ | -26892
SpaceInv. | $531$ | $\bf 1395$ | $650\bullet$ | 490
Tennis | $-22.4$ | $\bf 20.6$ | $20.5\bullet$ | -18.4
Timepilot | $1428$ | $13261\bullet$ | $\bf 17632$ | 1016
Tutankham | $3.6$ | $\bf 184$ | $179\bullet$ | 36.4
VideoPinball | $45683$ | $86942\bullet$ | $\bf 149712$ | 62151
$\\#$ Wins | 0/15 | $\bf 9/15$ | $6/15\bullet$ | 0/15
Table 5: DQN equipped with Recurrent Rational and Rational Networks always
outperform the more complex DDQN algorithm, equipped with the baseline. Mean
raw scores (averaged over $5$ reruns) are provided. DDQN agents are equipped
with Leaky ReLU (LReLU). The DQN baseline is also provided. The best results
are highlighted bold and runner-up with $\bullet$ markers. The last row
summarizes the total wins.

Rational Networks in DQN. We evaluate Rational Networks and Recurrent Rational
Networks on Deep Reinforcement Learning regression tasks, using the original
DQN architecture (Mnih et al., 2015) on 15 different games of the Atari 2600
domain. Our first experiments show that DQN agents equipped with Rational
Networks always outperform their baseline. Importantly, Recurrent Rational
Networks further outperform Rational Networks on most (9 out of 15) games.
Moreover, on 67% of the games where the DQN baseline does not, DQN agents
equipped with RN and RRN surpasses human experts (cf. Fig. 4).

Comparison to Swish/SiLU. The SiLU activation function
($\operatorname{SiLU}(x)=x\cdot\operatorname{sigmoid}(x)$) and its derivative
dSiLU were introduced by Elfwing _et al._ (2018). The authors showed that
using SiLU or a combination of SiLU (on convolutional layers) and its
derivative dSiLU (on fully connected layers) in DQN agents perform better on
several games than ReLU networks. Swish, a refined version of SiLU was later
discovered via Reinforcement Learning based automatic search in (Ramachandran
et al., 2018) (as $\operatorname{swish}(x)=x\cdot\operatorname{sigmoid}(\beta
x)$, with $\beta$ constant or as trainable parameter).

SiLU is —to our knowledge— the only activation function specifically designed
for Reinforcement Learning task. We train DQN agents equipped with SiLU and
with the combination (d+SiLU) proposed by Elfwing _et al._ on the same set of
games. Results are summarized in Tab. 4. Agents with SiLU were only able to
outperform Rational Networks on one game out of 15 (Kangaroo), whereas
Recurrent Rational Networks achieve better performance than every non-
learnable activation function for every game, doubling the score of the runner
up, which is the Rational Network, on Space Invaders.

Rational Flexibility to Solve Overestimation. A known problem of the DQN
algorithm is overestimation. Overestimation is due to the combination
Bootstrapping, Off-policy learning and a function approximator (Neural
Network) operated by DQN. To solve this problem, Van _et al._ (2016)
introduced DDQN, that uses an addition network to separate action selection
and action evaluation.

Q-learning overestimation was initially investigated in (Thrun & Schwartz,
1993) and notably attributed to insufficiently flexibility in the function
approximation, that lacks accuracy. Learnable rational functions provide
additional flexibility and accuracy while adding a minimal number of
parameters to the network, that might help tackle the overestimation problem.
To verify this hypothesis, we compare the performance of our DQN agent
equipped with (Recurrent) Rational Networks to a standard DDQN agent in Tab.
5. Our result show that for every tested game, indeed both Rational and
Recurrent Rational DQN agents outperform the DDQNs equipped with the baseline
activation.

DDQN only delays performance drops. As our (Recurrent) Rational DQN agents
continue learning for longer than $200$ epochs on several games (_cf._ Asterix
on Fig. 5, we let our agent train for $500$ epochs. This shows that for
several games (Breakout, JamesBond, Kangaroo, Seaquest, TimePilot), DDQNs do
not solve performance drops, but only delay them, sometimes lowering its
performance below DQNs. A complete graph, with the performance evolution of
every tested agent on every game is provided in the appendix.

Figure 5: DDQN delays performance drop instead of solving it. Evolution of the
scores of DQN agents with Rational Networks (RN) and Recurrent RN (RRN) and
DQN and DDQN with LReLU. The line corresponds to the mean score of every agent
and the shaded area to the std. dev. of five executed runs. A figure including
the evolution on every game is provided in the appendix.

##  6 Rationals: Beyond Existing Activations

Accross our rational journey, we found several best practices that we discuss
in this section.

Importance of Tracking the Input. Properties of different new activation
functions (Georgescu et al., 2020; Misra, 2020) were reported as crucial for
neural networks performances. To verify this hypothesis, the input
distribution of the function should be taken into consideration, as the
network can make use of the different parts of the activation function’s
domain. This is particularly relevant for highly learnable functions such as
rational functions. Therefore, in every activation function graph provided in
this work, we also provided the input distribution. An efficient and direct
way of tracking the distribution is provided in our rational framework.

Refining the Neural Equivalence Metric. Continuing along these lines, we
present in Equation 6 a refined version of our neural distance that takes into
account the input distribution $\rho$.

$\operatorname{rND}(f_{1},f_{2})=\displaystyle\min_{a,b,c,d}\int\rho(x)|f_{1}(x)-af_{2}(c.x+d)+b|\
dx\ .$ (6)

This new neural distance gives more importance to parts that are more used.
One could also use explainability techniques (Wang et al., 2019; Shao et al.,
2021) to track the importance across the input of the rational function.

Embedding Different Function. In Fig. 6 we provide the learned Rational and
Recurrent Rational functions for trained networks on several games. We observe
that several learned profiles are similar to the dSiLU one (notably Recurrent
Rationals), and that every activation function is non-monotonous (as the one
of supervised learning experiments, _cf._ Fig. 3), experimentally confirming
that non-monotonicity helps the learning process (Misra, 2020).

Figure 6: Learned rational activation functions of Reinforcement Learning
Agents automatically reduced their co-domains. Several input distribution has
converged to multimodal Gaussians (e.g. Asterix, Space Invaders and
Tutankham). (Left) activation functions at the successive layers in Rational
Networks. As for supervised learning, several desired properties (as
nonmoncity and bounded negative domains are also learned by these parametric
functions). (Right) Common activation function of every layer in a Recurrent
Rational Network. A Figure with parametric functions for every games is
provided in the appendix.

Furthermore, if we look at the inputs of those functions, we see that they
follow Gaussian distributions. For some of them (Asterix, Kangaroo, Space
Invaders, Tutankham), multimodality appears in the distribution along
learning. This lead us to think that the input distribution is split during
training, and that the rational learns several functions, one next to the
other for these different input groups. If these modes are produced by
different neurons of the layer, this means that several neuron groups need
different activation functions, and that the network could benefit from having
several rational activation functions across one layer. Detecting this need by
tracking the input distribution during learning could lead to performance
improvements.

The need for a new batch normalization. Consequently, using batch
normalization prevents this behavior, as at every layer, the input
distribution is normalized. Future work might benefit from a batch
normalization that would not prevent input distribution split.

Rationals’ Automatic Scaling. Another interesting property of both recurrent
rationals and rationals on Reinforcement Learning is that for all of them, the
size of the co-domain is lower than one, besides their long domain range.
While the supervised learning experiments consist in classifying, DQN requires
a precise regression for the Q-value estimation. This enforces the network to
accurately estimate the expected return of the current policy. We hypothesize
that this automatic vertical scaling mainly observed for reinforcement
learning (_cf._ Fig. 6 vs Fig. 3) helps the network to accurately predict the
Q-Value.

##  7 Conclusion

In this paper, we provided several explanations on the role of activation
functions and shed some light on the reasons why rational functions are so
efficient notably with their implicit residual connection embedding.
Importantly, we introduced Recurrent Rational Networks, whose nature self-
regularizing behavior allow to improve generalization performances, solving
the biggest flaw of Rational Networks. For the first time, the evaluation of
learnable rational activations was brought to Deep Reinforcement Learning,
exhibiting drastic performance improvements. Importantly, we show that
Recurrent Rational Networks further increase the performance of DQN agents.
Alongside releasing an efficient library for building Rational and Recurrent
Rational Networks, we discussed several best practices for their use, such as
tracking the input distribution and similarity of the functions across a
Rational Network for automatic merging.

Our work provides several interesting avenues for future work. To implement
Rational Networks, rational functions have so far been inserted in existing
neural networks architectures. To investigate this further, one should
incorporate Rational Networks into other architectures, _e.g._ Transformers
(Vaswani et al., 2017), and evaluate them on different tasks. Most interesting
is the incorporation of Rational Networks into Neural Architecture Search (Liu
et al., 2018) to discover new efficient Rational Network Architectures.
Finally, rich insight could be gained from the Linear System field (Lu et al.,
2018), linked with rationals through residual connections in this work.

## References

  * Al-Rawi & Al-Rawi (2010) Al-Rawi, M. S. and Al-Rawi, K. R.  On the equivalence between ordinary neural networks and higher order neural networks.  In _Artificial Higher Order Neural Networks for Computer Science and Engineering: Trends for Emerging Applications_ , pp. 138–158. IGI Global, 2010.
  * Boullé et al. (2020) Boullé, N., Nakatsukasa, Y., and Townsend, A.  Rational neural networks.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language models are few-shot learners.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Clevert et al. (2016) Clevert, D., Unterthiner, T., and Hochreiter, S.  Fast and accurate deep network learning by exponential linear units (elus).  In Bengio, Y. and LeCun, Y. (eds.), _4th International Conference on Learning Representations (ICLR)_ , 2016.
  * Eberhardt et al. (2016) Eberhardt, S., Cader, J. G., and Serre, T.  How deep is the feature analysis underlying rapid visual categorization?  In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , pp. 1100–1108, 2016.
  * Elfwing et al. (2018) Elfwing, S., Uchibe, E., and Doya, K.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Neural Networks_ , 107:3–11, 2018.
  * Georgescu et al. (2020) Georgescu, M.-I., Ionescu, R. T., Ristea, N.-C., and Sebe, N.  Non-linear neurons with human-like apical dendrite activations.  2020\.
  * Gidon et al. (2020) Gidon, A., Zolnik, T. A., Fidzinski, P., Bolduan, F., Papoutsi, A., Poirazi, P., Holtkamp, M., Vida, I., and Larkum, M. E.  Dendritic action potentials and computation in human layer 2/3 cortical neurons.  _Science_ , 367(6473):83–87, 2020.
  * Goyal et al. (2019) Goyal, M., Goyal, R., and Lall, B.  Learning activation functions: A new paradigm of understanding neural networks.  _CoRR_ , abs/1906.09529, 2019.
  * Greff et al. (2015) Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., and Schmidhuber, J.  LSTM: A search space odyssey.  _CoRR_ , abs/1503.04069, 2015.
  * Greff et al. (2017) Greff, K., Srivastava, R. K., and Schmidhuber, J.  Highway and residual networks learn unrolled iterative estimation.  In _5th International Conference on Learning Representations (ICLR)_. OpenReview.net, 2017.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778. IEEE Computer Society, 2016.
  * Kumar et al. (2019) Kumar, A., Serra, T., and Ramalingam, S.  Equivalent and approximate transformations of deep neural networks.  _CoRR_ , abs/1905.11428, 2019.
  * Li et al. (2020) Li, X., Tang, J., Zhang, Q., Gao, B., Yang, J. J., Song, S., Wu, W., Zhang, W., Yao, P., Deng, N., et al.  Power-efficient neural network with artificial dendrites.  _Nature Nanotechnology_ , 15(9):776–782, 2020\.
  * Liao & Poggio (2016) Liao, Q. and Poggio, T. A.  Bridging the gaps between residual learning, recurrent neural networks and visual cortex.  _CoRR_ , abs/1604.03640, 2016.
  * Liu et al. (2018) Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L., Fei-Fei, L., Yuille, A. L., Huang, J., and Murphy, K.  Progressive neural architecture search.  In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), _Computer Vision - ECCV 2018 - 15th European Conference_ , volume 11205 of _Lecture Notes in Computer Science_ , pp. 19–35. Springer, 2018.
  * Lu & Renals (2016) Lu, L. and Renals, S.  Small-footprint deep neural networks with highway connections for speech recognition.  In Morgan, N. (ed.), _17th Annual Conference of the International Speech Communication Association (INTERSPEECH)_ , pp. 12–16. ISCA, 2016.
  * Lu et al. (2018) Lu, Y., Zhong, A., Li, Q., and Dong, B.  Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations.  In _Proceedings of the 35th International Conference on Machine Learning, (ICML)_ , pp. 3282–3291, 2018.
  * Manessi & Rozza (2018) Manessi, F. and Rozza, A.  Learning combinations of activation functions.  In _2018 24th International Conference on Pattern Recognition (ICPR)_ , pp. 61–66. IEEE, 2018.
  * Misra (2020) Misra, D.  Mish: A self regularized non-monotonic activation function.  In _31st British Machine Vision Conference 2020 (BMVC)_. BMVA Press, 2020.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Molina et al. (2020) Molina, A., Schramowski, P., and Kersting, K.  Padé activation units: End-to-end learning of flexible activation functions in deep networks.  In _8th International Conference on Learning Representations (ICLR)_. OpenReview.net, 2020.
  * Naitzat et al. (2020) Naitzat, G., Zhitnikov, A., and Lim, L.  Topology of deep neural networks.  _CoRR_ , abs/2004.06093, 2020.
  * Ramachandran et al. (2017) Ramachandran, P., Zoph, B., and Le, Q. V.  Searching for activation functions.  _CoRR_ , abs/1710.05941, 2017.
  * Ramachandran et al. (2018) Ramachandran, P., Zoph, B., and Le, Q. V.  Searching for activation functions.  In _6th International Conference on Learning Representations (ICLR)_. OpenReview.net, 2018.
  * Redmon et al. (2016) Redmon, J., Divvala, S. K., Girshick, R. B., and Farhadi, A.  You only look once: Unified, real-time object detection.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 779–788. IEEE Computer Society, 2016.
  * Schulman et al. (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P.  Trust region policy optimization.  In Bach, F. R. and Blei, D. M. (eds.), _Proceedings of the 32nd International Conference on Machine Learning (ICML)_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pp. 1889–1897. JMLR, 2015\.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.
  * Shao et al. (2021) Shao, X., Skryagin, A., Schramowski, P., Stammer, W., and Kersting, K.  Right for better reasons: Training differentiable models by constraining their influence function.  In _Proceedings of Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI)_ , 2021.
  * Tavosanis (2012) Tavosanis, G.  Dendritic structural plasticity.  _Developmental neurobiology_ , 72(1):73–86, 2012\.
  * Telgarsky (2017) Telgarsky, M.  Neural networks and rational functions.  In Precup, D. and Teh, Y. W. (eds.), _Proceedings of the 34th International Conference on Machine Learning (ICML)_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 3387–3393. PMLR, 2017\.
  * Thrun & Schwartz (1993) Thrun, S. and Schwartz, A.  Issues in using function approximation for reinforcement learning.  In _Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum_ , 1993.
  * van Hasselt et al. (2016) van Hasselt, H., Guez, A., and Silver, D.  Deep reinforcement learning with double q-learning.  In Schuurmans, D. and Wellman, M. P. (eds.), _Proceedings of the Thirtieth Conference on Artificial Intelligence (AAAI)_ , pp. 2094–2100. AAAI Press, 2016.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need.  In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 5998–6008, 2017.
  * Veit et al. (2016) Veit, A., Wilber, M. J., and Belongie, S. J.  Residual networks behave like ensembles of relatively shallow networks.  In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , pp. 550–558, 2016.
  * Wang et al. (2019) Wang, H., Du, M., Yang, F., and Zhang, Z.  Score-cam: Improved visual explanations via score-weighted class activation mapping.  _CoRR_ , abs/1910.01279, 2019.
  * Xie et al. (2019) Xie, S., Zheng, H., Liu, C., and Lin, L.  SNAS: stochastic neural architecture search.  In _7th International Conference on Learning Representations (ICLR)_. OpenReview.net, 2019.
  * Zoph & Le (2017) Zoph, B. and Le, Q. V.  Neural architecture search with reinforcement learning.  In _5th International Conference on Learning Representations (ICLR)_. OpenReview.net, 2017.