Stella and Bauso

*Corresponding author,

# The Impact of Irrational Behaviours in the Optional Prisoner’s Dilemma with
Game-Environment Feedback

Leonardo Stella  Dario Bauso  Department of Computing, College of Science &
Engineering, University of Derby, United Kingdom  Jan C. Willems Center for
Systems and Control, ENTEG, Faculty of Science and Engineering, University of
Groningen, The Netherlands  Dipartimento di Ingegneria, University of Palermo,
Italy<EMAIL_ADDRESS>
###### Abstract

[Summary]In the optional prisoner’s dilemma (OPD), players can choose to
cooperate and defect as usual, but can also abstain as a third possible
strategy. This strategy models the players’ participation in the game and is a
relevant aspect in many settings, e.g. social networks or opinion dynamics
where abstention is an option during an election. In this paper, we provide a
formulation of the OPD where we consider irrational behaviours in the
population inspired by prospect theory. Prospect theory has gained increasing
popularity in recent times thanks to its ability to capture aspects such as
reference dependence or loss aversion which are common in human behaviour.
This element is original in our formulation of the game and is incorporated in
our framework through pairwise comparison dynamics. Recently, the impact of
the environment has been studied in the form of feedback on the population
dynamics. Another element of novelty in our work is the extension of the game-
environment feedback to the OPD in two forms of dynamics, the replicator and
the pairwise comparison. The contribution of this paper is threefold. First,
we propose a modelling framework where prospect theory is used to capture
irrational behaviours in an evolutionary game with game-environment feedback.
Second, we carry out the stability analysis of the system equilibria and
discuss the oscillating behaviours arising from the game-environment feedback.
Finally, we extend our previous results to the OPD and we discuss the main
differences between the model resulting from the replicator dynamics and the
one resulting from the pairwise comparison dynamics.

###### keywords:

Evolutionary games, game theory, pairwise comparison dynamics, nonlinear
dynamics, game-environment feedback, prospect theory.

##  1 Introduction

Evolutionary game theory studies the evolution of strategic interactions in a
population of rational decision makers based on suitable incentives, which are
given in the form of strategy-dependent payoffs 1, 2. The popularity of a
given strategy is influenced by the system dynamics and the incentives given
to an individual for choosing that strategy, thus leading to myopic behaviours
where the favoured option is not the global best for the population. This is
the case for the well-known prisoner’s dilemma (PD), in which the dominant
strategy is defection 3. In the setting of finite games, many evolutionary
dynamics have been used, including best response dynamics, the Brown-von
Neuman-Nash dynamics and replicator dynamics to mention a few. The latter is
recognised to be the one most widely used 4. However, regardless of the
mathematical model used, the common game setting involves fixed incentives
associated with each strategy. This limitation does not consider the role of
the environment in affecting the game dynamics and therefore the frequency of
the strategies in the population.

A recent line of research, initiated in the work by Weitz _et al._ 5, has
attracted a great attention as it proposes a new way to model the reciprocal
dependence of the payoffs on the environment and vice versa, taking into
account the enhancement and degradation effects on the environmental resource.
This is done through a direct dependence of the payoff matrix on the
environment, giving rise to interesting system dynamics characterised by
oscillatory behaviours and equilibria on the boundary of the phase space. The
resulting oscillations correspond to closed periodic orbits. Limit cycles have
been observed also when a time-scale difference between the game and the
environment dynamics is considered 6.

This feedback mechanism is believed to be able to explain the complexity of
real-world systems for which one often observes bi-directional feedback
between the state of the environment and the incentives associated to each
strategy 6. Interactions of this kind can be found in a wide range of
disciplines, including sociology, economics and animal behaviour 7, 8, 9, 10.
An example of this in human systems is the individuals’ decision to vaccinate
or not, which would result in otherwise-preventable infectious diseases in
children and then an increasing incentive to vaccinate 11, 12, 13. In ecology,
an example can be found in plant nutrient acquisition where the competitive
balance between the interactions of different species and the availability of
environmental resources alter the nature of the competitive process of
nitrogen fixation 14. Another example is in the context of collective
decision-making in honeybees where an initial attempt to incorporate game-
environment feedback within the parameters of the model has been proposed 15,
giving rise to oscillations.

The work by Su _et al._ has taken another direction by studying the impact of
the environment and its evolution through game transitions, namely a framework
where players move from playing one game and its associated payoff matrix to
another one based on the environmental resource 16. Chen and Szolnoki have
applied the concept of game-environment feedback to punishment and inspection
in the setting of governing the commons 17. Finally, another line of research
has explored the asymmetric aspects of this topic, both in the case of
asymmetric evolutionary dynamics and in the case of asymmetrical game-
environment feedback 18, 19.

Game-environment feedback, or in general coevolutionary game theory, has
generated increasing interest in the control community, where the design of
adequate control policies play a crucial role in achieving a specific system
behaviour 20, 21. An example of the use of game-environment feedback in
control is the work by Paarporn _et al._ where the controlled system exhibits
oscillatory behaviours due to deprecation of the environmental resource 22.
Examples from such a wide range of research areas show that an understanding
of the dynamics in the presence of game-environment feedback is a crucial
aspect for a deeper awareness of complex real-life systems.

Although several approaches have been adopted to study the prisoner’s dilemma,
including, e.g., the design of a strategy that is obtained by saturating a
polynomial function in an $n$-population game setting 23, incorporating the
evolution of the environment in the strategy profile through the payoff matrix
poses a risk in limiting the likelihood of cooperation, and gives rise to an
“oscillating tragedy of the commons” 5. An additional difficulty may come when
we take into consideration the corresponding optional game, i.e. the optional
prisoner’s dilemma (OPD), where players can choose to cooperate and defect,
but also abstain from playing the game 24. This formulation includes the
dynamics of many real-life decision-making settings, such as opinion dynamics
in the context of an election campaign and duopolistic competition in
marketing 24, 25.

By extending the work on game-environment feedback to the OPD, we derive two
models: one resulting from the replicator dynamics as an extension of the
prior work on game-environment feedback and another one resulting from the
pairwise comparison dynamics. The motivation that underpins the need for this
additional model comes from the surge of interest that prospect theory has had
in the past years and its continuous use in the past 30 years within
behavioural economics and beyond 26. Prospect theory has its foundations on
the work from two psychologists, Kahneman and Tversky, through which they
demonstrated a systematic misalignment between the predictions conducted with
utility theory and the real-life behaviour of individuals’ decision-making
under risk 27. The main elements that constitute this theory are 26:

  * •

reference dependence, which is a measure of utility to a reference point;

  * •

loss aversion, which captures the idea that people are more sensitive to
losses;

  * •

diminishing sensitivity, which models losses and gains in different ways, i.e.
the value function is convex in the region of losses and concave in the region
of gains, respectively;

  * •

probability weighting, which account for the addition weights that individuals
assign to probabilities of events.

Through the pairwise comparison dynamics, we capture some elements of
irrational or subjective behaviour that comes from the above elements of
prospect theory. Prospect theory has been applied in a range of settings,
including discrimination in obsessive-compulsive disorders 28, dynamics
pricing for shared mobility 29 and in the context of collective decision-
making through pairwise comparison dynamics 30. Finally, we include prospect
theory in the formulation of the OPS with game-environment feedback via
pairwise comparison dynamics through the use of a weighting function which
gives weight zero to the probability of unfavourable events in a risk-seeking
scenario.

Highlights of contributions. The contribution of this paper is threefold.
First, we propose a modelling framework where we capture the irrational
behaviours originating in the context of behavioural economics through
Kahneman and Tversky’s prospect theory. In the proposed framework, we study
the game-environment feedback on replicator dynamics5 and we extend it to a
different kind of dynamics, i.e. pairwise comparison dynamics. Second, we
carry out the stability analysis of the system resulting from these dynamics
and discuss its differences with the original model. Lastly, we extend the
work on the PD to account for the abstain strategy via the OPD. Motivated by
the importance of the abstain strategy in real-life contexts and the
irrational behaviours in many decision making settings, we investigate the
system resulting from the application of pairwise comparison dynamics to the
optional game.

This paper is organised as follows. In Section 2, we provide the problem
formulation for the PD with game-environment feedback in the case of pairwise
comparison dynamics. We conduct the stability analysis of the equilibria of
the system and we discuss the differences and similarities between our results
and the results in previous research works. In Section 3, we extend the
proposed framework with game-environment feedback to the case of the OPD,
where players can choose to abstain from playing the game as a third strategy.
We study the stability of the two models and discuss the main differences
between them and the added value of this work in relation to the rest of the
literature on this topic. Finally, in Section 4 we draw conclusions and
discuss future directions of research.

##  2 Feedback-Evolving Prisoner’s Dilemma via Pairwise Comparison Dynamics

Evolutionary game dynamics with game-environment feedback have become popular
in the past few years because of the induced dependency between the population
dynamics and the surrounding environment. This leads to a system where the
environment changes as a consequence of the frequency of the strategies and
where the population dynamics are affected by the environment in return.
Figure 1 shows a diagram representation of the game-environment feedback,
where the game dynamics and the environment form a closed-loop system.

$\dot{n}$Game-environment Feedback$A(n)$Payoff
Matrix$\dot{\mathbf{x}}$Population Dynamics Figure 1: Diagram representation
of the evolutionary dynamics for game-environment feedback. In general, the
payoff matrix $A$ determines the frequencies of strategies, $x_{i}$ for all
$i$, to derive the population dynamics. Through the game-environment feedback
the frequencies of the strategies change the state of the environment, $n$,
which, in turn, influences the values in the payoff matrix $A(n)$, with
explicit dependence on the environmental resource $n$.

###  2.1 Evolutionary Game Model via Pairwise Comparison Dynamics

In this section, we extend the initial feedback evolving game framework to
pairwise comparison dynamics, which are a form of innovative dynamics. We
compare our results to the framework obtained via replicator dynamics by Weitz
_et al._5 in the context of the prisoner’s dilemma (PD). The motivation for
this extension is twofold: first, through pairwise comparison dynamics we can
capture some aspects of irrational behaviour of individuals’ decision-making
under risk through prospect theory 26, 27; second, the results presented here
share similarities with the seminal work on game-environment feedback that are
worth investigating, especially in terms of equilibria and stability 5.

Let us consider the standard payoff matrix in the context of the PD where the
two-player game consists of the two strategies cooperation (C) and defection
(D):

$A=\left[\begin{array}[]{cc}R&S\\\ T&P\end{array}\right],$ (1)

where $R$ is the reward for cooperating, $S$ is the sucker’s payoff, $T$ is
the payoff associated with the temptation to cheat and $P$ is the punishment
for cheating. In the classical formulation of the game, $T>R>P>S$ so that
mutual defection is the Nash equilibrium associated with the game. In the
context of evolutionary game theory, the model associated with the above
symmetric two-player game via replicator dynamics can be written as 5

$\dot{x}=x(1-x)(r_{1}(x)-r_{2}(x)),$ (2)

where $x\equiv x_{1}$ because of the conservation of mass law for which
$x_{1}+x_{2}=1$, and where $r_{1}(x)$ is the fitness of player 1 and
$r_{2}(x)$ is the fitness of player two, respectively. These are defined as:

$\displaystyle r_{1}(x)$ $\displaystyle=$
$\displaystyle\sum_{j}a_{1j}x_{j}=(Ax)_{1}=Rx+S(1-x),$ (3) $\displaystyle
r_{2}(x)$ $\displaystyle=$
$\displaystyle\sum_{j}a_{2j}x_{j}=(Ax)_{2}=Tx+P(1-x).$ (4)

After substituting the above into equation (2), we obtain:

$\dot{x}=x(1-x)(Rx+S(1-x)-Tx-P(1-x))=-x(1-x)(\delta_{TR}x+\delta_{PS}(1-x)),$
(5)

where $\delta_{TR}=T-R$ and $\delta_{PS}=P-S$.

We can now formulate the analogous model for the PD via pairwise comparison
dynamics. To this end, let us consider the following dynamics:

$\dot{x}_{i}=\sum_{j}x_{j}\phi_{ji}-x_{i}\sum_{j}\phi_{ij},$ (6)

where $\phi_{ij}\geq 0$ is the rate at which a player $i$ changes his strategy
to strategy $j$ and $x_{i}\phi_{ij}$ is the flux from strategy $i$ to strategy
$j$. When $\phi_{ij}$ is given by the proportional rule
$\phi_{ij}=[a_{j}-a_{i}]_{+}=\sum_{k}[a_{jk}-a_{ik}]_{+}x_{j}$, and by
recalling that the notation $[\cdot]_{+}$ denotes the positive part as before,
the resulting dynamics are called pairwise comparison dynamics or pairwise
difference dynamics 31. After a simple calculation, the model resulting from
the pairwise comparison dynamics for the PD is the following:

$\dot{x}=(1-x)[r_{1}(x)-r_{2}(x)]_{+}-x[r_{2}(x)-r_{1}(x)]_{+},$ (7)

where $r_{1}(x)$ and $r_{2}(x)$ are the fitness of player 1 and the fitness of
player 2, respectively, as defined before. If we substitute the payoffs from
matrix (1), we get the following model:

$\dot{x}=-x(\delta_{TR}x+\delta_{PS}(1-x)).$ (8)

######  Lemma 2.1.

Consider system (8). This system has two fixed points: $x^{*}=0$ which is
asymptotically stable and $x^{*}=1-\delta_{PS}/\delta_{TR}$ which is unstable.

######  Proof 2.2.

The stability of these two equilibria can be identified by looking at the sign
of the square, i.e. $x^{*}=0$ is stable whereas
$x^{*}=\delta_{PS}/(\delta_{PS}-\delta_{TR})$ is unstable. Therefore, when $t$
is large, the population will converge to the point where all players choose
to defect and no cooperators are present.

Example 1. Consider the following payoff matrix

$A=\left[\begin{array}[]{cc}3&0\\\ 5&1\end{array}\right].$

We can calculate the fitness of player 1 and the fitness of player 2 as in the
following:

$r_{1}(x)=3x_{1},$ $r_{2}(x)=5x_{1}+x_{2},$

and the resulting model via replicator dynamics is

$\dot{x}=-x(1-x)(1+x),$

while the model resulting from the pairwise difference dynamics is

$\dot{x}=-x(1+x).$

In the above, regardless of the dynamics used, the only stable equilibrium
point in the domain $[0,\,1]$ is $x^{*}=0$, as it can be deduced from the sign
as before. Within the same domain, the model obtained via replicator dynamics
presents another equilibrium point $x^{*}=1$, which is unstable. The
equilibrium point $x^{*}=-1$, albeit common to both models, is not in the
given domain and therefore is not considered. The dynamics for this example
are depicted in Fig. 2 (top). $\square$

Figure 2: Time evolution of the model for the PD via replicator dynamics (RD)
and pairwise comparison dynamics (PCD). The first plot (top) shows the
evolution of the cooperation strategy for Example 1. The second plot (bottom)
shows the evolution of the cooperation strategy for Example 2.

Example 2. In this example, we consider a situation where cooperation is
favoured. Let the payoff matrix be

$A=\left[\begin{array}[]{cc}5&1\\\ 3&0\end{array}\right].$

The fitness of player 1 and the fitness of player 2 are calculated as

$r_{1}(x)=5x_{1}+x_{2},$ $r_{2}(x)=3x_{1},$

and the resulting model via replicator dynamics is

$\dot{x}=x(1-x)(1+x),$

while the model resulting from the pairwise difference dynamics is

$\dot{x}=(1-x)(1+x).$

As in the previous example, the model via replicator dynamics has two
equilibria in the domain $[0,\,1]$, namely $x^{*}=0$ and $x^{*}=1$, whereas
the model with pairwise comparison dynamics has only one equilibrium
$x^{*}=1$. In both cases, the only stable equilibrium point is $x^{*}=1$,
meaning that cooperation is the stable strategy. The dynamics for this example
are depicted in Fig. 2 (bottom). $\square$

###  2.2 Game-Environment Feedback

We can now extend the above results in the context of feedback-evolving games,
namely when the payoff matrix depends on an environmental resource $n$, in
line with the work by Weitz _et al._ 5. We define our game-environment
feedback dynamics as

$\begin{array}[]{lll}\epsilon\dot{x}=(1-x)[r_{1}(x,A(n))-r_{2}(x,(A(n))]_{+}-x[r_{2}(x,A(n))-r_{1}(x,A(n))]_{+},\\\
\dot{n}=n(1-n)[(1+\lambda)x-1].\end{array}$ (9)

The main difference between the above model and the one in the previous
section is the environmental dependency of the payoff matrix $A(n)$. For the
purpose of studying this model, we consider the following environment-
dependent matrix:

$A(n)=(1-n)\left[\begin{array}[]{cc}T&P\\\
R&S\end{array}\right]+n\left[\begin{array}[]{cc}R&S\\\
T&P\end{array}\right]=\left[\begin{array}[]{cc}T-n\delta_{TR}&P-n\delta_{PS}\\\
R+n\delta_{TR}&S+n\delta_{PS}\end{array}\right],$ (10)

where $\delta_{TR}=T-R$, $\delta_{PS}=P-S$ and we assume $T>R>P>S$ as before.
The resulting pairwise comparison dynamics (pairwise difference) for the PD
with game-environment feedback is therefore:

$\begin{array}[]{lll}\epsilon\dot{x}=(1-x)[\delta_{PS}+(\delta_{TR}-\delta_{PS})x](1-2n),\\\
\dot{n}=n(1-n)[(1+\lambda)x-1].\end{array}$ (11)

######  Theorem 2.3.

System (10) has five fixed points, namely: $(x^{*},n^{*})=(1,0)$,
$(x^{*},n^{*})=(1,1)$,
$(x^{*},n^{*})=(\delta_{PS}/(\delta_{PS}-\delta_{TR}),0)$,
$(x^{*},n^{*})=(\delta_{PS}/(\delta_{PS}-\delta_{TR}),1)$ and
$(x^{*},n^{*})=(1/(\lambda+1),1/2)$. The first four fixed points are unstable,
and the last one, namely $(x^{*},n^{*})=(1/(\lambda+1),1/2)$, is a neutrally
stable center.

######  Proof 2.4.

System (10) has five fixed points, namely: $(x^{*},n^{*})=(1,0)$,
$(x^{*},n^{*})=(1,1)$,
$(x^{*},n^{*})=(\delta_{PS}/(\delta_{PS}-\delta_{TR}),0)$,
$(x^{*},n^{*})=(\delta_{PS}/(\delta_{PS}-\delta_{TR}),1)$ and
$(x^{*},n^{*})=(1/(\lambda+1),1/2)$. The first two correspond to the scenario
of cooperators in a depleted environment and cooperators in a replete
environment. The third and fourth equilibria depend on the payoffs, whereas
the last equilibrium point is the only interior fixed point representing a
population of cooperators and defectors in a half-depleted environment.
Without loss of generality, we set $\epsilon=1$ and carry out the stability
analysis of the above equilibria. The Jacobian of system (10) is the
following:

$J(x,n)=\left[\begin{array}[]{cc}(1-2n)(\delta_{TR}-2\delta_{PS})+x(4n-2)(\delta_{TR}-\delta_{PS})&\quad-2\delta_{PS}-2x\delta_{TR}4x\delta_{PS}+2x^{2}(\delta_{TR}-\delta_{PS})\\\
n(\lambda+1)-n^{2}(\lambda+1)&\quad-1+2n+x(\lambda+1)-2nx(\lambda+1)\end{array}\right].$
(12)

The Jacobian linearised at the first four fixed points is:

$\begin{array}[]{ll}J(1,0)=\left[\begin{array}[]{cc}-\delta_{TR}&0\\\
0&\lambda\end{array}\right],&J\Big{(}\frac{\delta_{PS}}{\delta_{PS}-\delta_{TR}},0\Big{)}=\left[\begin{array}[]{cc}\delta_{TR}&0\\\
0&\frac{\lambda\delta_{PS}+\delta_{TR}}{\delta_{PS}-\delta_{TR}}\end{array}\right],\\\
J(1,1)=\left[\begin{array}[]{cc}\delta_{TR}&0\\\
0&-\lambda\end{array}\right],&J\Big{(}\frac{\delta_{PS}}{\delta_{PS}-\delta_{TR}},1\Big{)}=\left[\begin{array}[]{cc}-\delta_{TR}&0\\\
0&\frac{\lambda\delta_{PS}-\delta_{TR}}{\delta_{PS}-\delta_{TR}}\end{array}\right].\end{array}$

Each Jacobian has at least one positive eigenvalue and is therefore locally
unstable. We can now turn our attention to the interior fixed point and
calculate its Jacobian as in the following:

$J\Big{(}\frac{1}{\lambda+1},\frac{1}{2}\Big{)}=\left[\begin{array}[]{cc}0&\quad-2\delta_{PS}(\lambda+1)^{2}+2(\lambda+1)(2\delta_{PS}-\delta_{TR})-2(\delta_{PS}-\delta_{TR})\\\
\frac{1}{4}(\lambda+1)&\quad 0\end{array}\right],$

whose eigenvalues have no real parts and thus the interior fixed point is a
neutrally stable center.

######  Remark 2.5.

By comparing the above formulation of the PD with the one in Weitz _et al._ 5,
we obtain a model of degree 2 whose fixed points share the same stability
properties with the one resulting from the replicator dynamics. Two
interesting differences emerge from comparing the two models. The first one is
that our model does not directly include the fixed points corresponding to all
defectors in a depleted/replete environment, unless $P=S$. The second one can
be found by looking at the degree of our system, which is one lower than the
one by Weitz _et al._ This can be explained by looking at the rate of changing
strategy in the pairwise difference comparison: the value obtained by the
proportional rule $\phi_{ij}=[a_{j}-a_{i}]_{+}$ considers the probability of a
payoff increase only, thus assigning a different weighting factor to different
events. This aspect is one of the core elements in prospect theory and is the
motivation that underpins our approach 27.

Example 3. Given the payoff matrix in (9), for the sake of clarity we now
consider system (10) with the following parameters:

$\begin{array}[]{lll}&\epsilon=0.1,\quad\lambda=2,\\\ R=3,&S=0,\quad T=5,\quad
P=1.\end{array}$

The model resulting from the pairwise comparison dynamics is

$\begin{array}[]{lll}0.1\dot{x}=(1-x)[1+x](1-2n),\\\
n=n(1-n)[3x-1].\end{array}$

In accordance with the results, the above model has five fixed points:
$(x^{*},n^{*})=(1,0)$, $(x^{*},n^{*})=(1,1)$, $(x^{*},n^{*})=(-1,0)$,
$(x^{*},n^{*})=(-1,1)$ and $(x^{*},n^{*})=(1/3,1/2)$. The third and the fourth
are outside the domain $[0,\,1]$, and thus are not considered. The first two
fixed points are unstable, whereas the interior fixed point is a neutrally
stable center, and therefore we have an oscillatory behaviour. The time
evolution and phase plane dynamics for this example are depicted in Fig. 3.
$\square$

Figure 3: Feedback-evolving game model resulting from the pairwise comparison
dynamics for the PD (Example 3). The first plot (left) shows the evolution of
the fraction of cooperators and the environment over time. The second plot
(right) shows the phase plane dynamics in the $x$-$n$ plane for system (10):
the blue arrows indicate the direction of dynamics while the red dot denotes
the interior fixed point at (1/3, 1/2). Finally, the different trajectories
correspond to the following initial conditions: $(0.9,0.9)$, $(0.9,0.7)$,
$(0.1,0.3)$, $(0.1,0.1)$.

##  3 Optional Prisoner’s Dilemma with game-environment Feedback

A crucial aspect that is not captured by the standard PD is the possibility
that players can choose neither option. The optional prisoner’s dilemma (OPD)
game is an extension of the PD where players can choose to abstain, and
therefore neither cooperate nor defect 32. This translates to a game with
three possible strategies: cooperate, defect and abstain. The implications of
this additional strategy are crucial in many applications, whose most notable
examples are opinion dynamics during an election and duopolistic competition
in marketing 24, 25.

To describe the evolution of the strategy profile in the population under
game-environment feedback, let the environment-dependent payoff matrix be:

$A(n)=(1-n)\left[\begin{array}[]{ccc}T&P&L\\\ R&S&L\\\
L&L&L\end{array}\right]+n\left[\begin{array}[]{ccc}R&S&L\\\ T&P&L\\\
L&L&L\end{array}\right]=\left[\begin{array}[]{ccc}T-n\delta_{TR}&P-n\delta_{PS}&L\\\
R+n\delta_{TR}&S+n\delta_{PS}&L\\\ L&L&L\end{array}\right],$ (13)

where the parameters have the usual meaning, and $L$ is the loner’s payoff.
This payoff is given to both players if at least one chooses to abstain. As it
is common in the literature, we assume $T>R>L>P>S$.

In the following, we discuss the feedback-evolving OPD game model via
replicator dynamics and pairwise comparison dynamics. First, we provide a
general formulation of the OPD via replicator dynamics as in the following:

$\begin{array}[]{lll}\epsilon\dot{x}_{i}=x_{i}(1-x_{i})[r_{i}(x,A(n))]-x_{i}x_{j}[r_{j}(x,A(n))]-x_{i}L(1-x_{i}-x_{j}),\\\
\dot{n}=n(1-n)[(1+\lambda)x_{1}-1],\end{array}$ (14)

where $i=1,2$ and $j=2,1$. The specific model resulting from the payoff matrix
(13) where $x_{3}=1-x_{1}-x_{2}$ is the following:

$\begin{array}[]{lll}\epsilon\dot{x}_{1}=x_{1}(1-x_{1})[r_{1}(x,A(n))]-x_{1}x_{2}[r_{2}(x,A(n))]-x_{1}L(1-x_{1}-x_{2}),\\\
\epsilon\dot{x}_{2}=x_{2}(1-x_{2})[r_{2}(x,A(n))]-x_{1}x_{2}[r_{1}(x,A(n))]-x_{2}L(1-x_{1}-x_{2}),\\\
\dot{n}=n(1-n)[(1+\lambda)x_{1}-1].\end{array}$ (15)

In the above model, the evolution of the environment is subject to the
influence of the players choosing to cooperate, similarly to previous works 5,
6. To derive the corresponding model via pairwise comparison dynamics, recall
that the rate $\phi_{ij}$ is given by the proportional rule
$\phi_{ij}=[a_{j}-a_{i}]_{+}=\sum_{k}[a_{jk}-a_{ik}]_{+}x_{j}$. Therefore we
have:

$\begin{array}[]{lll}\epsilon\dot{x}_{i}=x_{j}[\phi_{ji}]_{+}+(1-x_{i}-x_{j})[\phi_{3i}]_{+}-x_{i}([\phi_{ij}]_{+}+[\phi_{i3}]_{+}),\\\
\dot{n}=n(1-n)[(1+\lambda)x_{1}-1],\end{array}$ (16)

where $i=1,2$ and $j=2,1$ as before. By substituting the values from the
payoff matrix (13), the resulting model is the following:

$\begin{array}[]{lll}\epsilon\dot{x}_{1}=(x_{2}-3nx_{1})(\delta_{TR}x_{1}+\delta_{PS}x_{2})+\delta_{TL}x_{1}(1-x_{1}-x_{2})+\delta_{PL}x_{1}x_{2},\\\
\epsilon\dot{x}_{2}=(x_{1}-2nx_{2})(\delta_{TR}x_{1}+\delta_{PS}x_{2})+(1-x_{1}-x_{2})((\delta_{RL}+\delta_{TR}n)x_{1}+\delta_{PS}nx_{2})+\delta_{TL}x_{1}x_{2},\\\
\dot{n}=n(1-n)[(1+\lambda)x_{1}-1],\end{array}$ (17)

where $\delta_{TL}=T-L$, $\delta_{PL}=P-L$ and $\delta_{RL}=R-L$, while the
other parameters have the usual meaning.

Table 1: List of all 11 fixed points for system (15). # | $x_{1}$ | $x_{2}$ | $n$
---|---|---|---
1 | 0 | 0 | 0
2 | 1 | 0 | 0
3 | 0 | 1 | 0
4 | 0 | 0 | 1
5 | 1 | 0 | 1
6 | 0 | 1 | 1
7 | $\delta_{PS}/(\delta_{PS}-\delta_{TR})$ | $-\delta_{TR}/(\delta_{PS}-\delta_{TR})$ | 0
8 | $\delta_{PS}/(\delta_{PS}-\delta_{TR})$ | $-\delta_{TR}/(\delta_{PS}-\delta_{TR})$ | 1
9 | $1/(1+\lambda)$ | $\lambda/(1+\lambda)$ | $1/2$
10 | $1/(1+\lambda)$ | $-(R-2L+T)/((P-2L+S)(1+\lambda))$ | $1/2$
11 | $1/(1+\lambda)$ | 0 | $(L-T)/(R-T)$

Results – Stability Analysis. System (15) has a total of eleven fixed points,
as reported in Table 1. The first one $(0,0,0)$, namely all players abstain,
is marginally stable as it can be seen by the Jacobian linearised about it:

$J(0,0,0)=\left[\begin{array}[]{ccc}0&0&0\\\ 0&0&0\\\
0&0&-1\end{array}\right],$

Depending on the initial conditions, the trajectories can converge to this
point or be attracted to the interior fixed points 9 and 10 from Table 1,
similarly to the PD with game-environment feedback. On a preliminary analysis,
this second behaviour can be seen when no players choose to abstain at the
start of the game, namely $x_{1}(0)+x_{2}(0)=1$.

On the other hand, system (17) has ten fixed points, but the analytical
expression for each of them is too large to fit in a table, albeit similar to
the one for the replicator dynamics. The important element in this case is
that, regardless of the initial conditions, all trajectories converge to the
interior fixed point at which the number of cooperators is equal to
$1/(1+\lambda)$, which turns out to be asymptotically stable.

######  Remark 3.1.

It is worth noting that by unilaterally choosing to assign weight zero to the
probability of unfavourable events and thus by considering only the
probability of a payoff increase via pairwise comparison dynamics, the
population converges to a mixed strategy equilibrium, where cooperation is
achieved by one part of the population. This formulation is in accordance to
the main elements that constitute the prospect theory 27. Finally, by
considering a situation where the degrading effect is stronger, namely
$\lambda<1$, the population will converge to a strategy profile where the
dominant strategy is cooperation.

Example 4. Let the payoff matrix be (13) with the following parameters:

$\begin{array}[]{lll}&\epsilon=0.5,\quad\lambda=2,\\\ R=3,\quad S=0,&\quad
L=2,\quad T=5,\quad P=1.\end{array}$

The models resulting from the replicator dynamics and pairwise difference
dynamics share many fixed points as before, but the stability behaviour of
these points changes in relation to the dynamics used and the initial
condition. For the replicator dynamics, we observe oscillations when
$x_{1}+x_{2}=1$, namely when no players abstain. The trajectories have an
oscillating behaviour about the internal fixed point, which can be calculated
as (0.33, 0.44, 0.5). The only stable fixed point is $(0,0,0)$, namely when
all players abstain. In the case of pairwise comparison dynamics, we observe a
completely different behaviour: all trajectories converge to the internal
fixed point, i.e. (0.33, 0.37, 0.51), regardless of the initial condition.
This internal point is asymptotically stable. These behaviours can be seen in
Figg. 4-5. $\square$

Figure 4: Feedback-evolving game model resulting from the replicator dynamics
for the OPD (Example 4). The first two plots (left and centre) show the
evolution of the fraction of cooperators and defectors and the environment
over time for initial conditions $(0.9,0.1,0.1)$ and $(0.1,0.9,0.9)$,
respectively. The last plot (right) shows the phase plane dynamics in the
$x_{1}$-$x_{2}$-$n$ space: the blue arrows indicate the direction of the
dynamics while the red dot denotes the interior fixed point at (0.33, 0.44,
0.5). For different initial conditions, the trajectories are either trapped in
a limit cycle or converge to $(0,0,0)$. Figure 5: Feedback-evolving game model
resulting from the pairwise comparison dynamics for the OPD (Example 4). The
first two plots (left and centre) show the evolution of the fraction of
cooperators and defectors and the environment over time for initial conditions
$(0.9,0.1,0.1)$ and $(0.1,0.9,0.9)$, respectively. The last plot (right) shows
the phase plane dynamics in the $x_{1}$-$x_{2}$-$n$ space: the blue arrows
indicate the direction of the dynamics while the red dot denotes the interior
fixed point at (0.33, 0.37, 0.51). Regardless of the initial conditions, all
trajectories converge to the interior fixed point.

##  4 Conclusions

Motivated by the emerging interest towards a game theoretic modelling
framework that can capture player’s irrational behaviours and the possibility
to abstain in a feedback-evolving game context, in this paper we have proposed
a unified framework to capture these three main aspects. The resulting
feedback-evolving optional prisoner’s dilemma under irrational behaviour is
obtained via pairwise comparison dynamics. We have discussed the differences
between our model and the prior works on the topic of game-environment
feedback within evolutionary game theory. Driven by the need for capturing
optional games, we have extended our analysis to the optional prisoner’s
dilemma by deriving the model corresponding to the replicator dynamics and the
pairwise comparison dynamics. Interestingly, the behaviour of these two
systems under game-environment feedback is notably different, despite the
presence of oscillations in both models. The latter model gives rise to an
interesting setting for partial cooperation, which provides an explanation for
the reasons underpinning the presence of cooperation in many biological
systems. Future work includes: i) the extension of this model to a structured
setting where a network topology is used to capture the interactions in the
population, ii) the study of different time scales for the population dynamics
and the evolution of the environment, and iii) the impact of time-varying
parameters on the population dynamics.

### Author contributions

L.S. and D.B. designed the research, L.S. performed the research and wrote the
paper.

### Financial disclosure

None reported.

### Conflict of interest

The authors declare no potential conflict of interests.

## References

  * 1 J. Maynard Smith and G. Price, “The Logic of Animal Conflict”, _Nature_ , vol. 246, no. 5427, pp. 15-18, 1973. Available: 10.1038/246015a0.
  * 2 J. Maynard Smith, _Evolution and the Theory of Games_. Cambridge: Cambridge University Press, 1982.
  * 3 R. Axelrod and W. D. Hamilton, “The Evolution of Cooperation”, _Science_ , vol. 211, no. 4489, pp. 1390-1396, 1981. Available: 10.1126/science.7466396.
  * 4 J. Hofbauer and K. Sigmund, _Evolutionary Games and Population Dynamics_. Cambridge: Cambridge University Press, 1998.
  * 5 J. Weitz, C. Eksin, K. Paarporn, S. Brown and W. Ratcliff, “An Oscillating Tragedy of the Commons in Replicator Dynamics with Game-Environment Feedback”, _Proceedings of the National Academy of Sciences_ , vol. 113, no. 47, pp. E7518-E7525, 2016. Available: 10.1073/pnas.1604096113.
  * 6 A. Tilman, J. Plotkin and E. Akçay, “Evolutionary games with environmental feedbacks”, _Nature Communications_ , vol. 11, no. 1, 2020. Available: 10.1038/s41467-020-14531-6.
  * 7 D. Pais, C. Caicedo-Núñez and N. Leonard, “Hopf Bifurcations and Limit Cycles in Evolutionary Network Dynamics”, _SIAM Journal on Applied Dynamical Systems_ , vol. 11, no. 4, pp. 1754-1784, 2012. Available: 10.1137/120878537.
  * 8 C. Mullon, L. Keller and L. Lehmann, “Social Polymorphism is Favoured by the Co-evolution of Dispersal with Social Behaviour”, _Nature Ecology & Evolution_, vol. 2, no. 1, pp. 132-140, 2017. Available: 10.1038/s41559-017-0397-y.
  * 9 J. Lee, Y. Iwasa, U. Dieckmann and K. Sigmund, “Social Evolution leads to Persistent Corruption”, _Proceedings of the National Academy of Sciences_ , vol. 116, no. 27, pp. 13276-13281, 2019. Available: 10.1073/pnas.1900078116.
  * 10 S. Estrela _et al._ , “Environmentally Mediated Social Dilemmas”, _Trends in Ecology & Evolution_, vol. 34, no. 1, pp. 6-18, 2019. Available: 10.1016/j.tree.2018.10.004.
  * 11 C. Bauch, A. Galvani and D. Earn, “Group Interest versus Self-Interest in Smallpox Vaccination Policy”, _Proceedings of the National Academy of Sciences_ , vol. 100, no. 18, pp. 10564-10567, 2003. Available: 10.1073/pnas.1731324100.
  * 12 C. Bauch and D. Earn, “Vaccination and the Theory of Games”, _Proceedings of the National Academy of Sciences_ , vol. 101, no. 36, pp. 13391-13394, 2004. Available: 10.1073/pnas.0403823101.
  * 13 A. Galvani, T. Reluga and G. Chapman, “Long-Standing Influenza Vaccination Policy is in Accord with Individual Self-Interest but not with the Utilitarian Optimum”, _Proceedings of the National Academy of Sciences_ , vol. 104, no. 13, pp. 5692-5697, 2007. Available: 10.1073/pnas.0606774104.
  * 14 D. Menge, S. Levin and L. Hedin, “Evolutionary Tradeoffs can Select against Nitrogen Fixation and thereby Maintain Nitrogen Limitation”, _Proceedings of the National Academy of Sciences_ , vol. 105, no. 5, pp. 1573-1578, 2008. Available: 10.1073/pnas.0711411105.
  * 15 W. Baar and D. Bauso, “Environmental Feedback incorporated on a Collective Decision Making Model”, _Proceedings of the 21st IFAC World Congress_ , 2020, vol. 53, no. 2, pp. 2832-2837. Available: https://doi.org/10.1016/j.ifacol.2020.12.953.
  * 16 Q. Su, A. McAvoy, L. Wang and M. Nowak, “Evolutionary Dynamics with Game Transitions”, _Proceedings of the National Academy of Sciences_ , vol. 116, no. 51, pp. 25398-25404, 2019. Available: 10.1073/pnas.1908936116.
  * 17 X. Chen and A. Szolnoki, “Punishment and Inspection for Governing the Commons in a Feedback-Evolving Game", _PLOS Computational Biology_ , vol. 14, no. 7, p. e1006347, 2018. Available: 10.1371/journal.pcbi.1006347.
  * 18 C. Hauert, C. Saade and A. McAvoy, “Asymmetric Evolutionary Games with Environmental Feedback”, _Journal of Theoretical Biology_ , vol. 462, pp. 347-360, 2019. Available: 10.1016/j.jtbi.2018.11.019.
  * 19 Y. Shao, X. Wang and F. Fu, “Evolutionary Dynamics of Group Cooperation with Asymmetrical Environmental Feedback”, _EPL (Europhysics Letters)_ , vol. 126, no. 4, p. 40005, 2019. Available: 10.1209/0295-5075/126/40005.
  * 20 T. Morimoto, T. Kanazawa and T. Ushio, “Subsidy-Based Control of Heterogeneous Multiagent Systems Modeled by Replicator Dynamics”, _IEEE Transactions on Automatic Control_ , vol. 61, no. 10, pp. 3158-3163, 2016. Available: 10.1109/tac.2015.2504519.
  * 21 X. Wang, Z. Zheng and F. Fu, “Steering Eco-Evolutionary Game Dynamics with Manifold Control”, _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 476, no. 2233, p. 20190643, 2020. Available: 10.1098/rspa.2019.0643.
  * 22 K. Paarporn, C. Eksin, J.S. Weitz and Y. Wardi, “Optimal Control Policies for Evolutionary Dynamics with Environmental Feedback”, _Proceedings of the 57th IEEE Conference on Decision and Control (CDC)_ , 2018, pp. 1905-1910.
  * 23 G. Giordano, D. Bauso and F. Blanchini, “A Robust Saturated Strategy for $N$-Player Prisoner’s Dilemma”, _SIAM Journal on Control and Optimization_ , vol. 56, no. 5, pp. 3478-3498, 2018. Available: 10.1137/17m1136328.
  * 24 J. Batali, and P. Kitcher, “Evolution of Altruism in Optional and Compulsory Games”, _Journal of Theoretical Biology_ , vol. 175, no. 2, pp. 161-171, 1995. Available: 10.1006/jtbi.1995.0128.
  * 25 L. Stella and D. Bauso, “Bio-inspired Evolutionary Dynamics on Complex Networks under Uncertain Cross-inhibitory Signals”, _Automatica_ , vol. 100, pp. 61-66, 2019. Available: 10.1016/j.automatica.2018.11.005.
  * 26 N. Barberis, “Thirty Years of Prospect Theory in Economics: A Review and Assessment”, _Journal of Economic Perspectives_ , vol. 27, no. 1, pp. 173-196, 2013. Available: 10.1257/jep.27.1.173.
  * 27 D. Kahneman and A. Tversky, “Prospect Theory: An Analysis of Decision under Risk”, _Econometrica_ , vol. 47, no. 2, p. 263-291, 1979. Available: 10.2307/1914185.
  * 28 S. George, J. Sheynin, R. Gonzalez, I. Liberzon and J. Abelson, “Diminished Value Discrimination in Obsessive-Compulsive Disorder: A Prospect Theory Model of Decision-Making Under Risk”, _Frontiers in Psychiatry_ , vol. 10, 2019. Available: 10.3389/fpsyt.2019.00469.
  * 29 Y. Guan, A. M. Annaswamy, and H. E. Tseng. “Cumulative Prospect Theory Based Dynamic Pricing for Shared Mobility on Demand Services”, _Proceedings of the IEEE 58th Annual Conference on Decision and Control (CDC)_ , 2019.
  * 30 L. Stella and D. Bauso. “Bio-Inspired Evolutionary Game Dynamics in Symmetric and Asymmetric Models”, _IEEE Control Systems Letters_ , vol. 2, no. 3, pp. 405-410, 2018. Available: 10.1109/lcsys.2018.2838445.
  * 31 J. Hofbauer, “Deterministic Evolutionary Game Dynamics”, _Proceedings of Symposia in Applied Mathematics_ , vol. 69, pp. 61-79, 2011.
  * 32 M. Cardinot, C. O. Riordan and J. Griffith, “The Optional Prisoners Dilemma in a Spatial Environment: Coevolving Game Strategy and Link Weights”, _Proceedings of the 8th International Joint Conference on Computational Intelligence (IJCCI)_ , 2016, vol. 1, pp. 86-93. Available: 10.5220/0006053900860093.


# Amortized Neural Networks for Low-Latency Speech Recognition

###### Abstract

We introduce Amortized Neural Networks (AmNets), a compute cost- and latency-
aware network architecture particularly well-suited for sequence modeling
tasks. We apply AmNets to the Recurrent Neural Network Transducer (RNN-T) to
reduce compute cost and latency for an automatic speech recognition (ASR)
task. The AmNets RNN-T architecture enables the network to dynamically switch
between encoder branches on a frame-by-frame basis. Branches are constructed
with variable levels of compute cost and model capacity. Here, we achieve
variable compute for two well-known candidate techniques: one using sparse
pruning and the other using matrix factorization. Frame-by-frame switching is
determined by an arbitrator network that requires negligible compute overhead.
We present results using both architectures on LibriSpeech data and show that
our proposed architecture can reduce inference cost by up to 45% and latency
to nearly real-time without incurring a loss in accuracy. ††First two authors
contributed equally.

Index Terms: automatic speech recognition, latency reduction, compute cost
optimization, on-device

##  1 Introduction

Production ASR systems using end-to-end, neural architectures such as the
RNN-T [1, 2, 3], have enabled the deployment of generalized ASR systems in
compute constrained settings. While ASR systems are under an ever-increasing
demand to perform in a more natural, responsive manner in a multitude of
domains, there has also been an increased desire to enable this technology to
execute locally on lower power edge devices where compute resources are
limited. When using lower-performance hardware that is unable to perform ASR
in real time, a backlog of audio data builds and can cause significant user-
perceived latency (UPL) [4].

A simple approach for minimizing UPL caused by resource constraints is to
reduce ASR model size such that the compute cost of a model does not throttle
local processing performance. Many strategies have been proposed for neural
ASR model size reduction such as moving from the LSTM [5] to more efficient
layers including the CIFG-LSTM [6], LSTM-p [7], or Simple Recurrent Unit (SRU)
[8]. Each of these architectural changes use fewer parameters while
maintaining accuracy. Other non-architectural methods of model size reduction
include sparse pruning [9, 10], matrix factorization [11], weight quantization
[12, 13], and knowledge distillation [14]. These strategies becomes less
attractive when the computational constraints of ever-shrinking edge devices
mandate extreme levels of model compression while society’s demands for ASR
are constantly expanding and require increasing model capacity.

During real-time ASR decoding, utterances are processed as a sequence of
feature frames that can vary wildly in their inherent complexity. For example,
silent frames often exist during pauses between acoustically rich words and
phonemes as well as before and after the main content of an utterance. Despite
this variability, typical speech processing systems use a predetermined
network with a fixed compute cost and model capacity to evaluate each
individual frame; this is a missed opportunity to accelerate ASR by
dynamically varying the amount of compute power used throughout an utterance.
Prior studies have established the applicability of variable approaches in
other domains including natural language processing [15], character-level
prediction and language modeling [16, 17], music modeling [18], and vision
[19].

In this work, we present a new neural network architecture for speech
processing that amortizes the cost of processing complex frames over an entire
sequence of frames to reduce latency. We apply our AmNets approach to the
RNN-T architecture for ASR applications. Here, we bifurcate the RNN-T encoder
into two branched encoder networks in which the branches are compressed to
different degrees to achieve varying levels of compute cost. For each frame, a
recurrent arbitrator network is trained to decide which encoder branch to use.
We optimize this decision making process by introducing a novel amortized loss
function that trains the arbitrator to deliberately choose variable compute
levels depending on the instantaneous accuracy-latency trade off within a
given sequence. Prior methods utilizing adaptive compute have instead focused
on reducing _average_ cost, which acts as an indirect regularizer for real-
time speech processing latency. We show that this is sub-optimal in comparison
to our architecture, which is driven by a direct optimization of the real-time
audio processing backlog. We illustrate the efficacy and flexibility of the
proposed methods by reporting results from two separate branch architectures
using sparse pruning and matrix factorization.

The key contributions of this paper are the amortized recurrent neural network
architecture (Section 2), the application of AmNets to on-device, low latency
speech processing (Section 3), and the amortized latency loss function
(Section 4). Section 5 discusses experimental details and results.

##  2 Amortized Network Architecture

Figure 1: Unfolded AmRNN layer with $N$ branches. Each block represents the
execution of an AmRNN layer at a single time step, i.e., a single cell. Left
and center blocks illustrate branch combining during training; right block
shows run-time hard branch switching.

The fundamental component of an AmNet is the Amortized RNN (AmRNN) layer shown
in Figure 1. An AmRNN layer consists of an arbitrator network, RNN cell
branches $\mathcal{B}_{1},\ldots,\mathcal{B}_{N}$, and a branch state
combiner. The arbitrator is the central mechanism for gating between branches,
and it is implemented as an RNN so that it is able to maintain knowledge of
prior decisions. The arbitrator is much smaller in size than any of the RNN
cell branches such that it does not add noticeable compute burden at inference
time. The arbitrator takes an input vector of pre-processed audio feature
frames $x_{t}$ and computes output branch arbitration decisions
$k_{t}=[\pi_{1},\ldots,\pi_{N}]$. The arbitrator can optionally take prior
branch output states $h_{t-1}$ and prior arbitration decisions $d_{t-1}$.
$k_{t}$ is passed to a sampler, which is responsible for converting those
decisions to a harder decision $d_{t}\in[0,1]$ sampled from $k_{t}$. During
training, we use the Gumbel-Softmax reparameterization trick to compute
$d_{t}$

$\noindent
d_{t}(n)=\frac{\textrm{exp}((\mathrm{log}(\pi_{n})+g_{n})/\tau)}{\sum_{j=1}^{N}\textrm{exp}((\mathrm{log}(\pi_{j})+g_{j})/\tau)},$
(1)

which allows gradients to flow through non-differentiable stochastic nodes
during backpropagation [20]; in this case, a discrete categorical
distribution. In Equation 1, $g_{1},...,g_{N}$ are i.i.d. samples taken from
the Gumbel distribution [21]. Hyperparameter $\tau\in(0,\infty)$ controls the
discreteness of the one-hot sampling, where values of $\tau$ close to zero
approach perfectly hard sampling. Throughout training, $\tau$ can be gradually
decreased to coerce the network towards more discrete decisions. During run-
time, the sampler is switched to one-hot encoding such that only one branch is
computed for each frame.

AmRNN branches can be comprised of any recurrent cells such as LSTM, GRU, and
SRU and can be of mixed dimensionality to achieve a balance between model
capacity and compute efficiency across branches. Branches can also consist of
stacked RNNs to increase model depth. Here, we only discuss AmNets with equal
state sizes across all branches. However, mixed dimensionality can be achieved
by including dense layers between branches to linearly project branch states
into the dimensions of other branches [22].

During training and at every time step $t$, each branch receives prior AmRNN
states $\\{h_{t-1},c_{t-1}\\}$ and input features $x_{t}$ and produces output
states $\\{h_{t,n},c_{t,n}\\}$ for each branch. These states are passed to the
combiner, which uses sampler decisions $d_{t}$ to return a linear combination
of states from each branch as a single set of output states
$\\{h_{t},c_{t}\\}$ for each time step:

$\displaystyle\noindent\\{h_{t},c_{t}\\}=\sum_{n=1}^{N}d_{t-1}(n)\\{h_{t,n},c_{t,n}\\}$
(2)

During run-time, the decision sequence from the sampler is treated as a hard
switch such that only one branch receives an input and produces an output for
the entire cell.

The AmRNN outputs branch decisions and associated costs to compute an
amortized loss $\mathcal{L}_{\text{compute}}$ over an entire sequence.

##  3 Candidate RNN-T Architectures

The RNN-T is an end-to-end neural architecture that is well-suited for
sequence-to-sequence modeling applications and has shown recent success for
ASR modeling, especially on-device [23]. This architecture consists of three
main components: two multi-layer RNNs, the encoder $\mathcal{F}$ and decoder
$\mathcal{G}$, and a joint network $\mathcal{J}$. For ASR, $\mathcal{F}$ takes
a vector of pre-processed audio feature frames $x_{1:T}$ as input and returns
output $h_{t}^{enc}$. $\mathcal{G}$ takes a vector embedding of previous non-
blank labels $y_{1:M}$ and returns intermediate output $h_{m}^{dec}$. Outputs
from the encoder and decoder are passed to the joint network. Here,
$\mathcal{J}$ is implemented as an additive operation between encoder and
decoder outputs but can also include a joint feedforward network. The
conditional output distribution $P\left(\hat{y}_{m+1}|x_{1:t},y_{1:m}\right)$
is obtained by applying a softmax to the result of $\mathcal{J}$.

Typically, the encoder network is significantly larger than the decoder.
Additionally, during run-time, the decoder is cacheable over states with the
same partial sub-word sequence and lends itself to batched execution (with
multi-threading). We therefore use AmNets to reduce encoder cost since it is
the primary latency bottleneck at run-time. In all AmNets in this study, we
opt for two branches: one with a higher cost and capacity, and one with a
lower cost and capacity. We demonstrate the flexibility of AmNets by
showcasing two candidate implementations, each using a different compression
scheme to achieve varied compute cost between branches.

In the first candidate architecture, we compress branch weights using sparse
pruning. Sparsity-inducing regularization of neural networks has shown to
achieve lowered compute costs while maintaining accuracy [24, 25, 26, 27, 28].
In this work, we achieve various encoder sparsity levels by gradually sparse
pruning weights throughout training [9]. Given an initial sparsity ratio of 0,
we achieve a final sparsity ratio of $s_{f}$ over $\nu$ pruning steps starting
at step 0 with a pruning frequency of $\Delta m$:

$\displaystyle\noindent s_{t}=s_{f}-s_{f}\left(1-\frac{m}{\nu\Delta
m}\right)^{3},m\in\\{0,\Delta m,\ldots,\nu\Delta m\\}$ (3)

Sparsity is enforced by including binary sparse masks alongside network weight
matrices within each layer. For example, a single LSTM cell undergoing sparse
pruning would include corresponding masks for the cell, input gate, output
gate, and forget gate matrices of equivalent dimension. Throughout training,
sparsity levels defined by Equation 3 are applied to sparse masks in a point-
wise fashion based on the magnitudes of model weights. At time $t$, the mask
indices set to zero correspond to the indices of weights with magnitude
closest to zero based on the ratio defined by $s_{t}$. At the beginning of
each training batch, weight matrices are point-wise multiplied by their masks
to set pruned weights to zero on each forward pass. By using sparse masks,
masked model weights are not dropped; hence, this policy enables the model to
recover from early bad masking decisions. For AmNets, separate sparsity
trackers are used for each branch to achieve different levels of target
sparsity.

In the second candidate architecture, we compress branches using low rank
matrix factorization (MF) by decomposing the weight matrices of each RNN layer
into two lower dimensional weight matrices via the singular value
decomposition (SVD). These methods have been used for model compression in
many domains [29, 30, 31, 32] including deep learning and speech recognition
[33, 34, 35]. In this method, each weight matrix $\bm{W}^{w\times v}$ in an
AmRNN layer is decomposed into two weight matrices
$\bm{P_{1}}=[\,P_{1}^{1},\ldots,P_{1}^{w}]\,$ and
$\bm{P_{2}}=[\,P_{2}^{1},\ldots,P_{2}^{v}]\,$. To achieve varied compute costs
across branches, we choose ranks
$\textbf{r}=\\{r_{1},\ldots,r_{N}\\}\leq\min(w,v)$ and truncate each weight
matrix such that for the $n^{th}$ branch, only the first $r_{n}$ column
vectors in $\bm{P_{1}}$ and $\bm{P_{2}}$ are used, e.g., for input $x$ and
output $y$,

$\displaystyle\noindent
y=[\,P_{1}^{1},\ldots,P_{1}^{r_{n}}]\cdot\left([\,P_{2}^{1},\ldots,P_{2}^{r_{n}}]^{\top}x\right)$
(4)

When using $r<(w\cdot v)/(w+v)$, the number of multiplications is less than
when using the original matrix $\bm{W}$. Given that the weights for each
branch are acquired from the same decomposition, MF lends itself to weight
sharing between branches to prevent additional memory overhead.

Both sparse pruning and MF introduce the ability to take advantage of pre-
training. Using a pre-trained RNN-T, one can extract the encoder and apply
either compression method to obtain AmRNN branches and reconstruct the model
using the remainder of the RNN-T network unchanged. The model can then be
fine-tuned to allow for structural adaptation. Such a feature allows AmNets to
be applied to existing ASR models without having to retrain from scratch.

Table 1: Baseline RNN-T and amortized RNN-T results. Compression ratio column indicates the overall compression for any given architecture. When two numbers are present in any column, the first indicates the slow (S) branch and the second indicates the fast (F) branch for AmNets. Params column indicates total number of parameters in the encoder network including the arbitrator. Model | Params | Compr. Ratio | WER | Branch Ratios (S/F) | FLOPs/Frame | Latency (ms)
---|---|---|---|---|---|---
Baseline | 42.7M | 0% | 8.5 | - | 42.7M | (0.0%) | 6154
Sparse Baseline | 36.3M | 15% | 8.7 | - | 37.1M | (-13.1%) | 5231
Sparse AmNet $\mathcal{L}_{\text{avg}}$ | 45.3M | 15%, 80% | 8.6 | 57% / 43% | 27.7M | (-35.2%) | 681
Sparse AmNet $\mathcal{L}_{\text{avg}}$ Biased | 45.3M | 15%, 80% | 9.2 | 48% / 52% | 25.3M | (-40.8%) | 234
Sparse AmNet $\mathcal{L}_{\text{amr}}$ | 45.3M | 15%, 80% | 8.7 | 48% / 52% | 25.3M | (-40.8%) | 1.87
MF Baseline | 32.3M | 35% | 8.5 | - | 30.4M | (-28.8%) | 3410
MF AmNet $\mathcal{L}_{\text{avg}}$ | 33.0M | 35%, 60% | 8.6 | 38% / 62% | 24.6M | (-42.3%) | 433
MF AmNet $\mathcal{L}_{\text{avg}}$ Biased | 33.0M | 35%, 60% | 9.1 | 23% / 77% | 23.2M | (-45.6%) | 29.4
MF AmNet $\mathcal{L}_{\text{amr}}$ | 33.0M | 35%, 60% | 8.6 | 23% / 77% | 23.2M | (-45.6%) | 9.00

##  4 Amortized Latency Loss

Figure 2: Amortized latency loss over two cost sequences. Both use the same
number of fast (F) and slow (S) decisions and result in the same average cost,
yet the (a) sequence often waits for new frames to arrive while (b) yields a
lower latency. The amortized latency loss accounts for this difference.

We now detail two implementations of $\mathcal{L}_{\text{compute}}$, which
regularizes the decisions of the arbitrator. For simplicity, we denote $q_{t}$
as the compute cost for the frame at timestep $t$ (determined by the
arbitrator’s decisions) in floating point operations (FLOPs). We consequently
define the loss as a function over the sequence costs,
$\mathcal{L}_{\text{compute}}\left(q_{1},\dots,q_{T}\right)$. A straight
forward approach for accounting the loss is to consider the total amount of
computation or a length normalized mean amount of computation used across the
sequence.

$\displaystyle\noindent\mathcal{L}_{\text{avg}}=\frac{1}{T}\sum_{i=1}^{T}{q_{i}}$
(5)

This type of regularization is common for other variable compute approaches
[15, 16, 17]. However, in interactive settings where response latency is
critical, such as voice assistants, the average compute loss serves as more of
a proxy regularizer than the true objective of minimizing response latency. To
understand why a decrease in average compute cost of an audio sequence does
not necessarily translate into a latency reduction, consider the two cost
sequences in Figure 2: (a) mostly fast (F), inexpensive operations followed by
a series of slow (S), expensive ones, and (b) the reverse, expensive followed
by inexpensive. Note that the F operations process frames faster than real-
time. While both sequences result in the same average cost, the first sequence
has a greater backlog of frames at the end of the utterance caused by
underutilized time when the model waited for the next frame. Ultimately, this
results in higher latency.

To address this distinction, we introduce a new measure, termed _amortized
latency loss_ , which accounts for the processing performance of the hardware
executing the computation, the audio feature frame rate, and the decision
sequence of the arbitrator. We denote $\mu$ as the effective FLOP rate (i.e.
FLOPs/s) representing the device processor speed and $\rho$ as the feature
frame rate in frames per second.

Defining $\ell_{t}$ to be the buffered FLOP backlog (i.e. _lag_ accumulated at
the $t$-th time step), we express the amortized latency loss
$\mathcal{L}_{\text{amr}}$ as the response delay in seconds

$\displaystyle\noindent\mathcal{L}_{\text{amr}}=\ell_{T}/\mu$ (6)

with $\ell_{t}$ expressed recursively as

$\displaystyle\ell_{t}=\left\\{\begin{array}[]{ll}0&t=0\\\
\max\left\\{\ell_{t-1}+q_{t}-\mu/\rho,0\right\\}&t>0\end{array}\right.$ (9)

The loss is convex over $\left(q_{1},\dots,q_{T}\right)$ and is efficiently
computable (along with its derivatives) in linear time using dynamic
programming. The amortized latency loss encourages the arbitrator to schedule
in a way that achieves high processor utilization while maintaining a short
response latency. In this study, $\mathcal{L}_{\text{amr}}$ is regularized and
combined with RNN-T sequence loss to achieve reduced latency without
sacrificing accuracy.

##  5 Experimental Results

We investigate model accuracy using the LibriSpeech corpus [36]. This data set
contains 960 hours of training data, and we evaluate using the “clean” test
data set. All data is pre-processed using a 64-dimensional log-filterbank
energy feature extractor. These feature frames are downsampled by three and
stacked with a stride size of two to produce 30-millisecond frames. All models
are trained to convergence with a warm-hold-decay learning rate schedule and
an Adam optimizer [37]. AmNets models are first trained with
$\mathcal{L}_{\text{avg}}$ loss, which is followed by a short fine-tuning
stage using $\mathcal{L}_{\text{amr}}$ loss.

Our baseline model is a lightweight, unidirectional LSTM-based RNN-T with five
1024-unit LSTM layers in the encoder, two 1024-unit LSTM layers in the
decoder, and an additive joint network with no trainable parameters. We use a
vocabulary of 4097 word pieces. The baseline model includes 63.5M parameters
with 42.7M belonging to the encoder. We use standard RNN-T beam search
decoding with a width of 16 to obtain word error rate (WER) measurements as
our accuracy metric.111We choose our baseline architecture because it has been
successfully tested in on-device, low-latency streaming settings while
delivering high accuracy on our internal data. Note that more advanced methods
such as language model rescoring [38], bidirectional and frequency LSTMs [39,
40], and attention-based transducers [41, 42, 43] exhibit superior WER on the
LibriSpeech data set.

In addition to the overall baseline model, we include baseline models for
sparsity and MF that use compression ratios of 15% and 35%, respectively.
These ratios were chosen by continuously increasing the compression for both
methods until a development set WER degradation was observed; all baselines
achieved 8.5% WER on LibriSpeech development clean data. All AmNet models
build upon the baseline model by substituting the encoder LSTM layers with a
two-branched AmRNN consisting of five stacked LSTM cells per branch and 1024
units per cell. The Sparse AmNets use slow and fast branches with 15% and 80%
sparsity, respectively, and the MF AmNets use ranks that result in 35% and 60%
compression, respectively. MF AmNets are seeded with weights acquired from the
baseline MF model and then trained for a small number of steps until
convergence. We also attempted to use pre-trained weights for the sparse
AmNets but found that training sparse models from scratch yields better
results. Sparse model branches are pruned with $\Delta m=100$ steps for the
first 30% of training using random branch sampling with a 50/50 slow/fast
ratio. After pruning, Gumbel softmax sampling is activated and used for the
remainder of training with $\tau$ linearly annealed from 1.0 to 0.5.

All arbitrator networks consist of two 128-unit LSTM layers and a single two-
unit output projection layer. The arbitrator is pre-trained on a task to
classify frames as high or low entropy. Binary targets for the arbitrator pre-
training are generated by thresholding a baseline model’s output distribution
entropy with a threshold chosen between 1.5-2.5 achieving similar results,
yielding 40/60-20/80% initial branch ratios (slow/fast). For all AmNets, no
more than 2.5% of the trainable encoder parameters are attributed to the
arbitrator network.

For all models, we report WER, average compute cost, and latency. Compute cost
is reported in terms of encoder FLOPs per frame including arbitrator cost.
Latency is reported in milliseconds and is the average across all test
utterances. To estimate latency, we use a single-threaded compute performance
of 650 million FLOPs per second, chosen for two reasons. First, this level of
performance is less than half the requirement for a device to maintain real-
time (i.e. zero latency) processing when using the baseline model. Second,
this level is commensurate with the compute power available when using off-
the-shelf embedded systems in the same class as Echo devices. Note that when
designing a model for on-device inference, one would know the compute
performance _a priori_ and would instead design encoder branch architectures
based on device capabilities.

The accuracy, compute cost, and latency for each model is shown in Table 1.
During the average compute cost loss ($\mathcal{L}_{\text{avg}}$) learning
phase, we use a regularization factor to strike a balance between RNN-T loss
and compute cost loss. This balance is achieved by finding the point at which
the AmNets branch ratios use the fast branch to the greatest extent without
incurring significant accuracy degradation. For both sparsity and MF, the
AmNets trained purely with $\mathcal{L}_{\text{avg}}$ significantly outperform
their baseline models with respect to compute cost and latency without
accuracy degradation. In this case, MF outperformed sparsity by both measures
due to the higher fast branch ratio for MF. Note that all sparse AmNets
outperformed the sparse baseline with respect to both FLOPS per frame and
latency despite having significantly more parameters.

During the amortized latency loss ($\mathcal{L}_{\text{amr}}$) fine-tuning
stage, we found that the branch ratios shifted towards the fast branch–from
43% to 52% for sparsity and 62% to 77% for MF. To illustrate that the latency
improvements realized by tuning with $\mathcal{L}_{\text{amr}}$ are not purely
the result of this ratio shift, we increase the $\mathcal{L}_{\text{avg}}$
regularization to achieve _biased_ branch ratios that match those observed
when using $\mathcal{L}_{\text{amr}}$. While the biased
$\mathcal{L}_{\text{avg}}$ models further reduce compute cost and latency,
they also incur noticeable accuracy reductions caused by forcing the
arbitrator to an unbalanced accuracy-latency trade-off. In contrast, the
$\mathcal{L}_{\text{amr}}$ models achieve nearly real-time latency with a
$>$99.9% reduction over all baselines while achieving baseline-level accuracy.

It is important to highlight that the biased MF $\mathcal{L}_{\text{avg}}$
AmNet reduced overall latency by a large margin over the biased sparse AmNet
model thus yielding a less significant latency improvement than sparsity when
moving to $\mathcal{L}_{\text{amr}}$ loss. This phenomenon emphasizes that,
while reducing the average compute cost _may_ result in a similarly reduced
and nearly real-time latency, this is not necessarily guaranteed. Depending on
the specific architecture, compression ratios, and device performance, the
unbounded $\mathcal{L}_{\text{avg}}$-trained arbitrator decision sequences
only promise an overall average compute cost reduction. The
$\mathcal{L}_{\text{amr}}$ loss tuning, however, delivers such a latency
guarantee for both candidate architectures by enforcing more deliberate branch
decisions and truly amortizing costs based on an awareness of the real-time
audio backlog.

##  6 Conclusion

We introduced Amortized Networks accompanied by amortized latency loss and
applied them to RNN-T for ASR. By dynamically switching between variable-
compute encoder branches on a frame-by-frame basis with amortized latency
loss, AmNets are able to yield significant reductions in compute cost and
latency. AmNets can be applied to a variety of scenarios and are especially
well-suited for sequence-to-sequence modeling.

## References

  * [1] A. Graves, A. R. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” _Proc. ICASSP_ , 2013.
  * [2] K. Rao, H. Sak, and R. Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer,” in _Proc. ASRU Workshop_ , Jan. 2018.
  * [3] A. Kannan, A. Datta, and T. N. S. et al., “Large-scale multilingual speech recognition with a streaming end-to-end model,” in _Proc. Interspeech_ , Sep. 2019.
  * [4] T. N. Sainath, Y. He, and B. L. et al., “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,” in _Proc. ICASSP_ , 2020.
  * [5] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, 1997.
  * [6] K. Greff, R. K. Srivastava, J. Koutnik, B. R. Steunebrink, and J. Schmidhuber, “LSTM: A search space odyssey,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 28, 2017.
  * [7] Y. K. Jia, Z. Wu, Y. Xu, D. Ke, and K. Su, “Long short-term memory projection recurrent neural network architectures for piano’s continuous note recognition,” _Journal of Robotics_ , 2017.
  * [8] T. Lei, Y. Zhang, S. I. Wang, H. Dai, and Y. Artzi, “Simple recurrent units for highly parallelizable recurrence,” in _Proc. EMNLP_ , 2018.
  * [9] M. H. Zhu and S. Gupta, “To prune, or not to prune: Exploring the efficacy of pruning for model compression,” in _6th International Conference on Learning Representations, ICLR - Workshop Track Proceedings_ , 2018.
  * [10] Y. Shangguan, J. Li, L. Qiao, R. Alvarez, and I. McGraw, “Optimizing speech recognition for the edge,” _arXiv preprint abs/1909.12408_ , 2019.
  * [11] R. Pang, T. N. Sainath, and R. P. et al., “Compression of end-to-end models,” in _Proc., Interspeech_ , Sep. 2018.
  * [12] R. Alvarez, R. Prabhavalkar, and A. Bakhtin, “On the efficient representation and execution of deep acoustic models,” in _Proc. Interspeech_ , Sep. 2016\.
  * [13] H. D. Nguyen, A. Alexandridis, and A. Mouchtaris, “Quantization aware training with absolute-cosine regularization for automatic speech recognition,” in _Proc. Interspeech_ , Oct. 2020.
  * [14] S. Panchapagesan, D. S. Park, C. C. Chiu, Y. Shangguan, Q. Liang, and A. Gruenstein, “Efficient knowledge distillation for RNN-transducer models,” _arXiv preprint abs/2011.06110_ , 2020.
  * [15] A. Tyagi, V. Sharma, R. Gupta, L. Samson, N. Zhuang, Z. Wang, and B. Campbell, “Fast intent classification for spoken language understanding systems,” in _Proc. ICASSP_ , May 2020.
  * [16] E. Bengio, P.-L. Bacon, J. Pineau, and D. Precup, “Conditional computation in neural networks for faster models,” _arXiv preprint abs/1511.06297_ , Nov. 2015.
  * [17] A. Graves, “Adaptive computation time for recurrent neural networks,” _arXiv preprint abs/1603.08983_ , 2016.
  * [18] Y. Jernite, E. Grave, A. Joulin, and T. Mikolov, “Variable computation in recurrent neural networks,” in _International Conference on Learning Representations, ICLR_ , Nov. 2017.
  * [19] T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama, “Adaptive neural networks for efficient inference,” in _34th International Conference on Machine Learning, ICML_ , vol. 2, 2017.
  * [20] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in _5th International Conference on Learning Representations, ICLR - Conference Track Proceedings_ , 2017.
  * [21] E. J. Gumbel, “Bivariate exponential distributions,” _Journal of the American Statistical Association_ , vol. 55, 1960.
  * [22] J. Macoskey, G. Strimel, and A. Rastrow, “Bifocal neural ASR: Exploiting keyword spotting for inference optimization,” in _Proc. ICASSP_ , Jun. 2021\.
  * [23] Y. He, T. N. Sainath, and R. P. et al., “Streaming end-to-end speech recognition for mobile devices,” in _Proc. ICASSP_ , May 2019.
  * [24] Z. Wu, D. Zhao, Q. Liang, J. Yu, A. Gulati, and R. Pang, “Dynamic sparsity neural networks for automatic speech recognition,” _arXiv preprint abs/2005.10627_ , 2020.
  * [25] T. Hastie, R. Tibshirani, and M. Wainwright, _Statistical learning with sparsity: The lasso and generalizations_. CRC Press, 2015.
  * [26] X. Zhou, Z. Du, and S. Z. et al., “Addressing sparsity in deep neural networks,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 38, 2019.
  * [27] T. Gale, E. Elsen, and S. Hooker, “The state of sparsity in deep neural networks,” _arXiv preprint abs/1902.09574_ , 2019.
  * [28] M. Hebiri and J. Lederer, “Layer sparsity in neural networks,” _arXiv preprint abs/2006.15604_ , 2020.
  * [29] H. C. Andrews and C. L. Patterson, “Singular value decomposition (SVD) image coding,” _IEEE Transactions on Communications_ , vol. 24, 1976.
  * [30] M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” _IEEE Transactions on Signal Processing_ , vol. 54, 2006.
  * [31] A. M. Rufai, G. Anbarjafari, and H. Demirel, “Lossy image compression using singular value decomposition and wavelet difference reduction,” _Digital Signal Processing: A Review Journal_ , vol. 24, 2014.
  * [32] D. F. McGivney, E. Pierre, D. Ma, Y. Jiang, H. Saybasili, V. Gulani, and M. A. Griswold, “SVD compression for magnetic resonance fingerprinting in the time domain,” _IEEE Transactions on Medical Imaging_ , vol. 33, 2014.
  * [33] R. Prabhavalkar, O. Alsharif, A. Bruguier, and L. McGraw, “On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition,” in _Proc. ICASSP_ , May 2016.
  * [34] H. Kim, M. U. K. Khan, and C. M. Kyung, “Efficient neural network compression,” in _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , Jun. 2019.
  * [35] S. Swaminathan, D. Garg, R. Kannan, and F. Andres, “Sparse low rank factorization for deep neural network compression,” _Neurocomputing_ , vol. 398, 2020.
  * [36] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _Proc. ICASSP_ , 2015.
  * [37] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR - Conference Track Proceedings_ , 2015.
  * [38] J. Li, R. Zhao, and Z. M. et al., “Developing RNN-T models surpassing high-performance hybrid models with customization capability,” in _Proc. Interspeech_ , Oct. 2020.
  * [39] A. Graves, N. Jaitly, and A. R. Mohamed, “Hybrid speech recognition with deep bidirectional LSTM,” in _Proc. ASRU_ , 2013.
  * [40] J. Li, A. Mohamed, G. Zweig, and Y. Gong, “Exploring multidimensional LSTMs for large vocabulary ASR,” in _Proc. ICASSP_ , May 2016.
  * [41] J. Li, Y. Wu, Y. Gaur, C. Wang, R. Zhao, and S. Liu, “On the comparison of popular end-to-end models for large scale speech recognition,” in _Proc. Interspeech_ , Oct. 2020.
  * [42] Y. Zhao, C. Ni, C. C. Leung, S. Joty, E. S. Chng, and B. Ma, “Cross attention with monotonic alignment for speech transformer,” in _Proc. Interspeech_ , Oct. 2020.
  * [43] Z. Tian, J. Yi, J. Tao, Y. Bai, and Z. Wen, “Self-attention transducers for end-to-end speech recognition,” in _Proc. Interspeech_ , Sep. 2019.