# Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning
with Latent Shielding

Chloe He,1, 2, 3 Borja G. León, 1 Francesco Belardinelli 1

###### Abstract

The growing trend of fledgling reinforcement learning systems making their way
into real-world applications has been accompanied by growing concerns for
their safety and robustness. In recent years, a variety of approaches have
been put forward to address the challenges of safety-aware reinforcement
learning; however, these methods often either require a handcrafted model of
the environment to be provided beforehand, or that the environment is
relatively simple and low-dimensional. We present a novel approach to safety-
aware deep reinforcement learning in high-dimensional environments called
latent shielding. Latent shielding leverages internal representations of the
environment learnt by model-based agents to “imagine” future trajectories and
avoid those deemed unsafe. We experimentally demonstrate that this approach
leads to improved adherence to formally-defined safety specifications.

## Introduction

The steady trickle of reinforcement learning (RL) systems making their way out
of the lab and into the real world has cast a spotlight on the safety and
robustness of RL agents. The motivation behind this should be relatively easy
to grasp: when training an agent in real-world settings, it is desirable that
some states are never reached as they could, for instance, cause permanent
damage to the hardware the agent is controlling. We can thus informally define
the notion of safety-aware RL in terms of the classical RL setup with the
added requirement that the number of unsafe states visited be minimised. Under
this definition, however, it has been found that many state-of-the-art RL
algorithms unnecessarily enter unsafe states despite safe alternatives being
available and there being a positive correlation between avoiding such states
and reward (Giacobbe et al. 2021).

The field of safety-aware RL encompasses a multitude of approaches ranging
from constrained policy optimisation (Chow et al. 2017; Achiam et al. 2017;
Yang et al. 2020) to safety critics (Srinivasan et al. 2020; Bharadhwaj et al.
2021; Thananjeyan et al. 2021) to meta-learning (Turchetta et al. 2020). In
this work, we focus on a particular family of approaches known as shielding
(Alshiekh et al. 2018; Anderson et al. 2020; Giacobbe et al. 2021; ElSayed-Aly
et al. 2021; Pranger et al. 2021). Central to shielding is the notion of a
shield, a filter that checks actions proposed by the agent’s existing policy
with reference to a model of the environment’s dynamics and some formal safety
specification. The shield overrides actions that may lead to an unsafe state
using some other safe (but by no means optimal) policy. A key advantage of
many shielding approaches is that the resulting shielded policies are formally
verifiable; however, a shortcoming is that they require a model of
environmental dynamics - typically handcrafted - to be provided in advance.
Providing such a model may prove difficult for complex real-world
environments, with inaccuracies and human biases creeping into handcrafted
models.

In this work, we propose a safe RL agent that makes uses of latent shielding,
an approach to shielding in environments where a formally-specified dynamics
model is not available in advance. At an intuitive level, the agent uses a
data-driven approach to learn its own latent world model (a component of which
is a dynamics model) which is then leveraged by a shield. The shield then uses
the agent’s model to “imagine” trajectories arising from different actions,
forcing the agent to avoid those it foresees leading to unsafe states. In
addition, the agent can be trained within its own latent world model thus
reducing the number of safety violations seen during training.

##### Contributions

The main contribution of this work is a framework for shielding agents in
complex, stochastic and high-dimensional environments without knowledge of
environmental dynamics a priori. We further introduce a new method to aid
exploration when training shielded agents. Though our framework loses the
formal safety guarantees associated with traditional symbolic shielding
approaches, our experiments illustrate that latent shielding reduces unsafe
behaviour during training and achieves testing performance comparable to
previous symbolic approaches.

## Preliminaries

In this section, we cover some relevant background topics. We begin by
introducing our problem setup for safety-aware RL and give an overview of the
specification language used in this work. This is followed by an outline of
the latent world model we make use of in this work as well as a discussion on
shielding.

### Problem Setup

We consider an agent interacting with an environment $\mathcal{E}$ modelled as
a partially observable Markov decision process (POMDP) with states
$s\in\mathcal{S}_{\mathcal{E}}$, observations
$o_{t}\in\mathcal{O}_{\mathcal{E}}$, agent-generated actions
$a_{t}\in\mathcal{A}_{\mathcal{E}}$ and scalar rewards $r_{t}\in\mathbb{R}$
over discrete time steps $t\in[0,1,...,T-1]$. We assume the environment has
been augmented with a labelling function
$L_{\mathcal{E}}^{\phi}:\mathcal{S}_{\mathcal{E}}\rightarrow\\{safe,unsafe\\}$
that, at each time step, informs us whether a violation has occurred with
respect to some formal safety specification $\phi$. For the avoidance of
doubt, we define a violation to have occurred whenever $\phi$ does not hold.
This is a weaker assumption than previous works in shielding (which assume
access to an abstraction of the environment) and can be thought as a secondary
safety-focused reward function with a binary output. Intuitively, the goal of
the agent is to learn a policy $\pi$ that maximises its expected cumulative
reward while minimising the number of violations of $\phi$.

### Syntactically Co-Safe Linear Temporal Logic

In this work, we use syntactically co-safe Linear Temporal Logic (scLTL)
(Kupferman and Vardi 2001) as our specification language. Valid scLTL formulae
over some set of atomic propositions $AP$ can be constructed according to the
following grammar:

$\phi::=true\mid d\mid\neg
d\mid\phi\lor\phi\mid\phi\land\phi\mid\bigcirc\phi\mid\phi\cup\phi\mid\diamond\phi$
(1)

where $d\in AP$, $\neg$ (negation), $\lor$ (disjunction), $\land$
(conjunction) are the familiar operators from propositional logic, and
$\bigcirc$ (next), $\cup$ (until) and $\diamond$ (eventually) are temporal
operators.

The main rationale behind our choice of specification language is the fact
that we can efficiently monitor a system’s adherence to an scLTL specification
using a technique known as progression (Bacchus and Kabanza 2000). This is
highly advantageous as it means that, given an scLTL specification,
$L_{\mathcal{\mathcal{E}}}^{\phi}$ can be straightforwardly synthesised.

### Recurrent State-Space Models

We refer to the predictive model of an environment maintained by a model-based
agent as its world model. World models can be learnt from experience and be
used both as a substitute for the environment during training (Ha and
Schmidhuber 2018; Hafner et al. 2021) and for planning at run-time (Hafner et
al. 2019b). Though many realisations of the notion of a world model exist, the
world model used in this work is based on the recurrent state-space model
(RSSM) proposed by (Hafner et al. 2019b).

An RSSM is composed of three key components: a latent dynamics model, a reward
model, and an observation model. These components act on compact states formed
from the concatenation of a deterministic latent state $h_{t}$ and stochastic
latent state $z_{t}$.

##### Latent Dynamics Model

The latent dynamics model is made up of a number of smaller models. First, the
recurrent model $h_{t}=f(h_{t-1},z_{t-1},a_{t-1})$ is used to compute the
deterministic latent state based on the previous compact state and action.
From $h_{t}$ and the current observation $o_{t}$, a distribution
$q(z_{t}|h_{t},o_{t})$ over posterior stochastic latent states $z_{t}$ is
computed by the representation model. At the same time, a distribution
$p(\hat{z}_{t}|h_{t})$ over prior stochastic latent states $\hat{z}_{t}$ is
computed by the transition model, based only on $h_{t}$. During training, the
transition model attempts to minimise the Kullback Leibler (KL) divergence
between the prior and posterior stochastic latent state distributions. In
doing this, the RSSM learns to predict future latent states (using the
recurrent and transition models) without access to future observations.

##### Observation Model

The observation model computes the distribution $p(\hat{o}_{t}|h_{t},z_{t})$
over observations $\hat{o}_{t}$ for a particular state. Though not strictly
needed, the observation model can prove useful for visualising predicted
future states and providing a richer training signal.

##### Reward Model

The reward model computes the distribution $p(\hat{r}_{t}|h_{t},z_{t})$ over
rewards $\hat{r}_{t}$ for a particular state.

In practice, the distributions $p$ and $q$ are implemented with neural
networks $p_{\theta}$ and $q_{\theta}$ respectively, parameterised by some set
of parameters $\theta$. These latent dynamics models define a fully-observable
Markov decision process (MDP) as the latent states in the agent’s own internal
model can always be observed by the agent (Hafner et al. 2019a). We denote the
state space of this MDP (comprised of compact latent states) as
$\mathcal{S}_{\mathcal{I}}$.

### Shielding

The classical formulation of shielding in RL is given by (Alshiekh et al.
2018). It assumes access to two ingredients: an LTL safety specification and
abstraction (a MDP model of the environment that captures the aspects of the
environment relevant for planning ahead with respect to the safety
specification). These ingredients are used to construct a formally verifiable
reactive system that monitors the agent’s actions, overriding those which lead
to violation states.

Proposed by (Giacobbe et al. 2021), bounded prescience shielding (BPS) avoids
the need for hand-crafted abstractions by exploiting the fact that some agents
are trained in computer simulations. The shield operates by leveraging access
to the program underlying the simulation to look ahead into future states
within some finite horizon. Using BPS over classical shielding does, however,
come with a few disadvantages. Firstly, it requires access to the simulation
at run-time which may prove difficult to provide (especially in cases where
running the simulation is computationally expensive). Moreover, an agent using
BPS, even when starting from a safe state, can find itself entering unsafe
states in cases where the number of steps between a violation being caused by
an action and the violation state itself exceeds the shield’s look-ahead
horizon. This is not the case for classical shielding which resembles BPS with
an infinite horizon.

### Bounded Safety

The notion of safety used by BPS is defined over MDPs. For an arbitrary MDP
with states $\mathcal{S}$ and actions $\mathcal{A}$, a bounded trajectory
$\rho$ of length $H$ is a sequence of states and actions
$s_{0}\xrightarrow[]{a_{0}}s_{1}\xrightarrow[]{a_{1}}\ldots\xrightarrow[]{a_{n-1}}s_{n}$
comprised of no more than $H$ states and with the final state $s_{n}$ either
being a terminal state or $n=H-1$. We further denote the set of all finite
trajectories starting from some arbitrary state $s\in\mathcal{S}$ by
$\varrho(s)$ and the set of all bounded trajectories of length $H$ that start
from $s$ by $\varrho_{H}(s)$.

We say a bounded trajectory $\rho$ of length $H$ satisfies $H$-bounded safety
with respect to safety specification $\phi$, written $S_{H}(\rho,\phi)$, if
and only if for all $s_{i}\in\rho$, $L_{\mathcal{E}}^{\phi}(s_{i})=safe$.
Moreover, we can extend the notion of $H$-bounded safety over the set of
policies: a policy $\pi$ is $H$-bounded safe with respect to $\phi$, denoted
as $S_{H}(\pi,\phi)$, if and only if for all s $\in\mathcal{S}$,

  * •

either there exists some $\rho\in\varrho_{H}(s)$ such that $S_{H}(\rho,\phi)$
and $\pi(s_{0})=a_{0}$;

  * •

or for all $\rho\in\varrho_{H}(s)$, $\neg S_{H}(\rho,\phi).$

In other words, the policy will choose a safe trajectory as long as one
exists. Finally, we formally define a violation of $\phi$ to be inevitable in
state $s_{0}\in\mathcal{S}$ if and only if for all $\rho\in\varrho(s_{0})$,
$\neg S_{H}(\rho,\phi)$.

## Shielded Dreams

We introduce the notion of latent shielding, a novel class of shielding
approaches that replace abstractions used by shields with learned latent
dynamics models thus allowing the enforcement of formal safety specifications
while avoiding the need for an explicitly-defined abstraction of the
environment. We further introduce the first such approach, approximate bounded
prescience shielding, a framework for latent shielding that leverages latent
world models learnt by model-based deep RL (DRL) agents. In this work, our
model-based agent of choice is Dreamer111In practice, any model-based agent
with a latent dynamics model can be used. (Hafner et al. 2021), which we
modify to incorporate shielding into its data collection, training and
deployment phases.

### Safety RSSM

We augment the standard RSSM with a labelling function
$L_{\mathcal{I}}^{\phi}:\mathcal{S}_{\mathcal{I}}\rightarrow\\{safe,unsafe\\}$
which maps latent states $s_{t}\in\mathcal{S}_{\mathcal{I}}$ to whether they
correspond to states in violation of $\phi$. As with the other models,
$L_{\mathcal{I}}^{\phi}$ is implemented with a neural network with a
categorical output $l_{\theta}$ also parameterised by $\theta$. This yields an
enhanced RSSM (illustrated in Figure 1) which we will refer to as a safety
RSSM (SRSSM).

Figure 1: Safety RSSM. In this example, the model observes the environment for
two time steps and predicts the subsequent two time steps. $a_{t}$ denotes the
action taken at step $t$, $o_{t}$ the observation, $h_{t}$ the deterministic
component of the latent state, $z_{t}$ the posterior stochastic component,
$\hat{z}_{t}$ the prior stochastic component, $\hat{\lambda}_{t}$ the
violation prediction (our novel contribution, highlighted in yellow),
$\hat{o}_{t}$ the predicted observation and $\hat{r}_{t}$ the predicted
reward. Circles represent stochastic variables whereas squares represent
deterministic variables. An arrow from one shape to another indicates that the
source is used in the calculation of the destination.

We train $l_{\theta}$ along with the other components of the SRSSM with the
objective

$\min_{\theta}\mathcal{L}_{model}=\mathcal{L}_{observation}+\mathcal{L}_{reward}+\mathcal{L}_{KL}+\mathcal{L}_{violation}$
(2)

where the first three terms are as described in (Hafner et al. 2019a, 2021).
$\mathcal{L}_{violation}$ is a new term that we introduce that acts as a
weighted binary cross-entropy loss over predictions by $l_{\theta}$.

### Approximate Bounded Prescience Shielding

We now integrate the SRSSM as part of a latent shielding approach which we
shall refer to as approximate bounded prescience shielding (ABPS). Though ABPS
is inspired by BPS, it differs in two key aspects: (1) we approximate the
labelling function $L_{\mathcal{E}}^{\phi}$ and environmental dynamics using
an SRSSM; and (2) we sample a fixed number of potential future trajectories as
opposed to exhaustively exploring all possibilities.

Thus, our approach can be thought of as an approximation of some “ideal”
bounded prescience shield that uses the true environmental dynamics and
labelling function. The advantage of the first difference should be obvious:
it enables the shield to learn its own abstraction, removing the need for
hand-crafting or access to a digital environment’s underlying program. Why the
second difference is advantageous may be slightly less obvious - it’s a
heuristic that allows us to increase the horizon $H$. By directing the
sampling of trajectories in accordance with states and actions the policy is
biased towards (as opposed to uniformly), it may be possible to achieve
comparable performance to a standard BPS in foreseeing unsafe states. The
intuition behind this is that by not sampling trajectories the agent is
unlikely to take, more of our computational budget can be dedicated into
ensuring that the agent’s most likely trajectories are safe. In other words,
we will not spend time planning to correct actions that the agent is unlikely
to take.

More formally, our shielded policy $\pi^{*}$ can be written:

$\pi^{*}(s_{t})=\begin{cases}\pi(s_{t}),\text{ if
}P(l_{\theta}(s_{t+1})=1|\pi(s_{t}))<\epsilon\\\ \pi_{alt}(s_{t}),\text{
otherwise}\\\ \end{cases}$ (3)

where $\pi(s_{t})$ is the agent’s policy without shielding;
$s_{t}\in\mathcal{S}_{\mathcal{I}}$ is some compact latent state; and
$\pi_{alt}$ is an alternative policy that ensures that, if a violation isn’t
already inevitable, the agent avoids a predicted unsafe state. Though more
complex candidates for $\pi_{alt}$ (such as selecting the unshielded policy’s
highest-ranked safe action (Alshiekh et al. 2018)) do exist, the
implementation of $\pi_{alt}$ in this work simply considers all the other
actions until a safe trajectory is found. In the event that no safe action is
found, the agent takes a random action.

Since the SRSSM models stochasticity, we can sample multiple futures arising
from an action being taken and derive probabilistic estimates of whether an
action will lead to a violation. In this way, we can estimate the safety of an
action taken in a given state by checking whether the probability of a
violation occurring (inferred by sampling) exceeds a fixed threshold
$\epsilon$ (see Algorithm 1). Moreover, the sampling process can, in practice,
be augmented to sample a wider range of trajectories (less likely to be taken
by the agent) by adding a small amount of noise
$\eta\sim\mathcal{N}(0,\sigma^{2})$ to actions suggested by the policy during
sampling.

Input: Current compact latent state $(h,z)$, unshielded action $a$, horizon
$H$, number of trajectories to sample $N$, random noise $\eta$, alternative
policy $\pi_{alt}$ and threshold $\epsilon$.

Output: A tuple containing an action and whether or not the shield had to
interfere.

1 Initialise array of zeros $\Lambda$ with length $N$.

2for  _n = 1..N_ do

/* Sample a trajectory and check if it is unsafe. */

3 Imagine trajectory
$\rho=(h,z)\xrightarrow[]{a}(\hat{h}_{1},\hat{z}_{1})\xrightarrow[]{a_{1}}...\xrightarrow[]{a_{H-1}}(\hat{h}_{H},\hat{z}_{H})$
where $a_{i}=\pi(\hat{h_{i}},\hat{z_{i}})+\eta$.

4 if $\exists i\in[1,2,...,H]$ such that $l_{\theta}(h_{i},z_{i})=unsafe$ then
$\Lambda[n]:=1$.

5 end for

6

/* Interfere if probability of violation exceeds $\epsilon$. */

7

8$interfered$ := $false$.

9if  _$\frac{1}{N}\sum_{\lambda\in\Lambda}{\lambda} >\epsilon$_ then

10 $a,interfered:=\pi_{alt}(h_{i},z_{i}),\;true$.

11 end if

12

13return _$\langle a,$ interfered$\rangle$._

Algorithm 1 Approximate Bounded Prescience Shielding in latent space.

### Training Regime

We extend the training regime proposed in (Hafner et al. 2019a) to include the
training and application of ABPS. Though the full training procedure is
detailed in Algorithm 2, we provide an overview below.

#### Overview

The training procedure can be split into three phases: data collection, latent
world model training and agent training (lines 10-22, 4-6 and 7-8 in Algorithm
2 respectively). These phases are cycled through until convergence.

##### Data Collection

In this phase, the agent interacts with the real environment to collect a
dataset
$\mathcal{D}\subseteq\mathcal{S}\times\mathcal{A}\times\mathcal{O}\times\mathbb{R}\times\\{0,1\\}$
of states, actions, observations, rewards and violations with which a latent
world model can be learned. At the very start of training, we collect
trajectories from $S$ seed episodes using a random policy; at all other times,
we use the agent’s shielded policy.

##### Latent World Model Training

The goal of this phase is to improve the model of the world so that the policy
has an accurate imagined environment to train in. To this end, we train the
latent world model with respect to the objective in Equation 2 on $B$ data
sequences of length $L$ sampled from $\mathcal{D}$.

##### Agent Training

Agent training is composed of two steps. First, starting from states from the
same $B$ data sequences from the latent world model training phase, the agent
imagines $I$ trajectories of length $H$ with actions chosen from its current
policy. Next, the unshielded policy $\pi$ is updated in an actor-critic
fashion as in (Hafner et al. 2019a, 2021).

#### Key Changes and Contributions

We describe and discuss the key changes we have made that differentiate our
agent from previous approaches.

Experience Dataset. Elements in the experience dataset $\mathcal{D}$ now
contain a binary variable representing whether a violation has occurred
$\lambda_{t}$.

Latent Shielding. Before being sent to the environment $\mathcal{E}$, actions
$a_{t}$ generated by the unshielded policy $\pi$ are routed through our newly-
proposed latent shield (described in Algorithm 1). The shield returns a new
approximately $H$-bounded safe action $a^{\prime}_{t}$.

Intrinsic Punishment. Whenever a violation occurs (whether detected by the
latent shield or by the environment), we override the environment’s reward
function by instead assigning a reward $p<0$. This discourages the agent’s
unshielded policy $\pi$ from taking actions that lead to unsafe states through
the standard RL setup. Over time, this means that the shield will have to
interfere less as $\pi$ becomes biased away from unsafe states. This is a
standard practice in the shielding literature (Alshiekh et al. 2018), however
new to the Dreamer family of agents (Hafner et al. 2019a, b, 2021).

Shield Introduction Schedule  We introduce the novel notion of a shield
introduction schedule $\Pi$ which can enable and disable shielding during
training. The rationale behind this is that a shield backed by an inaccurate
world model will incorrectly label some safe states as unsafe, and vice versa.
In some cases, this may prove detrimental to the training process: suppose
$l_{\theta}$ happens to be initialised in such a way that all states are
labelled as unsafe. As a result, the shield can prevent the agent from
exploring the environment and improving its internal model of the environment.
This, in turn, can prevent the labelling function from learning to correctly
differentiate safe states from unsafe states.

To give the agent time to learn a good world model before restricting
exploration through a learned shield, we augment the training procedure with
$\Pi$ which aims to gradually introduce shielding. To our knowledge, this is
the first time such a system has been proposed. Though there exist a wide
range of possible implementations of $\Pi$, in this work we use a simple
schedule that seems to work well empirically: start training with shielding
disabled and enable shielding once the world model loss (in particular, the
violation loss) begins to plateau. A detailed comparison of different shield
introduction schedules, however, is beyond the scope of this study and left
for further work.

1

Input: Punishment for violation $r_{punish}$, shield imagination horizon $H$,
number of steps to imagine ahead for training $I$, number of seed episodes
$S$, number of training steps per episode $C$, number of real environment
interaction steps per episode $T$, sequence length $L$, batch size $B$,
environment $\mathcal{E}$, labelling function $L_{\mathcal{E}}^{\phi}$ and
shield introduction schedule $\Pi$.

2

3Initialise dataset $\mathcal{D}$ with $S$ seed episodes and neural network
parameters $\theta$ randomly.

4while  _not converged_ do

/* Learning the internal world model and agent policy. */

5 for  _c = 1..C_ do

6 Draw $B$ data sequences
$\\{(a_{t},o_{t},r_{t},\lambda_{t})^{k+L}_{t=k}\\}\sim\mathcal{D}$.

7 Compute model states $h_{t}$, $z_{t}$ and $\hat{z}_{t}$.

8 Update $\theta$ using representation learning.

9 Imagine a trajectory $\rho$ of length $I$ for each state in each data
sequence.

10 Update the unshielded policy $\pi$ based on the imagined trajectories.

11 end for

12

/* Collecting data from the environment. */

13 for  _t = 1..T_ do

14

15 Compute model states $h_{t}$, $z_{t}$ from history.

16 Select action $a_{t}$ with the unshielded policy, adding exploration noise
if desired.

17 if  _$\Pi$ has enabled shielding_ then

18 Pass $h_{t}$, $z_{t}$ and $a_{t}$ into Algorithm 1 to obtain the tuple
$\langle a^{\prime}_{t},interfered\rangle$.

19 else

20 $a^{\prime}_{t},interfered$ := $a_{t},false$.

21 end if

22

23 $r_{t},o_{t}$ := $\mathcal{E}.$step ($a^{\prime}_{t}$).

24 Check if $L_{\mathcal{E}}^{\phi}$ has detected a violation and store the
result as 0 or 1 in $\lambda_{t}$.

25 if $\lambda_{t}=1$ or $interfered$ then $r_{t}:=r_{punish}$.

26 end for

27 Add experience to dataset
$\mathcal{D}:=\mathcal{D}\cup\\{(o_{t},a_{t},r_{t},\lambda_{t})_{t=1}^{T}\\}$.

28 end while

Algorithm 2 Training Dreamer with Approximate Bounded Prescience Shielding.

## Experiments

In this section, we compare our ABPS agent against Dreamer without shielding
(Hafner et al. 2019a, 2021), Dreamer with BPS (Giacobbe et al. 2021), and CPO
(Achiam et al. 2017). We also empirically investigate some aspects of the
internal workings of our agent. A summary of the environments used can be
found below.

##### Visual Grid World

The Visual Grid World (VGW) environment is a simple deterministic navigation
benchmark with high-dimensional ($64\times 64\times 3$) visual observations
(see Figure 2) and discrete actions (up, down, left, right and staying still).
The agent’s (in green) task is to navigate to randomly-placed targets (in
black) while avoiding any unsafe locations (in red). This yields a relatively
straightforward safety specification:

$\neg\tt{agent\\_in\\_red\\_square}\;\cup\;\tt{episode\\_ended}$

where $\tt{agent\\_in\\_red\\_square}$ is true if and only if the agent is in
an unsafe location, and $\tt{episode\\_ended}$ is only true at the end of an
episode.

The reward function used is also quite simple, with a small penalty term at
each time step to encourage movement (though indeed different forms of
intrinsic motivation may be also used):

$\mathcal{R}(s_{t},a_{t})=\begin{cases}100,&\text{agent reaches target,}\\\
-40,&\text{agent enters unsafe state,}\\\ -10,&\text{agent does not move,}\\\
-1,&\text{otherwise.}\end{cases}$

We experiment with both fixed and procedurally generated grids over episodes
consisting of 500 steps.

##### Cliff Driver

The Cliff Driver (CD) environment is a symbolic benchmark with stochastic
dynamics and continuous actions. The agent controls the forward acceleration
of a car and is tasked with driving to the edge of a cliff as quickly as
possible without overshooting and falling into the sea. The car exists on a
one-dimensional road with actions $a_{t}\in[-1,1]$ which correspond to the
acceleration of the car. The car cannot move backwards and its speed $v_{t}$
is lower-bounded at 0 (thus $a_{t}<0$ corresponds to braking as opposed to
reversing). The agent starts each episode stationary at a fixed distance
$x_{0}$ from the edge of the cliff with its distance at subsequent time steps
being written $x_{t}$. Stochasticity comes from the fact that at each time
step there is a probability $p_{stick}$ that the car’s controls get “stuck”
meaning that the current action is ignored and replaced with the previous
action.

Observations from the environment are given as two-dimensional vectors
encoding the distance from the edge of the cliff in one component and the
speed of the car in the other. The safety specification can thus be written:

$\neg\tt{agent\\_fallen\\_off\\_cliff}\;\cup\;\tt{episode\\_ended}$

where $\tt{agent\\_fallen\\_off\\_cliff}$ is true if and only if the agent has
overshot the cliff, and $\tt{episode\\_ended}$ is only true at the end of an
episode.

Finally, the reward function is given as:

$\mathcal{R}(s_{t},a_{t})=\begin{cases}1-\frac{x_{t}}{x_{0}},&\text{agent has
not fallen off cliff,}\\\ -5,&\text{otherwise.}\end{cases}$

We experiment with $p_{stick}=0.1$ and $p_{stick}=0.5$ on roads 10 units long
over episodes consisting of 20 steps.

Figure 2: Comparison of latent trajectories in the agents’ learned models of the environment. All trajectories begin from the leftmost observation and time progresses from left to right. The bottom row of arrows denote the actions taken just before each time step. From top to bottom, the three rows of images correspond to trajectories in the latent space of the BPS agent, the unshielded agent, and our agent. | Flavour | Metric | Latent | Unshielded | BPS | CPO
---|---|---|---|---|---|---
|  | Testing Reward | 15067 (434) | 13148 (249) | 12468 (620) | -2925 (1065)
| Fixed | Testing Violations | 0.30 (0.76) | 2.25 (1.60) | 0 (0) | 13.43 (19.25)
Visual |  | Training Violations | 1262 (172) | 2306 (833) | 0 (0) | 16455 (1435)
Grid World |  | Testing Reward | 8084 (2221) | 6825 (1427) | 1938 (3552) | -1588 (2051)
| Procedural | Testing Violations | 4.50 (3.59) | 33.7 (16.28) | 0 (0) | 19.60 (13.83)
|  | Training Violations | 14018 (1852) | 15309 (4686) | 0 (0) | 18705 (3756)
|  | Testing Reward | 8.57 (2.96) | 10.76 (3.29) | 10.50 (3.28) | 7.56 (2.86)
| $p_{stick}=0.1$ | Testing Violations | 0 (0) | 0 (0) | 0 (0) | 3.40 (1.91)
Cliff |  | Training Violations | 58.2 (9.60) | 90.0 (9.10) | 24.0 (13.02) | 973.0 (357.7)
Driver |  | Testing Reward | 8.10 (4.99) | 6.63 (8.07) | 7.10 (9.52) | 6.44 (3.00)
| $p_{stick}=0.5$ | Testing Violations | 0.18 (0.84) | 0.54 (1.53) | 0.22 (1.18) | 0.48 (1.24)
|  | Training Violations | 91.8 (16.85) | 157.6 (18.4) | 80.4 (17.43) | 3126 (2823)
Table 1: Comparison of trained agents with standard deviations in parentheses
where applicable.

### Performance Evaluation

We evaluate our agents on three metrics: (1) average reward per episode at
test-time; (2) average number of violations per episode at test-time; and (3)
total number of violations during training. Results are calculated by
averaging over five versions of each agent trained from different seeds.

##### Training Details

Experiments were carried out on a machine with a single NVIDIA RTX 2080 Ti
GPU, an AMD Ryzen 7 2700X processor and 64GB of RAM. For each environment, the
model-based agents were trained for the same number of steps; the model-free
CPO agents were allowed to train for longer (2$\times$ longer for the VGW
environment and 5$\times$ longer for the CD environments). The latent shield
horizon $H$ was 2 and 6 for the VGW and CD environments respectively.

We used the network architectures proposed in (Hafner et al. 2019a). The
number of nodes in each layer varied depending on the environment and can be
found in Table 2. We implemented our encoder for symbolic observations from
the CD environment as a feed-forward network with three fully-connected hidden
layers and ReLU activations. For the CPO agent, we used the same observation
encoders and policy networks as mentioned above. Moreover, since the action
space of the CD environment is continuous, we discretised the actions into
four bins when performing BPS (this boiled down to rounding the continuous
actions proposed by the agent to the nearest action in the set
$\\{-1,-0.1,0.1,1\\}$. Such modifications, however, were not needed for our
agent.

Shield introduction schedules were kept relatively simple. For the VGW
environment, the agent started with shielding disabled. After 10 episodes
(including 5 seed episodes), shielding was enabled every third episode. After
20 episodes, shielding was enabled every other episode. Shielding was fully
enabled after 30 episodes. In addition, the unsafe threshold $\epsilon$ was
decayed linearly from 0.5 to 0.125 over the course of training. For the CD
environment, shielding was enabled after 60 episodes (including 50 seed
episodes).

Figure 4: Training reward curves of agents using latent shielding with and
without a shield introduction schedule (SIS).

##### Results

As can be seen in Table 1, our agent collected more reward at test-time than
both the unshielded and BPS agents in the VGW environment. Moreover, our agent
saw a seven-fold reduction in test-time violations compared to the unshielded
agent across both the fixed and procedurally generated environments, averaging
at less than one violation per episode for the fixed VGW. This reduction in
training violations is even more dramatic when compared to the model-free CPO
agent. Our agent outperformed the other agents at test-time in the most
stochastic CD environment ($p_{stick}=0.5$). This is possibly due to the SRSSM
used by our latent shield being better able to capture non-determinism than
BPS. Nevertheless, the result is still rather impressive given that the agent
using BPS sampled 1024 trajectories at every time step whereas our agent only
sampled 20. Plots of training reward and violations can be seen in Figure 3.

### Qualitative Evaluation of Learned Dynamics

We evaluate the quality of each agent’s latent dynamics model by observing its
trajectory predictions given a particular starting state and action sequence.
Each agent was given the same starting observation and predefined sequence of
10 actions. The agents took these actions in their respective latent world
models with the states traversed being decoded for inspection. Results can be
seen in Figure 2 and indicate that, qualitatively, our agent’s decoder
performed the best, accurately predicting 10 frames into the future. One
possible reason for this observation is that the inclusion of the violation
detection objective in the SRSSM encourages the model to focus on accurately
capturing violation states. Another potential factor is that the BPS agent
never actually enters unsafe states and thus finds it difficult to represent
them.

### Do Shield Introduction Schedules Actually Help?

We compare the first 100 training episodes of our agent in the fixed VGW
environment with and without a shield introduction schedule. As with the
performance evaluation, results are calculated over five trained agents and a
plot of training reward can be seen in Figure 4. Though both agents start at
roughly the same reward, the agents with a shield introduction schedule
consistently outperforms their counterparts without shield introduction
schedules.

Figure 3: Reward curves (top row) and violations (bottom row) for the model-
based agents on the (from left to right) fixed VGW, procedurally generated
VGW, cliff driver ($p_{stick}=0.1$) and cliff driver ($p_{stick}=0.5$)
environments. The shaded area denotes one standard deviation. Dotted lines
denoting the performance of the CPO agent after training are given where
possible without distorting the scale of the y-axis.

## Related Work and Discussion

##### Latent World Models

Learning latent world models from visual observations has seen growing
interest from the RL community (Wahlström, Schön, and Deisenroth 2015; Watter
et al. 2015; Racanière et al. 2017; Ha and Schmidhuber 2018; Hafner et al.
2019a, b; Schrittwieser et al. 2020; Hafner et al. 2021). One key trend in the
literature is that of training agents in their own learned world models (Ha
and Schmidhuber 2018; Hafner et al. 2019a, 2021). In this work, we extend the
world model formulation used in (Hafner et al. 2019a, b, 2021) to encode a
notion of safety into state representations.

##### Safety By Filtering Actions

Overriding unsafe actions with safe ones has been a popular approach to
safety-aware RL. Introduced into the RL scene by (Alshiekh et al. 2018),
shielding has received much research interest and has seen applications in
real-world settings (Nikou et al. 2021). Various works have attempted to
address some of its shortcomings of the original formulation. These include
allowing the shield to be updated to aid exploration or correct model
imperfections (Anderson et al. 2020; Pranger et al. 2021); improving
performance in non-deterministic environments (Jansen et al. 2020; Li and
Bastani 2020); and extending shielding to multi-agent RL (ElSayed-Aly et al.
2021). To our knowledge, few works (Srinivasan et al. 2020; Thananjeyan et al.
2021; Bharadhwaj et al. 2021; Giacobbe et al. 2021) focus on removing the need
for handcrafting an abstraction (among the most time-consuming and error-prone
aspects of shielding), and only one of them (Giacobbe et al. 2021) achieves
this without getting rid of the abstraction altogether, albeit by providing
the agent with access to the program that controls the environment.

Hyperparameter | Value
---|---
VGW | CD
Deterministic State Size | 200 | 8
Stochastic State Size | 30 | 16
NN Hidden Layer Size | 200 | 16
Observation Embedding Size | 1024 | 32
Discount Factor ($\gamma$) | 0.99 | 0.99
Action Repeat | 1 | 1
Seed Episodes ($S$) | 5 | 50
Episode Length ($T$) | 500 | 20
Batch Size ($B$) | 50 | 250
Sequence Length ($L$) | 50 | 10
Training Steps ($C$) | 100 | 100
Exploration Noise Variance ($\sigma^{2}$) | 0.3 | 0.3
Imagination Horizon ($I$) | 15 | 15
KL Balancing Ratio ($\beta$) | 4:1 | 4:1
Violation Class Weighting | 3 | 3
Model Learning Rate | 1e-3 | 1e-4
Policy Learning Rate | 8e-5 | 8e-5
Value Learning Rate | 8e-5 | 8e-7
Bit Depth | 5 | -
Adam Epsilon | 1e-7 | 1e-7
Adam Beta | 0.9, 0.999 | 0.9, 0.999
ABPS Horizon ($H$) | 2 | 6
ABPS Sampled Trajectories ($N$) | 20 | 10
ABPS Unsafe Threshold ($\epsilon$) | 0.15 | 0.15
CPO Cost Limit | 5 | 0
Table 2: Hyperparameters for experiments in each environment.

In contrast, latent shielding tackles the problem head-on by directly learning
an abstraction for use by the shield. In this work, the abstraction we use is
an SRSSM, which captures stochasticity by design, a useful property for non-
deterministic environments. Though our latent shield satisfies an
approximation of $H$-bounded safety with respect to the learned abstraction,
its safety with respect to the true environmental dynamics is not guaranteed
and instead relies on the fidelity with which the true dynamics are captured.
Furthermore, unlike for its formally-verified predecessors, it is a necessary
sacrifice that the agent visits unsafe states during training in order to
learn a notion of safety (unless, of course, pre-training is possible).
Nevertheless, a learned abstraction may be advantageous in settings where
handcrafting an abstraction is not feasible and privileged access to some
simulation (as in (Giacobbe et al. 2021)) cannot be assumed. Moreover, latent
shielding may provide a greater degree of explainability over model-free
methods which get rid of the abstraction altogether (Srinivasan et al. 2020;
Thananjeyan et al. 2021; Bharadhwaj et al. 2021) \- if the shield overrides an
action, one can reconstruct the imagined unsafe trajectories that led to the
interference. Finally, it should be noted that the problem setup used in this
work assumes no prior knowledge on how safe behaviour might be achieved in the
environment. This may not be the case in many real-world settings and it is
conceivable that combining latent shield learning with curriculum learning
based on human knowledge (in a system such as that proposed by (Turchetta et
al. 2020)) may lead to improved safety during training.

## Conclusions

In this paper, we have presented latent shielding, a new framework for
shielding DRL agents without the need for a handcrafted abstraction of the
environment. Using this framework, we have designed a novel shield and
demonstrated that this method not only leads to improved adherence to safety
specifications on two benchmark environments with respect to an unshielded
agent, but also works out-of-the-box on both continuous and discrete
environments (unlike its predecessor, BPS). Furthermore, we have demonstrated
for the first time that shielding at inappropriate times may adversely impact
the performance of model-based DRL agents and showed how this phenomenon can
be counteracted using our novel notion of shield introduction schedules.

Our work opens several exciting avenues for future work. For instance, this
work uses a very simple shield introduction schedule; future work may provide
a richer investigation into the properties of different schedules. Moreover,
though demonstrating promising empirical results, our realisation of latent
shielding loses the formally verifiable safety guarantees enjoyed by many
symbolic shielding approaches - whether it is possible to construct a
verifiable implementation of latent shielding, or compensate for the loss of
formal guarantees, are open problems.

## Acknowledgements

The authors would like to thank Claudio Elgueta Karstegl for turning our GPU
machine off and on again during the national lockdowns.

## References

  * Achiam et al. (2017) Achiam, J.; Held, D.; Tamar, A.; and Abbeel, P. 2017.  Constrained Policy Optimization.  In Precup, D.; and Teh, Y. W., eds., _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , 22–31. PMLR.
  * Alshiekh et al. (2018) Alshiekh, M.; Bloem, R.; Ehlers, R.; Könighofer, B.; Niekum, S.; and Topcu, U. 2018.  Safe Reinforcement Learning via Shielding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Anderson et al. (2020) Anderson, G.; Verma, A.; Dillig, I.; and Chaudhuri, S. 2020.  Neurosymbolic Reinforcement Learning with Formally Verified Exploration.  In _Advances in Neural Information Processing Systems_ , volume 33, 6172–6183.
  * Bacchus and Kabanza (2000) Bacchus, F.; and Kabanza, F. 2000.  Using Temporal Logics to Express Search Control Knowledge for Planning.  _Artificial Intelligence_ , 116(1): 123–191.
  * Bharadhwaj et al. (2021) Bharadhwaj, H.; Kumar, A.; Rhinehart, N.; Levine, S.; Shkurti, F.; and Garg, A. 2021\.  Conservative Safety Critics for Exploration.  In _International Conference on Learning Representations_.
  * Chow et al. (2017) Chow, Y.; Ghavamzadeh, M.; Janson, L.; and Pavone, M. 2017.  Risk-Constrained Reinforcement Learning with Percentile Risk Criteria.  _J. Mach. Learn. Res._ , 18(1): 6070–6120.
  * ElSayed-Aly et al. (2021) ElSayed-Aly, I.; Bharadwaj, S.; Amato, C.; Ehlers, R.; Topcu, U.; and Feng, L. 2021\.  Safe Multi-Agent Reinforcement Learning via Shielding.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , AAMAS ’21, 483–491. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.  ISBN 9781450383073.
  * Giacobbe et al. (2021) Giacobbe, M.; Hasanbeig, M.; Kroening, D.; and Wijk, H. 2021.  Shielding Atari Games with Bounded Prescience.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , 1507–1509.  ISBN 9781450383073.
  * Ha and Schmidhuber (2018) Ha, D.; and Schmidhuber, J. 2018.  Recurrent World Models Facilitate Policy Evolution.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2455–2467.
  * Hafner et al. (2019a) Hafner, D.; Lillicrap, T.; Ba, J.; and Norouzi, M. 2019a.  Dream to Control: Learning Behaviors by Latent Imagination.  In _International Conference on Learning Representations_.
  * Hafner et al. (2019b) Hafner, D.; Lillicrap, T.; Fischer, I.; Villegas, R.; Ha, D.; Lee, H.; and Davidson, J. 2019b.  Learning Latent Dynamics for Planning from Pixels.  In _Proceedings of the 36th International Conference on Machine Learning_ , 2555–2565. PMLR.
  * Hafner et al. (2021) Hafner, D.; Lillicrap, T. P.; Norouzi, M.; and Ba, J. 2021.  Mastering Atari with Discrete World Models.  In _International Conference on Learning Representations_.
  * Jansen et al. (2020) Jansen, N.; Könighofer, B.; Junges, S.; Serban, A.; and Bloem, R. 2020.  Safe Reinforcement Learning Using Probabilistic Shields (Invited Paper).  In Konnov, I.; and Kovács, L., eds., _31st International Conference on Concurrency Theory (CONCUR 2020)_ , volume 171 of _Leibniz International Proceedings in Informatics (LIPIcs)_ , 3:1–3:16. Dagstuhl, Germany: Schloss Dagstuhl–Leibniz-Zentrum für Informatik.  ISBN 978-3-95977-160-3.
  * Kupferman and Vardi (2001) Kupferman, O.; and Vardi, M. Y. 2001.  _Formal Methods in System Design_ , 19(3): 291–314.
  * Li and Bastani (2020) Li, S.; and Bastani, O. 2020.  Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 7166–7172.
  * Nikou et al. (2021) Nikou, A.; Mujumdar, A.; Orlić, M.; and Vulgarakis Feljan, A. 2021.  Symbolic Reinforcement Learning for Safe RAN Control.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , AAMAS ’21, 1782–1784. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.  ISBN 9781450383073.
  * Pranger et al. (2021) Pranger, S.; Könighofer, B.; Tappler, M.; Deixelberger, M.; Jansen, N.; and Bloem, R. 2021.  Adaptive Shielding under Uncertainty.  In _2021 American Control Conference (ACC)_ , 3467–3474.
  * Racanière et al. (2017) Racanière, S.; Weber, T.; Reichert, D. P.; Buesing, L.; Guez, A.; Rezende, D.; Badia, A. P.; Vinyals, O.; Heess, N.; Li, Y.; Pascanu, R.; Battaglia, P.; Hassabis, D.; Silver, D.; and Wierstra, D. 2017.  Imagination-Augmented Agents for Deep Reinforcement Learning.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, 5694–5705. Red Hook, NY, USA: Curran Associates Inc.  ISBN 9781510860964.
  * Schrittwieser et al. (2020) Schrittwieser, J.; Antonoglou, I.; Hubert, T.; Simonyan, K.; Sifre, L.; Schmitt, S.; Guez, A.; Lockhart, E.; Hassabis, D.; Graepel, T.; Lillicrap, T.; and Silver, D. 2020.  Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model.  _Nature_ , 588(7839): 604–609.
  * Srinivasan et al. (2020) Srinivasan, K.; Eysenbach, B.; Ha, S.; Tan, J.; and Finn, C. 2020.  Learning to be Safe: Deep RL with a Safety Critic.  _ArXiv_ , abs/2010.14603.
  * Thananjeyan et al. (2021) Thananjeyan, B.; Balakrishna, A.; Nair, S.; Luo, M.; Srinivasan, K.; Hwang, M.; Gonzalez, J. E.; Ibarz, J.; Finn, C.; and Goldberg, K. 2021.  Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones.  _IEEE Robotics and Automation Letters_ , 6(3): 4915–4922.
  * Turchetta et al. (2020) Turchetta, M.; Kolobov, A.; Shah, S.; Krause, A.; and Agarwal, A. 2020.  Safe Reinforcement Learning via Curriculum Induction.  _Advances in Neural Information Processing Systems_ , 33.
  * Wahlström, Schön, and Deisenroth (2015) Wahlström, N.; Schön, T. B.; and Deisenroth, M. P. 2015.  From Pixels to Torques: Policy Learning with Deep Dynamical Models.  arXiv:1502.02251.
  * Watter et al. (2015) Watter, M.; Springenberg, J. T.; Boedecker, J.; and Riedmiller, M. 2015.  Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2_ , 2746–2754.
  * Yang et al. (2020) Yang, T.-Y.; Rosca, J.; Narasimhan, K.; and Ramadge, P. J. 2020.  Projection-Based Constrained Policy Optimization.  In _International Conference on Learning Representations_.


[affiliation=]JiuFeng [affiliation=]Mehmet HamzaErol [affiliation=]Joon
SonChung [affiliation=]ArdaSenocak

# ElasticAST: An Audio Spectrogram Transformer
for All Length and Resolutions

###### Abstract

Transformers have rapidly overtaken CNN-based architectures as the new
standard in audio classification. Transformer-based models, such as the Audio
Spectrogram Transformers (AST), also inherit the fixed-size input paradigm
from CNNs. However, this leads to performance degradation for ASTs in the
inference when input lengths vary from the training. This paper introduces an
approach that enables the use of variable-length audio inputs with AST models
during both training and inference. By employing sequence packing, our method
ElasticAST, accommodates any audio length during training, thereby offering
flexibility across all lengths and resolutions at the inference. This
flexibility allows ElasticAST to maintain evaluation capabilities at various
lengths or resolutions and achieve similar performance to standard ASTs
trained at specific lengths or resolutions. Moreover, experiments demonstrate
ElasticAST's better performance when trained and evaluated on native-length
audio datasets. Code is available at: https://github.com/JiuFengSC/ElasticAST
††This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (No. RS-2023-00212845).

###### keywords:

Audio Spectrogram Transformers, Audio Classification

##  1 Introduction

Until not long ago, convolution-based neural networks were the prominent
approach in both computer vision [1, 2] and audio processing [3]. More
recently, transformers [4] have made a significant impact on numerous computer
vision [5, 6, 7, 8, 9, 10, 11] and audio processing tasks [12, 13, 14, 15, 16,
17, 18, 19, 20], and CNN-based architectures are being replaced with these
attention-based architectures. Despite this replacement, transformers still
adhere to the fixed-size input paradigm as CNNs do. Similarly, Audio
Spectrogram Transformers (ASTs) [12] take a fixed-size input where the input
spectrograms are divided into fixed-size patches to create tokens as input for
the transformer encoder. To obtain a fixed-size input, audio spectrograms are
either trimmed or padded to a fixed size. Considering transformers can process
any sequence length, using varying input sizes rather than fixed ones, can be
a more optimal and natural choice for audio processing tasks.

This fixed input size paradigm poses several challenges and flaws: (1) Recent
datasets, such as VoxCeleb and Epic-Sounds, consist of audio recordings of
various lengths (see Figure 2). Considering the recent developments in using
in-the-wild data, self-supervised learning, and multimodal learning, having
data in various lengths becomes quite natural. (2) Trimming or padding the
input data is a suboptimal choice, as it can easily lead to the discarding or
contamination of information. (3) AST-based models lack flexibility when
evaluated with inputs of different lengths or temporal resolutions, both of
which result in varying sequence lengths, compared to those used during
training. This necessitates training different AST models for circumstances
with varying sequence length requirements, such as computation budget or
memory consumption. Overall, these considerations make standard ASTs
relatively limited. Therefore, it is appealing to explore the flexibility of
sequence lengths in Audio Spectrogram Transformers. Specifically, this
involves designing a single AST model capable of handling variable input
sequence lengths during both training and inference.

In this work, we present ElasticAST, an Audio Spectrogram Transformer model
that turns standard AST into a model capable of processing audio of any length
or temporal resolution during both training and inference without trimming or
padding. The resulting model is functionally superior to standard ASTs as it
maintains its performance across various lengths or resolutions of audio
during the inference stage and also delivers better performance on datasets
containing audio of various lengths.

---

Figure 1: Standard ASTs vs. ElasticAST. Standard ASTs' performance degrades
when evaluated on audio lengths different from their trained lengths, while
ElasticAST remains flexible to varying lengths.

There are previous approaches that explore different sequence lengths or
temporal resolutions in AST-based architectures. [21, 22] focus on providing
patch-size flexibility to ASTs and ViTs. Different patch sizes lead to
different sequence lengths. Patch sizes are randomly selected during training,
and a resizing algorithm is applied to convert the patch embedding weights
accordingly for the different patch sizes. As a result, models gain
flexibility with different patch sizes during the inference stage. Our method
follows a similar direction in terms of providing flexibility to sequence
length during both training and inference stages. However, instead of patch
sizes, our model focuses on variable lengths and temporal resolutions of audio
inputs. Another related work is [23], which uses mixed resolutions of audios
to train ASTs efficiently. The main idea is to process lower-resolution audio
(coarse) early in training, and then fine-tune with high-resolution data
(fine) later in a hierarchical manner. Rather than offering flexibility, this
work focuses on training efficiency. In contrast, our model randomly mixes
various resolutions of audios during training at once without employing any
curriculum learning strategy, making the model capable of seamlessly
processing any temporal resolution of audio in the inference. Moreover, both
of these previous works still use fixed length input. The work most similar to
ours is NaViT [24], which allows Vision Transformers to handle a variety of
image resolutions. Inspired by the discoveries in this study, we investigate
appropriate methods to make adaptations to the standard ASTs for flexibility
across all lengths and temporal resolutions of audio. To accomplish this task,
we take the following steps (shown in Figure 3): (1) Instead of using a fixed
audio length or resolution during training, we randomly select the resolution
or use the variable native length of the audios without trimming or padding.
(2) We pack these audios of different lengths (or resolutions) into a single
sequence to process them all at once during training. (3) We use the standard
AST architecture with minimal changes, such as limiting the scope of attention
to each individual sample to prevent it from attending to other samples within
a packed sequence, and replacing the class tokens with a masked attention
pooling mechanism. The rest of the model remains identical. We summarize the
contributions of our work as follows:

---

Figure 2: Variable Length Datasets and standard AST training input. Due to the
fixed-length processing constraints, ASTs discard informative tokens and
introduce non-informative tokens.

  * •

We demonstrate that standard Audio Spectrogram Transformers (ASTs) lack the
flexibility to be trained and evaluated on variable lengths or resolutions
different from those on which they were initially trained.

  * •

We introduce an approach that enables the creation of ElasticAST which allows
standard ASTs to be trainable with variable native lengths or temporal
resolutions of audio.

  * •

ElasticAST is a single AST model capable of operating across all lengths and
resolutions at the inference stage without significant performance
degradation, while achieving performance comparable to standard ASTs trained
at fixed lengths or resolutions.

  * •

We show that, in addition to flexible model usage, ElasticAST can surpass the
performance of standard ASTs on datasets of variable lengths, such as VoxCeleb
and Epic-Sounds, by leveraging the full semantic content of audio without the
need for cutting or padding.

##  2 Approach

---

Figure 3: Our ElasticAST framework.

###  2.1 Preliminaries

The Audio Spectrogram Transformer (AST [12]) utilizes a transformer-based
architecture to process audio spectrograms. It starts by splitting each
spectrogram $x\in\mathbb{R}^{f\times t}$ from a batch $X\in\mathbb{R}^{B\times
f\times t}$ into a sequence of $S$ smaller patches: $x\to
x_{i}\in\mathbb{R}^{p\times p}$, where $i$ ranges from $0$ to $S$ and
$S=(f/p)\times(t/p)$. These patches are then converted into embeddings via a
linear projection layer $e_{i}=P(x_{i})\in\mathbb{R}^{D}$. Next, a special
[cls] token is prepended to the sequence, increasing the length to $N=S+1$.
The learnable positional embeddings are added to provide the order information
of the tokens. The sequences are then transformed by the transformer encoder.
The encoder's output for the [cls] token acts as the audio spectrogram's
representation for downstream tasks, such as classification.

###  2.2 Architectural changes

ElasticAST is a conceptually simple extension of standard ASTs, designed to
accommodate the use of various audio spectrogram lengths during both the
training and inference stages. This flexibility is achieved with minimal
architectural modifications to conventional ASTs.

Sequence Packing. Unlike AST, which allocates $N$ tokens for all the $B$ input
spectrograms, our model employs a sequence packing method that accommodates
the varying lengths by organizing the spectrograms into token sequences
$X\in\mathbb{R}^{B^{\prime}\times N^{\prime}\times D}$ after patchification as
illustrated in Figure 3. This method initiates by sequentially filling the
first token sequence row with tokens derived from the samples until reaching
the preset token limit per row, $L^{\prime}$ (as a default set to 2048). If
the tokens from a sample about to surpass this limit, they are deferred to the
subsequent row, and the allocation process continues accordingly for the
remaining samples. This packing technique generates $B^{\prime}$ rows of token
sequences, each potentially varying in length but constrained by $L^{\prime}$.
To facilitate processing by the transformer encoder, as shown in Figure 3, we
introduce minimal padding tokens to each row, standardizing their lengths to
$N^{\prime}\leq L^{\prime}$, which corresponds to the longest sequence length
among the rows. Our approach to packing is deliberately straightforward,
prioritizing simplicity by processing samples sequentially in their original
order and filling rows on a first-come, first-served basis. Future work may
investigate more sophisticated packing algorithms to enhance efficiency.

Mask Self-Attention. In conventional patch-based transformer models [12, 5],
only the tokens from the same sample form a row, and these tokens are aware of
each other within the transformer encoder through global attention, formalized
as $\text{Attention}(Q,K,V)=\text{softmax}\left(QK^{T}\right)\cdot V$ where
$Q,K,V\in\mathbb{R}^{B\times N\times D}$. However, this mechanism becomes
impractical when sequences are comprised of tokens from different samples, as
it fails to constrain the attention within the tokens of the same sample. To
address this, our architecture is designed to prevent tokens from different
samples within the same sequence from attending to one another. We achieve
this by introducing a Masked Self-Attention mechanism. The main idea is to
introduce a boolean mask $M\in\mathbb{B}^{B^{\prime}\times N^{\prime}\times
N^{\prime}}$ for each sequence in a batch, where $M_{b,i,j}$ represents if the
$i$th token should attend to th $j$th token in $b$th sequence, thus encoding
within sample attention. Then, we modify the original attention mechanism as:
$\text{Attention}(Q,K,V)=\text{softmax}\left(\text{Mask}(QK^{T})\right)\cdot
V$ where $Q,K,V\in\mathbb{R}^{B^{\prime}\times N^{\prime}\times D}$, by
selectively preventing cross-sample attention, visually represented by
different colors in Figure 3.

Mask Attention Pooling. After the sequences are processed by the transformer
encoder without cross-sample attention contamination, the representation of
each sample before the linear classifier is obtained through Mask Attention
Pooling [25]. In AST, each sequence row has a prepended [cls] token used as
the representation of that sample. For the ElasticAST, rather than coupling a
[cls] token with each packed sample, we employ a Mask Attention Pooling layer
on the top of the encoder to derive sample representations, serving as an
effective alternative. This architecture mirrors the previously described Mask
Self-Attention mechanism, wherein the $K,V\in\mathbb{R}^{B^{\prime}\times
N^{\prime}\times D}$ matrices are obtained from the input, while another
$Q^{\prime}\in\mathbb{R}^{B^{\prime}\times n\times D}$ matrix is generated
from the repetition of a learnable query vector parameter
$q\in\mathbb{R}^{D}$, where $n$ refers to the maximum number of packed samples
in all rows. Then, with a generated mask $M\in\mathbb{B}^{B^{\prime}\times
n\times N^{\prime}}$ where $M_{b,i,j}$ corresponds to if $j$th token in the
$b$th sequence belongs to the $i$th sample. Then, the mask attention pooling,
$\text{AttnPool}=\text{softmax}\left(\text{Mask}(Q^{\prime}K^{T})\right)\cdot
V$, subsequently extracts representations $X\in\mathbb{R}^{B^{\prime}\times
n^{\prime}\times D}$, in a similar way of employing masking to ensure pooling
is confined to tokens within a sample, as depicted by the cone shadow areas in
Figure 3. Afterward, the obtained sequence is unpacked into
$X\in\mathbb{R}^{B\times D}$ by reordering the representations of each sample,
treating them as the [cls], and fed into the linear classifier.

##  3 Experiments

###  3.1 Datasets and Evaluation Metrics

Datasets. In our experiments, we utilize four datasets: AudioSet, VGGSound,
VoxCeleb, and Epic-Sounds. AudioSet [26] is a large multi-label dataset with
approximately 2 million 10-second clips spanning 527 labels across diverse
audio categories. VGGSound [27] includes around 200,000 10-second video clips,
labeled with 309 sound classes such as objects and human activities. VoxCeleb
[28] is an audio-visual dataset focused on human speech, featuring 1,251
speakers and approximately 145,000 utterances across a range of durations from
4 to 144 seconds. For experimental purposes, we impose a hypothetical upper
limit of 30 seconds on this dataset to simplify the experiments and manage the
memory usage effectively. Lastly, Epic-Sounds [29], derived from first-person
(egocentric) videos, includes 44 categories and a total of 75.9k audio files
of various lengths (see Figure 2). Similar to VoxCeleb, we set a provisional
maximum duration of 30 seconds for this dataset, though this limit can be
adjusted as needed.

Evaluation metrics. Given the presence of multi-labels in each AudioSet
sample, we use mean average precision (mAP) for evaluation across all
categories. For the other datasets, we report the Top-1 classification
accuracy (Acc) as our measure of evaluation since each sample has only a
single label.

###  3.2 Implementation Details

In this paper, the configuration for standard ASTs follows the same choices as
those in [21]. The batch and the patch size are set to 12 and 16 (B/16)
respectively, and all models are initialized with ViT [30] (ImageNet
Pretrained) for every dataset, except for VoxCeleb, where SSAST [13] weights
are employed. The learning rate is established at 1e-5 for all datasets, apart
from the Epic-Sounds dataset, which is set at 1e-4 by following [29]. To
accommodate varying audio lengths, we shifted from a 1D to a 2D positional
[31, 32] embeddings, encoding frequency and time positions separately. Our
ElasticAST uses the identical settings as the standard ASTs in this paper.
Additional minor architectural changes are already discussed in Section 2. By
default, we use a window of 25 ms with a frame shift of 10 ms to transform
waveforms into 128 mel-fbank. The resulting mel-spectrogram shapes for
10-second audio clips are as follows: for AudioSet and VGGSound, 128 × 1024.
For variable-length audio clips (in VoxCeleb and Epic-Sounds), the resulting
temporal length dimension is different according to the native length of the
audios. We adjust the frame shift (Fshift) hyperparameter for each different
audio resolutions so that the temporal length dimension changes accordingly.
All the results that are used to draw graphs in this paper are available
anonymously at https://sites.google.com/view/elasticast-interpseech24.

---

Figure 4: Results on variable native length audios.

###  3.3 Results of Various Length Training

We perform experiments with various natural lengths of audio to assess the
potential of ElasticAST. Thus, we use VoxCeleb and Epic-Sounds datasets, which
consist of various lengths of audio as shown in Figure 2, for the experiments
in this section. For the sake of memory, the maximum length of the audio is
limited to 30 seconds (see Section 3.1). ElasticAST is trained at the native
length of the audio by packing them into batches of sequences without cutting
or padding. Meanwhile, AST models are trained at fixed length by trimming and
padding as they can not accommodate varying input lengths during training. We
compare the performance of ElasticAST to various AST models in two
perspectives. First, We evaluate the performance of ElasticAST by using the
native length of the audios in the inference stage without any alteration,
while standard ASTs are evaluated with the audio lengths at which they are
trained. All of these results are depicted with scattered dots in Figure 4.
Second, we evaluate every model at a series of audio lengths $x_{i}$ from 256
to 3072, such that standard ASTs always apply cutting or padding whenever the
native length of the audio is not $x_{i}$. However, our model only cuts the
audio if the native length is longer than $x_{i}$; otherwise, no padding is
applied, and the native audio is processed as is.

Summary. (1) ElasticAST performs well with various data lengths. In contrast,
when AST models are evaluated at a different time length than the ones used
during training, their performance collapses. (2) ElasticAST on native length
audio can surpass the performance of all standard ASTs by leveraging the full
semantic content of audio without the need for cutting or padding (indicated
by scattered dots). (3) During inference, with the increasing audio length,
more semantics are provided, and ElasticAST can adapt to this change by
utilizing a larger semantic volume to boost its performance towards
saturation. The more data it can process, the better it performs. However, AST
models do not necessarily improve their performance. We hypothesize that the
semantic volume that AST can handle is fixed during training. (4) During the
training of ElasticAST, no information in the dataset is discarded by
trimming, and meanwhile, fewer padded tokens are used, as shown in Table 1.
However, the AST model has to trim longer audio or add non-informative padding
tokens to fit them into its length requirement. Given the same amount of
calculated tokens, ElasticAST consumes a larger meaningful semantic volume.
This highlights the resource consumption efficiency of our architecture.

###  3.4 Results of Various Temporal Resolutions Training

This section presents the results of ElasticAST using various temporal
resolutions. We parameterize the temporal resolution through the Fshift (frame
shift) value when generating spectrograms [33, 23]. This value determines the
length and, consequently, the detail/resolution of the spectrogram. We
temporally compress the mel-spectrograms by a factor of $C_{i}$, where the
frame-shift value is multiplied by randomly sampled $C_{i}$ from
$C=\\{1.0,1.2,1,4,...3.8,4.0\\}$ during training (mixed-resolution training).
Conventional ASTs are trained with a fixed resolution since they cannot
accommodate varying input lengths during training. We compare our ElasticAST
to standard ASTs during inference across multiple compression rates on
VGGSound and AudioSet, as illustrated in Figure 5. The rationale behind
choosing these two datasets is that they consist of fixed-length audios, and
we convert them into various lengths through different compression rates
(Fshift values).

Summary. (1) Our results show that ElasticAST possesses significant
flexibility in handling a wide range of resolutions across both datasets. In
comparison, the performance of standard ASTs declines when tested at
resolutions different from those used in training. (2) Beyond flexibility, our
model generally matches or surpasses the performance of standard ASTs, even on
the resolutions for which standard ASTs were specifically trained. (3) Our
model allows for the training of a single model adaptable to various
resolutions, unlike standard ASTs, which must be trained individually for each
resolution. This means that, within a given inference budget, ElasticAST can
seamlessly adjust to any computational budget, whereas standard ASTs would
require training a new model for each specific computational scenario.

---

Figure 5: Results on various resolutions.

###  3.5 Ablation Study

The main focus of ElasticAST is to provide flexibility to AST models for using
variable time length audio input in both the training and inference stages. In
Section 3.4, we present the results of using various temporal resolutions. The
time length is changed through the Fshift operation. Additionally, there can
be other methods for reducing temporal length [34, 23]. In this ablation
study, we investigate average pooling strategy in ElasticAST for using various
temporal lengths. By following [23], a mel-spectrogram is processed through an
extra layer of average-pooling. This layer uses a kernel and stride both sized
$1\times C$, effectively reducing the mel-spectrogram's temporal dimension by
a factor of $C$. The training and experimental settings are identical to those
described in Section 3.4, except for the new reduction method, average
pooling, which requires integer numbers. Therefore, only the values
$C=\\{1,2,3,4\\}$ are used as compression rates. To save computational time
and resource, we conduct this experiment on VGGSound. The results are shown in
the right side of Table 1. Similar to the Fshift approach, ElasticAST also
demonstrates flexibility in handling various range of temporal lengths with
this method.

###  3.6 Analysis on Token Usage

In this section we analyze the token usage efficiency of ElasticAST and
standard ASTs when various length audios are given during training. Due to
fixed-length processing constraints, StandardASTs either cut or pad the
inputs. In contrast, ElasticAST flexibly handles audios of varying lengths
without resorting to cutting or padding operations. However, as described in
Section 2, when packing audios of different lengths into a sequence, we use
padding to standardize the length of the token sequence rows, $L^{\prime}$.
Using the VoxCeleb dataset, our findings are presented in Table 1. The results
reveal that StandardAST introduces non-informative padding tokens, which
occupy 31.1% of the total training tokens, and cut 15.2% of the tokens in
training set. In contrast, ElasticAST minimizes padding and applies no
cutting. Moreover, as batch size increases, the amount of padding tokens used
for packing decreases. This analysis highlights ElasticAST's training
efficiency in processing informative content.

Models-Batch# | Pad Ratio(%) | Cut Ratio(%)
---|---|---
AST-All | 31.1 | 15.2
ElasticAST-12 | 19.2 | 0
ElasticAST-64 | 14.4 | 0
ElasticAST-128 | 13.9 | 0

---

Table 1: Token Usage and Ablation Result.

##  4 Conclusion

In conclusion, this paper focuses on enhancing Audio Spectrogram Transformers
(ASTs) to support training and inference with audios of various lengths. By
introducing a strategy that employs mixed-length training and requires only
minimal architectural adjustments to the AST framework, we develop ElasticAST.
This model is capable of being trained with audios of varying lengths and
demonstrates flexibility without performance loss across different lengths,
offering one single model for all audio lengths and resolutions. ElasticAST's
significance lies in its efficiency and adaptability to different
computational budgets without the need for re-training. The ability to handle
audio of various lengths is becoming increasingly important, considering the
recent developments in self-supervised multimodal learning that utilizes in-
the-wild data. ElasticAST's flexibility is particularly valuable for tasks
involving alignment, retrieval, and generation in multimodal contexts.

## References

  * [1] K. He, X. Zhang, S. Ren, and J. Sun, ``Deep residual learning for image recognition,'' in _Proc. CVPR_ , 2016.
  * [2] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, ``You only look once: Unified, real-time object detection,'' in _Proc. CVPR_ , 2016.
  * [3] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, ``Panns: Large-scale pretrained audio neural networks for audio pattern recognition,'' _TASLP_ , 2020.
  * [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _NeurIPS_ , 2017.
  * [5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani _et al._ , ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in _Proc. ICLR_ , 2021.
  * [6] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, ``Video action transformer network,'' in _Proc. CVPR_ , 2019.
  * [7] J. Lu, D. Batra, D. Parikh, and S. Lee, ``ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,'' in _NeurIPS_ , 2019.
  * [8] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, ``End-to-end object detection with transformers,'' in _Proc. ECCV_ , 2020.
  * [9] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, ``Emerging properties in self-supervised vision transformers,'' in _Proc. ICCV_ , 2021.
  * [10] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, ``Training data-efficient image transformers & distillation through attention,'' in _Proc. ICML_ , 2021.
  * [11] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr _et al._ , ``Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,'' in _Proc. CVPR_ , 2021.
  * [12] Y. Gong, Y.-A. Chung, and J. Glass, ``AST: audio spectrogram transformer,'' in _Proc. Interspeech_ , 2021.
  * [13] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass, ``SSAST: self-supervised audio spectrogram transformer,'' in _Proc. AAAI_ , 2022.
  * [14] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, ``Efficient training of audio transformers with patchout,'' in _Proc. Interspeech_ , 2022.
  * [15] A. Baade, P. Peng, and D. Harwath, ``MAE-AST: masked autoencoding audio spectrogram transformer,'' in _Proc. Interspeech_ , 2022.
  * [16] P.-Y. Huang, H. Xu, J. B. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, ``Masked autoencoders that listen,'' in _NeurIPS_ , 2022.
  * [17] D. Chong, H. Wang, P. Zhou, and Q. Zeng, ``Masked spectrogram prediction for self-supervised audio pre-training,'' _arXiv preprint arXiv:2204.12768_ , 2022.
  * [18] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, ``Attention bottlenecks for multimodal fusion,'' in _NeurIPS_ , 2021.
  * [19] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,'' _TASLP_ , 2021.
  * [20] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _NeurIPS_ , 2020.
  * [21] J. Feng, M. H. Erol, J. S. Chung, and A. Senocak, ``FlexiAST: Flexibility is what AST needs,'' in _Proc. Interspeech_ , 2023.
  * [22] L. Beyer, P. Izmailov, A. Kolesnikov, M. Caron, S. Kornblith, X. Zhai, M. Minderer, M. Tschannen, I. Alabdulmohsin, and F. Pavetic, ``FlexiViT: One model for all patch sizes,'' in _Proc. CVPR_ , 2023.
  * [23] J. Feng, M. H. Erol, J. S. Chung, and A. Senocak, ``From coarse to fine: Efficient training for audio spectrogram transformers,'' in _Proc. ICASSP_ , 2024.
  * [24] M. Dehghani, B. Mustafa, J. Djolonga, J. Heek, M. Minderer, M. Caron, A. Steiner, J. Puigcerver, R. Geirhos, I. M. Alabdulmohsin _et al._ , ``Patch n’pack: NaViT, a vision transformer for any aspect ratio and resolution,'' in _NeurIPS_ , 2024.
  * [25] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, ``Scaling vision transformers,'' in _Proc. CVPR_ , 2022.
  * [26] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, ``AudioSet: An ontology and human-labeled dataset for audio events,'' in _Proc. ICASSP_ , 2017.
  * [27] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, ``VGGSound: A large-scale audio-visual dataset,'' in _Proc. ICASSP_ , 2020.
  * [28] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, ``VoxCeleb: Large-scale speaker verification in the wild,'' _Computer Speech & Language_, 2020.
  * [29] J. Huh, J. Chalk, E. Kazakos, D. Damen, and A. Zisserman, ``EPIC-SOUNDS: A Large-Scale Dataset of Actions that Sound,'' in _Proc. ICASSP_ , 2023.
  * [30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in _Proc. ICLR_ , 2021.
  * [31] K. Lee, M. Joshi, I. R. Turc, H. Hu, F. Liu, J. M. Eisenschlos, U. Khandelwal, P. Shaw, M. Chang, and K. Toutanova, ``Pix2struct: Screenshot parsing as pretraining for visual language understanding,'' in _Proc. ICML_ , 2023.
  * [32] T. Ronen, O. Levy, and A. Golbert, ``Vision transformers with mixed-resolution tokenization,'' in _Proc. CVPR Workshops_ , 2023.
  * [33] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, ``Slow-fast auditory streams for audio recognition,'' in _Proc. ICASSP_ , 2021.
  * [34] X. Liu, H. Liu, Q. Kong, X. Mei, M. D. Plumbley, and W. Wang, ``Simple pooling front-ends for efficient audio classification,'' in _Proc. ICASSP_ , 2023.