# Joint Learning of Policy with Unknown Temporal Constraints for Safe
Reinforcement Learning

Lunet Yifru,1 Ali Baheri 2

###### Abstract

In many real-world applications, safety constraints for reinforcement learning
(RL) algorithms are either unknown or not explicitly defined. We propose a
framework that concurrently learns safety constraints and optimal RL policies
in such environments, supported by theoretical guarantees. Our approach merges
a logically-constrained RL algorithm with an evolutionary algorithm to
synthesize signal temporal logic (STL) specifications. The framework is
underpinned by theorems that establish the convergence of our joint learning
process and provide error bounds between the discovered policy and the true
optimal policy. We showcased our framework in grid-world environments,
successfully identifying both acceptable safety constraints and RL policies
while demonstrating the effectiveness of our theorems in practice.

## Introduction

RL has emerged as a powerful computational approach for training agents to
achieve complex objectives through interactions within stochastic environments
(Sutton and Barto 2018). RL algorithms have demonstrated significant success
in a wide range of applications and domains (Singh, Kumar, and Singh 2022;
Razzaghi et al. 2022). However, when deploying RL policies in real-world
scenarios, particularly those involving safety-critical operations, ensuring
the safety of the learning process becomes a paramount concern. Traditional RL
algorithms tend to focus on reward maximization, which may inadvertently lead
to violation of safety constraints. Safe RL aims to address this challenge by
learning policies that not only maximize the expected return but also respect
safety constraints throughout the learning process. One promising avenue of
research in safe RL involves the use of formal methods, such as temporal
logic, for specifying safety constraints in a mathematically rigorous manner.
The use of temporal logic constraints in the reward function can enable RL
agents to acquire policies that are not only efficient but also secure.
However, this approach assumes the availability of accurate temporal logic
specifications, which may not always be the case, especially in complex real-
world environments. In this brief, we propose a novel framework for jointly
learning RL policies and safety specifications. Our approach combines the
strengths of RL for policy optimization with computational techniques for
discovering temporal logic constraints from data. This joint learning strategy
allows us to efficiently derive an optimal policy and a suitable safety
constraint for a given environment, even in situations where the safety
constraints are not explicitly provided in advance.

## Related Work

Safe RL. Safe RL has garnered significant attention in recent years, as
researchers aim to address safety concerns associated with deploying RL agents
in safety-critical domains (Garcıa and Fernández 2015; Gu et al. 2022; Baheri
et al. 2020; Baheri 2022). A prevalent approach to safe RL involves
formulating the problem as a constrained optimization task, where the primary
objective is to maximize the expected return while satisfying given safety
constraints (Achiam et al. 2017). Another direction in safe RL is risk-
sensitive RL, which aims to balance the trade-off between exploration,
exploitation, and risk management (Mihatsch and Neuneier 2002). Risk-sensitive
RL algorithms incorporate risk measures, such as conditional value-at-risk
(CVaR) (Tamar, Glassner, and Mannor 2014) and risk envelope (Majumdar et al.
2017), to guide the learning process. An additional approach to ensure safety
in RL is through shielding, which intervenes in the agent’s actions when it
might violate safety constraints (Alshiekh et al. 2018). Integrating formal
methods, like temporal logic and Lyapunov-based techniques, into RL algorithms
has emerged as a promising direction for safe RL (Hasanbeig, Abate, and
Kroening 2018; Alur et al. 2023; Chow et al. 2018).

STL Mining. STL has emerged as an essential formalism for specifying complex
temporal properties and constraints in various applications, including
robotics and cyber-physical systems. In recent years, researchers have focused
on inferring or mining STL specifications from data, to facilitate the
development of safe and robust systems. A key approach to mining STL from data
is the use of algorithmic techniques, such as optimization-based algorithms
(Abbas et al. 2014) and machine learning methods (Fronda and Abbas 2022).
Optimization-based techniques seek to minimize an objective function that
captures the distance between the candidate STL formulas and the given data
traces. Data-driven techniques have shown promise in learning STL
specifications from data. Another direction in mining STL from data is the
development of automated, scalable, and robust techniques for the discovery of
interpretable STL specifications (Mohammadinejad et al. 2020). (Bartocci et
al. 2022) provides a comprehensive survey of the various techniques for mining
STL specifications from data.

## Methodology

We cast the joint learning of policy with unknown specifications as a bi-level
optimization problem (Sinha, Malo, and Deb 2017). In this formulation, the
upper level optimization aims to infer the correct STL safety constraint,
while the lower level optimization focuses on learning the optimal policy
under the inferred constraint. A human expert assists the learning by labeling
trajectories based on their safety. In this context, safety is attained when a
trajectory achieves the environmental objective without violating any safety
constraints, i.e., the trace should have a positive robustness value against
the true safety constraint. These components are iteratively called upon to
simultaneously identify the optimal policy and the suitable STL constraint
with the aid of the human expert. The outer loop, an evolutionary algorithm,
is designed to infer both the template and the parameters of an STL
specification that can classify the labeled dataset. This method is inspired
by the work in (Nenzi et al. 2018), which has been shown to result in simpler,
more interpretable outputs, as well as an improved misclassification rate
compared to those in (Bombara and Belta 2021; Kong et al. 2014). The algorithm
implements the following procedures: random generation of the initial STL
population, evaluation of fitness, $\mathcal{F}$, following the Eq. 1, ranking
population members based on fitness, discarding the bottom half of the
population, and applying genetic alterations such as mutation and crossover.
For a positively labeled trace, $X_{p}$, and a negatively labeled trace,
$X_{n}$, in their respective positive and negative datasets, $D_{p}$ and
$D_{n}$, the fitness function is,

$\mathcal{F}(\phi)=N_{\rho(\phi)\textsuperscript{+}\mid
X_{p}}+N_{\rho(\phi)\textsuperscript{-}\mid
X_{n}}+\mid\overline{\rho}(\phi)_{D_{p}}-\overline{\rho}(\phi)_{D_{n}}\mid$
(1)

where, $\mathcal{F}$ is the fitness function for STL $\phi$. The first term in
Eq. 1 represents the number of true positive classifications for the positive
samples, the second term represents the number of true negative
classifications for the negative samples, and the third term computes the
absolute value difference between the average of the robustness values,
$\overline{\rho}(\phi)$, for samples in $D_{p}$ and $D_{n}$.

The inner loop is comprised of a logically-constrained Q-learning in which the
reward is based solely on the robustness of a trajectory throughout an episode
against a given STL constraint. The definition of the reward function is shown
in Eq. 2.

$\mathcal{R}=\begin{cases}\rho(\phi_{[0:T]}),&\text{if
}\rho(\phi_{[0:T]})<0\\\ \rho(\phi_{[0:T]})+100,&\text{if
}\rho(\phi_{[0:T]})\geq 0\end{cases}$ (2)

where, $\mathcal{R}$ is the reward value determined by the robustness degree
$\rho(\phi)$ of the sample $s$ over the horizon of the STL, $T$.

The reward is sparse because it is given at the end of an episode, and not at
each step. This is due to the fact that, with timed STL constraints, the
robustness value cannot be quantified at every step and can only be computed
over a signal at least as long as the horizon, T, of the STL. After training,
the algorithm generates a certain number of rollout traces that are presented
to the human expert for labeling based on their safety, which is our final
component. In our experiments, we have automated the human labeling process by
computing the robustness value of the traces against the true safety
constraint of the environment. However, it is important to note that this is
only done for automation purposes, and as per the basis of our problem, this
true safety constraint is unknown, and traces should actually be labeled by an
expert. The labeled traces are then used as input to the evolutionary
algorithm. This process is repeated iteratively until convergence, which, in
this case, is defined by the number of rollout traces that are labeled safe by
the human expert. This metric is chosen because the safety of the rollout
traces reflects the quality of the STL used as a safety constraint as well as
the quality of the policy generated using that constraint. The framework is
depicted graphically in Fig. 1.

STL Mining (Genetic Algorithm)Policy Learning (Q-learning)Human ExpertSTL
ConstraintRollout TracesLabeled Data

Figure 1: Schematic representation of the integrated framework for
concurrently learning STL constraints and optimal policies. The framework
employs genetic algorithms for STL mining, Q-learning for policy learning, and
incorporates human expert feedback for refining the learned constraints and
policies.

## Results

We consider the problem of implementing RL algorithms in an environment where
the safety constraint is unknown in advance. Specifically, our goal is to
simultaneously infer the correct STL safety constraint and an optimal policy.
To evaluate our framework, we have designed grid-world environments of varying
sizes for an agent to navigate through to reach a goal at a specific location,
under temporal constraints. Initially, neither the goal location nor the time
constraint are known, making it impossible to design a traditional reward
function. The problem was initiated with $1000$ random $2$-dimensional
coordinate traces within the environment, which were then labeled by a human
to create a dataset for the evolutionary algorithm. The algorithm proceeds
with the steps outlined in the methodology until the number of safe traces, as
labeled by the human expert, meets a certain threshold. The experiment was
performed on 6$\times$6, 8$\times$8, and 10$\times$10 grid environments. The
outputs were evaluated based on two metrics: (i) the change in the percentage
of the number of unsafe traces from the first batch of rollout traces to the
last batch and (ii) the average misclassification rate (MCR) of the inferred
STL specification against a dataset labeled by the human expert. The first
metric evaluates how closely the inferred STL specification is able to capture
the true environmental constraints by assessing how the number of unsafe
samples in the rollout traces has decreased over the iterations, indicating
the STL specification is getting closer to the true (but unknown) constraint.
The second metric conveys the classification capability of the inferred STL
against labeled datasets. It quantifies how well the STL distinguishes between
safe and unsafe trajectories, as compared to that of a human expert. The
results are given in Table 1.

Table 1: Percentage of unsafe traces (per rollout sample) at the beginning and the end of the process, and average MCR for inferred STL. Size |  $\%$ of Unsafe Traces | Inferred STL
---|---|---
| First rollout | Last rollout | MCR
6$\times$6 | 73.2$\%$ | 1.2$\%$ | 0.02$\pm$0.0013
8$\times$8 | 86.7$\%$ | 4.3$\%$ | 0.04$\pm$0.001
10$\times$10 | 91.4$\%$ | 11.1$\%$ | 0.06$\pm$0.009

## Theoretical Results

To complement these empirical findings, we now delve into the theoretical
underpinnings of our approach. In this section, we will present the
theoretical results that support the convergence properties and error bounds
of our joint learning framework, providing a more rigorous understanding of
its performance.

### Joint Convergence of the Inner Loop and Outer Loop

Our objective is to demonstrate that the combined convergence of the inner and
outer loops results in the overall convergence of the framework. In essence,
we aim to prove that when the inner loop (Q-learning) reaches an optimal
policy and the outer loop (evolutionary algorithm for STL synthesis) attains
an optimal STL constraint, the entire framework converges to a stable
solution. By establishing these two implications, we can show that the
proposed framework effectively tackles the given problem and converges to a
solution that satisfies both the policy’s optimality and the STL constraint’s
optimality. To prove this joint convergence, we must address the following two
implications:

  * •

Convergence of the inner loop (Q-learning) to an optimal policy $\pi^{\ast}$
implies the convergence of the outer loop (evolutionary algorithm) to an
optimal STL constraint $\phi^{\ast}$.

  * •

Convergence of the outer loop (evolutionary algorithm) to an optimal STL
constraint $\phi^{\ast}$ implies the convergence of the inner loop
(Q-learning) to an optimal policy $\pi^{\ast}$.

Implication 1. This implication aims to show that if the inner loop converges
to an optimal policy, the outer loop converges to an optimal STL constraint.
This is achieved by demonstrating that the fitness function is maximized for
the optimal STL constraint and that the evolutionary algorithm converges to
this optimal constraint under certain conditions. Assume that the inner loop
converges to an optimal policy $\pi^{\ast}$. Under this assumption, we need to
prove that the outer loop converges to an optimal STL constraint
$\phi^{\ast}$. We approach this by showing that:

  * •

The fitness function $\mathcal{F}(\pi^{\ast},\phi)$ is maximized for the
optimal STL constraint $\pi^{\ast}$. This can be done by analyzing the
properties of the fitness function and the reward function
$\mathcal{R}(\pi,\phi)$ under the optimal policy.

  * •

The evolutionary algorithm for STL synthesis converges to the optimal STL
constraint $\pi^{\ast}$ under certain conditions, such as proper selection
pressure, sufficient exploration, and well-defined mutation and crossover
operators.

By showing that the fitness function is maximized for the optimal STL
constraint, and that the evolutionary algorithm converges to this optimal
constraint, we establish the convergence of the outer loop under the
assumption that the inner loop converges to the optimal policy.

Implication 2. This implication aims to show that if the outer loop converges
to an optimal STL constraint, the inner loop converges to an optimal policy.
This is achieved by demonstrating that the reward function provides the
necessary guidance under the optimal STL constraint, and that the Q-learning
algorithm converges to the optimal policy under this guidance. We assume that
the outer loop converges to an optimal STL constraint $\pi^{\ast}$. Under this
assumption, we need to prove that the inner loop converges to an optimal
policy $\pi^{\ast}$. We approach this by showing that:

  * •

The reward function $\mathcal{R}(\pi,\phi^{\ast})$ has the necessary
properties to guide the Q-learning algorithm towards the optimal policy. This
can be done by analyzing the reward function under the optimal STL constraint
and ensuring that it provides proper guidance and exploration-exploitation
trade-off.

  * •

The Q-learning algorithm converges to the optimal policy $\pi^{\ast}$ under
certain conditions, such as proper learning rates, sufficient exploration, and
well-defined state and action spaces.

We demonstrate the convergence of the inner loop by illustrating that the
reward function offers the needed guidance when operating under the optimal
STL constraint, and that the Q-learning algorithm converges to the optimal
policy with this guidance. This convergence is established based on the
assumption that the outer loop successfully converges to the optimal STL
constraint.

#### PROOF of Implication 1.

(a) Maximizing the fitness function for the optimal STL constraint.

Here, we provide a mathematical presentation of Implication 1. Let us first
define the key components of the framework: (i) policy:
$\pi:\mathcal{S}\to\mathcal{A}$, a mapping from states to actions, (ii) reward
function: $\mathcal{R}(\pi,\varphi):\Pi\times\Phi\to\mathbb{R}$, a function
that measures the reward for a given policy $\pi$ and STL constraint
$\varphi$, (iii) fitness function:
$\mathcal{F}(\pi,\varphi):\Pi\times\Phi\to\mathbb{R}$, a function that
measures the fitness of a given policy $\pi$ and STL constraint $\varphi$.
Now, let’s proceed with the proof of Implication 1:

Assume that the inner loop converges to an optimal policy $\pi^{\ast}$. We
want to show that the fitness function $\mathcal{F}(\pi^{\ast},\varphi)$ is
maximized for the optimal STL constraint $\varphi^{\ast}$. Let
$\mathcal{R}^{\ast}=\max_{\pi,\varphi}\mathcal{R}(\pi,\varphi)$ be the maximum
achievable reward. We know that the reward function $\mathcal{R}(\pi,\varphi)$
is maximized for the optimal policy $\pi^{\ast}$ and the optimal STL
constraint $\varphi^{\ast}$, i.e.,
$\mathcal{R}(\pi^{\ast},\varphi^{\ast})=\mathcal{R}^{*}$. We define the
fitness function $\mathcal{F}(\pi,\varphi)$ as a function of the reward
function $\mathcal{R}(\pi,\varphi)$:

$\mathcal{F}(\pi,\varphi)=\frac{R(\pi,\varphi)}{R^{*}}$

since $\mathcal{R}(\pi^{\ast},\varphi^{\ast})=\mathcal{R}^{*}$, we have:

$\mathcal{F}\left(\pi^{*},\varphi^{*}\right)=\frac{\mathcal{R}\left(\pi^{*},\varphi^{*}\right)}{\mathcal{R}^{*}}=\frac{\mathcal{R}^{*}}{\mathcal{R}^{*}}=1$

This result shows that the fitness function $\mathcal{F}(\pi^{\ast},\varphi)$
is indeed maximized for the optimal STL constraint $\varphi^{\ast}$, given
that the inner loop converges to the optimal policy $\pi^{\ast}$. The maximum
value of the fitness function is 1, which occurs when both the policy and the
STL constraint are optimal.

(b) Convergence of the evolutionary algorithm for STL synthesis.

Now, we provide insights into the convergence of the evolutionary algorithm
for STL synthesis. Let $\varphi_{i}$ be the STL constraint at iteration $i$ of
the evolutionary algorithm. We want to show that the evolutionary algorithm
converges to the optimal STL constraint $\varphi^{*}$ under certain
conditions. Let $P(\varphi_{i})$ be the probability distribution of the STL
constraint population at iteration $i$. The evolutionary algorithm updates
$P(\varphi_{i})$ through selection, mutation, and crossover operators. Let
$P_{\text{sel}}(\varphi_{i})$, $P_{\text{mut}}(\varphi_{i})$, and
$P_{\text{cross}}(\varphi_{i})$ be the updated probability distributions after
applying the selection, mutation, and crossover operators, respectively. Then,
the probability distribution at iteration $i+1$ is given by:

$P\left(\varphi_{i+1}\right)=P_{\text{cross }}\left(P_{\text{mut
}}\left(P_{\text{sel }}\left(\varphi_{i}\right)\right)\right)$

Under proper selection pressure, sufficient exploration, and well-defined
mutation and crossover operators, it can be shown that the evolutionary
algorithm converges to the optimal STL constraint $\varphi^{*}$ as the number
of iterations approaches infinity:

$\lim_{i\rightarrow\infty}P\left(\varphi_{i}\right)=\delta\left(\varphi-\varphi^{*}\right)$

where $\delta(\cdot)$ is the Dirac delta function, meaning that the
probability distribution converges to a distribution concentrated on the
optimal STL constraint $\varphi^{\ast}$. Through the demonstration of both
points (a) and (b), we confirm that the outer loop approaches the optimal STL
constraint, denoted as $\varphi^{\ast}$, provided that the inner loop
approaches the optimal policy, denoted as $\pi^{\ast}$. This evidence
indicates that the combined convergence of both the inner and outer loops
results in the comprehensive convergence of the entire framework.

### PROOF of Implication 2.

Assume that the outer loop converges to an optimal STL constraint
$\varphi^{\ast}$. Under this assumption, we need to prove that the inner loop
converges to an optimal policy $\pi^{\ast}$. We approach this by showing that:

  * •

The reward function $\mathcal{R}(\pi,\varphi^{\ast})$ has the necessary
properties to guide the Q-learning algorithm towards the optimal policy. This
can be done by analyzing the reward function under the optimal STL constraint
and ensuring that it provides proper guidance and exploration-exploitation
trade-off. Specifically, we show that $\mathcal{R}(\pi,\varphi^{\ast})$ is
Lipschitz continuous and has a unique maximum at the optimal policy $\pi^{*}$.
Moreover, the reward function should encourage sufficient exploration of the
state-action space while exploiting the knowledge acquired during the learning
process.

  * •

The Q-learning algorithm converges to the optimal policy $\pi^{\ast}$ under
certain conditions, such as proper learning rates, sufficient exploration, and
well-defined state and action spaces. According to the Q-learning convergence
theorem, the Q-learning algorithm converges to the optimal Q-function
$Q^{\ast}(s,a)$ if the following conditions are satisfied:

  1. 1.

Each state-action pair $(s,a)$ is visited infinitely often, i.e.,
$\lim_{t\to\infty}N_{t}(s,a)=\infty$, where $N_{t}(s,a)$ is the number of
visits to the state-action pair $(s,a)$ up to time $t$.

  2. 2.

The learning rate $\alpha_{t}(s,a)$ satisfies
$\sum_{t=1}^{\infty}\alpha_{t}(s,a)=\infty$ and
$\sum_{t=1}^{\infty}\alpha_{t}^{2}(s,a)<\infty$. This condition ensures that
the learning rate decays slowly enough to guarantee convergence.

Assuming that we have the optimal STL constraint $\varphi^{\ast}$, we consider
well-defined state and action spaces, along with an exploration strategy (such
as an $\epsilon$-greedy approach) that ensures each state-action pair is
visited infinitely often. If these conditions hold, the Q-learning algorithm
converges to the optimal policy $\pi^{\ast}$. The convergence of the inner
loop is based on two critical observations: first, the reward function
provides essential guidance when used under the optimal STL constraint, and
second, the Q-learning algorithm moves towards the optimal policy when steered
by this guidance. This convergence is contingent on the outer loop effectively
converging to the optimal STL constraint.

### Bounds on the Error

Deriving bounds on the error between the discovered policy and the true
optimal policy involves analyzing the mathematical relationship between the
error and various factors influencing it. Here’s a possible way to approach
this analysis: Let $\pi^{*}$ be the true optimal policy and $\pi^{\prime}$ be
the discovered policy. Define the error between these policies as:

$\epsilon\left(\pi^{\prime},\pi^{*}\right)=E\left[R\left(s,\pi^{*}(s)\right)-R\left(s,\pi^{\prime}(s)\right)\right]$

where $E[\cdot]$ denotes the expected value, $\mathcal{R}(s,a)$ is the reward
function for taking action $a$ in state $s$, and the expectation is taken over
all states $s$ in the state space. Now, consider the following factors that
may affect the error bounds:

  * •

Granularity of the state abstraction (denoted by $G$): A coarse state
abstraction may lead to a larger error between the discovered policy and the
true optimal policy. The impact of state abstraction granularity on the error
can be represented as:

$\epsilon_{G}(G)\leq C_{1}\cdot G$

where $C_{1}$ is a constant that depends on the problem’s specific
characteristics.

  * •

Quality of the learned STL specifications (denoted by $Q$): If the learned STL
specifications are not accurate or expressive enough, the error between the
discovered policy and the true optimal policy may be larger. The impact of the
quality of the learned STL specifications on the error can be represented as:

$\epsilon_{Q}(Q)\leq C_{2}\cdot(1-Q)$

where $C_{2}$ is a constant that depends on the problem’s specific
characteristics.

  * •

Amount of human feedback provided (denoted by $H$): Human feedback can help
guide the learning process and reduce the error between the discovered policy
and the true optimal policy. The impact of the amount of human feedback on the
error can be represented as:

$\epsilon_{H}(H)\leq C_{3}\cdot e^{-H}$

where $C_{3}$ is a constant that depends on the problem’s specific
characteristics, and $e^{-H}$ represents the exponential decay in error with
increasing human feedback.

Integrating these individual error bounds, we obtain an overall error bound:

$\epsilon\left(\pi^{\prime},\pi^{*}\right)\leq C_{1}\cdot
G+C_{2}\cdot(1-Q)+C_{3}\cdot e^{-H}.$

The error bound reveals how the error is influenced by various contributing
factors, including the level of detail in state abstraction, the accuracy of
the learned STL specifications, and the extent of human feedback received. By
carefully analyzing the error bound, we gain insight into the trade-offs
between these factors. Armed with this knowledge, we can formulate strategies
that minimize the error and ultimately improve the effectiveness of the bi-
level optimization framework.

## Conclusions

In this paper, we have studied a joint learning framework for the safety
constraint and the RL policy of an environment where the safety constraints
are unknown _a priori_. We have implemented an algorithm that optimizes the
safety constraint and the RL policy simultaneously. Our preliminary results
have shown that our framework is capable of identifying safety constraints
that are suitable for the environment and an optimal RL policy that results in
safe behavior. Future directions for this work will include testing our
algorithm in complex environments, assessing adaptability, and improving the
algorithm’s computational efficiency.

## References

  * Abbas et al. (2014) Abbas, H.; Winn, A.; Fainekos, G.; and Julius, A. A. 2014.  Functional gradient descent method for Metric Temporal Logic specifications.  In _2014 American Control Conference_ , 2312–2317.
  * Achiam et al. (2017) Achiam, J.; Held, D.; Tamar, A.; and Abbeel, P. 2017.  Constrained policy optimization.  In _International conference on machine learning_ , 22–31. PMLR.
  * Alshiekh et al. (2018) Alshiekh, M.; Bloem, R.; Ehlers, R.; Könighofer, B.; Niekum, S.; and Topcu, U. 2018.  Safe reinforcement learning via shielding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Alur et al. (2023) Alur, R.; Bansal, S.; Bastani, O.; and Jothimurugan, K. 2023.  Specification-Guided Reinforcement Learning.
  * Baheri (2022) Baheri, A. 2022.  Safe reinforcement learning with mixture density network, with application to autonomous driving.  _Results in Control and Optimization_ , 6: 100095.
  * Baheri et al. (2020) Baheri, A.; Nageshrao, S.; Tseng, H. E.; Kolmanovsky, I.; Girard, A.; and Filev, D. 2020.  Deep reinforcement learning with enhanced safety for autonomous highway driving.  In _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 1550–1555. IEEE.
  * Bartocci et al. (2022) Bartocci, E.; Mateis, C.; Nesterini, E.; and Nickovic, D. 2022.  Survey on mining signal temporal logic specifications.  _Information and Computation_ , 289: 104957.
  * Bombara and Belta (2021) Bombara, G.; and Belta, C. 2021.  Offline and Online Learning of Signal Temporal Logic Formulae Using Decision Trees.  _ACM Transactions on Cyber-Physical Systems_ , 5: 1–23.
  * Chow et al. (2018) Chow, Y.; Nachum, O.; Duenez-Guzman, E.; and Ghavamzadeh, M. 2018.  A lyapunov-based approach to safe reinforcement learning.  _Advances in neural information processing systems_ , 31.
  * Fronda and Abbas (2022) Fronda, N.; and Abbas, H. 2022.  Differentiable Inference of Temporal Logic Formulas.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 41: 1–1.
  * Garcıa and Fernández (2015) Garcıa, J.; and Fernández, F. 2015.  A comprehensive survey on safe reinforcement learning.  _Journal of Machine Learning Research_ , 16(1): 1437–1480.
  * Gu et al. (2022) Gu, S.; Yang, L.; Du, Y.; Chen, G.; Walter, F.; Wang, J.; Yang, Y.; and Knoll, A. 2022.  A review of safe reinforcement learning: Methods, theory and applications.  _arXiv preprint arXiv:2205.10330_.
  * Hasanbeig, Abate, and Kroening (2018) Hasanbeig, M.; Abate, A.; and Kroening, D. 2018.  Logically-constrained reinforcement learning.  _arXiv preprint arXiv:1801.08099_.
  * Kong et al. (2014) Kong, Z.; Jones, A.; Ayala, A.; Gol, E.; and Belta, C. 2014.  Temporal logic inference for classification and prediction from data.  273–282.
  * Majumdar et al. (2017) Majumdar, A.; Singh, S.; Mandlekar, A.; and Pavone, M. 2017.  Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models.  In _Robotics: science and systems_ , volume 16, 117.
  * Mihatsch and Neuneier (2002) Mihatsch, O.; and Neuneier, R. 2002.  Risk-sensitive reinforcement learning.  _Machine learning_ , 49: 267–290.
  * Mohammadinejad et al. (2020) Mohammadinejad, S.; Deshmukh, J. V.; Puranic, A. G.; Vazquez-Chanlatte, M.; and Donzé, A. 2020.  Interpretable classification of time-series data using efficient enumerative techniques.  In _Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control_ , 1–10.
  * Nenzi et al. (2018) Nenzi, L.; Silvetti, S.; Bartocci, E.; and Bortolussi, L. 2018.  A Robust Genetic Algorithm for Learning Temporal Specifications from Data.
  * Razzaghi et al. (2022) Razzaghi, P.; Tabrizian, A.; Guo, W.; Chen, S.; Taye, A.; Thompson, E.; Bregeon, A.; Baheri, A.; and Wei, P. 2022.  A Survey on Reinforcement Learning in Aviation Applications.  _arXiv preprint arXiv:2211.02147_.
  * Singh, Kumar, and Singh (2022) Singh, B.; Kumar, R.; and Singh, V. P. 2022.  Reinforcement learning in robotic applications: a comprehensive survey.  _Artificial Intelligence Review_ , 1–46.
  * Sinha, Malo, and Deb (2017) Sinha, A.; Malo, P.; and Deb, K. 2017.  A review on bilevel optimization: From classical to evolutionary approaches and applications.  _IEEE Transactions on Evolutionary Computation_ , 22(2): 276–295.
  * Sutton and Barto (2018) Sutton, R. S.; and Barto, A. G. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Tamar, Glassner, and Mannor (2014) Tamar, A.; Glassner, Y.; and Mannor, S. 2014.  Policy gradients beyond expectations: Conditional value-at-risk.  _arXiv preprint arXiv:1404.3862_.


# LIBRIHEAVY: A 50,000 HOURS ASR CORPUS WITH PUNCTUATION CASING AND CONTEXT

###### Abstract

In this paper, we introduce Libriheavy, a large-scale ASR corpus consisting of
50,000 hours of read English speech derived from LibriVox. To the best of our
knowledge, Libriheavy is the largest freely-available corpus of speech with
supervisions. Different from other open-sourced datasets that only provide
normalized transcriptions, Libriheavy contains richer information such as
punctuation, casing and text context, which brings more flexibility for system
building. Specifically, we propose a general and efficient pipeline to locate,
align and segment the audios in previously published Librilight to its
corresponding texts. The same as Librilight, Libriheavy also has three
training subsets small, medium, large of the sizes 500h, 5000h, 50000h
respectively. We also extract the dev and test evaluation sets from the
aligned audios and guarantee there is no overlapping speakers and books in
training sets. Baseline systems are built on the popular CTC-Attention and
transducer models. Additionally, we open-source our dataset creatation
pipeline which can also be used to other audio alignment tasks.

Index Terms—  Speech recognition, Corpus, Audio alignment, Librivox

##  1 Introduction

In the past decade, various system architectures, like Connectionist Temporal
Classification (CTC) [1], RNN-T [2] and encoder-decoder based model [3], have
been proposed, pushing the dominant framework from the hybrid Hidden Markov
Models (HMM) [4] to end-to-end models. In general, the neural network models
are supposed to be more data hungry than traditional systems.

A lot of work has been done on publishing open source datasets, for example,
the Wall Street Journal corpus [5], SwitchBoard [6], Fisher [7] and the famous
LibriSpeech corpus [8]. While these are all small or medium size datasets with
less than 2,000 hours, which is too small to train a good enough end-to-end
model. In recent years, there are also large-scale corpora like GigaSpeech
[9], People’s Speech [10] and MLS [11]. One drawback of these datasets is that
they only provide normalized transcriptions, making it impossible to train a
model that needs full-format texts, such as punctuation prediction.

Typical ASR corpora aim at training ASR systems to recognize independent
utterances. However, the preceding context of the current utterance may convey
useful information. Contextualized speech recognition utilizes the cross-
utterance context to improve the accuracy of ASR systems and yields promising
results [12, 13]. However, training such systems usually requires utterance-
level context for each training utterance, which is not available in most
existing ASR corpora. Therefore, such a dataset with textual context
information is highly desirable.

Motivated by the aforementioned points, we introduce Libriheavy, a large-scale
(50,000 hours) corpus containing not only fully formatted transcripts but also
textual context, which is suitable for various speech recognition related
tasks. In addition, unlike other open-source datasets that have their own
creating pipelines, we propose a general audio alignment method and release it
as a standard package. Our contributions are as follows:

  * •

We release a 50,000-hour of labeled audio containing punctuation casing and
preceding text;

  * •

We propose and open-source a general audio alignment pipeline, which makes it
easier to construct ASR corpora;

  * •

We provide solid evaluation results on Libriheavy, which demonstrate the high
quality of the corpus and the robustness of our pipeline.

##  2 Libriheavy corpus

In this section, we provide a detailed description of the Libriheavy corpus,
including audio files, metadata, data partitions, text styles, and other
aspects. Instructions and scripts are available in the Libriheavy GitHub
repository111https://github.com/k2-fsa/libriheavy.

###  2.1 Librilight

Librilight [14] is a collection of unlabeled spoken English audio derived from
open-source audio books from the LibriVox project 222https://librivox.org. It
contains over 60,000 hours of audio and aims for training speech recognition
systems under limited or no supervision. The corpus is free and publicly
available 333https://github.com/facebookresearch/libri-light.

###  2.2 Libriheavy

Libriheavy is a labeled version of Librilight. We align the audio files in
Librilight to their corresponding text in the original book and segment them
into smaller pieces with durations ranging from 2 to 30 seconds. We maintain
the original dataset splits of Librilight and have three training subsets
(small, medium, large). In addition, we further extract evaluation subsets
(dev, test-clean, test-other) for validation and testing. Table 1 shows the
statitics of these subsets.

Table 1: The dataset statistics of Libriheavy. subset | hours | books | per-spk hrs | total spks
---|---|---|---|---
small | 509 | 173 | 1.22 | 417
medium | 5042 | 960 | 3.29 | 1531
large | 50794 | 8592 | 7.54 | 6736
dev | 22.3 | 180 | 0.16 | 141
test-clean | 10.5 | 87 | 0.15 | 70
test-other | 11.5 | 112 | 0.16 | 72

####  2.2.1 Metadata

We save the metadata of the dataset as Lhotse [15] cuts in JSON lines. Each
line is a self-contained segment, including the transcript and its audio
source. Users can clip the corresponding audio segment with the given start
and duration attributes. Unlike other publicly available corpora that only
provide normalized transcripts, Libriheavy includes richer information such as
punctuation, casing, and text context. The text context is the transcription
of the preceding utterances, located in the pre_texts entry, with a default
length of 1000 bytes. There are also begin_byte and end_byte attributes, which
allow users to easily slice any length of text context from the original book
pointed to by the text_path attribute. Of course, there are other
supplementary entries that might be usefull for other tasks, such as id,
speaker, etc.

####  2.2.2 Evaluation Sets

As mentioned above, we have three evaluation sets in Libriheavy, namely dev,
test-clean, test-other. We ensure that the evaluation sets have no overlapping
speakers and books in the training set. To make the evaluation sets contain as
many speakers and books as possible while not dropping out too much training
data, we filtered out speakers and books with shorter durations as candidates.
We then determine the clean speakers and other speakers using the same method
as in [8] and divide the candidates into clean and other pool. We randomly
select 20 hours of audio from the clean pool, half of which forms the test-
clean set and the other half is appended to the dev set. We follow the same
procedure for the other pool. Librilight ensures that audio files from the
LibriSpeech evaluation sets are not present in the corpus, therefore, the
LibriSpeech evaluation sets can also be used as our evaluation sets.

##  3 Audio Alignment

This section describes the creation pipeline of the Libriheavy corpus. The key
task of audio alignment is to align the audio files to the corresponding text
and split them into short segments, while also excluding segments of audio
that do not correspond exactly with the aligned text. Our solution presented
here is a general pipeline that can be applied to other data generation tasks
as well. The implementation of all the following algorithms and corresponding
scripts are publicly available444https://github.com/k2-fsa/text_search.

###  3.1 Downloading text

To align the audio derived from audiobooks, we require the original text from
which the speaker read the audiobook. From the metadata provided by
Librilight, we can obtain the URL of the textbook for each audio file. We have
written scripts to automatically extract the text and download the sources for
all audiobooks. We then apply simple clean-up procedures such as removing
redundant spaces and lines to the text sources.

###  3.2 First alignment stage

The goal of this stage is to locate the audio to its corresponding text
segments (e.g. chapter) in the original book. First, we obtain the automatic
transcript of the audio file. Then we treat the automatic transcript as query
and the text in the original book as target 555We will normalize the text to
upper case and remove the punctuation, but keep the index into original text.,
and find the close matches (Sec 3.2.2) for each elements in the query over the
target. Finally, we determine the text segment of the audio by finding the
longest increasing pairs (Sec 3.2.3) of query elements and their close
matches. Note, we did not use the VAD tool provided by Librilight for audio
segmenting and as our algorithm requires a relatively long text to guarantee
its accuracy.

####  3.2.1 Transcribe audios

The audios in Librilight have a large variance in duration, from a few minutes
to hours. To avoid excessive computation on long audio files, we first split
the long audio into 30-second segments with 2 seconds of overlap at each side,
and then recognize these segments with an ASR model trained on Librispeech.
Finally, we combine the transcripts that belong to the same audio by
leveraging the timestamps of the recognized words.

####  3.2.2 Close matches

Now we have the automatic transcript and the original book for each audio. To
obtain the most similar text segment in the original book of the automatic
transcript roughly, we propose the close matches. First, we concatenate query
and target to a long sequence (target follows query), then a suffix array is
constructed on the sequence using the algorithm in [16]. The close matches of
the element in query position $i$ is defined as two positions in the original
sequence that are within the target portion, and which immediately follow and
precede, in the suffix array, query position $i$. This means that the suffixes
ending at those positions are reverse-lexicographically close to the suffix
ending at position $i$. Figure 1 shows a simple example of finding the close
matches of query “LOVE” over target “ILOVEYOU”.

Fig. 1: Example of finding close matches for a query (LOVE) over the target
(ILOVEYOU). The dash arrows point from the query elements to their close
matches.

####  3.2.3 Longest increasing pairs

Let us think of those close matches which we obtained above as a set of
$\left(i,j\right)$ pairs, where $i$ is an index into query sequence and $j$ is
an index into the target sequence. The query and its corresponding segment in
the target should be monotonic aligned, so we can get the approximate
alignment between the two sequences by finding the longest chain of pairs:
$\left(i_{1},j_{1}\right),\left(i_{2},j_{2}\right),...\left(i_{N},j_{N}\right)$,
such that $i_{1}<=i_{2}<=...<=i_{N}$, and $j_{1}<=j_{2}<=...<=j_{N}$.

###  3.3 Second alignment stage

From the longest chain obtained from the previous step, we can roughly locate
the region in the target sequence relative to the query. At this stage, we use
the Levenshtein alignment [17] to find the best single region of alignment
between the recognized audio (query) and the text segment (obtained by the
longest chain pairs). Since Levenshtein alignment is a quadratic time
complexity algorithm, and will be very inefficient for long sequences. We can
use the traceback through the pairs in the longest chain as the backbone for
the Levenshtein alignment, so that we limit the Levenshtein alignment into
blocks defined by the $\left(i,j\right)$ positions in this traceback. By
concatenating the Levenshtein alignments of all the blocks along the query
index, we obtain the Levenshtein alignment of the whole query.

###  3.4 Audio segmentation

The goal of audio segmentation is to break long audio into shorter segments,
ranging from 2 seconds to 30 seconds, which are more suitable for ASR
training. We use a two-stage scoring method to search for good segmentations
666The scores mentioned below will be normalized to the same scale, so none of
the scores would dominate the final score.. All books in LibriVox have
punctuation, so we decided to split the sentence only at punctuation
indicating the end of a sentence, namely, “.”, “?” and “!” 777Our toolkit also
supports splitting sentences at a certain threshold of silence.. We select the
positions in the alignment that follow chosen punctuations as Begin Of a
Segment (BOS) and the positions followed by chosen punctuations as End Of a
Segment (EOS), then we compute scores for these positions:

  * •

The number of silence seconds this position follows or is followed by, up to 3
seconds.

  * •

The score corresponding to the number of insertions, deletions and
substitutions within a certain region of this position.

Each pair of BOS and EOS forms a segment. The following rule is applied to
assign scores to potential segments:

  * •

The score of BOS plus the score of EOS.

  * •

A score related to the duration of the segment, which guarantees the duration
is in the range of 2 to 30 seconds and encourages a duration between 5 to 20
seconds.

  * •

A bonus for the number of matches in the alignment.

  * •

A penalty for the number of errors in the alignment.

For each BOS, we find the 4 best-scoring EOS and vice versa. We then append
the preceding 2 sets of segments to get a list of candidate segments. We
determine the best segmentations by getting the highest-scoring set of
segments that do not overlap. In practise, to avoid dropping out too much
audio, we allow some kind of overlap if the overlapping length is less than a
quarter of the segment.

##  4 Experiments

In this section, we present the baseline systems and experimental results for
two popular models, namely CTC-Attention [18] and neural transducer [2]. We
then compare the performance between the models trained on normalized text and
texts with punctuation and casing.

###  4.1 CTC-Attention baseline system

We build the CTC-Attention baseline using the Wenet [19] framework. We use the
classic setup of Wenet toolkit which consists of a 12-layer Conformer [20]
encoder and a 6-layer Transformer decoder. The embedding dimension is set to
512. The kernel size of the convolution layers is set to 31. The feedforward
dimension is set to 2048. The modeling units are 500-class Byte Pair Encoding
(BPE) [21] word pieces. The loss function is a logarithmic linear combination
of the CTC loss (weight = 0.3) and attention loss with label smoothing (weight
= 0.1). The input features are 80-channel Fbank extracted on 25 ms windows
shifted by 10 ms with dither equal 0.1. SpecAugment [22] and on-the-fly Speed
perturbation [23] are also applied to augment the training data. During
training, we use the Adam optimizer [24] with the maximum learning rate of
0.002. We use the Noam [25] learning rate scheduler with 25k warm-up steps.

The model is trained for 90, 60 and 15 epochs on the small, medium and large
subsets, respectively. Table 2 shows the Word Error Rate (WER) of the models
on Libriheavy test sets. As a reference, we also show the WER on the
LibriSpeech test sets. The N-Best hypotheses are first generated by the CTC
branch and then rescored by the attention branch. Note that for the
LibriSpeech results, we apply some simple text normalization, such as
converting numbers to their corresponding text and converting abbreviations
(e.g “Mr.” to “Mister”) on the hypotheses to make it compatible with the
LibriSpeech transcripts. We also apply these normalization procedures in the
following experiments.

Table 2: The WERs of LibriSpeech (ls) and Libriheavy (lh) test sets on CTC-Attention system. subset | ls-clean | ls-other | lh-clean | lh-other
---|---|---|---|---
small | 5.76 | 15.60 | 6.94 | 15.17
medium | 3.15 | 7.88 | 3.80 | 8.80
large | 2.02 | 5.22 | 2.74 | 6.68

###  4.2 Transducer baseline system

We build the transducer baseline system using the icefall
888https://github.com/k2-fsa/icefall which is one of the projects in the Next-
gen Kaldi toolkit. Icefall implements a transformer-like transducer system,
which consists of a encoder and a stateless decoder [26]. Different from the
setting in [26] which only has an embedding layer, an extra 1-D convolution
layer with a kernel size of 2 is added on top of it. The encoder used in this
baseline is a newly proposed model called Zipformer. The Zipformer paper has
not been released yet, but the implementation details and training pipeline
can be found in icefall
999https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/zipformer/zipformer.py.
We use the default setting in the Zipformer LibriSpeech recipe in icefall for
all the following experiments.

The same as CTC-Attention baseline system, we train the model for 90, 60 and
15 epochs for the small, medium and large subsets, respectively. Table 3 shows
the decoding results of the models trained on different training subsets, and
the WERs on LibriSpeech and Libriheavy test sets are presented. We use the
beam search method proposed in [27] which limits the maximum symbol per frame
to one to accelerate the decoding.

Table 3: The WERs of LibriSpeech (ls) and Libriheavy (lh) test sets on transducer system. subset | ls-clean | ls-other | lh-clean | lh-other
---|---|---|---|---
small | 4.05 | 9.89 | 4.68 | 10.01
medium | 2.35 | 4.82 | 2.90 | 6.57
large | 1.62 | 3.36 | 2.20 | 5.57

###  4.3 Training with punctuation and casing

This section benchmarks the performance of models trained on texts with
punctuation and casing, and compares them with the performance of models
trained on normalized texts. The system setting is almost the same as the
transducer baseline system mentioned above. The only difference is that we
adopt 756-class BPE word pieces rather than 500 for modeling, because we open
the fallback_bytes flag when training the BPE model to handle rare characters,
so we need an additional 256 positions for bytes. Table 4 shows the WERs and
Char Error Rate (CER) of models trained on texts with punctuation and casing.

Table 5 compares the results of systems trained on normalized texts (upper
case without punctuation) and unnormalized texts (casing with punctuation). In
this experiment, we normalized both the transcripts and decoding results to
upper case and removed the punctuation when calculating the WERs. From the
results, the performance gap between two types of training texts is large when
the training set is small, but as the training set grows, the gap becomes
negligible. This indicates that when the training set is large enough, the
style of training texts will not make much difference on performance, while
training with texts with punctuation and casing brings us more information and
flexibilities.

Table 4: The Libriheavy WERs and CERs on transducer system trained on texts with punctuation and casing. subset | WER | CER
---|---|---
lh-clean | lh-other | lh-clean | lh-other
small | 13.04 | 19.54 | 4.51 | 7.90
medium | 9.84 | 13.39 | 3.02 | 5.10
large | 7.76 | 11.32 | 2.41 | 4.22
Table 5: The comparison of WERs between models trained on Upper case No
Punctuation (UNP) and Casing with Punctuation (C&P).

subset text ls-clean ls-other lh-clean lh-other small UNP 4.05 9.89 4.68 10.01
C&P 4.51 10.84 5.16 11.12 medium UNP 2.35 4.82 2.90 6.57 C&P 2.45 5.03 3.05
6.78 large UNP 1.62 3.36 2.20 5.57 C&P 1.72 3.52 2.28 5.68

##  5 Conclusion

We release a large-scale (50,000 hours) corpus containing punctuation, casing
and text context, which can be used in various of ASR tasks. We also propose
and open-source a general and efficient audio alignment toolkit, which makes
constructing speech corpora much easier. Finally, we conduct solid experiments
on the released corpus, and the results show that our corpus is of high
quality and demonstrates the effectiveness of our creation pipeline.

## References

  * [1] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proc. ICML, 2006.
  * [2] A. Mohamed A. Graves and G. Hinton,  “Speech recognition with deep recurrent neural networks,”  in Proc. ICASSP, Vancouver, 2013.
  * [3] William Chan, Navdeep Jaitly, et al.,  “Listen, attend and spell,”  in Proc. ICASSP, Shanghai, 2016.
  * [4] George E Dahl, Dong Yu, Li Deng, and Alex Acero,  “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,”  IEEE/ACM Trans. on Audio, Speech, and Language Processing, 2011\.
  * [5] Douglas B. Paul and Janet M. Baker,  “The design for the Wall Street Journal-based CSR corpus,”  in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, 1992.
  * [6] Godfrey, John J., and Edward Holliman.,  “Switchboard-1 release 2 ldc97s62.,” 1993.
  * [7] Cieri, Christopher, et al.,  “Fisher english training speech part 1 transcripts ldc2004t19.,” 2004\.
  * [8] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an ASR corpus based on public domain audio books,”  in Proc. ICASSP, Brisbane, 2015.
  * [9] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, et al.,  “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,”  in Proc. Interspeech, Brno, 2021.
  * [10] Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, et al.,  “The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,” 2021.
  * [11] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert,  “MLS: A large-scale multilingual dataset for speech research,”  in Proc. Interspeech, 2020.
  * [12] Kai Wei, Thanh Tran, Feng-Ju Chang, et al.,  “Attentive contextual carryover for multi-turn end-to-end spoken language understanding,”  in Proc. ASRU, 2021.
  * [13] Shuo-Yiin Chang, Chao Zhang, Tara N Sainath, Bo Li, and Trevor Strohman,  “Context-aware end-to-end ASR using self-attentive embedding and tensor fusion,”  in Proc. ICASSP, Rhodes, 2023.
  * [14] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, et al.,  “Libri-light: A benchmark for asr with limited or no supervision,”  in Proc. ICASSP, Barcelona, 2020.
  * [15] Piotr Żelasko, Daniel Povey, Jan ”Yenda” Trmal, and Sanjeev Khudanpur,  “Lhotse: a speech data representation library for the modern deep learning ecosystem,” 2021.
  * [16] Juha Kärkkäinen, Peter Sanders, and Stefan Burkhardt,  “Linear work suffix array construction,”  Journal of the ACM, 2006.
  * [17] Vladimir I Levenshtein et al.,  “Binary codes capable of correcting deletions, insertions, and reversals,”  in Soviet physics doklady. Soviet Union, 1966.
  * [18] Shinji Watanabe et al.,  “Hybrid CTC/attention architecture for end-to-end speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, 2017.
  * [19] Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, et al.,  “Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit,”  in Proc. Interspeech, 2021.
  * [20] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, et al.,  “Conformer: Convolution-augmented Transformer for Speech Recognition,”  in Proc. Interspeech, Shanghai, 2020.
  * [21] Rico Sennrich, Barry Haddow, and Alexandra Birch,  “Neural machine translation of rare words with subword units,”  in Proc. ACL, Berlin, 2016.
  * [22] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, et al.,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  in Proc. Interspeech, Graz, 2019.
  * [23] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “Audio augmentation for speech recognition,”  in Proc. Interspeech, Dresden, 2015.
  * [24] D. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  Computer Science, 2014.
  * [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, et al.,  “Attention is all you need,”  in Proc. NIPS, Long Beach, 2017.
  * [26] Mohammadreza Ghodsi, Xiaofeng Liu, James Apfel, et al.,  “RNN-transducer with stateless prediction network,”  in Proc. ICASSP, Barcelona, 2020.
  * [27] Wei Kang, Liyong Guo, Fangjun Kuang, Long Lin, Mingshuang Luo, Zengwei Yao, Xiaoyu Yang, Piotr Żelasko, and Daniel Povey,  “Fast and parallel decoding for transducer,”  in Proc. ICASSP, Rhodes, 2023.