# Bridging Intelligence and Instinct: A New Control Paradigm for Autonomous
Robots

Shimian Zhang1 Qiuhong Lu2 1Shimian Zhang is with Shanghai Hrstek Co., Ltd
<EMAIL_ADDRESS>Lu is with Shanghai Hrstek Co., Ltd
<EMAIL_ADDRESS>
###### Abstract

As the advent of artificial general intelligence (AGI) progresses at a
breathtaking pace, the application of large language models (LLMs) as AI
Agents in robotics remains in its nascent stage. A significant concern that
hampers the seamless integration of these AI Agents into robotics is the
unpredictability of the content they generate, a phenomena known as
“hallucination”. Drawing inspiration from biological neural systems, we
propose a novel, layered architecture for autonomous robotics, bridging AI
agent intelligence and robot instinct. In this context, we define Robot
Instinct as the innate or learned set of responses and priorities in an
autonomous robotic system that ensures survival-essential tasks, such as
safety assurance and obstacle avoidance, are carried out in a timely and
effective manner. This paradigm harmoniously combines the intelligence of LLMs
with the instinct of robotic behaviors, contributing to a more safe and
versatile autonomous robotic system. As a case study, we illustrate this
paradigm within the context of a mobile robot, demonstrating its potential to
significantly enhance autonomous robotics and enabling a future where robots
can operate independently and safely across diverse environments.

##  I Introduction

The rapid evolution of artificial general intelligence (AGI) technologies,
particularly large language models (LLMs) like GPT-4 [15] and LLaMA [19], has
catalyzed a new wave of potential applications within the realm of robotics
[20]. As AI agents, LLMs can generate high-level decisions and instructions
that guide a robot’s behavior , demonstrating potential in tool utilization
[11, 17], task planning [13],and task creation [4]. However, despite the
promises, there remains a dearth of substantive applications within robotics,
largely due to the unpredictability of LLMs’ outputs, a phenomenon often
referred to as “hallucination” [15, 2].

Figure 1: Layered Hierarchy Design. The pyramid structure of our proposed
architecture, comprising four layers: External, Decision, Instinct, and
Device. The External layer represents high-level entities interacting with the
system, the Decision layer acts as the ‘brain’ making high-level decisions,
the Instinct layer continuously maintains safety acting as the ‘brainstem,’
and the Device layer executes the commands by controlling the robot’s physical
actions.

Mitigation strategies such as Chain-of-Thoughts [8], Self-Reflection [18] and
etc., have been proposed to manage this issue, but these interventions target
the LLMs themselves rather than considering potential architectural solutions
within the robotic system. Traditionally, robotic control systems have adhered
to either cognitive models [9], which mimic human cognitive processes, or
behavior-based models that produce direct responses to sensory inputs [3].
Yet, none of these traditional architectures have fully considered the
incorporation and interaction of AI agents.

In response to these challenges, we introduce a novel architecture for robotic
control systems designed to bridge high-level intelligence with low-level
instinctual protocols, as Fig. 1 shows. Inspired by the human nervous system’s
’brain and brainstem’ paradigm, this architecture proposes four distinct
layers: External, Decision, Instinct, and Device. This new paradigm treats the
AI agent as the ’brain,’ handling advanced decision-making, while a ’Robot
Instinct’ module acts as the ’brainstem,’ overseeing basic survival-essential
tasks.

Our architectural design presents a systemic solution to the hallucination
risks posed by LLM-based AI agents. By integrating high-level decision-making
AGIs with robust low-level safety mechanisms, we limit the harm of potential
incorrect decisions. Moreover, we emphasize the need for robots, even as they
gain sophisticated AGI capabilities, to retain robust instinctual reactions
akin to human survival instincts, ensuring they can effectively serve in
various tasks and environments.

The primary contribution of this paper is the introduction of this innovative
four-layered architecture for robotic control systems. We demonstrate its
efficacy and versatility through a specific case study, marking a significant
stride towards a new era of robotics. We envision this approach not only
propelling the development of robotic control systems but also fostering a
tighter synergy between AI and robotics, empowering more nuanced interaction
between robots and their environment.

##  II Related Works

###  II-A AI Agents for Robotics

The application of AI agents in robotics has become a burgeoning field of
research with the advancement of machine learning techniques, especially Large
Language Models (LLMs).

####  II-A1 LLMs as AI Agents

Recent research findings indicate that LLMs can be effectively used as AI
agents to address complex tasks in robotics [4, 11, 17, 13]. Such capabilities
enable LLMs to interact with both structured and unstructured environments,
making them valuable for the development of versatile and adaptable robotic
systems.

####  II-A2 AI Agents as the Robot Brain

Despite these advancements, the application of AI agents as the primary
”brain” of a robot is still in the early stages [12], largely limited by the
complexities of integrating AI with physical systems. Some work has integrated
models like Chat-GPT into a robot’s operational flow [20], albeit in a
simulated environment with the need for human oversight to ensure accuracy and
appropriateness of the AI’s outputs.

In light of these challenges and opportunities, our proposed design framework
represents a pioneering step towards fully integrating AI agents into robotic
systems. We view this as an inevitable trend in the evolution of robotics,
given the increasing demand for intelligent and autonomous operation in real-
world environments. Our approach aims to create a truly anthropomorphic
robotic entity, capable of sophisticated interaction with the world while
ensuring the safety and reliability of operations.

###  II-B Robot System Architectures

Historically, the Sense-Plan-Act (SPA) architecture was proposed as a tiered
model with distinct modules [16], each responsible for a specific function or
task. While this top-down architecture was effective in dividing labor, it was
also inherently sequential, resulting in difficulties in rapidly adapting to
dynamic environments.

An alternative approach was put forth by [3], who proposed the Subsumption
Architecture. This model popularized Behavior-Based Robotics (BBR), which
emphasized decentralized control and modularity. Complex behaviors were seen
as emergent from the interaction of simple, concurrently operating modules.
However, the practical implementation of this architecture has proven to be
complex, particularly in intelligent robots where the decoupling of high-level
and low-level behaviors can be challenging.

The Hybrid Architecture was then introduced to bridge the strengths of the SPA
and BBR architectures. In a hybrid model, a robot’s behavior can be generated
either by a central plan (akin to SPA) or by a set of independent behavior
modules (akin to BBR). [5] proposed a notable instance of this paradigm, a
three-tiered architecture consisting of a reactive Controller layer, a
reactive Sequencer layer, and a Deliberator layer.

Our proposed paradigm control architecture could be seen as a new form of
Hybrid Architecture, introducing unique components designed to maximize the
benefits of recent advancements in AI. It innovatively blends the deliberative
power of AI Agent decision-making capabilities with the reactivity of an
Instinct layer witth low-level safety mechanisms, coupled with a clear
definition of human-machine interactions within the External Layer.

###  II-C Safety Mechanisms

Safety mechanisms are crucial to the design and implementation of any robotic
system, ensuring safe interactions with the environment and reliable execution
of tasks. Various techniques have been developed over the years.

####  II-C1 Model-Based Safety Mechanisms

Techniques such as Control Lyapunov Function (CLF) and Control Barrier
Function (CBF) have been extensively used to enforce safety in robotic systems
[1, 14, 6]. These techniques are highly effective in systems where precise
mathematical models of the system dynamics and environment are known. However,
the performance of these methods can be significantly degraded if the model is
inaccurate, or in the presence of unknown system dynamics.

####  II-C2 Model-Free Safety Control

With the advent of powerful Reinforcement Learning (RL) algorithms, model-free
safety control mechanisms have emerged as promising solutions on complex and
high-dimensional systems [7, 10]. These mechanisms are particularly useful
when dealing with unknown environments or unmodeled system dynamics. However,
RL-based mechanisms typically require a large amount of sample data for
training and can be time-consuming. Such mechanism are usually not easy to
migrate from one robot platform to another.

####  II-C3 AI Agents Safety Concerns

Given the rapidly growing capabilities of LLMs, it is imperative to address
the potential safety concerns that come with their integration into robotic
systems. Techniques such as Chain of Thoughts [8], Self-Reflection [18], and
human-in-the-loop mechanisms [20] have been used to prevent or reduce the rate
of ”hallucination”. However, these methods do not entirely eliminate the
propensity of LLMs to generate incorrect responses due to their inherent
nature.

Our architecture distinctly addresses these safety concerns through two
primary mechanisms: First, our Instinct Layer operating independent,
uninterrupted safety mechanisms, effectively caters to safety requirements.
The implementation of these safety protocols can leverage established safety
control methods such as CBF, RL-learning, etc. Second, our Decision Layer has
a multi-tiered interaction mechanism including a feedback loop with the
Instinct Layer for continual self-reflection and interaction with the External
Layer for incorporating human-in-the-loop to prevent the possibility of
incorrect planning from an LLM as much as possible.

##  III Proposed Framework

###  III-A Layered Hierarchy Design

Our proposed control framework revolutionizes the conventional robotic
architecture by incorporating a hierarchically layered design based on the
level of intelligence, as Fig. 1 shows. This hierarchical model emulates the
human cognitive process, enabling more human-like behavior and interaction in
robotic systems.

####  III-A1 External Layer

At the top of this hierarchy, as an external layer, we have the (Artificial)
General Intelligence, represented by humans and other high-level intelligent
agents. This level provides high-level goals and instructions and serves as a
means of interpreting the robot’s behavior and feedback in the broader context
of the world.

####  III-A2 Decision Layer

Below this, we have the Decision Layer, the highest level within the robot’s
architecture. This layer is composed of multiple AI Agents as the ’brain’ of
the robot, each potentially an instance of high-level generative models such
as GPT-4, LLaMA, etc. The tasks carried out by these AI Agents encompass a
range of high-level goals that are fundamental for a fully autonomous robot.
Such tasks include complex interaction with humans or other agents, autonomous
decision making based on the environment or the tasks at hand, goal-directed
behavior based on both short-term and long-term objectives, and even tool use
and learning.

Realizing these high-level tasks is often achieved by converting the tasks
into a text form that the AI models understand [citation]. The models can then
generate a chain of thoughts or decisions [citation], similar to how humans
would reason. This makes use of AI Planners and AI Executors [citation], which
respectively deal with deciding what to do and implementing the decision.

####  III-A3 Instinct Layer

Following the Decision Layer is the Instinct Layer. This level of our
framework represents the low-level intelligence within the robotic system as
the ‘brainstem’ and is composed of a collection of Robot Instinct modules.
Unlike the AI Agents that make strategic decisions, the role of the Robot
Instinct modules is to ensure the robot’s safety and operability in the face
of immediate environmental challenges. Each of these modules can be powered by
a Discriminative Model, or a more traditional control scheme depending on the
specific task.

These survival-essential tasks are abilities intrinsic to the robot’s survival
and functionality, such as obstacle avoidance, random roaming, and overload
protection. These functions remain operational even in the absence or failure
of the AI Agents, whether due to hallucination phenomena, signal interference,
or other unexpected scenarios. Consequently, even when higher-level decision-
making capabilities are compromised, the robot can still provide basic
services and ensure its own safety and that of its surroundings.

####  III-A4 Device Layer

Finally, at the base of the hierarchy, we have the Device Layer. This layer is
devoid of any form of intelligence and comprises the fundamental hardware
components of the robot, such as the motor, camera, laser, and DRAM. These
devices serve as the robot’s sensors, executors, and memory, forming the
’physical body’ of the robot that interacts with the environment.

By adopting such a layered, hierarchical design, our framework allows for the
distribution of control and decision-making processes at different levels of
intelligence. This not only enables robots to exhibit more human-like behavior
but also opens new avenues for safer and more reliable robot operation.

###  III-B Modules and Flows

After having delved into the detailed discussion on the layered design of the
architecture, we now turn to delineating the functionalities of the various
modules within the architecture and the data, control, and feedback flows
between them, as Fig. 2 shows. This modular design and flow analysis are key
to realizing our layered architecture, jointly forming the backbone of the
architecture.

Figure 2: Modules and Flows. Drawing from our layered hierarchy, we can
distill our framework into a few key modules: the (Artificial) General
Intelligence module, which includes Human and/or Other Agents, the AI Agent
module, the Robot Instinct module, and the Device Layer modules. The Device
Layer is further subdivided into the Sensor, Executor, and Memory modules.
This modular breakdown allows for targeted interactions and efficient data and
control flows within the robotic system.

####  III-B1 Data Flow

All real-time data (such as image and audio signals and motor encoder
information) are first acquired by the Sensor module and then delivered to the
Robot Instinct module. The Robot Instinct is responsible for processing this
raw sensory data, executing survival-essential tasks, and producing a
simplified version for the AI Agent. This design eases the computational
burden of the AI Agent and allows it to focus more on high-level decision-
making, similar to how higher cognitive functions in humans rely on processed
sensory inputs.

The processed data from Robot Instinct is then transferred to the AI Agent for
advanced decision-making processes. Both the Robot Instinct and AI Agent
generate data based on their operation, which is stored in the Memory module
for future recall and learning. The real-time nature and reliability of the
Robot Instinct processing are critical to the successful operation of this
framework.

####  III-B2 Control Flow

The AI Agent, using generative AI, generates high-level commands, often as
function API calls in high-level languages, even pseudocode in complex
conditions or loops. These commands, which encapsulate complex tasks or
behaviors, are then sent to the Robot Instinct module.

The Robot Instinct module is responsible for the execution of these high-level
commands. To do so, it translates high-level commands into a series of low-
level API calls such as specific motor movements or stops, in languages closer
to the hardware, like C 111The Robot Instinct’s safety check should ensure low
latency, allowing the high-level commands to ”penetrate” through the Robot
Instinct and reach the Device Layer directly while maintaining safety..

Importantly, the Robot Instinct module provides a standardized interface for
the AI Agent, abstracting the details of the underlying Executor devices. This
abstraction allows the AI Agent to focus on high-level decision-making without
worrying about the specific characteristics of the individual devices, which
greatly enhances the scalability and versatility of the AI system.

####  III-B3 Feedback Flow

Feedback data from the Robot Instinct module is returned to the AI Agent,
promoting a process of self-reflection and adjustment. This feedback mechanism
allows the AI Agent to make informed next-step decisions, thus enabling more
effective learning and adaptation to dynamic environments.

###  III-C Feedback Mechanism

In our proposed robotic control framework, the AI Agent and the Robot Instinct
interact with each other not merely through one-way command issuing but also
through bidirectional feedback communication, as Fig. 3 shows. This closed-
loop structure aids in refining the decision-making process and ensures the
safety of the robot’s operations.

Figure 3: Feedback Mechanism.

####  III-C1 Feedback to the AI Agent

The AI Agent, after issuing high-level commands, receives feedback from the
Robot Instinct. This feedback includes the status of the commands executed and
possibly additional sensor data. By incorporating modern techniques such as
chain of thoughts/decision, self-reflection via in-context learning, the AI
Agent can optimize subsequent commands without the need for retraining the
model. This is a critical feature for the robot to learn and adapt to new
environments and situations.

####  III-C2 Safety Protocols and Instinctive Refusal

On the other side, the Robot Instinct module is not simply a recipient of
high-level commands from the AI Agent. It has built-in safety protocols that
allow it to check every incoming command. The safety check utilizes either a
discriminative model or traditional methods to ensure the commands will not
violate any survival-essential tasks. If a command is deemed unsafe, the Robot
Instinct module has the authority to instinctively refuse it. This feature
ensures that the robot’s fundamental safety and self-preservation are always a
priority.

###  III-D Architecture Revisit: Intelligence and Necessity

We now revisit our layered design from dual perspectives. We delve further
into the framework’s intelligence and necessity, providing a comprehensive
understanding and evaluation of our design.

Firstly, our layered architecture presents a top-down distribution in terms of
the level of intelligence. At the highest level, in the External Layer, we
find humans or other advanced intelligent agents capable of sophisticated
decision-making and high-level task planning. The Decision Layer houses the AI
Agent, responsible for the robot’s high-level decisions. Following is the
Instinct Layer, accountable for some critical survival tasks such as obstacle
avoidance and overload protection. Finally, at the base in the Device Layer,
we have basic devices and systems such as sensors and actuators.

Secondly, our layered architecture demonstrates a bottom-up distribution in
terms of necessity. At the base, the Device Layer is the necessary structural
part of the robot, without which the robot cannot achieve basic perception and
action. The Instinct Layer is the necessary control part of the robot, without
which the robot is unsafe and incapable of performing basic survival tasks.
The Decision Layer is a necessary part of the intelligent robot, without which
the robot cannot perform advanced decision-making and task planning. Finally,
at the highest level, the External Layer is a necessary part for group robots
or human-robot interaction, without which the robot cannot perform cooperation
and interaction.

##  IV Case Study: Mobile Robot

###  IV-A The Traditional Behavior-based Structure

In this section, we examine the traditional behavior-based structure by means
of a mobile robot case study. This existing control framework is structured
around distinct parallel behavior modules, each with their direct input from
sensors and direct output to actuators. These behavior modules, acting like
individual ’mini-brains,’ coordinate to control the robot. Key behavioral
modules include target tracking, map building (e.g., based on LiDAR SLAM),
path planning, motor operation, and obstacle avoidance modules.

The behavior-based structure has its advantages. It stands out for its
simplicity, which leads to efficiency in design and testing stages. This
design approach also ensures that individual modules work independently,
minimizing mutual interference and allowing for independent optimization of
each.

However, the main drawback of the behavior-based structure emerges in the
context of system generalization and scalability. In this architecture, every
behavioral module receives its input directly from sensors and sends its
output directly to actuators. As a result, each module needs to be intimately
familiar with the particular sensors and actuators of the robot it is
implemented on. This specificity restricts the reusability of these modules
across different robotic platforms and types, hindering the potential for
broad-based solutions and extensions to various robotic platforms. Moreover,
due to the tight coupling between modules and specific hardware, making
adjustments for new sensors or actuators, or adding new behaviors, may require
significant system rewrite, adding complexity to the system’s maintenance and
expansion.

###  IV-B Transition to the Proposed Framework

Transiting from traditional behavior-based structures, we elucidate our novel
framework from bottom to top.

####  IV-B1 Device Layer

The Device Layer comprises the fundamental hardware necessary for the
operation of the robot, including motors, actuators, sensors, and other
physical components that directly interact with the environment.

####  IV-B2 Instinct Layer

The Instinct Layer acts as a critical hub, orchestrating survival-essential
tasks such as obstacle avoidance, overload protection, and safety assurance to
prevent harm to humans. Different modules within this layer, each running on a
dedicated chip, handle these tasks in parallel, achieving both efficiency and
robustness. This design enables the layer to adapt to escalating complexities
as mobile robot tasks increase in their scope and intricacy.

To ensure the robust and timely execution of these tasks, the Instinct modules
are implemented on low-latency, high-stability platforms such as micro-
controllers or FPGAs. As depicted in Alg. 1, each instinct module has the
highest thread priority. This priority assignment ensures that survival-
essential tasks are always addressed promptly, even in the face of concurrent
high-level command executions from the Decision Layer.

A significant feature of our architecture is the dual-layer safety mechanism
incorporated into the Instinct Layer. Every high-level command received from
the Decision Layer must pass a rigorous safety check before execution. This
procedure guarantees that any command, whether survival-essential or derived
from higher-level decision-making, conforms to the pre-defined safety
protocol. Thus, our architecture harmonizes advanced decision-making
capabilities with rigorous safety assurances, providing a reliable and safe
control system for mobile robots.

Input: High-level commands from Agent, Devices (Motor, Lidar)

Output: Low-level command execution, Feedback to Agent

initialization;

while  _True_ do

// Survival Essential Tasks

if  _device.status == ”safe”_ then

performSurvivalTasks();

end if

else

enterSafeMode();

sendFeedback();

continue;

end if

// Handling High-level Commands

commands = getHighCommands();

foreach  _command_ do

lowCommands = convert(command);

foreach  _lowCommand_ do

if  _safetyCheck(lowCommand)_ then

switch  _lowCommand.type_ do

case  _Motor_ do

motor.execute(lowCommand);

end case

case  _Lidar_ do

lidar.acquire(lowCommand);

end case

end switch

sendFeedback();

sendData();

end if

else

refusal();

sendFeedback();

end if

end foreach

end foreach

end while

Algorithm 1 Instinct Module

####  IV-B3 Decision Layer

The Decision Layer serves as the ’brain’ of the robot, where intricate
decision-making tasks are carried out. As highlighted in Algorithm 2, this
layer is not operated by a single LLM, but rather, it consists of multiple LLM
modules that collaborate to perform the AI Agent’s tasks. These tasks include
interactions with humans or other agents (facilitated by the Interaction
Agent), task planning and order arbitration (handled by the Task Planning
Agent), tool utilization (enabled by the Tool Utilization Agent), and self-
reflection based on feedback from the Instinct Layer (performed by the Self-
Reflection Agent).

The Interaction Agent enables the AI to comprehend and respond to external
instructions, facilitating cooperative tasks between the robot and humans or
other agents. The Task Planning Agent, working closely with the Self-
Reflection Agent, enables the AI to analyze dynamic environments and make
autonomous decisions. By interpreting the Instinct Layer’s API
calls/documentation, the Tool Utilization Agent enhances the system’s
versatility.

Incorporated within the Decision Layer is a safety mechanism that reduces the
risk of incorrect AI agent decisions. By assigning distinct roles to each LLM
module and establishing feedback loops, the AI is less prone to making
mistakes. The safety checks at the Decision Layer further reinforce the safety
precautions implemented at the Instinct Layer, providing a comprehensive
safety system for the robot.

Input: Tasks from External Layer, Feedback from Instinct Layer

Output: High-level commands to Instinct Layer

initialization;

while  _True_ do

// LLM for interaction with External Layer

tasks = getTasks();

foreach  _task_ do

while  _task not complete_ do

// LLM for getting feedback from Instinct Layer

feedback = getFeedback();

status = getData();

// LLM for self-reflection

reflect = selfReflection(feedback, status);

// LLM for task planning and order arbitration

highCommands = plan(task, reflect, status);

// LLM for tool utilization

sendCommand(highCommands);

task.update();

end while

end foreach

end while

Algorithm 2 Agent Algorithm with Multiple LLMs

####  IV-B4 External Layer

The External Layer represents high-level (artificial) general intelligence,
including humans and other AI agents that send out commands to the robot. In
an era demanding advanced human-machine cooperation and swarm coordination,
our External Layer is designed to meet these needs by facilitating complex and
sophisticated interactions and commands.

The transition to this proposed framework presents significant advantages.
Firstly, it improves the scalability and adaptability of the system.
Algorithms in the AI Agent module can be designed and deployed across
different types of robots without needing to understand the intricacies of
each specific hardware component. Secondly, the framework encourages
modularity, making it easier to add, remove, or upgrade specific functions.
Lastly, the introduction of the External Layer opens up new possibilities for
human-robot interaction and cooperative intelligence, fostering the
integration of the robot into larger, more complex systems.

##  V Conclusion

In this work, we have proposed a novel control paradigm for autonomous robots,
unifying the intellectual capabilities of large language models (LLMs) with
instinctual functionalities. Our layered design approach inherently caters to
the evolving demand for intelligent, safe, and adaptive robotic systems.

The burgeoning development of AI agents, particularly LLMs, has been a game-
changer in the realm of robotics, offering immense benefits such as intuitive
task comprehension, adept human-robot interaction, and complex decision-
making. However, with these opportunities come inherent risks, especially
regarding safety. By integrating an Instinct Layer in our paradigm, we ensure
the prompt execution of survival-essential tasks and maintain an upper hand in
safety matters.

Despite the promising blueprint of our new architecture, there are avenues
left unexplored in this paper. Currently, we are planning to conduct
experimental validations of our paradigm on mobile robot platforms and multi-
axis robotic arms. This will further substantiate the enhanced intelligence
and safety aspects brought by our design. Furthermore, the inter-layer
communication, although not covered extensively in this paper, is a crucial
area of future research. We envision a medium akin to Robot OS between the
Decision and Instinct Layers to bolster system transparency, reduce design
complexity, and increase scalability.

In addition, we barely scratched the surface of long/short term memory in this
paper. A more profound exploration is planned for future studies, elucidating
how AI Agents and Robot Instincts can learn and optimize strategies from their
memory.

While our work indeed poses challenges and has room for refinement, the
proposed architecture unarguably opens up new frontiers in the landscape of
autonomous robotics. By bridging intelligence and instinct, we hope to harness
the immense potential of AI and robotics and anticipate a future where robots
can operate independently, intelligently, and safely in a wide range of
environments.

## References

  * [1] Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada.  Control barrier functions: Theory and applications.  In 2019 18th European control conference (ECC), pages 3420–3431. IEEE, 2019.
  * [2] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  arXiv preprint arXiv:2302.04023, 2023.
  * [3] Rodney Brooks.  A robust layered control system for a mobile robot.  IEEE journal on robotics and automation, 2(1):14–23, 1986.
  * [4] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [5] Erann Gat, R Peter Bonnasso, Robin Murphy, et al.  On three-layer architectures.  Artificial intelligence and mobile robots, 195:210, 1998.
  * [6] Yukai Gong, Ross Hartley, Xingye Da, Ayonga Hereid, Omar Harib, Jiunn-Kai Huang, and Jessy Grizzle.  Feedback control of a cassie bipedal robot: Walking, standing, and riding a segway.  In 2019 American Control Conference (ACC), pages 4559–4566. IEEE, 2019.
  * [7] Jemin Hwangbo, Inkyu Sa, Roland Siegwart, and Marco Hutter.  Control of a quadrotor with reinforcement learning.  IEEE Robotics and Automation Letters, 2(4):2096–2103, 2017.
  * [8] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  Advances in neural information processing systems, 35:22199–22213, 2022.
  * [9] Hector Levesque and Gerhard Lakemeyer.  Chapter 23 Cognitive Robotics, volume 3 of Foundations of Artificial Intelligence, page 869–886.  Elsevier, 2008.
  * [10] Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath.  Reinforcement learning for robust parameterized locomotion control of bipedal robots.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2811–2817. IEEE, 2021.
  * [11] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al.  Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis.  arXiv preprint arXiv:2303.16434, 2023.
  * [12] Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard Ghanem.  Llm as a robotic brain: Unifying egocentric memory and control.  arXiv preprint arXiv:2304.09349, 2023.
  * [13] Y Nakajima.  Task-driven autonomous agent utilizing gpt-4, pinecone, and langchain for diverse applications.  See https://yoheinakajima. com/task-driven-autonomous-agent-utilizing-gpt-4-pinecone-and-langchain-for-diverse-applications (accessed 18 April 2023), 2023.
  * [14] Quan Nguyen and Koushil Sreenath.  Safety-critical control for dynamical bipedal walking with precise footstep placement.  IFAC-PapersOnLine, 48(27):147–154, 2015.
  * [15] OpenAI.  Gpt-4 technical report, 2023.
  * [16] George Saridis.  Intelligent robotic control.  IEEE Transactions on Automatic Control, 28(5):547–557, 1983.
  * [17] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  arXiv preprint arXiv:2303.17580, 2023.
  * [18] Noah Shinn, Beck Labash, and Ashwin Gopinath.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  arXiv preprint arXiv:2303.11366, 2023.
  * [19] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [20] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor.  Chatgpt for robotics: Design principles and model abilities.  Microsoft Auton. Syst. Robot. Res, 2:20, 2023.


# COMBINING UNSUPERVISED AND TEXT AUGMENTED SEMI-SUPERVISED LEARNING FOR LOW
RESOURCED AUTOREGRESSIVE SPEECH RECOGNITION

###### Abstract

Recent advances in unsupervised representation learning have demonstrated the
impact of pretraining on large amounts of read speech. We adapt these
techniques for domain adaptation in low-resource—both in terms of data and
compute—conversational and broadcast domains. Moving beyond CTC, we pretrain
state-of-the-art Conformer models in an unsupervised manner. While the
unsupervised approach outperforms traditional semi-supervised training, the
techniques are complementary. Combining the techniques is a 5% absolute
improvement in WER, averaged over all conditions, compared to semi-supervised
training alone. Additional text data is incorporated through external language
models. By using CTC-based decoding, we are better able to take advantage of
the additional text data. When used as a transcription model, it allows the
Conformer model to better incorporate the knowledge from the language model
through semi-supervised training than shallow fusion. Final performance is an
additional 2% better absolute when using CTC-based decoding for semi-
supervised training compared to shallow fusion.

Index Terms—  seq2seq, unsupervised learning, semi-supervised training, domain
adaptation

##  1 Introduction

Epistemic uncertainty [1], due to limited training data, is a universal
problem for automatic speech recognition (ASR), and machine learning in
general. Transcribed data is difficult to obtain because it is either
expensive or restricted due to privacy or proprietary reasons. Some companies
do have access to tremendous amounts of transcribed data, but it cannot be
shared beyond the company. It is also possible to obtain large amounts of
transcribed data in certain domains like read speech [2], but that data is
unlikely to be useful in building models for more difficult domains like far-
field and conversational speech. While gathering large quantities of labeled
data is difficult, unlabeled data is easier to find. Assuming the model’s
purpose is to process large amounts of data, then this same data can serve as
untranscribed training data. Even if a human is not allowed to inspect the
data for privacy or security reasons, an automated system may still use the
data for training.

A common approach to incorporating untranscibed data in ASR training is
through semi-supervised training (SST)—the use of pseudotranscripts from an
initial model for supervision. A strong lexicon and language model—provided
externally to a hybrid model or implicit in a sequence-to-sequence
model—restricts the output to sequences of likely words in the language.
Because of the extra information added by the language model, SST works
remarkably well for ASR. When trying to improve a well-trained supervised
model, it can take an order of magnitude more unsupervised data to provide an
improvement [3, 4]. In the case of domain adaptation, SST can produce large
gains with just a few hours of data [5].

While SST performs well for ASR, it does have drawbacks. The output from the
transcription model is a discrete word sequence that has removed uncertainty
in the output. Alternatives that utilize the lattice have been proposed, but
additional gains are minimal [6]. Most work uses academic datasets where the
untranscribed data has already been filtered. We can assume the data is of
high quality and matched to the domain of interest. With wild data, the data
itself is questionable. Portions of the data may be from the wrong domain,
wrong language, or even contain no discernible speech at all. Using wild data
indiscriminately can harm the final model. Even in cases where the data is
curated, large errors in the hypothesized transcripts, especially in the form
of deletions, can harm the final model [5].

An alternative, and potentially complementary approach is unsupervised
learning. Unsupervised representation learning can address some of the
concerns with SST. While SST requires curation of the untranscribed data,
unsupervised training should be more robust to low quality, or out-of-domain
data. Since we are not assigning real labels, poor hypothesized transcripts
cannot corrupt the model. The model might still waste capacity modeling
irrelevant data, but that is less detrimental than hallucinating words for
out-of-domain data.

Unsupervised learning approaches work by defining a _pretext_ task. We can
largely separate the approaches into two categories, contrastive and
reconstruction-based. The fundamental difference between the two approaches is
that the contrastive approaches require negative samples. The model generates
a candidate output and compares it with the true target and a number of
negative samples. The objective function is based on the candidate’s
similarity with the target and distance from the negative samples. In [7],
future states are predicted from the current state. The more recent _wav2vec
2.0_ approach predicts a state given surrounding context [8], similar to the
way BERT [9] is trained for NLP tasks.

In reconstruction-based approaches have no negative samples; the model
directly attempts to reconstruct the input. Autoencoder-style approaches are
typical reconstruction-based approaches [10, 11, 12]. A variant of wav2vec
2.0, DeCoAR 2.0 [13] replaces the contrastive loss with a reconstruction loss.
The recent HuBERT [14] approach can also be considered a reconstruction-based
approach. We discuss the HuBERT approach in more detail in the next section.

Reconstruction has the potential downside that the model has no prior
knowledge of what parts of the representation are more important than others,
so all errors are weighted equally. When dealing with high-dimensional signals
(e.g. speech or images), modeling the underlying relationships between the
dimensions in the original feature space is difficult and costly.
Reconstructing every detail is likely unnecessary as there are many nuisance
dimensions that are irrelevant for the eventual task. We know the number of
relevant classes/categories are orders of magnitude smaller than the number of
unique signals. We also know the objective function cannot distinguish between
irrelevant differences in the reconstructed signal. Approaches like CPC [7],
wav2vec 2.0 [8], DeCoAR 2.0 [13] use quantized or latent representations
representations to allow the model to focus on the more relevant dimensions
during reconstruction.

In this work we use the HuBERT approach to unsupervised learning to pretrain
the encoder of a Conformer model for the purposes of domain adaptation. The
model is then fine-tuned on out-of-domain supervised data. We compare
unsupervised learning with semi-supervised learning, both separately and in
combination. We also have access to large amounts of text data in the new
domain. Despite the advance of techniques like shallow fusion, deep fusion
[15], cold fusion [16], and internal language model estimation [17], there is
a limit to how much an autoregressive model can gain from additional language
model data. We experiment with utilizing the encoder from the Conformer model
as a CTC model in a non-autoregressive framework in order to take full
advantage of the additional text data. Our contributions include:

  * •

Application of unsupervised learning in a low-resource domain adaptation task
with an autoregressive model

  * •

Integration of external language model information through semi-supervised
training

  * •

Confirmation that unsupervised pretraining is both more powerful and
complementary with semi-supervised learning.

In Section 2 we describe our approaches to unsupervised and semi-supervised
training. Section 3 details the data and experimental setup. Our results are
presented and discussed in Section 4, and conclusions are presented in Section
5.

##  2 Learning from Untranscribed Data

###  2.1 Unsupervised Pretraining Approach

The structure of the network in the HuBERT approach [14] is similar to wav2vec
2.0 [8]. Both use the same convolutional waveform encoder followed by a large
transformer network, however, HuBERT does not use a quantization component.
The first step in the training process requires a k-means clustering of the
features to generate unsupervised targets. The initial features are MFCC
features, but they are used to bootstrap the initial model and later
iterations use an internal representation from the transformer network. Given
the clusters as targets, the model is trained using frame-level cross-entropy.
Learning the cluster targets would be trivial for a sophisticated model. To
combat a degenerative solution, some of the inputs to the transformer network
are masked and only updates related to the masked frames are used; instead of
directly predicting the target from the input, the model predicts the target
based on context.

While we are inspired by the HuBERT approach [14], our implementation and use
differs in several aspects from the original work. Our design decisions stem
from a desire to reduce computational and data requirements. Instead of
building the initial clusters from MFCC features, we first train a Conformer
[18] model on a small amount of supervised out-of-domain data. Once the model
has been trained, the decoder portion of the network is discarded and only the
encoder is kept. We use the final layer of the encoder to generate embeddings
for the untranscribed data. After embedding the acoustic signal with the
encoder, we cluster the embedded features using k-means. As in [14], the
clusters serve as labels for the unsupervised training. We remove the
convolutional waveform encoder and train a Conformer encoder from filterbank
features instead. During this step, there is no decoder involved. The model is
trained using frame-level cross-entropy using the clusters as targets. After
the encoder has been pretrained using the untranscribed data, a randomly
initialized decoder is appended and the entire model is fine-tuned using the
original transcribed audio.

Our work also differs from HuBERT in terms of the task. Our data consists of a
mix of CTS and broadcast news, more difficult domains than read speech. The
HuBERT paper used a minimum of 960 hours and as much as 60k hours, much more
data than we have available. We consider the task of domain adaptation with
unsupervised learning. The models trained in the HuBERT paper were CTC models
with external LMs. We focused on encoder-decoder style models, both with and
without external LMs. The HuBERT paper mentions that the number of GPUs used
during unsupervised pretraining is critical because of batch size. A minimum
of 16 GPUs was required to achieve good results. We found performance to be
stable with respect to the number of GPUs used, with a single GPU being
sufficient. The original HuBERT paper focuses only on masked frames during
unsupervised learning. We perform the same update on the model regardless of
whether the original frame was masked.

We do not report these results, but our model failed to improve over the
baseline when the clusters were generated using MFCC features. While best
performance was seen by clustering features extracted from an encoder, the
original HuBERT paper was able to learn using the MFCC features for
clustering. The reason for our failure to learn using MFCC-based clusters
could be due to a variety of factors including more difficult, limited data,
and limited GPUs.

###  2.2 Semi-Supervised Learning

The standard approach to semi-supervised training uses an initial model to
transcribe the unlabeled data and then treats the hypothesized transcripts as
truth. We start with the supervised model as our transcription model. During
SST we use both the supervised data and the untranscribed data, making no
distinction between the two during the training process. For hybrid models
keeping only a subset of the hypothesized transcripts can be important. The
common approach is to simply filter by confidence, but more sophisticated
methods are sometimes used [5, 19]. For autoregressive models, we found
selection to be less of an issue. The models are data hungry and tend to
perform better with more data, even if it is lower quality.

The recent _noisy student_ approach [20, 21] is also related. That approach
extends the classical approach to SST by applying more data augmentation and a
more careful filtering of the pseudotranscripts. Typically the noisy student
work also increases the model size as the amount of data increases. Since the
untranscribed data in our sets is significantly less than the amount used in
the noisy student work, we do not apply these additional improvements.

##  3 Experimental Setup

###  3.1 MATERIAL Data

Table 1:  Amount of transcribed CS and untranscribed CS and BN data for each language. Most of the untrascribed data is BN and no transcribed BN data is available. Language | Transcribed | Untranscribed
---|---|---
CS | BN | CS | BN
Bulgarian | 41.1 | 0 | 33.3 | 149.9
Swahili | 68.3 | 0 | 57.6 | 149.0
Tagalog | 127.9 | 0 | 48.4 | 153.8

We use three languages from the IARPA
MATERIAL111https://www.iarpa.gov/index.php/research-programs/material program:
Bulgarian, Swahili, and Tagalog. The languages are a representative sample of
the nine languages from the program in terms of difficulty. All transcribed
training data in the program consists of conversational telephone speech (CS).
The test data contains a small amount of CS data, but mostly consists of news
broadcast (NB) and topical broadcast (TB) data. The same is true of the
additional untranscribed data used for training. For the remainder of the
paper, we combine the two broadcast sets into a single set labeled broadcast
(BN). This represents a large domain shift between the training and test data.
In addition to the domain shift, the out-of-vocabulary (OOV) rate is high for
all languages. To address the domain shift from the text side we augment that
text data with source-side parallel data provided by the MATERIAL program,
ParaCrawl [22] data (where available), and automatically collected web data
[23]. The total amount of text data ranges from 80 to 120 million words per
language. Note that the text data is still out-of-domain, though likely a
closer match than the original acoustic transcripts. The distribution of
domain for the acoustic data is shown in Table 1. For more details about the
data, see [24]. Note that the transcribed CS data for Swahili and Tagalog is
identical to the data from the IARPA BABEL program, available through the LDC
(LDC2017S05, LDC2016S13).

###  3.2 Conformer Model Training

Our sequence-to-sequence models are conformer-based [18] encoder-decoder
models trained in EspNet [25]. The configuration is similar to the ones
described in [26]. The encoder has 12 layers with four attention heads, an
embedding dimension of 256, and a FFN dimension of 2048. The decoder uses 6
layers with identical parameters. In addition to the standard cross-entropy
objective function, we also use the CTC objective function [27]. Our output
units are characters.

During unsupervised pretraining, we use frame-level cross entropy to train the
Conformer encoder with the k-means clusters as targets. The encoder is trained
for a maximum of 40 epochs. The minibatch size is 128 utterances,
corresponding to approximately 500 seconds of audio. We use SpecAugment [28]
during both supervised and unsupervised training.

###  3.3 Combining the Encoder with N-Gram Language Models

Since our Conformer models are jointly trained with a cross-entropy and CTC
objective function, the encoder can be used separately from the decoder for
ASR. Using the encoder alone makes the model non-autoregressive. One benefit
of a non-autoregressive model is it will generate posteriors for each output
unit at each frame, making it easy to combine the model with an external word-
level LM and lexicon.

We combine a weighted finite state transducer (WFST)-based decoder with the
Conformer encoder in order to utilize an expanded lexicon and n-gram language
model. For our experiments, we use trigram LMs. This has been done before in
[29][30], and in particular our approach is similar to WFST decoding in [30]
using a decoder implemented in Kaldi [31]. The WFST is a simple composition of
three transducers: a token transducer, which removes blank symbols output by
CTC and collapses repeated characters; a lexicon transducer, which maps
sequences of collapsed characters into words; and a grammar transducer, which
contains the n-gram LM.

##  4 Results

###  4.1 Impact of Cluster Type in Unsupervised Training

Table 2:  Performance (WER) of unsupervised clusters from K-Means compared to using position-dependent phone labels. Language | Cluster Type | CS | BN
---|---|---|---
Bulgarian | Position Dependent Phones | 40.1 | 38.7
Bulgarian | K-Means (5000) | 35.0 | 40.2
Swahili | Position Dependent Phones | 38.1 | 57.9
Swahili | K-Means (5000) | 33.7 | 49.0
Tagalog | Position Dependent Phones | 44.8 | 52.8
Tagalog | K-Means (5000) | 38.8 | 53.3

For all three languages we compared the unsupervised targets with using
position-dependent phones as targets. Results are in Table 2. Results are
similar to using the unsupervised clusters, except for Swahili where the
unsupervised clusters are significantly better. Note that we do not have
ground truth transcripts for the unsupervised data, so the phone targets come
from forced alignment with the hypothesized transcripts from a hybrid model.
We did tune the number of clusters on Bulgarian, cluster size has little
impact on overall performance. In all cases we are using the output from a
supervised encoder for clustering. We were unable to improve over a baseline
model when using MFCC features for clustering.

###  4.2 Performance without Additional LM Data

Table 3:  Comparing WER performance using unsupervised and semi-supervised training without external language model data. Model Description | Bulgarian | Swahili | Tagalog
---|---|---|---
CS | BN | CS | BN | CS | BN
Supervised | 39.1 | 50.7 | 37.5 | 58.1 | 43.5 | 65.0
SST iter. 1 | 35.5 | 41.5 | 35.7 | 49.7 | 42.4 | 57.6
SST iter. 2 | 34.8 | 36.9 | 36.3 | 47.0 | 42.6 | 53.7
Unsup iter. 1 | 35.0 | 40.2 | 33.7 | 49.0 | 38.8 | 53.3
Unsup iter. 2 | 32.1 | 35.0 | 33.6 | 44.2 | 38.6 | 49.4
SST from Unsup iter. 1 | 31.9 | 33.4 | 33.4 | 36.9 | 39.2 | 47.8

Results without an external language model can be seen in Table 3. All models
are Conformer encoder-decoder models; the external LM is a two-layer LSTM
model. The supervised model, trained only on CS, sets the baseline for all
subsequent models. While WER for BN is typically lower than CS in most
languages, that is not that case in our results. The supervised model has
never seen broadcast data and the BN data has a high OOV rate, explaining the
discrepancy.

We directly compare the WER reduction from semi-supervised and unsupervised
training. The semi-supervised model shows a clear improvement across all
conditions, but the unsupervised model consistently outperforms the semi-
supervised model (41.7% average WER vs. 43.7%). When considering multiple
iterations, two rounds of unsupervised training is better than two rounds of
SST. In fact, the gap between the unsupervised and semi-supervised approach
grows from 2.0% absolute to 3.1%. We also consider combining the two
approaches by applying SST using the initial unsupervised model for
transcription. Combining the two approaches gives the best overall
performance—4.8% absolute improvement compared to multiple iterations of SST
alone—demonstrating the complementarity of the unsupervised and semi-
supervised approaches.

The overall improvement on BN data is not surprising given the lack of BN data
in the supervised model, but the gain in CS is also significant. Unsupervised
pretraining improves the CS result of Tagalog by almost five points absolute.
Tagalog had the largest amount of supervised CS data ($\approx$ 120 hours) and
there was only an additional 50 hours in the untranscribed set. The
unsupervised pretraining clearly provides benefits for both in and out-of-
domain data.

###  4.3 Performance with additional LM data

Table 4:  Comparing performance using supervised, unsupervised, and semi-supervised training with external language data. For each model comparisons are given using shallow fusion and CTC decoding with an N-Gram LM Supervision | Transcription | Bulgarian | Swahili | Tagalog
---|---|---|---|---
Model | CS | BN | CS | BN | CS | BN
Supervised | none |  |  |  |  |  |
ab Fusion |  | 37.6 | 41.8 | 38.6 | 52.6 | 46.0 | 62.5
ab CTC |  | 39.2 | 36.4 | 38.6 | 48.1 | 46.5 | 56.8
Unsup. | none |  |  |  |  |  |
ab Fusion |  | 35.1 | 32.9 | 35.6 | 44.2 | 41.4 | 51.1
ab CTC |  | 37.5 | 29.1 | 36.4 | 40.0 | 42.0 | 48.7
SST | Sup+Fusion |  |  |  |  |  |
ab Fusion |  | 33.7 | 32.2 | 37.3 | 43.0 | 43.4 | 55.4
ab CTC |  | 38.4 | 30.9 | 39.0 | 39.3 | 45.3 | 49.6
SST | Unsup+Fusion |  |  |  |  |  |
ab Fusion |  | 31.6 | 26.4 | 35.3 | 38.3 | 38.6 | 44.9
ab CTC |  | 37.5 | 26.9 | 36.3 | 35.6 | 41.2 | 41.8
SST | Unsup+CTC |  |  |  |  |  |
ab Fusion |  | 30.8 | 24.5 | 33.4 | 33.7 | 38.2 | 41.0
ab CTC |  | 36.1 | 27.3 | 35.1 | 34.9 | 41.8 | 44.1

Table 4 contains the full set of results comparing unsupervised and SST with
the use of external LMs. Initially, we consider the use of shallow fusion and
ignore the CTC results. Overall, we can see a dramatic improvement in the
performance on BN for all languages and models. The external LM has a much
lower OOV rate and the additional language data is more similar to BN data
than CS data. Once the external LM has been incorporated, the superiority of
the unsupervised approach over the SST approach is no longer seen; they each
outperform the other half the time. However, best performance is still
achieved by combining the two approaches. Average performance of the
supervised model is 46.5%. A single round of SST improves the WER to 40.8%,
while SST on top of the unsupervised pretraining further improves the WER to
35.9%.

###  4.4 CTC-based Decoding with N-Gram LMs

In addition to shallow fusion, Table 4 contains results using the CTC encoder
with an external lexicon and LM. Entries marked with _CTC_ use a trigram LM
trained on the same data as the model used for shallow fusion. CTC only
decoding is significantly worse than using the full Conformer model. Due to
space, full results are not shown, but CTC decoding is about 5 to 8 percent
absolute worse than the Conformer model when no external LM is used for
either. However, CTC decoding is able to recover the gap and improve upon the
Conformer model through the use of an external lexicon and LM. For the
supervised, unsupervised, and semi-supervised models (using a transcription
model with shallow fusion), the CTC decoding outperforms the full Conformer
model with shallow fusion on BN data. For example, the supervised model is
improved by 4.5% to 5.7% absolute on BN data when using CTC decoding.
Performance on CS data is worse though, likely due the external lexicon and LM
providing less additional information for the CS data.

The conclusion from the final semi-supervised model, where the transcription
comes from a CTC decode, shows a different pattern. Even for the BN domain,
CTC decoding no longer provides an improvement over shallow fusion. When the
transcription model uses CTC decoding, the resulting semi-supervised model
will have largely incorporated that information into the model. The additional
information in the lexicon and LM is no longer beneficial. The CTC decoding is
helpful, but only for the initial transcription. Using CTC decoding for
transcription in semi-supervised training yields an average WER of 33.6%
across all conditions when the full Conformer model is used for decoding with
shallow fusion, a 2.3% average improvement over using shallow fusion for
pseudotranscription.

##  5 Conclusions

Unsupervised representation learning is a powerful approach to using
untranscribed data in an ASR system. Prior work has focused on large amounts
of read speech. We demonstrate the applicability to smaller amounts of more
difficult data—CTS and BN. The approach works on a single GPU and does not
require multiple GPUs to increase the minibatch size. Previous work has
focused on non-autoregressive models trained with CTC, but we show the
approach can improve an autoregressive Conformer model. In a direct comparison
with SST approaches, the unsupervised approach proves more effective; though
we achieve our best performance by combining the techniques—5% absolute better
than SST alone.

We demonstrated that external language models can significantly improve
performance on out-of-domain test sets. While an autoregressive model like the
Conformer model is generally superior to a non-autoregressive CTC approach, we
show that CTC decoding can make more effective use of external language model
data. By using CTC decoding with an external LM as a transcription model for
semi-supervised training, we are able to incorporate the information into the
Conformer model, leading to an overall additional improvement of 2% absolute.
Once we have used CTC decoding in semi-supervised transcription, it becomes
better to use the full autoregressive semi-supervised model with shallow
fusion for decoding compared to decoding with CTC.

We plan to continue to improve our unsupervised learning approach. While semi-
supervised training is mature, unsupervised learning is a less explored space.
We expect further optimization will lead to even larger gains.

##  6 ACKNOWLEDGEMENTS

This work was supported by the Intelligence Advanced Research Projects
Activity (IARPA) via Department of Defense US Air Force Research Laboratory
contract number FA8650-17-C-9118. This document does not contain technology or
Technical Data controlled under either the U.S. International Traffic in Arms
Regulations or the U.S. Export Administration Regulations.

## References

  * [1] Eyke Hüllermeier and Willem Waegeman,  “Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction,”  arXiv preprint arXiv:1910.09457, 2019.
  * [2] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in IEEE ICASSP, 2015, pp. 5206–5210.
  * [3] Yan Huang, Dong Yu, Yifan Gong, and Chaojun Liu,  “Semi-supervised gmm and dnn acoustic model training with multi-system combination and confidence re-calibration.,”  in Interspeech, 2013, pp. 2360–2364.
  * [4] Sree Hari Krishnan Parthasarathi and Nikko Strom,  “Lessons from building acoustic models with a million hours of speech,”  in IEEE ICASSP, 2019, pp. 6670–6674.
  * [5] Shannon Wotherspoon, William Hartmann, Matthew Snover, and Owen Kimball,  “Improved data selection for domain adaptation in asr,”  in IEEE ICASSP, 2021.
  * [6] Vimal Manohar, Hossein Hadian, Daniel Povey, and Sanjeev Khudanpur,  “Semi-supervised training of acoustic models using lattice-free mmi,”  in IEEE ICASSP, 2018, pp. 4844–4848.
  * [7] Aäron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,” 2018.
  * [8] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [10] Michael Neumann and Ngoc Thang Vu,  “Improving speech emotion recognition with unsupervised representation learning on unlabeled speech,”  in IEEE ICASSP, 2019, pp. 7390–7394.
  * [11] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM TASLP, vol. 29, pp. 2351–2366, 2021.
  * [12] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass,  “An unsupervised autoregressive model for speech representation learning,”  arXiv preprint arXiv:1904.03240, 2019.
  * [13] Shaoshi Ling and Yuzong Liu,  “Decoar 2.0: Deep contextualized acoustic representations with vector quantization,”  arXiv preprint arXiv:2012.06659, 2020.
  * [14] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  arXiv preprint arXiv:2106.07447, 2021.
  * [15] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio,  “On using monolingual corpora in neural machine translation,”  arXiv preprint arXiv:1503.03535, 2015.
  * [16] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates,  “Cold fusion: Training seq2seq models together with language models,”  arXiv preprint arXiv:1708.06426, 2017.
  * [17] Zhong Meng, Sarangarajan Parthasarathy, Eric Sun, Yashesh Gaur, Naoyuki Kanda, Liang Lu, Xie Chen, Rui Zhao, Jinyu Li, and Yifan Gong,  “Internal language model estimation for domain-adaptive end-to-end speech recognition,”  in IEEE SLT, 2021, pp. 243–250.
  * [18] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and R. Pang,  “Conformer: Convolution-augmented transformer for speech recognition,” 2020.
  * [19] Félix de Chaumont Quitry, Asa Oines, Pedro Moreno, and Eugene Weinstein,  “High quality agreement-based semi-supervised training data for acoustic modeling,”  in IEEE SLT, 2016, pp. 592–596.
  * [20] Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le,  “Improved noisy student training for automatic speech recognition,” 2020\.
  * [21] Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu,  “Pushing the limits of semi-supervised learning for automatic speech recognition,”  arXiv preprint arXiv:2010.10504, 2020.
  * [22] Miquel Esplà-Gomis, Mikel L Forcada, Gema Ramírez-Sánchez, and Hieu Hoang,  “Paracrawl: Web-scale parallel corpora for the languages of the eu,”  in Proceedings of Machine Translation Summit XVII, 2019, pp. 118–119.
  * [23] Le Zhang, Damianos Karakos, William Hartmann, Roger Hsiao, Richard Schwartz, and Stavros Tsakalidis,  “Enhancing low resource keyword spotting with automatically retrieved web documents,”  in Interspeech, 2015.
  * [24] Chak-Fai Li, Francis Keith, William Hartmann, Matthew Snover, and Owen Kimball,  “Overcoming domain mismatch in low resource sequence-to-sequence asr models using hybrid generated pseudotranscripts,”  arXiv preprint arXiv:2106.07716, 2021.
  * [25] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al.,  “Espnet: End-to-end speech processing toolkit,”  arXiv preprint arXiv:1804.00015, 2018.
  * [26] Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, et al.,  “Recent developments on espnet toolkit boosted by conformer,”  in IEEE ICASSP, 2021, pp. 5874–5878.
  * [27] Suyoun Kim, Takaaki Hori, and Shinji Watanabe,  “Joint ctc-attention based end-to-end speech recognition using multi-task learning,”  in IEEE ICASSP, 2017, pp. 4835–4839.
  * [28] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [29] Yajie Miao, Mohammad Gowayyed, and Florian Metze,  “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,”  in IEEE ASRU, 2015, pp. 167–174.
  * [30] Thomas Zenkel, Ramon Sanabria, Florian Metze, Jan Niehues, Matthias Sperber, Sebastian Stüker, and Alex Waibel,  “Comparison of decoding strategies for ctc acoustic models,”  arXiv preprint arXiv:1708.04469, 2017.
  * [31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.,  “The kaldi speech recognition toolkit,”  in IEEE ASRU, 2011.