# Towards vision-based dual arm robotic fruit harvesting

Ege Gursoy1,2 Benjamin Navarro1 Akansel Cosgun2,3 Dana Kulić2 Andrea
Cherubini1 1LIRMM, Univ. Montpellier, CNRS, Montpellier, France 2Monash
University, Australia 3Deakin University, Australia

###### Abstract

Interest in agricultural robotics has increased considerably in recent years
due to benefits such as improvement in productivity and labor reduction.
However, current problems associated with unstructured environments make the
development of robotic harvesters challenging. Most research in agricultural
robotics focuses on single arm manipulation. Here, we propose a dual-arm
approach. We present a dual-arm fruit harvesting robot equipped with a RGB-D
camera, cutting and collecting tools. We exploit the cooperative task
description to maximize the capabilities of the dual-arm robot. We designed a
Hierarchical Quadratic Programming based control strategy to fulfill the set
of hard constrains related to the robot and environment: robot joint limits,
robot self-collisions, robot-fruit and robot-tree collisions. We combine deep
learning and standard image processing algorithms to detect and track fruits
as well as the tree trunk in the scene. We validate our perception methods on
real-world RGB-D images and our control method on simulated experiments.

##  I INTRODUCTION

As one of the most important processes in agriculture, manual harvesting
remains challenging due to its physically demanding nature. With the expected
global labor shortage[1], robotic harvesting becomes an important research
area to explore. Robotic harvesting can improve productivity by: reducing
labor costs, increasing yield quality and enabling better control of
environmental implications [2]. The key challenges in robotic harvesting are:

  1. 1.

Fruit localization: Determining the location of each fruit wrt the robot.

  2. 2.

Scene understanding: Localizing potential obstacles in the scene such as
branches, tree trunk, leaves etc.

  3. 3.

Motion control: Reaching the fruit while avoiding collisions, to detach and
collect the fruit.

In this work, we address fruit localization by detecting fruits in RGB-D
images using machine learning and tracking them by assigning the same ID over
successive frames. Scene understanding is necessary to avoid robot collisions
while reaching the fruit. On each image, we segment tree trunk and branches by
combining a mask in Hue Saturation Value (HSV) color space with one in the
depth image. We assume that tree branches are symmetric around their trunk.
Then, we define a cylinder around the trunk, which the robot will avoid during
operation. For motion control, two arms (cutting and collecting) move towards
the closest fruit, according to the cooperative task representation [3] while
avoiding the trunk and other fruit. We are using a Hierarchical Quadratic
Programming (HQP) based control algorithm to fulfill the constraints related
to the robot and environment such as robot joint limits, robot self-collisions
and robot-tree collisions. In the last phase, the cutting arm moves to the
stalk bearing of the fruit (peduncle) while the collecting arm moves below the
fruit. The same cycle continues until there are no more reachable fruits left
in the scene. An overview of our system is shown in Figure 2

Figure 1: Dual arm robot BAZAR [4] with cutting and collecting arms.

A dual-arm robot compared to a single-arm, can perform more complex and
delicate movements, ensuring careful handling of fruits, particularly those
that are fragile or require precise detachment. The flexibility of the dual-
arm increases the adaptability to different fruit types with the same set of
end-effector tools. Moreover, the use of dual-arm robots enhances safety by
minimizing damage to fruits and surrounding plant structures through increased
precision.

The main contribution of this study is the preliminary validation of a dual-
arm harvesting robot with vision and control, including : (i) real-time
detection and tracking of fruit using deep learning methods (ii) dual-arm
control in cooperative-task space based on Hierarchical Quadratic Programming
(iii) collision avoidance of fruits and tree trunk by 2D RGB images and depth
information (iv) generalization to round fruits grow on trees.

Figure 2: Overview of our robotic harvesting system. Perception and Motion
Control blocs are detailed in Section III and IV, respectively.

The remainder of the paper is structured as follows. A review of the state of
art in autonomous fruit harvesting is given in Section II. The perception of
our robot is presented in Section III. The motion control and picking steps
are explained in Section IV. Finally, results of the experiments are presented
in Section V, followed by a discussion and propositions for future work (Sec.
VI).

##  II RELATED WORK

Robotic fruit harvesting has been demonstrated for various fruits such as
apples [5], aubergines [6], citrus [7], cucumbers [8], tomatoes [9],
strawberries [10] and sweet peppers [11]. Apart from [5][6][9], the cited
systems only use one robotic arm, whose end-effector is responsible for both
grasping and detaching the fruit, using rigidly linked gripper and cutter.
This rigid link is specific for each fruit, limiting versatility. To address
the broad diversity of fruit shapes and sizes, we propose the use of two arms.

Detecting fruits and locating them is the first challenge for the robot. Deep
learning detectors are popular among agricultural robotics researchers. In
[12], Santos et al. introduce an interactive instance mask generation on RGB
images based on graph matching and compare the performance of learning based
object detection algorithms on grapes. Kang et al. [13] introduce an automatic
label generation module for the detection of apples in RGB images of orchards.
In [14], Koirala et al. investigate popular object detection frameworks for
mango detection and design a custom detection network. Liu et al. [15] follow
a multi-class approach in the detection of citrus by splitting their samples
into classes. Kirk et al. [16] use a custom neural network which takes a
6-channel RGB + CIELAB representation as input. In [17] Ganesh et al. train a
neural network with RGB and HSV 6-channel input to detect oranges. However,
the majority of these approaches have complex pipelines which are difficult to
train and hard to generalize to other crop types.

Fruit tracking is another key requirement for autonomous fruit harvesting. Roy
et al. [18] use apple contours in an Structure from Motion (SfM) pipeline, to
generate dense matches and reconstruct the apples in 3D. However, SfM may have
difficulty distinguishing between fruit and non-fruit objects that have
similar contours. Das et al. [19] use a Support Vector Machine to detect
fruits, optical flow to associate the fruits between frames. However, optical
flow may suffer in case of occlusions. Liu et al. present a convolutional
network to separate fruit and non-fruit pixels and use a Kalman Filter based
Hungarian algorithm to correct Kanade–Lucas–Tomasi tracker which is
computationally expensive.

Motion control is an equally important part of the harvesting robot. Ling et
al. [9] propose a dual-arm robot for harvesting tomatoes. The vacuum cup-type
tool attached to one of the end-effectors grasps the target fruit, while the
other arm (equipped with a cutter) cuts the stem. However, the vacuum cup-type
tool may not work well with fruits that have softer or more delicate skin,
which may get damaged during the harvesting process. Sepulveda et al. [6]
introduce a dual-arm aubergine harvesting robot configured similarly to
humans. The planning algorithm decides the movement modes which are the
simultaneous harvesting of two fruits or harvesting a single fruit with one
arm, depending on the fruit location. Davidson et al. [5] propose a dual-arm
coordination strategy for fruit collection to reduce the harvesting cycle
time. Rather than displacing the fruit to a bin, the picking arm drops the
apple into a robotic collector near the point of fruit detachment. Their
approach is hard to generalize to different fruits. Additionally, the dropping
of the fruit may result in damage or bruising, which can reduce the quality
and value of the harvested fruit.

##  III PERCEPTION

Here, we present our perception system for detecting, tracking and localizing
fruits, the fruit selection strategy, and the tree location estimation
methodology.

In this work, we assume that fruits are ellipsoids (see Figure 3), which
appear in a 2D image as ellipses with axes aligned with the image height and
width. The ellipsoid surrounding the fruit has a circular symmetry i.e. it has
two equal semi-diameters along the $x$ and $z$ axes in the camera frame. Since
most round fruits (e.g., apples, lemons) have this property, we can assume
that our approach is valid in those cases.

###  III-A Fruit Localization

Fruit detection consists in determining the pixel coordinates of the corners
of the rectangle which encloses the fruits in the image. The specifications of
each detected fruit are fed to the tracking algorithm, which assigns a unique
ID to each fruit and correlates the same fruit data in different instances to
determine whether or not new detected fruits can be associated with the
existing IDs. Finally, fruit localization refers to calculating the 3D spatial
position of the fruits the in camera frame shown in Figure 4.

Figure 3: Fruit ellipsoid assumption (left) and its ellipse projection
(right).

We use a deep learning object detection algorithm, YOLOv5 [20] because of its
speed and accuracy. We conduct domain adaptation and transfer learning of
YOLOv5. We initialize the network with pre-trained weights on COCO dataset
[21] to benefit from their feature extractors and use 300 orange images from
AgriPlant Dataset [22] to train. We split the dataset in 70:30 ratio for
training and validation. The fruit tracking task relies on DeepSORT [23], an
improved version of SORT [24], based on frame-to-frame association with a deep
association metric.

With the detected fruits in bounding boxes, fruit 3D positions are computed by
incorporating the depth information from the RGB-D camera. We denote
$(u_{TL},v_{TL})$ and $(u_{BR},v_{BR})$ the top-left and bottom-right pixel
coordinates of the bounding box detected by YOLO. Fruit center in pixels is
calculated as:

$\left(u_{C},v_{C}\right)=\left(\frac{u_{BR}–u_{TL}}{2},\frac{v_{BR}–v_{TL}}{2}\right)$
(1)

Under this hypothesis, we compute the horizontal and vertical semi-axes of the
ellipse surrounding the fruit $r_{h}=u_{BR}-u_{C}$ and $r_{v}=v_{BR}–v_{C}$

We denote the projection of a pixel in color information $(u,v)$ to a 3D
position in camera frame $(x,y,z)$ as follows:

$x,y,z=\mathrm{proj}(u,v)\vspace{-0.01cm}$ (2)

Figure 4: Coordinate frames definitions. Absolute Frame links to fixed World
Frame, while Relative Frame links to moving Absolute Frame.

For each detected fruit, we compute its center
$\mathbf{P}_{f}=x_{f},y_{f},z_{f}$ and distance to the peduncle from the
center $d_{v}$. Under the ellipsoidal fruit assumption, the point visible at
$(u_{C},v_{C})$ is the nearest to the camera optical center. We denote its
depth $z_{min}$. the fruit depth is calculated by:

$z_{f}=z_{min}+\mathrm{dist}[\>\mathrm{proj}(u_{C},v_{C}),\;\mathrm{proj}(u_{C}+r_{h},v_{C})\>]\vspace{-0.01cm}$
(3)

with $\mathrm{dist}$ as the distance between two points in 3D Cartesian space.

The distance to the peduncle from the center is considered to be equal to the
3D transformation of the vertical semi-axis of the ellipse surrounding the
fruit:

$d_{v}=\mathrm{dist}[\>\mathrm{proj}(u_{C},v_{C}),\;\mathrm{proj}(u_{C},v_{C}+r_{v})\>]\vspace{-0.01cm}$
(4)

These results will be used in the following to determine the cutting point.

###  III-B Fruit Selection

Fruit selection describes the process of determining the best candidate fruit
to harvest. We compute the distance of the fruits in the scene to the optical
center of the camera using the color and depth information as described in
Section III-A. For each image, these distances with their associated IDs are
stored in a list. We update the list if the position error between the
previous state of a fruit ID is smaller than a threshold, otherwise the
previous state is kept. This ensures a robust approach to counter temporary ID
errors caused by occlusions. We delete IDs if they are not detected in the
scene for 100 image frames. Finally, we select the ID with smallest distance
for harvesting.

###  III-C Tree Localization

We must estimate the tree position to avoid collisions between the robot and
trunk to prevent damaging the plant or the robot while reaching for a fruit.
Instead of machine learning, we use a simpler, computationally efficient
method, based only on color and depth information.

A HSV color space segmentation mask and a depth mask separate tree branches
and trunk from the rest of the scene while assuming that they are not occluded
in the scene. We convert the RGB information into the more consistent HSV
space. The HSV space is chosen since its dependence upon hue is assigned to a
single value (H) rather than across all three RGB values.

On top of the HSV mask, we create a second mask using the depth information to
filter out pixels that exceed a certain depth threshold. The secondary mask
ensures pixels in the HSV range are not too far away and do not correspond to
3D points which do not belong to the tree.

The mean of the $(u,v)$ pixel coordinates in the RGB image and the depth value
of the remaining pixels give the position of the tree. We denote the
estimation of 3D coordinates of the tree in camera frame as
$\mathbf{P}_{t}=x_{t},y_{t},z_{t}$. The mean depth of the remaining pixels
after the mask gives us $z_{t}$, while $x_{t}$ and $y_{t}$ are calculated
using (2).

##  IV MOTION CONTROL

For a dual-arm fruit harvesting robot, both manipulator arms need to approach
the fruits located at different positions with high accuracy and flexibility,
while safely manipulating the object. To achieve this, we design a dual-arm
control strategy paired with a cooperative representation method.

###  IV-A Dual-arm cooperative control

We describe our fruit picking task using the cooperative task representation
[25], which fully characterizes the operational space for dual-arm control.
This representation allows us to specify the operations in terms of an
absolute task with pose attached to a fixed world frame and a relative task
attached to the absolute task, as shown in Figure 4. The main reasons for
using this representation are the ability to adapt control algorithms
originally designed for single manipulators (e.g., quadratic programming based
approaches) to dual-arm robots, and to perform safe dual-arm object
manipulation without hitting the obstacles nor damaging the fruits. Our tasks
include a cutting phase to detach the fruit from its stem and a collecting
phase to catch the falling fruit. We believe that this representation is a
proper fit to our tasks since the cutting is best to describe in a fixed frame
while catching is more appropriately described in the relative frame.

We define fruit cutting as the absolute task which controls the motion of the
robot in the workspace and fruit collecting as the relative task, which
regulates the relative motion between the two arms.

Let us now formally define these absolute and relative tasks. We define a
desired pose of the absolute task
$\mathbf{x}_{a}=[x_{a},y_{a},z_{a},\phi_{a},\theta_{a},\psi_{a}]$ for the
cutting (left) arm in a fixed frame, (in our case, the camera frame) and a
desired pose of the relative task
$\mathbf{x}_{r}=[x_{r},y_{r},z_{r},\phi_{r},\theta_{r},\psi_{r}]$ for the
collecting (right) arm w.r.t. the frame attached to the cutting (left) arm
during the harvesting phase. Once we select the fruit to harvest in Section
III-B, both arms should go towards that fruit. While harvesting, the tip of
the cutting tool should be placed at the peduncle to detach the fruit while
the center of the collecting tool is slightly below the fruit to catch it with
the basket when it falls. We compute the peduncle location using the vertical
semi-axis $d_{v}$ and fruit position $\mathbf{P}_{f}$ as described in Section
III-A.

Figure 5: Trunk obstacle cylinder in gray and fruit obstacle in orange.

As for the orientations, we do not impose any hard constraint on $\theta_{a}$
to maximize the reachable points in the workspace since cutting only requires
the tip of the cutting tool to reach the peduncle. $\theta_{r}$ is constrained
to $\pi/2$ radian, resulting in $\pi/2$ radian angle between two arms at all
times, to prevent collisions between the arms. Other orientations are set to
stay unchanged from the natural pose of the robot while its arms are in front.
The desired poses to reach a fruit relative to the camera frame are described
by:

$\displaystyle\mathbf{x}_{a}=[x_{f},y_{f}-d_{v}-d_{s},z_{f},0,*,\pi]$
$\displaystyle\mathbf{x}_{r}=[0,-d_{v}-d_{s},0,0,\pi/2,\pi]\vspace{-0.1cm}$
(5)

with $d_{s}$ a security distance while cutting the fruit, and $*$ indicating
unconstrained task component.

In the cooperative dual-arm task representation, both the relative and
absolute tasks have to be realized at the same time. However, a relevant
choice is to prioritize the absolute task over the relative one. Indeed, it is
crucial to fulfill the absolute task since the cutting arm position must be
more precise than the collecting arm.

Thus, we use Hierarchical Quadratic Programming (HQP) as described in [26]. It
consists in solving a sequence of quadratic programming problems for which the
solution space of a given task is restricted to the null space of higher
priority tasks. This ensures that low priority tasks to do not interfere with
the others. The relationship between joint velocities $\mathbf{\dot{q}}$ and
task space velocities $\mathbf{\dot{x}}$ is given by
$\mathbf{\dot{x}}=\mathbf{J}\mathbf{\dot{q}}$ with $\mathbf{J}$ the Jacobian
matrix.

The Inverse Kinematics module based on HQP outputs the joint velocity vector
which will be sent to the robot. HQP includes a set of hard constraints to be
fulfilled at all times, defined both at joint space level (joint limits) and
task space level (collision avoidance). We define the set of constraints under
the standard form of linear inequality equations:

$\mathbf{A}\mathbf{\dot{q}}\leq\mathbf{b}\vspace{-0.1cm}$ (6)

where $\mathbf{A}$ is the linear coefficients matrix and $\mathbf{b}$ the
constant vector in the inequality constraints.

Therefore, in the cooperative dual-arm task space and taking into account
constraints, the highest and lowest priority tasks are solved through the
following optimization problems:

$\displaystyle\mathbf{\dot{q}^{*}_{\mathit{a}}}=\arg\min_{\mathbf{\dot{q}}}||\mathbf{J_{\mathit{a}}}\mathbf{\dot{q}}-\mathbf{\dot{x_{\mathit{a}}}}||_{2}$
$\displaystyle\textit{s.t.}~{}~{}~{}~{}\mathbf{A}\mathbf{\dot{q}}\leq\mathbf{b}$
$\displaystyle\mathbf{\dot{q}^{*}_{\mathit{a,r}}}=\arg\min_{\mathbf{\dot{q}}}||\mathbf{J_{\mathit{r}}}\mathbf{\dot{q}}-\mathbf{\dot{x_{\mathit{r}}}}||_{2}$
(7)
$\displaystyle\textit{s.t.}~{}~{}~{}~{}\mathbf{A}\mathbf{\dot{q}}\leq\mathbf{b}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\mathbf{J_{\mathit{a}}\dot{q}=J_{\mathit{a}}\dot{q}^{*}_{\mathit{a}}}\vspace{-0.1cm}$

where the $\mathbf{a},\mathbf{r}$ subscripts define the high priority
(absolute) and low priority (relative) tasks and $\mathbf{\dot{q}}$ refers to
the vector concatenating the joint velocities of the two arms.

###  IV-B Obstacle Avoidance

To prevent collisions with the fruit and branches which can damage the robot
and the plant, we approach and leave the fruit from the front before and after
the harvesting phase. We define additional tasks for before and after
harvesting, named approach and leave tasks. The desired poses for the approach
and leaving task are as (IV-A) except $z_{a}=z_{f}-d_{z}$ with $d_{z}$ as a
parameter defining the distance to the fruit before passing to the harvesting
phase.

To prevent any dangerous situation, we define the branches and trunk of the
tree as an obstacle to avoid. In a real-world scenario, the branches of a tree
have very complex shapes. Hence, we opt for a computationally efficient
obstacle which covers the tree branches and trunk. We consider an infinite
height cylinder collision object around the trunk of the tree with a variable
radius to prevent the robot from entering the area between the selected fruit
and the trunk. The cylinder radius depends on the fruit position. Variable
radius provides us a safe approach distance towards the fruit which prevents
the robot from entering the area behind the fruit. Let $r_{obs}$ be the radius
of the cylinder. We place the center of the cylinder at $\mathbf{P}_{t}$. The
height of the cylinder lays on the $y$ axis in the camera frame. The radius is
defined as $r_{obs}=\mathrm{dist}[\>\mathbf{P}_{f},\;\mathbf{P}_{t}\>]-d_{h}$
with $\mathrm{dist}$ the distance between two points in 3D Cartesian space.
For each new fruit, $r_{obs}$ is updated.

Since we model each fruit as an ellipsoid with circular section in the $xz$
plane, the fruit is characterized by: its position $\mathbf{P}_{f}$ and semi-
axis lengths $d_{h}$ and $d_{v}$.

The tree and fruit obstacles are shown in Figure 5. It is important to note
that since the fruit and tree positions are calculated in real-time, the
location and size of the obstacles is also updated accordingly.

After modeling the obstacles, a collision avoidance constraint is generated
for all pairs of potentially colliding parts which are separated by a distance
$d\leq d_{i}$. The obstacle avoidance strategy is expressed as linear
constraints using Velocity Damper [27]. The definition of the Velocity Damper
is given by:

$\dot{d}^{*}=-\xi\frac{d-d_{s}}{d_{i}-d_{s}},d\leq d_{i}\vspace{-0.1cm}$ (8)

Figure 6: Simulation with scanned tree, detected oranges and BAZAR.

with $d_{i}$ the activation distance, $d_{s}$ the safety distance, $d$ the
current distance between bodies, and $\xi$ the convergence speed parameter.
The derived inequality constraint is given by $\dot{d}\geq\dot{d}^{*}$. When
two bodies are separated by a distance smaller than $d_{i}$, their relative
speed becomes bounded. The motion is slowed down until $d_{s}$. If $d$ becomes
larger than 0, the only possible motion between the bodies is to maintain or
increase their relative distance. The relative distance can be expressed as:

$\dot{d}=\mathbf{nJ_{\mathit{p}}\dot{q}_{\mathit{d}}}\vspace{-0.1cm}$ (9)

with $\mathbf{n}$ the normalized vector joining the points on the objects and
$\mathbf{J_{\mathit{p}}}$ the Jacobian matrix computed at one object and takes
as reference the other one. Using (8) and (9), we can specify a constraint in
function of joint velocities under the standard form of linear inequalities
(6) with $\mathbf{A}=\mathbf{nJ_{\mathit{p}}}$ and $\mathbf{b}=\dot{d}^{*}$.

###  IV-C Detaching and Collecting

Although we do not cite any specific tools, they can be imagined as shears to
detach the fruit and a basket to catch them while falling. Once the desired
poses are reached by both arms, the cutting tool is triggered and the
collecting arm waits the falling fruit. Then, we select a new fruit with the
method explained in Section III-B and repeat the process.

##  V EXPERIMENTS

We conduct a case study to validate our methodology on artificial orange trees
in our laboratory. The fruit detection and tracking strategies are tested on
images coming from the RGB-D camera, in real-time. We create a simulation
scene with the 3D model of our orange tree as well as the dual arm robot
kinematic model to validate our motion control strategy. The video of the
experiments attached to the paper is also available online at:
https://youtu.be/ObCR5biqlOM.

Figure 7: RGB image with the fruit detection and depth information are shown
in top-left and top-right. Bottom-left and bottom-right images show the HSV
and depth masks.

###  V-A Platform Design

The setup consists of dual-arm mobile cobot BAZAR [4] shown in Figure 1
equipped with two 7-DOF Kuka LWR4 arms and an Intel Realsense D435 RGB-D
camera. We implement the robot controller with the Robot Kinematic Control
Library (RKCL). Hyper-parameters for YOLO are set to default, for DeepSORT
model type: OSnet x0.25 [28], cosine distance: 0.2, overlap distance: 0.5,
age: 300, n init: 10, nn budget: 30. Intel realsense SDK is used for the image
and camera parameters acquisition and the 2D to 3D projection functions. We
use OpenCV 111https://opencv.org for image processing operations. HSV values
used for the segmentation mask are H: 10-20, S: 70-130, V: 80-140 according to
OpenCV’s HSV conventions. Finally,
CoppeliaSim222https://www.coppeliarobotics.com is used to simulate the robot
in real-time.

###  V-B Autonomous Harvesting

Fruit detection and tracking are realized using the camera on BAZAR. Then, to
simulate robot actions, we created a CoppeliaSim scene shown in Figure 6. The
scene includes a functional BAZAR robot and a 3D scan of the artificial orange
tree. We detect the oranges in our lab using the RGB-D camera and add them to
the simulation according to their ID, positions and dimensions wrt the camera
frame with a sampling frequency of 15 Hz. These fruits are considered as
harvesting targets for the robot.

In the first experiment, the camera detects 5 oranges in the scene as shown in
Figure 7. There is 1 undetected orange which is heavily occluded by leaves. We
intentionally create obstacles and move fruits with our hands to test the
performance of the fruit tracking. We validate our detection and tracking
method on 1935 images. The fruit ID assignment is correct in 83% of the 266
occluded images and it is 100% accurate on the rest. Picking and collecting
tools are simulated by a black cylinder and a yellow disk in CoppeliaSim. We
assume that if both of the end-effectors reach their goal pose for the target
fruit, the harvest is successful. The norms of the position error of the
absolute and relative tasks in Figure 8 show that the robot successfully
harvests all the detected fruits in the scene. The experiment is finished in
32 seconds.

In the second experiment, the camera detects all 5 oranges in the scene and
track them accurately at all times. The robot failed to harvest one orange due
to its unreachable position. The rest of the oranges are successfully
harvested.

In both experiments, we do not observe any collision between the robot and the
tree trunk or oranges.

Figure 8: Results of the harvesting experiment. Top: norm of the position
error in meters. Bottom: norm of the orientation error in radian.

##  VI CONCLUSION

This paper describes a dual-arm autonomous harvesting system that uses state-
of-the-art detection, tracking and control methods, which validated in an
orange case study. Fruit detection shows promising results since all fruits
except a heavily occluded one are detected correctly. Tracking results are
positive despite the occlusions in the scene. Detected fruits in real world
are successfully harvested in simulation. However, further experiments with
the real robot and different fruits are necessary for the validation of our
methods. During our real-robot experiments, we have encountered occlusions
caused by robot arms which we could not test in our simulation. Also, since we
do not consider branches as obstacles, interactions with them cause fruit
displacements.

In future, a more dynamic scene is needed to test the robustness of the
tracking algorithm. In some real-world cases, fruits are not always facing
downwards. To face this issue, a more precise detection strategy for the
peduncle can be explored. A planning algorithm should efficiently determine
harvesting path which can also reduce harvest time. Cutting and collecting
strategies with tools are to be determined after real-robot experiments due to
the difficulties on modeling physical interactions in robot simulators.

The methods presented in this paper provide a step towards the goal of fully
autonomous fruit harvesting systems. In this context, one arm solutions have
already shown promising results. We believe that using dual-arm robots will
improve harvesting performance and efficiency.

## References

  * [1] E. Karan and S. Asgari, “Resilience of food, energy, and water systems to a sudden labor shortage,” _Environment Systems and Decisions_ , vol. 41, no. 1, pp. 63–81, 2021.
  * [2] G. Kootstra, X. Wang, P. M. Blok, J. Hemming, and E. Van Henten, “Selective harvesting robotics: current research, trends, and future directions,” _Current Robotics Reports_ , vol. 2, no. 1, pp. 95–104, 2021.
  * [3] P. Chiacchio, S. Chiaverini, and B. Siciliano, “Direct and Inverse Kinematics for Coordinated Motion Tasks of a Two-Manipulator System,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 118, no. 4, pp. 691–697, 12 1996\.
  * [4] A. Cherubini, R. Passama, B. Navarro, M. Sorour, A. Khelloufi, O. Mazhar, S. Tarbouriech, J. Zhu, O. Tempier, A. Crosnier, _et al._ , “A collaborative robot for the factory of the future: Bazar,” _The International Journal of Advanced Manufacturing Technology_ , vol. 105, no. 9, pp. 3643–3659, 2019.
  * [5] J. R. Davidson, C. J. Hohimer, C. Mo, and M. Karkee, “Dual robot coordination for apple harvesting,” in _2017 ASABE annual international meeting_. American Society of Agricultural and Biological Engineers, 2017, p. 1.
  * [6] D. SepúLveda, R. Fernández, E. Navas, M. Armada, and P. González-De-Santos, “Robotic aubergine harvesting using dual-arm manipulation,” _IEEE Access_ , vol. 8, pp. 121 889–121 904, 2020.
  * [7] S. Mehta and T. Burks, “Vision-based control of robotic manipulator for citrus harvesting,” _Computers and Electronics in Agriculture_ , vol. 102, pp. 146–158, 2014.
  * [8] E. J. Van Henten, J. Hemming, B. Van Tuijl, J. Kornet, J. Meuleman, J. Bontsema, and E. Van Os, “An autonomous robot for harvesting cucumbers in greenhouses,” _Autonomous robots_ , vol. 13, no. 3, pp. 241–258, 2002.
  * [9] X. Ling, Y. Zhao, L. Gong, C. Liu, and T. Wang, “Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision,” _Robotics and Autonomous Systems_ , vol. 114, pp. 134–143, 2019.
  * [10] S. Hayashi, K. Shigematsu, S. Yamamoto, K. Kobayashi, Y. Kohno, J. Kamata, and M. Kurita, “Evaluation of a strawberry-harvesting robot in a field test,” _Biosystems engineering_ , vol. 105, no. 2, pp. 160–171, 2010.
  * [11] C. Lehnert, A. English, C. McCool, A. W. Tow, and T. Perez, “Autonomous sweet pepper harvesting for protected cropping systems,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 2, pp. 872–879, 2017.
  * [12] T. T. Santos, L. L. de Souza, A. A. dos Santos, and S. Avila, “Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association,” _Computers and Electronics in Agriculture_ , vol. 170, p. 105247, 2020.
  * [13] H. Kang and C. Chen, “Fast implementation of real-time fruit detection in apple orchards using deep learning,” _Computers and Electronics in Agriculture_ , vol. 168, p. 105108, 2020.
  * [14] A. Koirala, K. Walsh, Z. Wang, and C. McCarthy, “Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘mangoyolo’,” _Precision Agriculture_ , vol. 20, no. 6, pp. 1107–1135, 2019.
  * [15] Y.-P. Liu, C.-H. Yang, H. Ling, S. Mabu, and T. Kuremoto, “A visual system of citrus picking robot using convolutional neural networks,” in _2018 5th International Conference on Systems and Informatics (ICSAI)_. IEEE, 2018, pp. 344–349.
  * [16] R. Kirk, G. Cielniak, and M. Mangan, “L* a* b* fruits: A rapid and robust outdoor fruit detection system combining bio-inspired features with one-stage deep learning networks,” _Sensors_ , vol. 20, no. 1, p. 275, 2020.
  * [17] P. Ganesh, K. Volle, T. Burks, and S. Mehta, “Deep orange: Mask r-cnn based orange detection and segmentation,” _IFAC-PapersOnLine_ , vol. 52, no. 30, pp. 70–75, 2019.
  * [18] P. Roy and V. Isler, “Surveying apple orchards with a monocular vision system,” in _2016 IEEE International Conference on Automation Science and Engineering (CASE)_. IEEE, 2016, pp. 916–921.
  * [19] J. Das, G. Cross, C. Qu, A. Makineni, P. Tokekar, Y. Mulgaonkar, and V. Kumar, “Devices, systems, and methods for automated monitoring enabling precision agriculture,” in _2015 IEEE International Conference on Automation Science and Engineering (CASE)_. IEEE, 2015, pp. 462–469.
  * [20] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, NanoCode012, Y. Kwon, TaoXie, K. Michael, J. Fang, imyhxy, Lorna, C. Wong, Z. Yifu, A. V, D. Montes, Z. Wang, C. Fati, J. Nadar, Laughing, UnglvKitDe, tkianai, yxNONG, P. Skalski, A. Hogan, M. Strobel, M. Jain, L. Mammana, and xylieong, “ultralytics/yolov5: v6.2 - YOLOv5 Classification Models, Apple M1, Reproducibility, ClearML and Deci.ai integrations,” Aug. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.7002879
  * [21] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision (ECCV)_. Springer, 2014, pp. 740–755.
  * [22] P. Pawara, E. Okafor, O. Surinta, L. Schomaker, and M. Wiering, “Comparing local descriptors and bags of visual words to deep convolutional neural networks for plant recognition,” in _International Conference on Pattern Recognition Applications and Methods (ICPRAM)_ , vol. 2. SciTePress, 2017, pp. 479–486.
  * [23] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 3645–3649.
  * [24] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” in _2016 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2016, pp. 3464–3468.
  * [25] B. V. Adorno, P. Fraisse, and S. Druon, “Dual position control strategies using the cooperative dual task-space framework,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2010, pp. 3955–3960.
  * [26] S. Tarbouriech, B. Navarro, P. Fraisse, A. Crosnier, A. Cherubini, and D. Sallé, “An admittance based hierarchical control framework for dual-arm cobots,” _Mechatronics_ , vol. 86, p. 102814, 2022.
  * [27] B. Faverjon and P. Tournassoud, “A local based approach for path planning of manipulators with a high number of degrees of freedom,” in _Proceedings. 1987 IEEE International Conference on Robotics and Automation (ICRA)_ , vol. 4. IEEE, 1987, pp. 1152–1159.
  * [28] K. Zhou, Y. Yang, A. Cavallaro, and T. Xiang, “Omni-scale feature learning for person re-identification,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3702–3712.


Jinzi Qi1, Hugo Van hamme1

# Parameter-efficient Dysarthric Speech Recognition Using Adapter Fusion and
Householder Transformation

###### Abstract

In dysarthric speech recognition, data scarcity and the vast diversity between
dysarthric speakers pose significant challenges. While finetuning has been a
popular solution, it can lead to overfitting and low parameter efficiency.
Adapter modules offer a better solution, with their small size and easy
applicability. Additionally, Adapter Fusion can facilitate knowledge transfer
from multiple learned adapters, but may employ more parameters. In this work,
we apply Adapter Fusion for target speaker adaptation and speech recognition,
achieving acceptable accuracy with significantly fewer speaker-specific
trainable parameters than classical finetuning methods. We further improve the
parameter efficiency of the fusion layer by reducing the size of query and key
layers and using Householder transformation to reparameterize the value linear
layer. Our proposed fusion layer achieves comparable recognition results to
the original method with only one third of the parameters.

Index Terms: dysarthric speech recognition, parameter efficiency, adapter
fusion, Householder transformation

##  1 Introduction

The latest speech technologies, including Automatic Speech Recognition (ASR),
have become increasingly popular and provide great convenience in everyday
life. These technologies have traditionally focused on clear, canonical
speech. However, in recent years, there has been growing interest in
developing ASR models for dysarthric speech [1, 2, 3, 4], a neurological
disease characterized by poor phoneme articulation. Dysarthric speakers face
difficulties in daily communication, highlighting the importance of automatic
dysarthric speech recognition.

State-of-the-art ASR models [5, 6, 7] typically use a Transformer architecture
[8] which employs millions of trainable parameters and needs hundreds of hours
data for model training. However, for dysarthric speech, data scarcity has
been a constant issue, due to difficulties in recruitment, collection and
labeling. Thus, compared to training an E2E ASR model from scratch, finetuning
an E2E ASR model [9, 10, 11] pretrained on abundant canonical speech seems
more feasible. However, finetuning the entire model containing a massive
number of parameters with the limited available dysarthric data can lead to
over-fitting and parameter inefficiency. Moreover, dysarthric speech is
influenced by mixed factors, like gender, pathogenesis (diagnosis) and
severity level. The huge diversity between dysarthric speakers requires
personalized model adaption [2, 3, 12, 13, 14]. Finetuning the entire model
would require a personalized model to be stored for each user, which would
occupy valuable on-device storage space or require a significant upgrade in
server storage for a large number of users [15].

Adapters [16, 17], which only contain a limited number of parameters, can
provide a solution for both data scarcity and limited storage size. An adapter
is a bottleneck module that is injected between layers of a pretrained model
(figure 1(a)). It is trained while the other parts of the pre-trained model
are frozen. Previous studies [18, 19, 20, 21] have demonstrated the
effectiveness of Adapters in parameter-efficient transfer learning. For
dysarthric speech, personalized adapters are trained and tested on atypical
speech data in [15], which has resulted in a similar Word Error Rate (WER)
compared to finetuning. Furthermore, in [14], an auxiliary net that studies
speaker information is added to boost the personalized adapter performance. In
this work, we use a model pretrained on canonical speech and then train the
inserted adapter with dysarthric speech.

Figure 1: (a) Transformer encoder layer with adapters and fusion layer, (b)
details of the fusion layer.

Due to the simplicity and the small size of adapter modules, multiple adapters
can be easily deployed for different groups, individuals and tasks. When
trained adapters from a source domain are available, adapting to a target
domain could benefit from existing adapters. To maximize the transferred
knowledge among different trained adapters, Adapter Fusion [22] (figure 1(b))
is promoted, which utilizes an attention mechanism, called fusion layer, to
combine representations from source adapters and improve the performance of
the target task. Specifically, the attention score is calculated and assigned
to value linear layer output. For dysarthric speech, we could employ adapters
trained on source speaker data and use the fusion layer to maximize the
model's performance for target speaker adaptation. However, this may lead to
an increase in the number of trainable parameters, which could contradict the
goal of having a small storage size for personalized models.

In this work, we apply the Adapter Fusion method to dysarthric speaker
adaptation and speech recognition and investigate the feasibility of improving
its parameter efficiency. Firstly, we train personalized adapters using source
dysarthric speakers' data and then train the fusion layer using target speaker
data for the dysarthric speech recognition task. Secondly, we trace the source
of the performance improvement when using the fusion layer by ablating its two
components: the attention score and the value linear layer. Furthermore, we
inspect the influence of the rotation and scaling operation in the value
linear layer by applying Singular Value Decomposition (SVD) to its weight
matrix. Finally, we explore the possibility of improving the parameter
efficiency of the fusion layer by reducing the key and query linear layer size
and using Householder transformation [23] to reformulate the rotation
operation in the value linear layer.

In section 2, we introduce the Adapter Fusion method and the application of
Householder transformation. Section 3 describes the databases we used and the
experimental settings. Results and analysis will be provided in section 4, and
section 5 gives conclusions.

##  2 Methods

In this section, the method we use is introduced in detail. We use a encoder-
decoder model with hybrid loss [24] as the base ASR model.It contains a
transformer encoder and two decoders: A transformer decoder and a
Connectionist Temporal Classification (CTC) [25] decoder. For simplicity, we
only insert the Adapter module in the last encoder layer and thus the fusion
layer is also used in the last encoder layer.

###  2.1 Adapter Fusion method

The transformer encoder pretrained on canonical speech
$\mathcal{M}_{O}(\cdot)$ maps a target speaker's input sequence $\mathbf{X}$
of duration $T$, to $\mathbf{Y}_{O}=\mathcal{M}_{O}(\mathbf{X})$. Suppose we
have $N$ dysarthric source speakers, and the personalized adapter trained by
each source speaker data is $\mathcal{M}_{a_{n}}(\cdot)$, $n\in[1,N]$. Then
the output of adapter $n$ is
$\mathbf{Y}_{a_{n}}=\mathcal{M}_{a_{n}}(\mathcal{M}_{O}(\mathbf{X}))$, the
stacked adapter output is
$\mathbf{Y}_{A}=[\mathbf{Y}_{a_{1}},\mathbf{Y}_{a_{2}},...,\mathbf{Y}_{a_{N}}]$.

For the fusion layer, we name the whole layer by $\mathcal{M}_{F}(\cdot)$,
weight matrix of value linear layer by $\mathbf{W}$ (no bias used in this
layer), the key linear layer by $\mathbf{K}(\cdot)$ and query linear layer by
$\mathbf{Q}(\cdot)$. Then in the original fusion layer [22], the fusion layer
output is
$\mathbf{Y}_{F}=\sum^{N}_{n=1}\alpha_{n}\mathbf{Y}_{a_{n}}\mathbf{W}+\mathcal{R}$,
where $\alpha_{n}$ is the attention score for adapter $n$,
$\alpha_{n}=softmax_{over~{}n}(\mathbf{Q}(\mathbf{Y}_{O})\times\mathbf{K}(\mathbf{Y}_{a_{n}})^{\prime})$,
$\mathcal{R}$ is a residual term (see figure 1(a)).

To avoid gradient vanishing, different from the original version, we add a
layer-normalization layer $LN(\cdot)$ for key and query term, then the
attention score could be written as
$\alpha_{n}=Softmax_{over~{}n}(LN(\mathbf{Q}(\mathbf{Y}_{O}))\times
LN(\mathbf{K}(\mathbf{Y}_{a_{n}}))^{\prime})$. In the following experiments,
when we ablate the attention score in the fusion layer, the fusion output
becomes
$\mathbf{Y}_{F}=\frac{1}{N}\sum^{N}_{n=1}\mathbf{Y}_{a_{n}}\mathbf{W}+\mathcal{R}$.
When eliminating the value linear layer, the fusion output is
$\mathbf{Y}_{F}=\sum^{N}_{n=1}\alpha_{n}\mathbf{Y}_{a_{n}}+\mathcal{R}$.

The weight matrix of value linear layer $\mathbf{W}$ is initialized with an
all-one-diagonal and the rest with small weights ($1e-6$) [22, 19]. To
guarantee stable adapter outputs and avoid overtraining, $\mathbf{W}$ is
regularized to the identity matrix by introducing an additional loss term:

$L_{reg}=||\mathbf{I_{W}}-\mathbf{W}||^{2}$ (1)

where $\mathbf{I_{W}}$ is an identity matrix of same size as $\mathbf{W}$.
Then the total loss function during target speaker adaptation training is:

$L=(1-\lambda_{1})*L_{ASR,Trans}+\lambda_{1}*L_{ASR,CTC}+\lambda_{2}*L_{reg}$
(2)

where $L_{ASR,Trans}$ is the loss from transformer decoder, $L_{ASR,CTC}$ is
the loss from CTC decoder.

If we train the fusion layer on target speaker data, the performance could
benefit from fusing learned knowledge from trained source adapters. However,
compared to a single personalized adapter, the fusion layer may have no
advantage in the number of trainable parameters. In this work, we explore
methods to improve the parameter efficiency of the fusion layer. A natural
choice is to reduce the size of $\mathbf{K}(\cdot)$ and $\mathbf{Q}(\cdot)$.
Further on, in the next subsection, we will work on the value linear layer.

###  2.2 Adapter Fusion with Householder transformation

The weight matrix $\mathbf{W}$ of value linear layer acts on the adapter
output. Through SVD, we obtain two orthogonal matrices $\mathbf{U}$ and
$\mathbf{V}$ representing a rotation/reflection and a diagonal matrix
$\mathbf{\Sigma}$ for scaling, where $\mathbf{W}=\mathbf{U\Sigma V}^{T}$.
After training is complete, we can evaluate the effectiveness of the rotation
and scaling operations on the trained $\mathbf{W}$ by forming a new matrix
$\mathbf{W_{UV}}=\mathbf{UV}^{T}$ for rotation and another
$\mathbf{W_{\Sigma}}$ for scaling. $\mathbf{W_{\Sigma}}$ is a diagonal matrix
and its diagonal vector is calculated by
$(\mathbf{\Sigma}_{max}-\mathbf{\Sigma}_{min})\frac{\textbf{c}-max(\textbf{c})}{max(\textbf{c})-min(\textbf{c})}+\mathbf{\Sigma}_{max}$,
where $\mathbf{\Sigma}_{max}$ and $\mathbf{\Sigma}_{min}$ are the maximum and
minimum of diagonal of $\mathbf{\Sigma}$, and $\mathbf{c}$ is a vector
containing l-2 norm of the rows of $\mathbf{W}$.

Therefore, we can reformulate the matrix $\mathbf{W}$ as a scaling vector and
a rotation matrix. Since the matrix is regularized to an identity matrix,
there is redundancy in its parameters, and it should be full rank. To preserve
the rank of the matrix while reducing the number of trainable parameters, we
employ Householder transformation [23] to reparameterize the rotation part of
$\mathbf{W}$.

The Householder transformation $\mathbf{P}$ describes a reflection about a
hyperplane orthogonal to $\mathbf{v}$, a length-preserving orthogonal
transform :

$\mathbf{P}=\mathbf{I_{W}}-2\mathbf{v}\mathbf{v}^{T}$ (3)

where $\mathbf{v}$ is column unit vector. Since the product of orthogonal
matrices is again orthogonal, we can use it as a reparamerization of
$\mathbf{UV}^{T}$. Suppose $\mathbf{W}$ has $d$ dimensions and $d^{2}$
trainable elements. Then to reparameterize $\mathbf{W}$, the
$\mathbf{v}$-vector should have $d$ entries, and since it's a unit vector, it
has $d-1$ degrees of freedom. If we employ $d$ Householder matrices and use
one scaling vector $\mathbf{s}$ ($d$-dimension), the total degrees of freedom
($d\times(d-1)+d$) would be the same as $\mathbf{W}$. If the model obtains
comparable performance using less than $d$ Householder matrices, the fusion
layer could achieve higher parameter efficiency.

To ensure the transform can be initialize close to the identity matrix, we use
couples of $\mathbf{v}$-vectors:

$\begin{split}\mathbf{P}_{c}=&(\mathbf{I_{W}}-2\frac{(\mathbf{v}_{c,1}-\mathbf{v}_{c,2})(\mathbf{v}_{c,1}-\mathbf{v}_{c,2})^{T}}{||\mathbf{v}_{c,1}-\mathbf{v}_{c,2}||^{2}})\times\\\
&(\mathbf{I_{W}}-2\frac{(\mathbf{v}_{c,1}+\mathbf{v}_{c,2})(\mathbf{v}_{c,1}+\mathbf{v}_{c,2})^{T}}{||\mathbf{v}_{c,1}+\mathbf{v}_{c,2}||^{2}})\end{split}$
(4)

where $\mathbf{v}_{c,1}$ and $\mathbf{v}_{c,2}$ form the $c$-th couple. We
initialize them with standard normally distributed values and rescale
$\mathbf{v}_{c,1}$ to unit length and $\mathbf{v}_{c,2}$ to length
$\frac{0.01}{\sqrt{d}}$. Then the final rotation matrix $\mathbf{P}_{C}$ using
$C$ $\mathbf{v}$-vector couples is written as:

$\begin{split}\mathbf{P}_{C}=\prod_{c=1}^{C}\mathbf{P}_{c}\end{split}$ (5)

Finally, we can add a diagonal scaling matrix $\mathbf{\Sigma_{s}}$ using
scaling vector $\mathbf{s}$ on its diagonal:

$\mathbf{W}_{C}=\mathbf{\Sigma_{s}}*\mathbf{P}_{C}$ (6)

Notice that this does not allow to build any square matrix, but seemed to
suffice for obtaining good performance.

##  3 Experiments

###  3.1 Datasets

We use CGN dataset [26] (excluding the ``a,c,d,e'' components) as the
canonical speech data to pretrain the ASR model. It contains more than
300-hour Dutch speech. For dysarthric speech, Domotica dataset [27] is used,
which contains around 9-hour dysarthric Dutch speech from 17 speakers, in
total $4173$ utterances. The content is commands related to home automation,
such as ``turn on the light in the kitchen''. The intelligibility scores of
speakers are provided and we categorize speakers into 3 severity levels: high
(score$\textgreater$80, 5 speakers), medium (80$\geq$score$\textgreater$70, 7
speakers), low(70$\geq$score, 5 speakers).

###  3.2 Training strategy

Training proceeds through the following steps (see figure 1(a)):

  * Step 1

Pretrain the transformer ASR model with canonical speech. Freeze the
pretrained model.

Step 2

Insert one adapter into the last encoder layer and train it with $N-3$ source
dysarthric speakers' data jointly.

Step 3

For each of $N$ source dysarthric speakers, initialize the adapter from the
one trained in Step 2 and train each adapter using each source speaker's data
only. Then freeze the $N$ adapters.

Step 4

Insert the fusion layer after the $N$ adapters and train with the target
dysarthric speaker's data.

In the experiment, we divide 15 dysarthric speakers into 5 subsets and each
subset includes 3 speakers with different severity levels. In each trial, we
use four subsets as source speakers and one subset as the target speakers. The
remaining two speakers (pp34, pp35) are consistently used as source speakers.
Thus number of source speakers is $N=14$. In Step 2 we use 3 subsets of the
source speakers and pp34, pp35 for training and 1 subset for validation. In
Step 3, for each of the $N$ source speakers, we use $90\%$ data for training
and $10\%$ for validation. In Step 4, data of each target speaker is divided
into five parts, and during fusion layer training, we utilize three parts
($60\%$) for training, one for validation and one for testing in each fold. On
average, $5\%$ is around $1.5$ minute per speaker and $60\%$ is about $19$
minutes. Our metric for speech recognition is Character Error Rate (CER) as it
is more universal across tasks. 73 characters are used. The provided results
are CER averaged over 15 speakers and five data-folds of each speaker.

###  3.3 Network and training setup

The speech features used in the model are $83$-dimensional filter bank and
pitch features. We implement the method based on the ESPnet toolkit [24]. The
transformer encoder has 12 layers and the transformer decoder has 6 layers.
For Step 1, we use a batch size of $64$ and the ``Noam'' optimizer [8] with a
learning rate of $10$ and $25000$ warm-up steps. The total number of training
epochs is 230 and the final pretrained model is averaged over the 10 epochs
with the highest validation accuracy. For Step 2-4, as well as in case of
finetuning the model, we use the Adam optimizer with a learning rate of
$0.001$ and early stopping with a patience of $20$. The batch size is $32$.
The final model is averaged over three checkpoints with highest validation
accuracy. The dimension of the transformer encoder layer output is $256$. The
inner dimension (size of ``Down projection'') of the Adapter module is also
chosen as $256$ since this size gives the best performance in our preliminary
experiments, meaning that the module is not a traditional bottleneck shape.
$\mathbf{K}(\cdot)$ and $\mathbf{Q}(\cdot)$ in the fusion layer have an
original size of $256$ [22], and we use $64$ as the reduced size. In the loss
function, we set $\lambda_{1}=0.3$ and $\lambda_{2}=0.01$. Beam size $4$ is
employed for joint decoding.

##  4 Results

In this section, we provide the results of using Adapter Fusion with/without
Householder transformation.

###  4.1 Adapter Fusion performance

We first compare the dysarthric speech recognition performance using different
models:

  * Pretrain:

Test the pretrained model obtained in Step 1.

FT-Enc:

Finetune the pretrained model encoder with target speaker data.

FT-EncDec:

Finetune the whole pretrained model with target speaker data.

Pretrain-Adpt:

Test the model in Step 2.

Source-Adpt-avg:

Test the model in Step 3, average outputs of all source adapters as the target
speaker's output.

Target-Adpt:

Finetune the pretrained adapter with target speaker data.

Fusion-256dAtt+W:

Test the model in Step 4.

Fusion-64dAtt+W:

Use fusion layer with reduced dimension 64 of $\mathbf{K}(\cdot)$ and
$\mathbf{Q}(\cdot)$ in Step 4 and test the model.

Fusion-64dAtt/W:

Use fusion layer with eliminated value linear layer / attention score in Step
4 and test the model.

Fusion-$\mathbf{W_{UV}}$/$\mathbf{W_{\Sigma}}$:

Do SVD on matrix $\mathbf{W}$ in Fusion-W, replace it with $\mathbf{W_{UV}}$ /
$\mathbf{W_{\Sigma}}$ and test model.

Table 1 provides the trainable parameter count and CER of these models. If the
model is trained with target speaker data, the training data amounted to
$60\%$ of all data. Figure 2 shows the CER as a function of the amount of
target training data.

To assess the effectiveness of the pretrained model in speech recognition, we
test it on a new canonical speech dataset [28] and achieved a reasonable CER
of $6.3\%$. When testing on dysarthric speech, we obtained a high CER due to
domain mismatch. Finetuning methods yield the best recognition results when
$60\%$ data is used. However, as shown in figure 2, when only $5\%$ of the
data is used, the model is poorly trained with finetuning, resulting in a
higher CER than other methods.

When we test the model with existing adapters, the pretrained adapter performs
better than averaging source speaker adapters, as simple averaging might cause
a performance drop due to the existence of individuality among adapters.
Training a personalized adapter for the target speaker results in reasonable
recognition results adding only $0.5\%$ parameters, while a fusion layer
produces an even better CER with more parameters than Target-Adpt. Both
methods could not surpass the finetuning methods in our experiment setting
when more than $5\%$ data is used, except for the very-low-resource case
($5\%$) due to the small parameter count. By reducing the size of the
$\mathbf{K}(\cdot)$ and $\mathbf{Q}(\cdot)$ to $64$, the CER is even further
reduced while the layer has fewer parameters. This might be because of
overtraining of the larger model.

Table 1: Number of target-specific trainable parameters and CER in % when using at most 60 % of data. Name | #para | CER
---|---|---
Pretrain | - | 49.98
FT-Enc | 17.7M | 1.62
FT-EncDec | 27.2M | 1.39
Pretrain-Adpt | - | 13.27
Source-Adpt-avg | - | 14.08
Target-Adpt | 131.5k | 4.40
Fusion-256dAtt+$\mathbf{W}$ | 197.6k | 2.85
Fusion-64dAtt+$\mathbf{W}$ | 98.6k | 2.61
Fusion-64dAtt | 33.0k | 8.38
Fusion-$\mathbf{W}$ | 65.5k | 3.03
Fusion-$\mathbf{W_{UV}}$ | - | 6.62
Fusion-$\mathbf{W_{\Sigma}}$ | - | 13.34
Fusion-$\mathbf{P_{64}}$ | 32.8k | 3.28
Fusion-$\mathbf{W_{64}}$ | 33.0k | 3.19
Fusion-64dAtt+$\mathbf{W_{64}}$ | 66.1k | 2.79
Figure 2: CER results of different models when training on different amounts
of target speaker data.

To trace the performance improvement brought by the fusion layer, we ablate
the value linear layer (Fusion-64dAtt) or the attention score (Fusion-W) in
the fusion layer and then train it. The case Fusion-W gives a lower CER,
indicating the greater importance of the value linear layer. This might be
because the adaptation among dysarthric speakers has smaller differentiation
than other tasks' adaptation [22, 19]. By performing SVD on the weight matrix
$\mathbf{W}$ of the value linear layer, we separate it into rotation and
scaling operations and evaluate the impact of each operation by testing the
model Fusion-W with one single operation. Comparing Fusion-$\mathbf{W_{UV}}$
and Fusion-$\mathbf{W_{\Sigma}}$, we find that the rotation plays a more
important role in modifying the adapter outputs $\mathbf{Y}_{A}$. However,
even using only the scaling vector still results in a performance improvement
over no action (Source-Adpt-avg).

###  4.2 Adapter Fusion with Householder Transformation

To enhance parameter efficiency, we aim to reformulate the rotation operation
in the matrix $\mathbf{W}$ using Householder transformation. Table 1 provides
the number of trainable parameters and CER results using $64$
$\mathbf{v}$-vector couples to form an orthogonal $\mathbf{P}_{64}$ matrix. In
Fusion-$\mathbf{P}_{64}$, we use $\mathbf{P}_{64}$ only as the weight matrix,
achieving $3.28\%$ CER, while in Fusion-$\mathbf{W_{64}}$, we add the scaling
as in equation (6), which further improves the CER. In
Fusion-64dAtt+$\mathbf{W_{64}}$, we complete the model by adding the
attention, yielding a CER of $2.79\%$, which is very close to the baseline
model Fusion-64dAtt+$\mathbf{W}$ with two-thirds of its parameters and is even
higher than baseline model Fusion-256dAtt+$\mathbf{W}$ with only one-third of
its parameters.

Table 2 compares the CER for using different $C$ values in model
Fusion-$\mathbf{W_{C}}$ with the baseline case Fusion-W. Our results show that
using $C=64$, the model Fusion-$\mathbf{W_{64}}$ achieves similar CER as the
baseline while using only half of the parameters. Table 2 also demonstrates
that when we have sufficient training data ($60\%$), increasing the number of
$\mathbf{v}$-vectors ($C$ value) always benefits performance, and it will
reach an upper limit and doesn't outperform the baseline. When training data
is limited ($5\%$), increasing the $C$ value will initially improve the model
performance but will then suffer from a lack of training data as well. The
results show that the Householder factorization is scalable way to trade off
the target-specific model size for accuracy. Notice also that applying $d$
Householder factors has a similar complexity as multiplication with a $d\times
d$ matrix.

Table 2: CER in % when using different $C$ values in model Fusion-$\mathbf{W_{C}}$, compared with baseline case (bl) Fusion-$\mathbf{W}$. Training data % | CER of bl/Fusion-$\mathbf{W_{C}}$ when C =
---|---
bl | 1 | 2 | 8 | 64 | 128
5% | 8.42 | 10.90 | 10.48 | 9.62 | 8.90 | 9.20
60% | 3.03 | 7.81 | 6.62 | 4.53 | 3.19 | 3.12

##  5 Conclusions

Dysarthric speech recognition and speaker adaptation face challenges due to
data scarcity and huge diversity between dysarthric speakers. Finetuning, a
common method of transferring knowledge from a rich resource domain (in our
case canonical speech recognition), has drawbacks such as overfitting and high
storage requirements for personalized use. Thanks to their small size and ease
of use, Adapter modules offer a suitable solution. Adapter Fusion can boost
knowledge transfer between learned source speaker adapters, but it may
increase the number of parameters used.

In this study, we apply Adapter Fusion to target speaker adaptation for speech
recognition, achieving acceptable CER results with significantly fewer
trainable parameters than classical finetuning methods. We also analyze the
performance improvement brought by the fusion layer and identify the critical
role played by the rotation operation of the value linear layer weight matrix
$\mathbf{W}$. Finally, we improve the parameter efficiency of the fusion layer
by reducing the size of the query and key linear layer and reformulating
$\mathbf{W}$ using Householder transformation. The proposed fusion layer
achieves comparable recognition results as our starting point with only one
third of the parameters.

In the future, we plan to further validate the generality of the proposed
methods on additional datasets and different tasks. Additionally, we will
explore solutions for the zero-shot case [29] of dysarthric speech
recognition, taking practical scenarios into consideration where the model has
no access to the target speakers during training.

##  6 Acknowledgements

The research was supported by KU Leuven Special Research Fund grant
C24M/22/025 and the Flemish Government under the ``Onderzoeksprogramma
Artificiële Intelligentie (AI) Vlaanderen'' programme.

## References

  * [1] D. Wang, J. Yu, X. Wu, L. Sun, X. Liu, and H. Meng, ``Improved end-to-end dysarthric speech recognition via meta-learning based model re-initialization,'' in _2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP)_. IEEE, 2021, pp. 1–5.
  * [2] M. Geng, X. Xie, Z. Ye, T. Wang, G. Li, S. Hu, X. Liu, and H. Meng, ``Speaker adaptation using spectro-temporal deep features for dysarthric and elderly speech recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 2597–2611, 2022.
  * [3] J. Tobin and K. Tomanek, ``Personalized automatic speech recognition trained on small disordered speech datasets,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 6637–6641.
  * [4] Z. Yue, E. Loweimi, Z. Cvetkovic, H. Christensen, and J. Barker, ``Multi-modal acoustic-articulatory feature fusion for dysarthric speech recognition,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7372–7376.
  * [5] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, ``Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7829–7833.
  * [6] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , ``Conformer: Convolution-augmented transformer for speech recognition,'' _Proc. Interspeech 2020_ , pp. 5036–5040, 2020\.
  * [7] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [9] J. Shor, D. Emanuel, O. Lang, O. Tuval, M. Brenner, J. Cattiau, F. Vieira, M. McNally, T. Charbonneau, M. Nollstadt _et al._ , ``Personalizing asr for dysarthric and accented speech with limited data,'' _Proc. Interspeech 2019_ , pp. 784–788, 2019.
  * [10] Y. Takashima, R. Takashima, T. Takiguchi, and Y. Ariki, ``Knowledge transferability between the speech data of persons with dysarthria speaking different languages for dysarthric speech recognition,'' _IEEE Access_ , vol. 7, pp. 164 320–164 326, 2019.
  * [11] P. Wang, B. BabaAli, and H. Van hamme, ``A study into pre-training strategies for spoken language understanding on dysarthric speech,'' _Proc. Interspeech 2021_ , 2021.
  * [12] R. Takashima, T. Takiguchi, and Y. Ariki, ``Two-step acoustic model adaptation for dysarthric speech recognition,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6104–6108.
  * [13] R. Turrisi and L. Badino, ``Interpretable dysarthric speaker adaptation based on optimal-transport,'' _arXiv preprint arXiv:2203.07143_ , 2022.
  * [14] M. K. Baskar, T. Herzig, D. Nguyen, M. Diez, T. Polzehl, L. Burget, J. Černockỳ _et al._ , ``Speaker adaptation for wav2vec2 based dysarthric asr,'' _Proc. Interspeech 2022_ , pp. 3403–3407, 2022.
  * [15] K. Tomanek, V. Zayats, D. Padfield, K. Vaillancourt, and F. Biadsy, ``Residual adapters for parameter-efficient asr adaptation to atypical and accented speech,'' in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021, pp. 6751–6760.
  * [16] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, ``Learning multiple visual domains with residual adapters,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [17] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, ``Parameter-efficient transfer learning for nlp,'' in _International Conference on Machine Learning_. PMLR, 2019, pp. 2790–2799.
  * [18] A. Kannan, A. Datta, T. N. Sainath, E. Weinstein, B. Ramabhadran, Y. Wu, A. Bapna, Z. Chen, and S. Lee, ``Large-scale multilingual speech recognition with a streaming end-to-end model,'' _Proc. Interspeech 2019_ , pp. 2130–2134, 2019.
  * [19] W. Hou, H. Zhu, Y. Wang, J. Wang, T. Qin, R. Xu, and T. Shinozaki, ``Exploiting adapters for cross-lingual low-resource speech recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 317–329, 2021.
  * [20] R. Karimi Mahabadi, J. Henderson, and S. Ruder, ``Compacter: Efficient low-rank hypercomplex adapter layers,'' _Advances in Neural Information Processing Systems_ , vol. 34, pp. 1022–1035, 2021.
  * [21] R. Wang, D. Tang, N. Duan, Z. Wei, X.-J. Huang, J. Ji, G. Cao, D. Jiang, and M. Zhou, ``K-adapter: Infusing knowledge into pre-trained models with adapters,'' in _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , 2021, pp. 1405–1418.
  * [22] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych, ``Adapterfusion: Non-destructive task composition for transfer learning,'' in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 2021, pp. 487–503.
  * [23] A. S. Householder, ``Unitary triangularization of a nonsymmetric matrix,'' _Journal of the ACM (JACM)_ , vol. 5, no. 4, pp. 339–342, 1958.
  * [24] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen _et al._ , ``Espnet: End-to-end speech processing toolkit,'' _Proc. Interspeech 2018_ , pp. 2207–2211, 2018.
  * [25] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [26] N. Oostdijk, W. Goedertier, F. Van Eynde, L. Boves, J.-P. Martens, M. Moortgat, and R. H. Baayen, ``Experiences from the spoken dutch corpus project,'' in _LREC 2002_. European Language Resources Association, 2002, pp. 340–347.
  * [27] B. Ons, J. F. Gemmeke, and H. Van hamme, ``The self-taught vocal interface,'' _EURASIP Journal on Audio, Speech, and Music Processing_ , vol. 2014, no. 1, pp. 1–16, 2014.
  * [28] L. Bollens, B. Accou, H. Van hamme, and T. Francart, ``A large auditory eeg decoding dataset,'' 2023. [Online]. Available: https://doi.org/10.48804/K3VSND
  * [29] Q. Xu, A. Baevski, and M. Auli, ``Simple and effective zero-shot cross-lingual phoneme recognition,'' _arXiv preprint arXiv:2109.11680_ , 2021.