# Accessible Torque Bandwidth of a Series Elastic Actuator Considering the
Thermodynamic Limitations

Bhanuka Silva1 and Navinda Kottege2
1Department of Electronic and Telecommunication Engineering, University of
Moratuwa, Sri Lanka
2Robotics and Autonomous Systems Group, CSIRO, Pullenvale, QLD 4069, Australia

###### Abstract

Within the scope of the paper, electromechanical and thermodynamic models are
derived for a series elastic actuator and open loop and closed loop torque
bandwidth parameters are analysed considering the thermodynamic behaviour of
the actuator. It was observed that the closed loop torque bandwidth of the
electromechanical subsystem of the actuator was not accessible in the entire
torque reference amplitude range due to thermodynamic limitations. Therefore,
a stator winding temperature estimation based adaptive controller is utilised
and analysed to improve the accessibility of the controller based torque
bandwidth. This paper implements the methodology on a HEBI Robotics X5-9
actuator as a case study.

##  1 Introduction

In many robotic applications, it is required to control robot-environment
interaction forces. For this purpose, it is essential to have high performance
actuators providing the ability of controlling applied torque to the joints of
mechanisms.

In series elastic actuators (SEAs), an elastic element is placed between the
load side and the gear reduced electric motor side. Many previous works like
[?], [?] and [?] discuss the inherent pros and cons of such actuators. The
deflection in the elastic element produces the required amount of torque to
the load side of the actuator, therefore, force control based strategies can
be converted to position control based strategies of the elastic element’s
deflection which is one of the main advantage of SEA based control systems
[?]. A careful parameter identification of the SEA electromechanical subsystem
allows the utilisation of model based feedback and disturbance rejection
controllers for the actuator output’s robust torque tracking accuracy.

Neglecting thermal limitation analysis in torque control applications has a
risk factor of driving the internal motor to extreme stator winding
temperatures resulting in irreversible internal demagnetisation damages [?].
Torque control algorithms and thermodynamic models are often treated as
separate branches for an actuator, therefore, the literature rarely adopts the
analysis of closed loop torque bandwidth of an actuator considering the
thermodynamic performance limitations. This paper assesses the methodology in
modelling electromechanical and thermodynamic subsystems and conducting
experiments to bridge the gap.

Figure 1: X5-9 series elastic actuator from HEBI Robotics. Casing is mounted
to a fixed metal base while the actuator output is fixed for an experiment.

###  1.1 Related work

For the purpose of modelling of a SEA, [?] exploit a two-mass model which
includes the interaction of the load side dynamics that is modelled using a
simple inertia and a damping coefficient. [?] add an additional layer of
impedance interaction for modelling any interaction with the environment.
Therefore, the two mass model will effectively characterise most of the linear
dynamics of the actuator. They improve force control performance using a
disturbance observer (DOB) and a feedforward controller and explore the robust
stability of the system against changes in environmental parameters. [?]
discuss a novel method to use an adaptive disturbance observer to estimate the
parameters of the nominal plant inverse model which in turn minimises the
errors of the DOB. [?] and [?] further discuss the observer design and [?]
reveal that an open loop DOB where the observer is applied only to the plant
model and a closed loop DOB where the observer is applied to the plant as well
as the controller would behave in a similar manner in enforcing the nominal
plant model. The former implementation is associated in this paper.

For a model based control system design, careful parameter identification of
SEA is important. Brushless direct current (BLDC) motors, due to their high
efficiency, less maintenance, wide range of operating speed, high power
density, small size and good torque-speed characteristics are preferred over
brushed DC motors. However, the dynamic behaviour of BLDC motors is non-linear
[?]. The torque ripple effect caused by multiple phases and magnetic fields is
considered one of the main reasons for such non-linearities [?]. A combination
of feedback linearisation strategy and an observer is often used to enforce
the nominal plant model by compensating for such effects.

[?] assess the physically feasible torque control bandwidth of an integrated
toque controllable actuator and the controller independent physical upper
bound for the bandwidth of the actuator when the output of the actuator is
fixed as well as varied in inertia. A safe operation time is computed for each
torque-frequency condition by looking at the thermal actuator properties. For
the inclusion of thermal limitations, thermodynamic modelling of an actuator
is important and lumped parameter modelling technique is argued to have good
accuracy in correctly predicting the thermal characteristics in a wide range
of operating conditions [?], [?] and [?].

The organisation of the paper is as follows. Section 2 discusses the
methodology of SEA modelling and controller implementation. Section 3
emphasises on the torque tracking results that are obtained in conjunction
with thermodynamic behaviour. The paper concludes in section 4 discussing the
effectiveness of the novel methodology in maximising the accessible torque
bandwidth.

##  2 Methodology

This section discusses the electrical, mechanical and thermodynamic subsystems
of a SEA emphasising the modelling approximations and parameter identification
methodologies. A brief theoretical analysis on DOBs and model based controller
designing are also discussed in this section for applied torque’s robust
tracking accuracy.

Figure 2: Linear model of the SEA.

###  2.1 Electrical subsystem

The SEA input voltage is regulated to the nominal value and an internal
controller regulates the voltage to the terminals of the BLDC motor to
generate electromagnetic torque $\tau_{m}(t)$. Readings of three hall sensors
at 120∘ offset are used allowing six different control voltage combinations
for the three phases.

Linear dynamics of the electrical subsystem are characterised by effective
winding resistance $R$ and winding inductance $L$. The following equations
represent the dynamics of the electrical subsystem of the actuator. $K_{e}$
and $K_{\tau}$ represent the back electromotive force constant and motor
torque constant respectively. $V(t)$ is the applied winding voltage and
$i_{m}(t)$ is the winding current of the linear model. $\delta(t)$ is the
Dirac delta function.

$V(t)-K_{e}\dot{\theta}_{m}(t)=Ri_{m}(t)+L(\dot{i_{m}}(t)+i_{m}(0)\delta(t))$
(1)

$\tau_{m}(t)=K_{\tau}i_{m}(t)$ (2)

###  2.2 Mechanical subsystem

Time domain linear equation given in equation 3 characterises the mechanical
subsystem of the BLDC motor. The rotor inertia is $J_{m}$ and the viscous
friction is $B_{m}$. The effects of cogging torque, frictional torque and
coulomb frictional torque are neglected as non linearities to be compensated
by the DOB. $K_{s}$ represents the torque constant of the elastic element and
the generated torque is considered to have a static characteristic as in
equation 4. Deflection of the elastic element is represented by $\theta_{d}$
and the BLDC motor angle by $\theta_{m}$. Inertia, damping and frictional
torques of the gearbox are neglected for the simplicity of the actuator model
and has a gear reduction of $N$ to $1$.

$\tau_{m}(t)-\frac{K_{s}\theta_{d}(t)}{N}=J_{m}(\ddot{\theta}_{m}(t)+\dot{\theta}_{m}(0)\delta(t))+B_{m}\dot{\theta}_{m}(t)$
(3)

$\tau_{spring}=K_{s}\theta_{d}$ (4)

Linear dynamics of the load $A(s)$ are parameterised by the load inertia
$J_{l}$ and the damping coefficient $B_{l}$. In general, load inertia can be
represented by a time varying functional depending on the application of the
actuator. The stiffness of the load side is assumed to be zero.

$A(s)=J_{l}s+B_{l}$ (5)

Figure 2 depicts the linear model of the SEA. Note that Laplace domain
notation ‘s’ is removed for simplicity. Transfer functions represented by
$P_{e}(s)$, $P_{m}(s)$ and $P_{l}(s)$ are shown in the same figure. For the
electrical subsystem;

$\begin{bmatrix}\tau_{m}\\\ \end{bmatrix}=\begin{bmatrix}P_{e}&-sK_{e}P_{e}\\\
\end{bmatrix}\begin{bmatrix}V\\\ \theta_{m}\\\ \end{bmatrix}$ (6)

For the mechanical subsystem;

${\begin{bmatrix}\theta_{d}\\\ \theta_{l}\\\ \theta_{m}\\\
\end{bmatrix}=\begin{bmatrix}\frac{P_{m}N^{-1}}{D}&-\frac{N^{-2}P_{m}+P_{l}}{D}&-\frac{P_{l}}{D}\\\
\frac{K_{s}P_{l}P_{m}N^{-1}}{D}&\frac{P_{l}}{D}&\frac{P_{l}(1+K_{s}N^{-2}P_{m})}{D}\\\
\frac{P_{m}(1+K_{s}P_{l})}{D}&-\frac{N^{-1}P_{m}}{D}&\frac{K_{s}N^{-1}P_{m}P_{l}}{D}\\\
\end{bmatrix}\begin{bmatrix}\tau_{m}\\\ d_{d}\\\ d_{l}\\\ \end{bmatrix}}$ (7)

Here $D(s)=1+K_{s}N^{-2}P_{m}(s)+K_{s}P_{l}(s)$.

###  2.3 System identification of the electromechanical subsystem

An open loop frequency sweep signal is fed while the output of the actuator is
fixed simulating an infinite load side inertia ($J_{l}\rightarrow\infty$). The
system transfer function $\frac{\theta_{d}(s)}{V(s)}$ obtained from equations
7 and 6 then reduces to;

$\frac{\theta_{d}(s)}{V(s)}=\frac{\frac{K_{\tau}}{N}}{A_{3}s^{3}+A_{2}s^{2}+A_{1}s+A_{0}}$
(8)

Denominator coefficients are simplified in the equation 8 and stand for;
$A_{0}=\frac{K_{s}R}{N^{2}}$,
$A_{1}=K_{e}K_{\tau}+\frac{K_{s}L}{N^{2}}+B_{m}R$, $A_{2}=B_{m}L+J_{m}R$ and
$A_{3}=J_{m}L$. Therefore, the transfer function identification involves in a
third order model fit and experimental data are obtained varying the amplitude
of the commanded PWM reference signal. However, for each amplitude value,
there can be mismatches in the model fit results and it implies the presence
of system non-linearities as well as time variant behaviours. A nominal plant
model $P_{n}$ is derived from the results such that it minimises the
mismatches. Uncertainty estimation of the model fit results with respect to
the selected nominal plant model can be obtained from equation 9. $G_{x}(s)$
represents the experimental model fit for the reference PWM amplitude value
$x$.

$\Delta_{x}(s)=\frac{G_{x}(s)-G_{nominal}(s)}{G_{nominal}(s)}$ (9)

###  2.4 Thermodynamics

Heat energy flow per unit time within the thermodynamic subsystem of the
actuator is modelled using the analogy of an equivalent electrical circuit.
The electrical currents in the Figure 3 represent the power flow as a result
of the heat flux transfer from the identified thermal bodies of the actuator.
A five body electro-analogous network consisting motor stator winding, motor
stator housing, outside of the motor, the controller electronics and the
metallic case of the enclosed actuator could be observed. The network can be
further simplified to a four body analogy with the assumption that the casing
is equal to the ambiance as long as it is fixed to a large metal base
consisting an approximated infinite heat capacity in it. The ambiance $T_{A}$
is controlled to be 25∘C. Actuator’s electronic circuit board temperature
sensor in general reads a higher temperature ($T_{B}$) than the ambiance of
25∘C due to solid-state electronic heat flux, therefore, the resultant power
flow is modelled using a current source $i$ in the model.

Figure 3: Four body electro analogous circuit diagram.

In Figure 3, thermal resistances $R_{1}$ to $R_{3}$ characterise the heat
dissipation between the different bodies inside the actuator. $C_{1}$, $C_{2}$
and $C_{3}$ represent the heat capacity constants. $R_{1}C_{1}$ to
$R_{3}C_{3}$ characterise the thermal time constants $\tau_{1}$ to $\tau_{3}$.
$T_{W}$, $T_{H}$ and $T_{M}$ stands respectively for the instantaneous
temperatures of the stator winding, stator housing and BLDC motor proximity at
a given time.

The power loss due to the stator winding current is called Joule power loss
and is the significant source of heat flux of the model. It can be governed by
the following equation where the electrical winding resistance is also
temperature dependent which in turn cannot be neglected in thermal behaviour.
$R_{A}$ is the electrical winding resistance at 25∘C and $i_{m}$ is the stator
winding current at a given time. $\alpha_{cu}$ represents the temperature
coefficient of resistance.

$P_{J}(T_{w})=R_{A}(1+\alpha_{cu}(T_{W}-T_{A})){i_{m}}^{2}$ (10)

The differential equations 11, 12 and 13 are obtained from the law of
conservation of charge entering and leaving a junction in equivalent
electrical circuit model of Figure 3.

$R_{A}(1+\alpha_{cu}.(T_{W}-T_{A})){i_{m}}^{2}=\\\
\frac{T_{W}-T_{H}}{R_{1}}+C_{1}\frac{d}{dt}(T_{W}-T_{A})$ (11)

$\frac{T_{W}-T_{H}}{R_{1}}=\frac{T_{H}-T_{M}}{R_{2}}+C_{2}\frac{d}{dt}(T_{H}-T_{A})$
(12)

$\frac{T_{H}-T_{M}}{R_{2}}+i=\frac{T_{M}-T_{A}}{R_{3}}+C_{3}\frac{d}{dt}(T_{M}-T_{A})$
(13)

Temperature variation of the thermal bodies is shown in Figure 4 for a test
case with the output fixed and commanded with an open loop PWM value resulting
in a scaled step input of winding current.

Figure 4: General four body temperature variation observation.

According to the temperature curves in Figure 4, $T_{W}$ has the smallest rise
time compared to $T_{H}$ and $T_{M}$ and can have a higher probability in
reaching the operative ceiling defined by the manufacturer. The maximum
permissible winding temperature $T_{MAX}$ is 130∘C for the test actuator
therefore an arbitrary continuous winding current must not result in exceeding
the particular value, or otherwise, the actuator’s internal processor would
limit the PWM value to the BLDC motor, making any operation unpredictable.

Experimental identification of the electro analogous circuit parameters is
conducted with the data gathered for similar step input experiments with
different open loop commanding PWM values. Steady state response for equations
11, 12 and 13 as well as model fitting for the experimental data resulted in
an accuracy of 99.98 percentage in correctly interpreting the observed
behaviour. Results for the model parameters are given in appendix Table 1.

Expected stator winding temperature at the steady state ($T_{W|s.s.}$) for a
known continuous winding current $i_{m}$ can be calculated from the thermal
model when the capacitors are saturated; thus open circuited.

$T_{W|s.s.}=T_{A}+\frac{i_{m}^{2}R_{A}(R_{1}+R_{2}+R_{3})+iR_{3}}{1-\alpha_{cu}i_{m}^{2}R_{A}(R_{1}+R_{2}+R_{3})}$
(14)

The nominal winding current $i_{N}$ is defined as the continuous stator
winding current value that results in having the maximum permissible winding
temperature reached in a steady state operation; i.e. $i_{m}=i_{N}$ when
$T_{W|s.s.}=T_{MAX}$. According to the equation 14, any decrease in $T_{A}$
would yield a higher value for $i_{N}$ and vice versa implying that $i_{N}$
depends on $T_{A}$ in practical applications.

When the actuator is undergoing some periodic operations, equation 14 will be
governed by the root mean square (RMS) value of the stator winding current. In
a practical application with multiple and identical actuators, the actuator
with the highest RMS stator winding current or usually the actuator that
contributes to the maximum joint torque value would thermodynamically limit
the maximum capabilities of the entire system.

It is possible to overload the internal BLDC motor beyond its nominal winding
current for a brief period of time. This situation is characterised by the
stator housing behaving as the ambiance to form a parallel $R_{1}$ and $C_{1}$
circuit subsection. The equation 15 assumes $T_{\beta}$ as the expected steady
state stator winding temperature at an overloaded continuous current. In
practice, as $T_{\beta}>T_{MAX}$, extended overloading would result in
demagnetisations in the motor making it permanently unusable. Therefore, the
overloading should be brief and another parameter should be defined for the
maximum permissible time for a particular overloading condition.

$T_{W}-T_{H}=(T_{\beta}-T_{H})(1-e^{-t/\tau_{1}})$ (15)

A better intuition can be obtained in comparing the overloading current
$i_{O}$ with the nominal winding current $i_{N}$. Equation 16 defines an
overload constant ‘$K_{o}$’ with two approximations made in the thermal model.
Electrical stator winding resistance at $T_{MAX}$ winding temperature is
considered equal to the stator winding resistance at $T_{\beta}$ winding
temperature. Also the losses due to the solid state electronic heat flux $i$
are assumed to be negligible.

$K_{o}=\dfrac{i_{O}}{i_{N}}\sqrt{\dfrac{(T_{MAX}-T_{A})R_{1}}{(T_{MAX}-T_{H})(R_{1}+R_{2}+R_{3})}}$
(16)

Therefore, the actuator’s brief overloading can be characterised by the
following equation that is now dependent on the defined overloading factor.
$t_{on}$ is the maximum safe on-time to overload the actuator for a given
$K_{o}$ value.

$t_{on}=\tau_{1}\,ln(\frac{{K_{o}}^{2}}{{K_{o}}^{2}-1})$ (17)

Figure 5: Maximum permissible on time.

Figure 5 represents the maximum allowable on time for a brief overloading
session. The time is upper bounded to a five times of winding time constant
(5$\times$1.492 s) in order to be defined as a short term operation. It should
be noted that the overloading factor depends on the initial housing
temperature as well.

As the model parameters were calculated with respect to a heating up in the
winding, in order to validate the results for a cooling down instance, the
differential equations can be modified with a $U(-t)$ step function. For
$t>0$, winding current is therefore zero resulting in no Joule power losses
for the rest of the experimental duration. The governing time domain equations
can be simulated and validated with respect to the actual experimental
results.

###  2.5 Stator winding temperature estimation

Once the thermodynamic model is established from experimental results, it can
be utilised along with the ambient temperature feedback and stator winding
current feedback from the actuator to estimate the winding temperature inside
the motor as in equations 18 and 19. This approach is suitable to avoid the
impracticality as well as the complexity of sensing the actual temperature
inside any drive motor. Simplifying the thermodynamic model, it is assumed
that the ambient temperature is equal to the temperature just outside the BLDC
motor and $\tau_{1}<<\tau_{2}$.

$T_{W}(t)=T_{H}(t)-T_{W}(0)e^{\frac{-1}{R_{1}C_{1}}t}+\int_{0}^{t}P_{J}(\tau)g(t-\tau)d\tau$
(18)

$g(t)=\frac{1}{C_{1}}e^{-\frac{1}{R_{1}.C_{1}}t}$ (19)

In the equation 18, $T_{W}(0)$ is the initial condition and $P_{J}(t)$ is the
Joule losses due to the stator winding current as in the equation 10.
Instantaneous housing temperature can also be derived in a similar equation
according to the simplified thermodynamic model.

###  2.6 Model based controller design

The concept of the disturbance observer has been long utilised by researchers
due to its performance in precise and robust control in force related
applications. Following relationship in equation 20 represents the input
output behaviour of the system with reference to the terms given in Figure 2
and Figure 6. $P_{n}(s)+\Delta(s)$ is the nominal plant model added with model
uncertainties. Transfer functions $P_{d1}(s)$ and $P_{d2}(s)$ represent the
contribution of external disturbances $d_{d}(s)$ and $d_{l}(s)$ into the
observed output $\theta_{d}(s)$.

$\begin{bmatrix}\theta_{d}(s)\\\
\end{bmatrix}=\begin{bmatrix}P_{n}(s)+\Delta(s)&P_{d1}(s)&P_{d2}(s)\\\
\end{bmatrix}\begin{bmatrix}V(s)\\\ d_{d}(s)\\\ d_{l}(s)\\\ \end{bmatrix}$
(20)

Figure 6: Proposed control system inclusive of the feedback controller and the
disturbance rejection controller.

The observer highlighted in grey in the Figure 6 corresponds to an open loop
DOB system where the compensation is applied only to the actuator plant
excluding the feedback controller $C_{fb}(s)$. $X(s)$ is the output coming
into the actuator from feedback controller. One of the most common observer
type of nominal plant pseudo inversion is used here and the transfer function
for the observer block can be deduced as;

$\frac{\theta_{d}(s)}{X(s)}=\frac{P_{n}(s)+\Delta(s)}{1-Q(s)+[P_{n}(s)+\Delta(s)][Q(s)P_{n}^{-1}(s)]}+\\\
[1-Q(s)]\frac{P_{d1}(s)d_{d}(s)+P_{d2}(s)d_{l}(s)}{X(s)}$ (21)

Here $Q(s)$ is a Butterworth low pass filter with a degree greater than or
equal to the relative order of $P_{n}(s)$. Considering the behaviour of the
system, at low frequencies, $Q(s)=1$ and results in;

$\frac{\theta_{d}(s)}{X(s)}=P_{n}(s)$ (22)

meaning the system will behave like the nominal plant model. For high
frequencies, $Q(s)=0$ and the system would behave like the open loop plant
with non modelled dynamics and disturbances.

$C_{fb}(s)$ is the feedback controller intended to compensate for the tracking
error. It controls the reference of the pulse width modulated input to the
actuator using a PID controller with suitable gains and clamps. MATLAB
Simulink based PID tuning is conducted according to the response time and
robustness required. Tuned controller augments the open loop cut off frequency
of the actuator to a higher value and would additionally suppress the
resonances as well. In the realisation, discrete time counterparts
$C_{fb}(z)$, $Q(z)$ and $P^{-1}_{n}(z)$ are used according to the selected
sampling rate. Inclusion of the Q filter is to stabilise the model inversion
transfer function. In order to properly realise the filter, the Q filter must
be stable alone and the magnitude should be less than the magnitude of the
inverse of model-uncertainty upper bound. Also the cut off frequency should
lie away from the interested torque bandwidth region.

##  3 Experimental results

Torque tracking of a SEA stands for the accurate control of the output torque
to the load dynamics via the deflection of the elastic element. As only the
stiffness of the elastic element is considered, it inherits static
characteristics therefore, the scenario of torque control could be considered
as a position control problem. Experimental actuator uses encoder readings to
measure the deflection and return the torque value as a feedback. Within the
scope of this section, torque tracking results in conjunction with
thermodynamic behaviour of the actuator are discussed.

###  3.1 Open loop torque response

In open loop configuration, the fixed-output SEA is commanded with a
sinusoidal reference PWM signal which linearly increases in the frequency. No
feedback based control algorithm is enabled, therefore, the torque is
effectively generated according to the open loop plant dynamics. Stator
winding temperature behaviour is monitored and different experiments are
conducted varying the amplitude of the reference signal.

Figure 7: Torque magnitude variation with the frequency.

In Figure 7, time domain behaviour is omitted and instead, the torque
magnitude variation at each instantaneous frequency is plotted along with the
instantaneous winding temperature value. Figure contains the actual data
points as well as some interpolated data points for better visualisation. The
reference signal is duly terminated if the winding temperature is reached
$T_{MAX}$, the maximum permissible value of 130∘C and no more data points are
acquired for higher frequency values. As the winding temperature values
relatively become higher with higher reference amplitudes, behaviours
corresponding to 0.4 and 0.2 reference amplitudes are not visible in the
figure and are emphasised in the figure 8.

Figure 8: Torque magnitude variation graph emphasising the lower reference
values.

The results from figures 7 and 8 emphasise the torque generation
characteristics of the actuator with respect to the frequency of the reference
signal. From the respective transfer function analysis, minimum open loop
torque bandwidth is obtained as 1.54 Hz and no thermal limitations are
observed before the particular value. The surfaces are not plotted beyond 20
Hz.

A noticeable trench that portrays lower winding temperatures can be observed
in the figure 7. Initially at the lower frequencies, the actuator can generate
torque values with a higher gain, therefore, generates more Joule losses and
it later decreases as the torque gain reduces with the frequency but
eventually reaches higher Joule losses again as the frequency increases.

Output locked scenario is optimal for benchmarking instances but in reality
the torque utilisation is much more relevant for the output non-locked
scenarios. It corresponds to having a non-infinite load side inertia
($J_{l}$).

Figure 9: Torque magnitude variation for non-zero load side dynamics.

Figure 9 shows the torque characteristics for four different load side
dynamics and the open loop input signal is a 0.6 reference PWM signal. The
frequency where the highest torque magnitudes are produced, drifts from left
to right when the load dynamics given by $P_{l}(s)$ starts from $0$ (output
fixed) and increased (load side inertia $J_{l}$ is reduced). According to the
winding temperature based colour mapping in the figure, it can be concluded
that the highest winding temperatures are predominantly obtained from the
output fixed instance during higher frequencies, therefore, the possible
thermal limitations would be visible there first.

###  3.2 Closed loop torque tracking

The torque tracking capabilities in a closed loop system will be governed by
the feedback controller algorithm and is subjective to change with alterations
in the controller parameters. For example, re-tuning proportional, integral
and derivative gains of a PID controller will change the response of a system
to a given input. In order to provide a good estimation on the accessible
torque bandwidth of the actuator, model based control algorithm parameters are
obtained using Simulink PID tuning.

Figure 10: Closed loop output torque gain variation with frequency.

Figure 10 visualises the torque tracking results for the closed loop system
superimposed with the stator winding temperature variation throughout the
frequency scale. Maximum permissible value of 130∘C is reached around 5 Hz
implying a possible thermal limitation occurrence in prior to the reaching of
the closed loop torque bandwidth.

Figure 11: Comparison of the closed loop torque bandwidth and the frequency at
thermal limitations.

The Figure 11 shows the gain transfer functions derived for each input
reference torque amplitude shown in Figure 10. For 6, 8 and 10 Nm reference
values, the actuator could not reach the characteristic closed loop torque
bandwidth due to the occurrence of thermal extremes in a prior frequency
value. The actuator could reach the torque bandwidth prior to the thermal
extreme frequency only in the 12 Nm amplitude reference signal. Unlike in the
open loop system, the general thermal behaviour for each of the reference
signals remains the same in closed loop configuration as the feedback
controller drives the PWM reference command signal to the saturation regions
(+1 and -1) after a certain frequency value.

Occurrence of thermal limitations prior to the closed loop torque bandwidth
imposes a disadvantage over the wide range of usability of an actuator.
Nevertheless, considering only the closed loop electromechanical bandwidth of
the actuator while excluding the thermodynamics could result in a shorter
lifespan of the actuator when operated in thermal extreme regions. Following
section discusses on the inclusion of a thermal controller which would
regulate the inputs to the actuator.

###  3.3 Regulation of PWM reference commands to avoid thermal limitations

According to the experimental results, the expected torque bandwidth is not
accessible due to thermal limitations, therefore, a controller should regulate
the input PWM reference signal beyond a certain winding temperature threshold.
A predefined amplitude restriction as well as low pass filtering would ideally
delay the thermal extremes but as it alters the input signal waveform, would
consequently lower the torque bandwidth as well. However the trade-off is
justifiable in order to harness the maximum accessible torque bandwidth.

Figure 12: Example thermal control for the torque amplitude of 8 Nm. Solid
lines represent a transfer function reshape from gain adjustment and dashed
lines represent a transfer function reshape from non linear filtering.

The thermal controller is triggered at the threshold value of $95\%$ of the
maximum permissible winding temperature $T_{MAX}$ (Starting point of arrow
‘A’). As the temperature increases beyond the value, the controller lowers the
PWM input reference signal’s gain and removes some of the high frequency
components. Consequently, the input-output transfer function would also
deviate from the original transfer function as shown from the arrow ‘C’ of the
figure 12. An example scenario is emphasised from the arrow ‘A’. The end point
of the arrow corresponds to the refined closed loop torque bandwidth with no
thermal limitations and such a bandwidth improvement is significant in
comparison to the interested frequency range. Transfer function reshaping by
the controller within the range given by the arrow ‘B’ will have a closed loop
bandwidth less than the original accessible value and the remaining region
will have an improved accessible torque bandwidth.

##  4 Discussion and conclusions

Torque tracking capabilities of an actuator depend on the characteristics of
the electromechanical subsystem, thermodynamic subsystem as well as on the
implemented controller algorithms, therefore, the limitations could occur with
any or all of them. Within the scope of the paper, the detailed methodology in
evaluating the accessible torque bandwidth value of an actuator is emphasised
while taking all such subsystems into the parallel consideration.

Experimental results show the presence of thermodynamic limitations near the
electromechanical torque bandwidth. The experiments were conducted in a
practical scenario of closed loop configuration and it was identified that the
output fixed instance will capture the thermodynamic limitations first in the
frequency range. The authors have further evaluated an approach in enhancing
the accessible torque bandwidth with incorporation of a thermal controller. In
the example shown in subsection 3.3, the electromechanical torque bandwidth of
7.25 Hz was not accessible due to the stator winding temperature exceeding the
maximum permissible value at 5.57 Hz. Inclusion of the thermal controller
improves the accessible torque bandwidth by $8.25\%$ up to 6.03 Hz. Authors
therefore encourage the readers in inclusion of such an analysis for
maximising the accessible torque bandwidth as well as the lifespan of
actuators in torque control applications. Opportunities for further research
are identified and should prioritise in evaluating the accessible torque
bandwidth with respect to the implemented controller characteristics such as
load resonances. Within the scope, only feedback and disturbance rejection
based control systems are utilised and future work can include more robust
algorithms such as inclusion of a feedforward control layer to the system.
External input to the electrical subsystem is considered to be only the PWM
reference signal, therefore, the dependency of the stator winding current is
considered to be a multiple input function of the reference PWM value and of
the instantaneous motor velocity. However, if the thermal controller is
capable enough to regulate the winding current as well, the transfer functions
and control systems derived within the paper’s scope should be revised
accordingly.

##  5 Acknowledgement

The authors would like to thank Arash Kh. Sichani for his guidance and advice
during this work. Authors would also like to thank Isuru Kalhara and Thomas
Molnar for their support. This work was fully funded by Commonwealth
Scientific and Industrial Research Organisation (CSIRO), Australia.

## Appendix

The actuator under experimental consideration is a HEBI robotics professional
grade series elastic actuator named X5-9. Electromechanical model parameters
are given in Table 2 and thermodynamic model parameters are given in Table 1.

Table 1: Four body thermodynamic model parameters of the actuator Parameter | Value
---|---
Thermal time constant $\tau_{1}$ | $1.49\,s$
Thermal time constant $\tau_{2}$ | $13.66\,s$
Thermal time constant $\tau_{3}$ | $>60\,s$
Thermal resistance $R_{1}$ | $5.368\,KW^{-1}$
Thermal resistance $R_{2}$ | $1.253\,KW^{-1}$
Thermal resistance $R_{3}$ | $0.357\,KW^{-1}$
Table 2: Electromechanical model parameters of the actuator Parameter | Value
---|---
Gear ratio $N:1$ | $1742.222:1$
Spring constant $K_{s}$ | $130\,Nmrad^{-1}$
Torque constant |  $7.1\,NmA^{-1}$ (actuator)
Speed constant |  $0.9\,rpmV^{-1}$ (actuator)
Terminal resistance |  $9.99\,\Omega$ (phase to phase)
Terminal inductance |  $0.163\,mH$ (phase to phase)
Torque constant |  $6.26\,mNmA^{-1}$ (BLDC)
Speed constant |  $1530\,rpmV^{-1}$ (BLDC)
Experimental $J_{m}$ | $5.615*10^{-8}\,kgm^{2}$
Experimental $B_{m}$ | $8.726*10^{-7}\,Nms{rad}^{-1}$
Experimental $K_{\tau}$ | $5.484*10^{-3}\,NmA^{-1}$
Experimental $K_{e}$ | $2.8*10^{-3}\,Vs{rad}^{-1}$
Experimental L | $0.794\,mH$
Experimental R | $6.840\,\Omega$

## References

  * [Choi et al., 2014] J. Choi, B. Na, S. Oh, and K. Kong.  A disturbance observer for robust position tracking control and ground contact detection of a Cheetaroid-I leg.  In IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pages 72–75, July 2014.
  * [Choi et al., 2016] J. Choi, H. Choi, K. Kong, and D. Jin Hyun.  An adaptive disturbance observer for precision control of time-varying systems.  IFAC-PapersOnLine, 49(21):240 – 245, 2016.  7th IFAC Symposium on Mechatronic Systems MECHATRONICS.
  * [Demetriades et al., 2008] G. D. Demetriades, H. Zelaya De La Parra, E. Andersson, and H. Olsson.  A real-time thermal model of a permanent magnet synchronous motor based on geometrical measures.  In IEEE Power Electronics Specialists Conference, pages 3061–3067, June 2008.
  * [Fussell, 1993] B. K. Fussell.  Thermal effects on the torque-speed performance of a brushless DC motor.  In Electrical/Electronics Insulation Conference, pages 403–411, Oct 1993.
  * [Malzahn et al., 2017] J. Malzahn, N. Kashiri, W. Roozing, N. Tsagarakis, and D. Caldwell.  What is the torque bandwidth of this actuator?  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4762–4768, Sept 2017.
  * [Oh et al., 2015] S. Oh, C. Lee, and K. Kong.  Force control and force observer design of series elastic actuator based on its dynamic characteristics.  In 41st Annual Conference of the IEEE Industrial Electronics Society (IECON), pages 004639–004644, Nov 2015.
  * [Park et al., 2016] Y. Park, S. Oh, and H. Zoe.  Dynamic analysis of reaction force sensing series elastic actuator as unlumped two mass system.  In 42nd Annual Conference of the IEEE Industrial Electronics Society (IECON), pages 5784–5789, Oct 2016.
  * [Pratt and Williamson, 1995] G. A. Pratt and M. M. Williamson.  Series elastic actuators.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), volume 1, pages 399–406 vol.1, Aug 1995.
  * [Ridwan et al., 2016] M. Ridwan, M. N. Yuniarto, and Soedibyo.  Electrical equivalent circuit based modeling and analysis of brushless direct current (BLDC) motor.  In International Seminar on Intelligent Technology and Its Applications (ISITIA), pages 471–478, July 2016.
  * [Roozing et al., 2016] W. Roozing, J. Malzahn, D. G. Caldwell, and N. G. Tsagarakis.  Comparison of open-loop and closed-loop disturbance observers for series elastic actuators.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3842–3847, Oct 2016.
  * [Shim et al., 2016] H. Shim, G. Park, Y. Joo, J. Back, and N. H. Jo.  Yet another tutorial of disturbance observer: robust stabilization and recovery of nominal performance.  Control Theory and Technology, 14(3):237–249, Aug 2016.
  * [Urukalo and Blazevic, 2011] D. Urukalo and P. Blazevic.  In situ non-invasive identification of thermal resistances for small dc motor.  In IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pages 582–587, July 2011.
  * [Xiang et al., 2015] C. Xiang, X. Wang, Y. Ma, and B. Xu.  Practical modeling and comprehensive system identification of a BLDC motor.  Mathematical Problems in Engineering, 2015(879581):11, 2015.


# Multi-Dialect Arabic Speech Recognition

Abbas Raza Ali  Faculty of Science and Technology
Bournemouth University
Poole BH12 5BB, United Kingdom
<EMAIL_ADDRESS>
###### Abstract

This paper presents the design and development of multi-dialect automatic
speech recognition for Arabic. Deep neural networks are becoming an effective
tool to solve sequential data problems, particularly, adopting an end-to-end
training of the system. Arabic speech recognition is a complex task because of
the existence of multiple dialects, non-availability of large corpora and
missing vocalization. Thus, the first contribution of this work is the
development of a large multi-dialectal corpus with either full or at least
partial vocalized transcription. Additionally, the open-source corpus has been
gathered from multiple sources that bring non-standard Arabic alphabets in
transcription which are normalized by defining a common character-set. The
second contribution is the development of a framework to train an acoustic
model achieving state-of-the-art performance. The network architecture
comprises of a combination of convolutional and recurrent layers. The
spectrogram features of the audio data are extracted in the frequency vs time
domain and fed in the network. The output frames, produced by the recurrent
model, are further trained to align the audio features with its corresponding
transcription sequences. The sequence alignment is performed using a beam
search decoder with a tetra-gram language model. The proposed system achieved
a 14% error rate which outperforms previous systems.

###### Index Terms:

Automatic Speech Recognition, Corpus Development, Deep Neural Networks, Multi-
Dialect Arabic, Recurrent Neural Networks

##  I Introduction

Deep Neural Networks (DNN) has become a de-facto standard in the Computer
Vision task. In recent years, they are gaining traction in speech and language
processing tasks with a significant improvement in acoustic modelling [18].
Despite human speech varies with respect to pitch, amplitude, duration and the
combination of phonemes and morphemes, yet how they tie together to create
words. The acoustic modeling is a contextual problem which generates
sequential information. The Recurrent Neural Networks (RNN), a class of DNN,
is considered as the most powerful tool for this kind of problem [12]. Instead
of combining RNNs with Hidden Markov Model (HMM) and Gaussian Mixture Model
(GMM), [14] introduces a concept of end-to-end automatic speech recognition
(ASR) using solely RNN. Thus, end-to-end training replaces the traditional
acoustic modeling approach to predict different states efficiently. Also, it
makes the overall task simplified and combined with a language model (LM)
gives a state-of-the-art performance.

Arabic is the $5^{th}$ most widely spoken language even though it is an under
resource language. Although, there are few open-source Arabic speech corpora
available which can be used to build a baseline speech recognition model.
However, for the phonetically rich language like Arabic, these resources are
not sufficient to build a reasonably accurate acoustic model. The key
challenge in the case of the Arabic language is to obtain diacritized
transcription of the audios which is not very common in open-source speech
corpora. The native Arabic speakers understand the meaning of the text from
the context rather than expecting diacritized text. Another challenge
associated with the Arabic language is the existence of several dialects with
a multitude of pronunciations [6].

This paper addresses a number of the challenges faced during the design and
development of multi-dialect Arabic ASR. Typically, the DNNs require a large
amount of relevant data to train a model. Hence, a large multi-dialectal
corpus has been developed which is one of the contributions of this work.
Additionally, the open-source corpus has been gathered from multiple sources.
The corpus brings variants of character-set and their transcripts are either
fully-, partially- or un-diacritized. Thus, the transcription of all the
sources is normalized to a common character-set, defined in the next section,
whereas automatic diacritization followed by a manual review has been
performed. The automatic diacritization systems are quite effective for
Arabic, Urdu [9], and a few other languages.

The rest of the paper is organized as follows. The development of speech
corpus, from various sources, is discussed in Section II. Section III is
devoted to elaborate on the overall approach of this work. Section IV reports
the experimental results followed by their analysis. Finally, the paper is
concluded in Section V.

TABLE I: List of speech corpora used to build acoustic model Dataset | Dialect | Duration | Prompt
---|---|---|---
Aldiri [3] | MSA | 2 hours | Read speech
KACST [5] | MSA | 2 hours | Read speech
Isolated Words [16] | MSA | 10 hours | Read speech
Arabic News Channel [19] | MSA, Egyptian, Gulf, Levantine, North African | 57 hours | Conversations, interviews, and news documentary
KSU [13] | MSA | 120 hours | Read speech
MGB-2 [20] | MSA, Egyptian, Gulf, Levantine, North Africa | 1,200 hours | Conversations, interviews, and news documentary
This work | MSA, Egyptian, Gulf | 70 hours | Read speech

##  II Speech Corpus

In order to train an acoustic model, a large amount of speech corpus has been
gathered. The corpus, covering multiple dialects, is recorded from 450-500
Arabic speakers possessing different accents. The distribution of the gender
is around 35-65 for female and male speakers respectively. Additionally, the
corpus is a mixture of various speaking styles including a) read speech, b)
answer speech (or interviews), c) descriptive speech (or documentary), and d)
spontaneous speech (or conversation). However, irrespective of the speech
style, the audio is segmented on average between 3 and 50 seconds which is
suitable to train a good speech recognizer. The quality of the transcription
varied significantly in different sources. The variations in the transcript
are in the form of un-diacritized or semi-diacritized transcript. Also, the
conversational speech contains overlapping of phrases and the usage of non-
standard Arabic alphabets [25].

The speech corpus is gathered from multiple sources which are listed in Table
I. The Aldiri corpus consists of 4,740 utterances spoken by 6 speakers with
the distribution of 3 males and 3 females. The KACST dataset consists of 4,573
utterances in Modern Standard Arabic (MSA) dialect. The ‘Isolated words’ data-
source contains 9,992 utterances of 20 words spoken by 50 male Arabic
speakers. The ‘Arabic News Channel’ corpus is composed of conversational
speech style with around 27,000 utterances. There are few challenges
associated with this corpus because of its speaking style, such as overlapping
speech and usage of non-standard alphabets. The non-standard alphabets that
appear in the transcript are normalized to a common character-set and the
overlapping phrases are omitted from the final corpus. The KSU database is
recorded from more than 200 speakers belonging to different ethnic groups. The
transcription of the corpus is composed of phonetically rich and balanced
words, digits, sentences, and paragraphs. The audio is recorded from low and
high-quality microphones and telephony mediums in different environments
including office, sound-proof room, and cafeteria. The Multi-Genre Broadcast
(MGB)-2 corpus is recorded from 19 different programs of an Arabic TV channel
where the content covers three speaking styles including conversations,
interviews, and documentaries. The MSA dialect was most dominating in the
programs along with Egyptian, Gulf, Levantine and North African.

###  II-A Character-set

The raw Arabic corpora are gathered from multiple sources using different
character-sets in their transcripts. A common character-set has been defined
to normalize all the sources. Initially, for some experiments, the regular
Arabic short vowels called: fatha, damma, and kasra including a diacritical-
mark shadda are part of the character-set. The remaining Arabic diacritical-
marks were rear in most of the data-sources, hence, they were excluded
comprising of fathatan, kasratan, dammatan, and sukun. Three long vowels are
part of the character-set including alef, waw and ya. The rest of the
alphabets are standard consonants of Arabic. In the final experiments, the
short vowels and shadda are also excluded which were causing dataset
imbalance. The Arabic character-set used in training the system is shown in
Fig. 1.

Figure 1: Arabic character-set with phonemes and description

###  II-B Data Cleansing

The data cleansing and normalization become critical when developing a corpus
from different sources that possess different rules. The audio files are
recorded using the low and high quality of microphones with different
background noise. A high-pass filter is applied to the audios with a cutoff
frequency of 150 Hz to cancel the DC offset. The audio files are converting
into a single-channel 16-bit 16,000 Hz sample rate encoded using Linear Pulse
Code Modulation (LPCM). The long audio files are split between 2-30 seconds
segments. Similarly, the transcript is normalized accordingly by segmenting
long sentences into multiple phrases to synchronize them with their
corresponding audios. Also, different sources bring a different combination of
diacritical-marks which were normalized by, initially, limiting the text to
fatha, damma, kasra, and shadda. In the later experiments, all the
diacritical-marks were removed except ‘hamza above’. The ‘hamza above’ mark
was used with alef, waw and ya. The transcripts with non-standard Arabic
characters have been completely removed from the corpus.

###  II-C Development of New Corpus

In addition to the collection of open-source Arabic speech corpora, an effort
has been made to design and develop a phonetically rich and balanced corpus.
The development of the corpus consists of three key stages: 1) the content
preparation, 2) speech recording and validation and 3) verification. The high-
level steps of collecting and developing Arabic speech corpus are shown in
Fig. 2. The idea is to first utilize all the available open-source resources
to build a basic corpus followed by the development of a new corpus. Following
are the three key areas of new corpus development:

####  II-C1 Transcription

A phonetically rich and balanced transcription is prepared which covers all
the possibilities of Arabic phonemes and their different positions in a word.
Thus, the transcript of Saudi Accented Arabic Voice Bank (SAAVB) [7] was
chosen. The transcript includes 1,033 unique phrases including numbers, words,
sentences, and paragraphs that were designed to differentiate the number of
dialects.

####  II-C2 Corpus Recording and Validation

The corpus recording process started with the recruitment of a large number of
Arabic speakers possessing different dialects. The recruited speakers were
required to read 190 phrases that were assigned to them via an automatic
system. A recorded phrase is first evaluated using speech processing tools and
is stored in the database only if it passes the quality test. In case of
failure, because of either noise, unnecessary silence, repetition, etc., the
speaker is required to re-record the same phrase until it gets approved by the
speech analyst. This process is repeated on all the phrases and speakers.

####  II-C3 Verification

A detailed verification of the recording setup has been performed by the
linguists before the initiation of a recording session. The verification
process is not limited to the verification of recording equipment, it also
checks the quality of the recorded speech. Further, prior to the recording
sessions, the speakers are instructed on the usage of recording hardware and
software.

Figure 2: Steps to build Arabic Speech Corpus

##  III Methodology

Arabic is a Semitic language having unique phonemes and phonetic features,
multiple dialects, and a complicated morphological word structure [2]. In
order to train a speech recognizer using this kind of complex data, the DNN
capacity has been increased via depth. Two models are trained independently
namely; acoustic model and language model, see Equation 1. The speech corpus,
comprising of transcribed audios, is used to train the acoustic model. The
acoustic model is used to predict phonemes, given a segment of audio features.
On the other hand, an n-gram based language model predicts the next word in a
sequence. It trains on text corpus comprising of transcripts of conversations.

ASR $\displaystyle=\operatorname*{arg\,max}_{w\in vocab}\textrm{
}(\textrm{language model})\textrm{x}(\textrm{acoustic model})$ (1)
$\displaystyle=\operatorname*{arg\,max}_{w\in vocab}\textrm{ }(words)\textrm{
x }(features|words)$ $\displaystyle=\operatorname*{arg\,max}_{w\in
vocab}\textrm{ }(w)\textrm{ x }(x|w)$

The network consists of a combination of convolutional and recurrent layers
followed by a fully-connected layer. The convolutional layers are limited to
2-3 with a stack of 4-5 bi-directional recurrent layers. Various sizes of
network width are tried including 768 and 1024. Also, two types of RNNs have
experimented which are Long short-term memory (LSTM) and Gated Recurrent Units
(GRUs). A clipped Rectified Linear Unit (ReLU) function has been applied as
formulated in Equation 2. The End-to-End speech recognition architecture is
shown in Fig. 3.

$\sigma(x)=min(max(x,0),20)$ (2)

Three convolutional layers have been used with 32, 32 and 96 filter sizes
having dimensions 41x11, 21x11 and 21x11 respectively. The strides of 2x2, 2x1
and 2x1 have been applied to down-sample the input. The regularisation is key
to better performance by avoiding over-fitting when using RNN. Hence, batch
normalization (BatchNorm) is used with convolutional layers whereas a dropout
of 20% is used for recurrent layers. The BatchNorm also enhances the training
speed of this kind of very deep networks with marginally improved accuracy. To
avoid over-fitting, early stopping is also used while training the network.
The sequence alignment is performed through the Connectionist Temporal
Classification (CTC) which is found to be very successful for speech and
language processing tasks [24]. The training examples of varying lengths are
sorted in increasing order of length since short examples are comparatively
less costly.

Figure 3: The architecture of end-to-end Arabic speech recognition. In the
experiments 2-3 convolution and 4-5 recurrent layers have been tested.

The inference process applies normalization and extracts features of the new
audio. The trained acoustic model produces phoneme probabilities over time.
These probabilities are fed to the decoder, along with a language model, to
compute the most likely sequence of words for the given audio input.

###  III-A Feature Extraction

The Mel-frequency Cepstral Coefficients features (MFCC) have demonstrated
promising performance for speech recognition task [27]. However, for this
work, the spectrograms are extracted from the audio files which are fed as
input to the convolutional layers of the network. The spectrogram is a two-
dimensional representation of audio data consisting of frequency spectrum over
time which can be processed as a two-dimensional image. The filter banks are
computed over windows of 0.02 seconds with strides of 0.01 seconds [14].

###  III-B Acoustic Modeling

The acoustic model is trained using more than 1,400 hours of speech data. A
deep and wide network has been designed to train the acoustic model along with
sequence alignment. The optimal set of hyper-parameters is found by performing
a number of experiments with different values of filters and strides of
convolutional layers, and the type and width of recurrent layers. The weights
are initialized with Xavier-initialization [8]. A dropout of 20% has been used
for recurrent layers whereas BatchNorm is used for convolutional layers. The
network is trained with Adam optimizer [17] with a variable learning rate
which decays by a factor of 10 on every alternative epoch with an initial
value of 0.001.

The recurrent networks are trained as frame-level classifiers for speech
recognition tasks [15]. These frames further require training to align the
input audio with corresponding output transcription sequences. The CTC allows
a sequence training without requiring any prior alignment between the input
and target sequences. In this work, the CTC beam search along with a trained
word-level LM has been used. The LM guides the CTC beam search decoder on
domain-specific and miss-spelled terms. The CTC is used with a beamwidth of
512.

###  III-C Language Model

The word-level LM predicts the upcoming word given the previous sequence of
words [28]. A word-level tetra-gram LM is trained on a large Arabic corpus.
The short vowels and shadda diacritical-mark are also removed from the
language model corpus to synchronize with the character-set of the acoustic
model. The LM has been trained over millions of unique utterances using KenLM
with Kneser-Ney smoothed tetra-gram model [11]. The beam search is used to
find the optimal word sequence. The trained model is quite effective for
spelling and implicitly disambiguate homophones.

The corpus that is used to build LM is collected from multiple sources listed
in Table II. The MGB-2 has diversity in broadcast news comprising of 19
distinct programs of an Arabic TV channel. The text is crawled from the
website of the channel. The Tashkeela corpus contains 75.6 million fully
vocalized terms that are crawled from 97 classical and modern Arabic books.
The consolidated text is normalized using the same rules which are applied to
the transcription of the audio data.

TABLE II: List of corpora used to train language model Dataset | Vocabulary | Domain
---|---|---
MGB-2 LM [20] | 130 million | News
Tashkeela [23] | 75.6 million | Islamic classical books

##  IV Analysis and Results

The proposed architecture is tested with different combinations of hyper-
parameters using a trial-and-error approach. The corpus is divided into
training, development and validation sets with a ratio of 70%, 20%, and 10%
respectively. The experiments are conducted in a distributed training
framework for 32-35 epochs using 10 Nvidia 1080ti GPUs. The total batch size
of 80 (8 batches per GPUs) is used with a learning rate of 0.0001 and a
momentum of 0.99 to train multiple models. There are 8 different models
trained varying along 2-3 convolutional and 4-5 recurrent layers, and RNN
width (768 or 1024) and type (LSTM or GRU). The models which perform well
during the training phase on the development set were chosen for further
evaluation. Their performance is measured using word error rate (WER) which is
formulated in Equation 3.

$\textrm{WER}=\frac{\textrm{substitutions (S)}+\textrm{deletions
(D)}+\textrm{insertions (I)}}{\textrm{total words in the reference (N)}}$ (3)

TABLE III: Comparison of different arrangements of convolutional and recurrent layers along with network width and RNN type Architecture | Neurons | RNN Type | WER (%)
---|---|---|---
2-CNN, 4-RNN | 768 | GRU | 14.07
2-CNN, 4-RNN | 768 | LSTM | 14.67
2-CNN, 4-RNN | 1024 | GRU | 15.32
2-CNN, 4-RNN | 1024 | LSTM | 15.54
3-CNN, 5-RNN | 768 | GRU | 14.88
3-CNN, 5-RNN | 768 | LSTM | 15.30
3-CNN, 5-RNN | 1024 | GRU | 14.23
3-CNN, 5-RNN | 1024 | LSTM | 14.61

The outcome of the acoustic model is decoded using the CTC beam search with
LM. The final WER of the different models is reported in Table III. The model
with 2 convolutional layers followed by 4 bi-directional recurrent layers,
composed of GRU cell and 768 neurons, significantly outperformed the rest of
the variants. The statistical significance is computed using a paired t-test
of the top-performing architecture’s WER with the rest of the WERs. Further,
the high-performing model is using a fully-connected layer followed by a CTC
beam search decoder with LM.

Table IV provides a comparison of the proposed system with the prior research.
The performance of several DNN variants, such as Feed-forward, CNN, LSTM, Time
Delayed Neural Networks (TDNN) and GMM have been evaluated.

TABLE IV: Comparison of the proposed approach with existing systems Research | Acoustic/Lang. Model |  Speech Corpus | WER %
---|---|---|---
[4] | TDNN/3-gram |  MGB-2 | 20.5
[22] | RNN/4-gram |  MGB-2 | 18.3
[25] | RNN/3-gram |  MGB2 | 14.70
[26] | RNN/4-gram |  Nemlar1 & NetDC2 | 14.42
This work | CNN, RNN/4-gram |  MGB-2 & custom | 14.07
1http://catalog.elra.info/product_info.php?products_id=874
2http://catalog.elra.info/product_info.php?products_id=13

##  V Conclusions

An end-to-end automatic speech recognition system for multi-dialectal Arabic
language has been presented in this paper. The key contributions include the
development of large multi-dialectal corpus and an end-to-end speech
recognition model with accuracy close to the state-of-the-art for the Arabic
language. This accuracy is the result of numerous trials to explore various
network architectures and their hyper-parameters, accompanied by several
techniques including sorting of training samples, BatchNorm, early stopping
and decoding with LM.

Two key contributions are claimed in this work. The first contribution is the
development of a large multi-dialectal Arabic speech corpus consisting of
1,400 hours of speech data. The corpus has been gathered from multiple sources
and normalized on a common character-set. The second contribution is the
development of a framework to train the Arabic speech recognition acoustic
model with close to state-of-the-art accuracy. The system has achieved an
overall WER of 14%.

## References

  * [1] M. Boudraa, B. Boudraa, and B. Guerin, “Twenty Lists of Ten Arabic Sentences for Assessment,” in ACUSTICA, ACTA-ACUSTICA, vol. 86, no. 5, 1998.
  * [2] S. Selouani, and J. Caelen, “Arabic phonetic features recognition using modular connectionist architectures,” in Proceedings of the IEEE Interactive Voice Technology for Communication, IVTTA, 1998.
  * [3] B. Al-Diri, A. Sharieh, and T. Hudaib, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in Interspeech, 2015.
  * [4] V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in Interspeech, 2015.
  * [5] M. A. Mansour, “KACST arabic phonetics database,” Riyadh, Kingdom of Saudi Arabia, 2004.
  * [6] A. Sankaranarayanan, S. Bangalore, and S. S. Narayanan, “Automatic diacritization of Arabic transcripts for automatic speech recognition,” in Proceedings of the International Conference on Natural Language Processing (ICON), 2005.
  * [7] M. Alghamdi, F. Alhargan, M. Alkanhal, A. Alkhairy, M. Eldesouki, and A. Alenazi, “Saudi Accented Arabic Voice Bank,” in Journal of King Saud University, Computer and Information Sciences, vol. 20, pages 45–64, 2008.
  * [8] X. Glorot, and Y. Bengio, “Understanding the difficulty of training deep feed-forward neural networks,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (PMLR), 2010.
  * [9] A. R. Ali, and S. Hussain, “Automatic Diacritization for Urdu,” in Proceedings of the Conference on Language and Technology (CLT), 2010.
  * [10] M. A. M. Abushariah, R. N. Ainon, R. Zainuddin, M. Elshafei, and O. O. Khalifa, “Phonetically rich and balanced text and speech corpora for Arabic language,” in Journal Language Resources and Evaluation, vol. 46, no. 4, pages 601–634, 2012.
  * [11] K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn, “Scalable modified Kneser-Ney language model estimation,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013.
  * [12] A. Graves, A. Mohamed, and G. Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” in Computing Research Repository (CoRR), vol. abs/1303.5778, 2013.
  * [13] M. Alsulaiman, Z. Ali, G. Muhammed, M. Bencherif, and A. Mahmood, “KSU Speech Database: Text Selection, Recording and Verification,” in European Modelling Symposium, 2013.
  * [14] A. Hannun, C. Case, J. Casper, and B. Catanzaro, “Deep Speech: Scaling up end-to-end speech recognition,” in Computing Research Repository (CoRR), vol. abs/1412.5567, 2014.
  * [15] A. Graves, and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in Proceedings of the 31st International Conference on International Conference on Machine Learning (ICML), 2014.
  * [16] A. Alalshekmubarak, and L. S. Smith, “On Improving the Classification Capability of Reservoir Computing for Arabic Speech Recognition,” in 24th International Conference on Artificial Neural Networks, 2014.
  * [17] D. P. Kingma, and J. B. Adam, “A method for stochastic optimization,” International Conference on Learning Representations (ICLR), 2015.
  * [18] D. Amodei, S. Ananthanarayanan, R. Anubhai, and J. Bai, “Deep Speech 2: End-to-End Speech Recognition in English and Mandarin,” in Proceedings of Machine Learning Research (PMLR), vol. 48, 2015.
  * [19] S. Wray, and A. Ali, “Crowdsource a little to label a lot: Labeling a Speech Corpus of Dialectal Arabic,” in Interspeech, 2015.
  * [20] S. Khurana, and A. Ali, “QCRI advanced transcription system (QATS) for the Arabic Multi-Dialect Broadcast media recognition: MGB-2 challenge,” in IEEE Spoken Language Technology Workshop (SLT), 2016.
  * [21] A. Ahmed, Y. Hifny, K. Shaalan, and S. L. Toral, “Lexicon Free Arabic Speech Recognition Recipe,” in International Conference on Advanced Intelligent Systems and Informatics, vol. 12, pages 147–159, 2016.
  * [22] T. AlHanai, W. N. Hsu, and J. Glass, “Development of the MIT ASR system for the 2016 Arabic Multi-genre Broadcast Challenge,” in IEEE Spoken Language Technology Workshop (SLT), 2016.
  * [23] T. Zerrouki, and A. Balla, “Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems,” in Data in Brief, vol. 11, 2017.
  * [24] M. Najafian, W. N. Hsu, A. Ali, and J. Glass, “Automatic speech recognition of Arabic multi-genre broadcast media,” in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2017.
  * [25] A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition,” in Computing Research Repository (CoRR), vol. abs/1609.05625, 2017.
  * [26] M. A. Menacer, O. Mella, and D. Fohr, “An enhanced automatic speech recognition system for Arabic,” in the 3rd Arabic Natural Language Processing Workshop (EACL), 2017.
  * [27] L. Bouchakour, and M. Debyeche, “Improving Continuous Arabic Speech Recognition over Mobile Networks DSR and NSR Using MFCCs Features Transformed,” in International Journal of Circuits, Systems and Signal Processing, vol. 12, 2018.
  * [28] A. R. Ali, “Cognitive Computing to Optimize IT Services,” in IEEE 17th International Conference on Cognitive Informatics and Cognitive Computing (ICCI*CC), Berkeley, CA, 2018, vol. 1, pages 54–60.