# Atomistic aspects of load transfer and fracture in CNT-reinforced aluminium

Samaneh Nasiri1, Kai Wang2, Mingjun Yang2,
Julien Guénolé3,4, Qianqian Li5, and Michael Zaiser1 1WW8-Materials
Simulation, Department of Materials Science, Friedrich-Alexander Universität
Erlangen-Nürnberg, 90762 Fürth, Germany  2School of Materials Science and
Engineering, Southwest Petroleum University, Chengdu, Sichuan, PR China
3Université de Lorraine, CNRS, Arts et Métiers ParisTech, LEM3, 57070, Metz,
France  4Labex Damas, Université de Lorraine, 57070, Metz, France  5Department
of Aeronautics, Imperial College, Prince Consort Road, London SW7 2BZ, UK
<EMAIL_ADDRESS>
###### Abstract

This paper describes atomistic simulations of deformation and fracture of Al
reinforced with carbon nanotubes (CNTs). We use density functional theory
(DFT) to understand the energetics of Al-graphene interfaces and gain
reference data for the parameterization of Al-C empirical potentials. We then
investigate the load transfer between CNTs and Al and its effect on composite
strengthening. To this end, we perform uniaxial tensile simulations of an Al
crystal reinforced with CNTs of various volume fractions. We also study the
interaction of the embedded CNTs with a crack. We show that the interaction
between CNTs and Al is weak such that, under tensile loading, CNTs can easily
slide inside the Al matrix and get pulled out from the cracked surface. This
effect is almost independent of CNT length and volume fraction. Little load
transfer and consequently no crack bridging are observed during the simulation
of pristine CNTs threading the crack surfaces. CNTs that are geometrically
fixated inside Al, on the other hand, can increase the fracture stress and
enhance plastic dissipation in the matrix. CNTs located in front of a growing
crack blunt the crack and induce plastic deformation of the Al matrix.
Depending on the CNT orientation, these processes can either increase or
decrease the failure stress of the composite.

##  1 Introduction

Carbon nanoparticles (CNPs), due to their excellent mechanical and structural
properties, can significantly improve the mechanical properties of metal or
polymer matrices [1, 2, 3, 4, 5, 6, 7, 8, 9]. Carbon nanotubes (CNTs) exhibit
a length-to-diameter ratio (aspect ratio) as high as 1.3e+08 ([10, 11, 12]),
Young’s modulus values of $\sim$ 1 TPa, tensile strength $>$ 100 GPa, and
failure strains of 15-30% depending on chirality for defect-free single-walled
CNTs. These properties are incomparable to any other material. Well dispersed
CNTs inside metal matrices, therefore, can potentially carry part of the load
and increase effective elastic moduli [13, 14, 15, 16, 17, 18, 19].
Experimental studies also showed that embedded CNPs bridge incipient cracks
[16] and impose strong constraints on dislocation motion [20], thus
potentially increasing yield strength, fracture strength, and possibly also
toughness[21, 22, 23]. Experimental observations indicate that CNTs hinder
crack propagation by deflecting the crack from its growth direction. CNTs can
also bridge between two crack faces, restrain the crack from opening, and
eventually prevent crack growth. Figure 1 depicts the possible mechanisms of
crack-CNT interactions. Crack bridging and CNT pull-out phenomena from an Al
matrix have been observed experimentally by Chen et al. [16, 15] and Park et
al. [24]. Similar observations are documented in several studies on different
CNT-reinforced ceramics [25].

Figure 1: Suggested fracture modes in fiber/matrix composites [26]

.

For low-melting lightweight metals such as Al or Mg, the benefit of embedding
CNTs is still under discussion since these metals do not wet $sp^{2}$ bonded
carbon nanoparticles. Experimental and simulation studies reported a small
value between 24.0 and 35.0 MPa ([27, 28]) for the interfacial shear stress
(IFSS) between Al matrix and CNTs, indicating a weak interface between Al and
pristine CNTs. Therefore, the immersed CNTs into the Al melt tend to bond to
themselves rather than Al atoms and agglomerate. At the same time, the
interfacial bonding between the nanoparticles and the matrix remains modest
after solidification. In a worst-case scenario, agglomerated nanoparticles
with weak bonding into the surrounding metal matrix might act as flaws that
deteriorate, rather than improve, the mechanical properties[22]. An extended
review of tensile properties of Al reinforced with CNTs can be found in [29].

There exist several simulation studies regarding the mechanical behavior of
metals with embedded CNTs. Molecular dynamics (MD) studies of CNT-reinforced
aluminum composites under uniaxial tensile and compressive loading have been
performed by several researchers [30, 31, 32, 33, 34]. We find these
simulations problematic for several reasons. In some simulations, to model
long CNTs, the authors considered CNTs that span the system and cross periodic
boundaries in the loading direction. We demonstrate in Section 3.2 that such a
set-up is prone to artifacts as it enforces isostrain loading of matrix and
CNT even if no load transfer at the metal/CNT interface is present. In
addition, to define the $sp^{2}$ bonded C-C interactions in CNTs, researchers
used the Adaptive Intermolecular Reactive Empirical Bond Order (AIREBO)
potential [35] which is a reliable potential for simulation of most physical
and mechanical properties of hydrocarbons. However, AIRBEO is inadequate to
correctly describe C-C bond breaking under tensile loads because the cut-off
function of AIRBEO induces an artificial bond stiffening at large bond
elongations, which delay bond breakage and induce a ductile failure mode.
Several authors have discussed that this is an artifact of the cut-off
function [36, 37]. In simulations with an improved cut-off function that
accounts for changes in the local atomic environment, CNTs found to behave
rather brittle at room temperature [37]. Accordingly, reports of ductile
behavior of an Al matrix reinforced by long CNTs may represent artifacts of
the potentials and boundary conditions used by the authors. It is also
worrying that all the mentioned investigations define the interfacial
interactions between the metal and carbon atoms by Lennard-Jones pair
potentials which are parameterized by combining Lorentz-Berthelot rules which
have been criticized for inaccurate prediction of properties by molecular
simulation [38, 39].

In the present work, we report the results of a multiscale simulation study of
the mechanical properties of Al composites reinforced with CNTs. Firstly, we
performed density functional theory (DFT) calculations to establish the
energetics of different types of Al-graphene interfaces. The DFT results are
then used as reference data for parametrizing the Van der Waals-like
potentials for interfacial interactions between Al and $sp^{2}$ bonded carbon
nanoparticles. We then employed these potentials in molecular dynamics (MD)
simulations to establish the strengthening mechanisms in CNT-reinforced
aluminum. We investigated the behavior of a periodically continued Al crystal
reinforced with single-walled CNTs(5,5) under uniaxial tensile load, aiming to
understand the effect of CNT length and volume fraction on tensile strength of
the Al-CNT composite. To study the interaction of cracks with CNTs under Mode
I loading, we considered an Al crystal containing a CNT and an edge crack. We
investigated crack deflection and crack bridging mechanisms by CNTs in Al by
changing the CNT length, position, entanglements, and angle of the CNT with
respect to the crack front.

##  2 Density-functional calculations: Methods and results

We performed DFT calculations to parametrize the interatomic potentials
between Al and $sp^{2}$ bonded carbon. To this end, we considered a planar
geometry amenable to periodic continuation containing six-layer Al(111) slabs
adhering to monolayer graphene (single-walled CNT with an infinite diameter).
Both Al slabs and graphene are oriented parallel to the $xy$ plane of the
Cartesian coordinate system. Periodic boundary conditions are applied in all
three spatial directions with a 20 Å vacuum gap between the periodic replicas
in the $z$ direction. Two variants of this structure are constructed as shown
in Figure 2. For model (a), p $(7\times 7)$ graphene and p $(6\times 6)$ Al
slabs were used in calculations. For model (b), we used p $(2\times 2)$
graphene with three atoms per Al layer as in the work of Gong et al. [40] and
Khomyakov et al. [41]. Model (a) approximates an incoherent interface, while
model (b) approximates a coherent interface. In both models, the lattice
parameters were set to their experimental values, and the metal surface was
strained to fit the graphene supercell.

The DFT calculations were carried out by the Quantum Espresso package version
6.1 [42], employing PBE-based projected augmented wave(PAW) potentials [43].
Previous studies have shown that the optB88-vdW [44] exchange-correlation
functional gives a reasonably accurate prediction of both interlayer distance
and binding energy for graphite [45], so here we choose the optB88-vdW
functional in all our DFT calculations. In our calculations, we used the
kinetic energy cutoffs of 40 Ry and 450 Ry for the wave function and charge
density calculations, respectively. A Methfessel-Paxton smearing of 0.01 Ry
was used for the electronic convergence, and the convergence for self-
consistency calculations was less than 0.0001 Ry. In the calculations of model
(a), only the gamma k-point was used due to the large supercell, while an
$8\times 8\times 1$ k-points grid was used in the calculations of model (b).
For additional verification of the method, we simulated the same configuration
(hpc-fcc) as in the DFT calculations by Christian et al. [46] of graphene
adhering to an Al(111) slab and found good agreement with their work.

Figure 2: (a) and (b) top views (left) and side views (right) of the geometry
for model a and model b respectively. (c) Al-graphene binding energy for both
models calculated with DFT (dotted data) and empirical potentials (Morse
potential for Al-C interactions)(lines).

Since our aim is to obtain data for parameterizing an interaction potential,
rather than performing an accurate analysis of the interface structure and
energy using DFT, we do not carry out any structural relaxation but simply
separate the graphene sheet rigidly from the Al(111) slab without geometry
relaxation of either the Al slab or the graphene. We evaluate the binding
energy per carbon atom $E_{\mathrm{b}}$ as

$\displaystyle
E_{\mathrm{b}}=[E_{\mathrm{tot}}-(E_{\mathrm{Al}}+E_{\mathrm{gr}})]/n$ (1)

$E_{\mathrm{tot}}$ is the total energy of the graphene-Al(111) system,
$E_{\mathrm{Al}}$ represents the energy of the six-layer Al(111) slab, and
$E_{\mathrm{gr}}$ and $n$ represent the energy of the free-standing graphene
sheet and the number of carbon atoms in the graphene sheet, respectively. As
depicted in Figure 2, Al manifest weak interaction (physisorption) interfaces
with graphene sheets with adsorption energy of around 0.03–0.05 eV/carbon
atom, and equilibrium interfacial distances larger than 3.0
$\mathrm{\SIUnitSymbolAngstrom}$ [47, 40]. In these interfaces, the energy
differences between different conformations of the graphene on top of the Al
slab are minor and hence the energy ’landscape’ is rather flat.

##  3 Molecular simulations

The atomistic simulations presented in this work were performed with the
molecular dynamics code LAMMPS [48] (version 16 Mar 2018). Atomsk package
[49], and VNL builder in QuantumATK [50, 51] were used to create the initial
atomic configurations. Data visualization and post-processing were obtained
with the help of the Open Visualization Tool (OVITO) [52]. Common neighbour
analysis (CNA) [53, 54] was used to analyze the crystallographic structure
around atoms and to identify defects and dislocations in the Al matrix.

###  3.1 Interaction potentials

The chosen interatomic potential is based upon combining potentials of
different types that are separately optimized to describe the energetics and
mechanical properties of covalently bonded carbon, and of aluminum. We usend
Embedded Atom model (EAM) potential of Mishin et al. [55] for describing Al-Al
interactions and the modified screened reactive empirical bond-order (scr-
REBO) potential by Pastweka et al. [37] for C-C interactions. The scr-REBO
potential uses a local environment-dependent cut-off function that leads to
smooth changes in force and a much better description of bond-breaking and re-
forming comparing to conventional truncated REBO potentials. The weak
interactions in a physisorbed interface between Al and $sp^{2}$ bonded carbon
can be described by a Morse potential as in the approach used by [56] for
simulating Ni-graphene interactions. We used a Morse potential of the form
$E_{\mathrm{M}}=D_{\mathrm{M}}(\exp[-2\alpha(r-r_{0})]-2\exp[-\alpha(r-r_{0})])$
for Al-C which has been parametrized using our DFT calculations and fitted to
represent the physisorbed interface state between graphene and Al, where $r$
is the bond distance, $r_{0}$ the equilibrium bond distance, $D_{\mathrm{M}}$
is the well depth, and $\alpha$ controls the stiffness of the potential. To
parametrize the Morse potential, we used the same configuration as in the DFT
simulations, i.e., we separate the graphene sheet rigidly from the Al(111)
surface, which allows us to directly compare the MD and DFT energies (Figure
2). The resulting parameters for the Al-C Morse potential are $D_{\mathrm{M}}$
= 0.0048 eV, $r_{0}$ = 4.52 $\mathrm{\AA}$ and $\alpha$ = 1.0
$\mathrm{\AA}^{-1}$. We emphasize that a combination of structurally
dissimilar potentials — EAM, scr-REBO, and Morse — as used here can produce
adequate results only as long as the interactions described by the respective
terms can be considered as additive. That is approximately true for
physisorption governed by van der Waals forces, which is not expected to
significantly alter the bonding characteristics of either metals or covalently
bonded carbon. On the other hand, situations where chemical bonds are
established at the metal-carbon interface or where reactive processes or
mechanical mixing occur, cannot be adequately described in this manner. In
particular, we note that our approach a priori disallows carbide formation.

###  3.2 Uniaxial tensile deformation of CNT-reinforced Al composites

This section studies the behavior of a periodically continued Al crystal
reinforced by a single-walled CNT(5,5) under uniaxial tensile loading, aiming
to understand the effect of length and volume fraction of CNTs on the tensile
strength of Al-CNT composite.

####  3.2.1 Sample preparation

We created a monocrystalline Al block with faces aligned with the [100],
[010], and [001] directions of an fcc crystal lattice. Al atoms are initially
located on the sites of a perfect fcc lattice. The simulation box is periodic
in all spatial directions. The energy of the system is minimized through
structural relaxation using a CG algorithm. The system is accepted as relaxed
when both the change of energy between successive iterations divided by the
energy magnitude is less than or equal to the tolerance ($10^{-8}$) or the
2-norm (length) of the global force vector is less than the threshold value of
$10^{-8}$ eV/$\mathrm{\AA}$ as in [57]. A single-walled CNT(5,5) is then
embedded into the center of the Al block, such that its longitudinal axis is
parallel to the [010] lattice direction, which aligns with the $y$ direction
of the Cartesian coordinate system. The embedded CNT has either equal length
as the periodic cell (i.e., it extends across the periodic boundary) denoted
as ”long CNT” or it has a 2.46 $\mathrm{\SIUnitSymbolAngstrom}$ shorter length
than the Al matrix such that it terminates within the periodic cell denoted as
”short CNT” in Figure 3. To create the requisite space between Al and C atoms,
we removed the Al atoms that were located inside the CNT or within a distance
of less than 3.0 $\mathrm{\SIUnitSymbolAngstrom}$ from C atoms. To equilibrate
the system, Al atoms around the CNT were annealed at 1500K (i.e., above the
melting temperature of Al) for 20 ps in the $NpT$ ensemble at zero pressure by
using the Nosé-Hoover thermostat and barostat, then quenched to 0.1K at a rate
of $10^{-12}K.ps^{-1}$. During the melt-quench cycle, the boundary atoms were
fixed on the sites of the original crystal lattice. In this case,
crystallization starts from the boundary and leads to a monocrystalline Al
block surrounding the CNT, which remains aligned with a [110] lattice
direction. Carbon atoms are fixed during this process. A final structural
minimization is then performed to remove any excess energy in the system. We
then calculated the Al/CNT interface energy ($E_{\mathrm{int}}$) as
$E_{\mathrm{int}}=[E_{\mathrm{Al-CNT}}-(E_{\mathrm{Al-
void}}+E_{\mathrm{CNT}})]/n$, where $E_{\mathrm{Al-CNT}}$ is the total energy
of the Al-CNT system, $E_{\mathrm{Al-void}}$ represents the energy of the Al
block with a cylindrical void in the middle induced by the CNT, and
$E_{\mathrm{CNT}}$ and $n$ represent the energy of the free-standing CNT and
the number of carbon atoms in the CNT, respectively. The calculated interface
energy for the minimized structure is equal to $E_{\mathrm{int}}$ = 0.053
eV/carbon atom that matches well to the equilibrium binding energy from our
DFT calculations (Figure .2).

Table 1: Geometrical parameters of the simulated samples for investigation of CNT length effect on composite strength. The unit of the length and diameter is $\mathrm{\SIUnitSymbolAngstrom}$. Samples | $L_{x}$ | $L_{z}$ | $L_{y}$ | $D$ |  $f_{\mathrm{CNT}}$[%]
---|---|---|---|---|---
1 | 113.40 | 113.40 | 101.25 | 6.78 | 0.27
2 | 113.40 | 113.40 | 202.50 | 6.78 | 0.27
3 | 113.40 | 113.40 | 303.75 | 6.78 | 0.27
4 | 113.40 | 113.40 | 405.00 | 6.78 | 0.27
Table 2: Geometrical parameters of the simulated samples to study the effect of CNT volume fraction on composite strength. The unit of the length and diameter is $\mathrm{\SIUnitSymbolAngstrom}$. Samples | $L_{x}$ | $L_{z}$ | $L_{y}$ | $D$ |  $f_{\mathrm{CNT}}$[%]
---|---|---|---|---|---
1 | 113.40 | 113.40 | 101.25 | 6.78 | 0.27
2 | 68.85 | 68.85 | 101.25 | 6.78 | 0.76
3 | 56.70 | 56.70 | 101.25 | 6.78 | 1.10
4 | 44.55 | 44.55 | 101.25 | 6.78 | 1.72
5 | 28.35 | 28.35 | 101.25 | 6.78 | 4.40

####  3.2.2 Simulation set-ups and results

Uniaxial tensile tests are performed by stretching the simulation box with a
constant strain rate of 0.001 $\mathrm{ps^{-1}}$ parallel to the $y$
direction. Temperature is kept at 0.1 K, and cross-contraction of the
simulation box is allowed in both lateral dimensions by enforcing zero
pressure using a Nosé-Hoover thermostat and barostat in $NpT$ ensemble. We
studied the effect of CNT length by systematically varying the size of the
simulation box in the loading direction ($L_{y}$) for both short and long CNT
configurations (Table 1). In this set of simulations, the CNT volume fraction
$f_{\mathrm{CNT}}$ is kept at a constant value of 0.27%. The effect of CNT
volume fraction on composite strength is then studied by varying the lateral
sizes ($L_{x}$, and $L_{z}$) of the system according to Table 2 while keeping
the $L_{y}$ fixed.

Figure 3: Left: atomic configurations of ”short” and ”long” CNTs embedded
inside an Al single crystal. Right: (a) Comparison between the stress-strain
curves of an Al single crystal and Al-CNT composites with CNT volume fraction
$f_{\mathrm{CNT}}$ = 0.27 %. (b) Influence of the CNT volume fraction
$f_{\mathrm{CNT}}$ on composite strength for ”short” CNTs and (c) for ”long”
CNTs. (d) Dependency of peak stress on CNT volume fraction for both long and
short CNTs.

Figure 3 (left) shows the initial configurations of both short and long CNTs
in the Al matrix, whereas Figure 3 (a-c) shows stress-strain curves compared
with that of an Al single crystal. In the case of Al reinforced with short
CNTs, the composite strength $\sigma_{\mathrm{p}}$, defined by the highest
value of stress reached on the stress-strain curve, is slightly lower than the
Al single crystal strength $\sigma_{\mathrm{Al}}$ as shown in Figure 3 (a).
According to Figure 3 (b) and (d), increasing the volume fraction
$f_{\mathrm{CNT}}$ of ”short” CNTs leads to a further decrease in strength. As
we observed from our DFT calculations, the interface between Al and CNT is
weak - the interface energy is low, and the interface shear strength (IFSS) is
of the order of 30 MPa or less [28, 27]. Therefore, when the Al-CNT system is
loaded in tension, the pristine short CNT aligned parallel to the loading
direction can easily slide inside the matrix without carrying a significant
part of the applied load. Therefore, under the axial load, the Al atoms at the
Al/CNT interface are stretched, while the short CNT is sliding inside the Al
and are not stretched. Consequently, the short CNT acts like a void that is
elongated along the loading direction, leading to an approximate decrease in
composite strength $\sigma_{\mathrm{p}}$, the peak stress in the stress-strain
curve, like

$\sigma_{\mathrm{p}}=(1-f_{\mathrm{CNT}})\sigma_{\mathrm{Al}}$ (2)

as shown in Figure 3 (d), where $\sigma_{\mathrm{Al}}$ is Al failure strength.

In the case of Al reinforced with long CNTs, we observed an increase in the
composite strength as we increase the CNT volume fraction as shown in Figure 3
(c-d). Upon loading, one observes two load drops in Figure 3(c), first at a
strain $\approx 0.13$ when the Al matrix undergoes profuse dislocation
nucleation ($\epsilon_{\mathrm{Al}}^{\mathrm{f}}$), and then again at a strain
$\approx 0.2$ when the CNT ruptures ($\epsilon_{\mathrm{CNT}}^{\mathrm{f}}$).
We should emphasize that the increase in the composite strength is not a
consequence of any load transfer at the Al/long CNT interface but a direct
result of the applied boundary conditions enforcing isostrain deformation of
the Al block and the embedded CNT. The isostrain deformation, before the first
load drop, forces both Al and CNT to deform in parallel. Thus the peak stress
$\sigma_{\mathrm{p}}$ fulfills the simple relationship

$\sigma_{\mathrm{p}}=\epsilon_{\mathrm{Al}}^{\mathrm{f}}[(1-f_{\mathrm{CNT}})Y_{\mathrm{Al}}+f_{\mathrm{CNT}}Y_{\mathrm{CNT}}]$
(3)

as demonstrated in Figure 3(d) where $Y_{\mathrm{Al}}$ and $Y_{\mathrm{CNT}}$
represent Young’s modulus of Al and CNT, respectively. However, this finding
is physically doubtful since, as demonstrated by our simulation of ”short”
CNTs, even a very small change of the CNT configuration at the boundary of the
simulation supercell suffices to destroy the enforced isostrain deformation
and the strengthening effect which, in consequence, must be deemed a
simulation artifact caused by the improper choice of boundary conditions.

###  3.3 Crack propagation in CNT-reinforced Al crystals under tensile loading

This section studies the interaction of cracks with CNT embedded into the Al
matrix under tensile loading (mode I). We investigated the effects of CNT
positions with respect to the crack tip, CNT angles to the normal vector of
the crack plane, and geometries of entangled CNTs on the crack propagation and
fracture toughness. Figure 4 schematically depicts models of Al-CNT systems
with an edge crack. Figure 4(b-e) are models for examination of the effect of
CNT on the crack blunting (deflection), where a CNT is placed ahead of the
crack tip. Figure 4(h-j) are models to study the effect of CNT on the crack
bridging, where a CNT is placed behind the crack tip. A pure Al matrix with an
edge crack as shown in Figure 4(a) is the reference model used for comparison.

Figure 4: Simulation set-ups for crack–CNT interactions (schematic). The crack
surfaces are parallel to the xz-plane and the crack line runs the $z$
direction. (a) Al with an edge crack, (b-e) models for the effect of CNT on
the crack deflection. In (b), (c) and (d) a long CNT, a short CNT, and a
cylindrical cavity are located ahead of the crack tip, respectively. In (e) a
short CNT is located ahead of the crack tip and makes an angle $\theta$ with
the normal vector of the crack plane. (f-j) models for crack bridging by CNTs.
In (f) a long CNT is located behind the crack tip. (g) an inclined short CNT
is located behind the crack tip. (h-j) correspond to geometrically anchored
CNT located behind the crack tip. (h) Al–CNT anchored by 60$\degree$ side
branches, (i) Al–CNT anchored by 90$\degree$ side branches, (j) Al–CNT
anchored by 120$\degree$ side branches.

####  3.3.1 Sample preparation and simulation set-up

We considere an Al single crystal with faces aligned with the [100], [010],
and [001] directions of the fcc crystal lattice. Al atoms are initially
located put on the sites of a perfect fcc lattice. The system size is 300.64
$\times$ 200.52 $\times$ 202.48 $$\mathrm{\SIUnitSymbolAngstrom}$^{3}$ in $x$,
$y$, $z$ directions, respectively. The created Al matrix is relaxed using the
CG algorithm. A sharp edge crack is introduced into the crystal by removing
the interaction between the atoms on both sides of the crack plane as in [58].
The crack plane is the $xz$ plane and the crack line runs parallel to the $z$
direction. The crack length (=72.86 $\mathrm{\AA}$) is chosen to be
sufficiently long to avoid any dislocation nucleation in the bulk before a
crack extension or dislocation nucleation at the crack tip take place. Surface
boundary conditions are imposed in the $x$ and $y$ directions, whereas
periodic boundary conditions are applied in the $z$ direction (along the crack
front). This model is an example of a brittle crack system with a [001](010)
crack system which can not be contained within any
$\left\\{111\right\\}$-glide plane and therefore, the natural dislocation in
an fcc lattice cannot be emitted from the crack front [59]. A single-walled
CNT(10,10) is embedded into the Al crystal with its longitudinal parallel to
the [010] direction. The embedded CNTs have either equal length, indicated as
”long CNT” or have a 60 $\mathrm{\SIUnitSymbolAngstrom}$ shorter length than
the Al matrix, which is denoted as ”short CNT”. The energy of the system is
minimized as described in section 3.2.1.

The system is loaded in tension (mode I) in the $y$ direction as follows. We
used a Cartesian coordinate system with its origin at the center point of the
block. Velocities of $v_{y}$ and $-v_{y}$, constant in time, are imposed in
$y$ direction on the atoms of the uppermost and lowermost (010) planes
(boundary atoms), initially located at $y=\pm L_{y}/2$. Al atoms between the
lower and upper boundaries are assigned initial velocities, which vary
linearly as functions of the $y$ coordinate, $v(y)=2v_{y}y/L_{y}$. The strain
rate $\dot{\epsilon_{yy}}$ of the tensile test is thus given by
$\dot{\epsilon}_{yy}=2v_{y}/L_{y}$. The tensile deformation is performed at a
constant strain rate of 0.001 $\mathrm{ps^{-1}}$. The temperature of the
system is set to 0.1 K.

###  3.4 Crack bridging by CNTs

Figure 5: (a-c) Stress patterns in pure Al, Al-short CNT, and Al-long CNT,
respectively, (left) at the critical tensile strain and (right) after the
onset of crack propagation. (d) Stress-strain curves for pure Al, Al-long CNT,
and Al-short CNTs with different inclination angles ($\theta$).

Figure 5 (a-c) shows the edge crack conformation at (left), and after (right)
the critical tensile strain, where the system reaches its maximum stress and
the crack starts to propagate. Figure 5 (a) depicts a crack propagating in the
pure Al matrix, whereas (b) shows a matrix reinforced with a short CNT and (c)
with a long CNT. These snapshots represent sections along the $xy$ plane,
which includes the CNT center axis, and atoms are colored based on the
magnitude of their virial stress $\sigma_{yy}$ in the loading direction. As
depicted in this figure, all systems show the same failure behavior. The crack
starts to extend at a critical tensile displacement, corresponding to a sudden
load drop in the stress-strain curves. The crack extension occurs at the
critical strain $\epsilon^{\mathrm{c}}_{\mathrm{Al}}\approx 0.055$ in all
simulations. The simulations of Al–short CNTs show that the CNT easily slides
out of the Al matrix due to the low interfacial shear strength and, as a
consequence, it carries little tensile load. In the case of Al–long CNT, as
the CNT is loaded geometrically parallel with the Al block, the CNT is under
much higher stress compared to the short CNT, the long CNT does not affect the
crack propagation process either, since there is again no shear stress
transfer at the Al/CNT interface. Hence, the presence of the CNT leaves the
crack tip stress field and the crack face opening almost unchanged. The peak
stress of the Al–long CNT composite is slightly higher than that of the pure
Al and Al–short CNT, which is a direct consequence of the isostrain
deformation. The inclination of the short CNT with respect to the crack plane
normal has little influence on the stress-strain curves, as evidenced in
Figure 5 (d).

Figure 6: Stress patterns around cracks bridged by CNTs that are anchored in
the Al matrix by side branches under angles of (a) Al-CNT 60$\degree$, (b) Al-
CNT 90$\degree$, and (c) Al-CNT 120$\degree$, and (d) corresponding stress-
strain curves.

From experimental observations, it is known that CNTs are normally not
isolated inside the matrix. Owing to their high surface area and energy, CNTs
are usually found entangled with each other. This entanglement may anchor them
in the Al matrix and delay the pullout process. Here we investigated the
effect of geometrically anchored CNTs on the fracture process in terms of a
few idealized initial configurations where CNTs are anchored in the matrix by
inclined branches at angles of 60$\degree$, 90$\degree$, and 120$\degree$ as
depicted in Figure 4 (h-j). The energy minimization steps during the sample
preparation and the deformation procedure are the same as in the simulation of
straight CNTs. Stress patterns for each model at the critical strain where the
model systems reach their maximum stress, as well as the respective stress-
strain curves, are shown in Figure 6.

Upon loading, the anchored CNTs carry part of the tensile load, but more
importantly, the presence of the side branches modifies the overall stress
pattern such that additional stress concentrations emerge at the intersections
between CNT and side branches, as well as at the side branches endings, see
Figure 6 (a-c), where the black-colored regions represent the regions with a
high magnitude of tensile stress. On the other hand, the crack tip stress
concentration is reduced. This modification of the stress pattern leads to an
increase in the overall stress and strain when crack propagation occurs. The
effect is most pronounced for the CNT anchored by the side arms under an angle
of 90$\degree$ where fracture stress and strain are increased by about 20% and
35%, respectively, compared to pure Al.

###  3.5 Crack blunting by CNT

This section investigates how CNTs can blunt a sharp, brittle edge crack in an
aluminum matrix. We studied the effect of CNT length, and CNT inclination
angle with respect to the normal vector of the crack plane. The initial
configurations and deformation procedure are as explained in section 3.3.1.
The only difference is that in this section, the CNTs are located ahead of the
crack tip as depicted in Figure 4(b-e).

Figure 7: (a) Stress strain curves of a short and a long CNT, and a short
cavity placed ahead of the crack tip in Al to model crack blunting. (b) Stress
strain curves for short CNTs placed at the crack tip with various inclination
angles ($\theta$).

The presence of a CNT located at the crack tip introduces multi-axial stresses
that may facilitate dislocation nucleation and, as a consequence, induce
plasticity and crack front blunting. We, therefore, analysed the effects of a
CNT intersecting the crack tip. In order to understand whether these effects
are due to the CNT or due to the cylindrical void in the Al lattice induced by
the CNT, we also study the situation where the CNT is removed from its
surrounding cavity. In this manner, we aim to clarify whether the CNT itself
changes the fracture mode or the crack-front blunting is caused by the
existence of a cylindrical void surrounding the CNT.

Figure 7 (a) shows that the Al matrix with a short CNT, a cavity, and a long
CNT have in essence the same stress-strain curves. They all restrain the crack
in a similar manner from expanding, leading to an increase of 10% and 20% in
fracture stress and fracture strain, respectively. Figure 8 gives a closer
look at the crack growth process in the case of a short CNT intersecting the
crack tip. As the applied strain increases, small dislocation loops (which are
split into Shockley partials) are emitted from the Al/CNT interface and expand
on the crack surface as well as into the bulk. The CNT thus acts as a catalyst
that facilitates dislocation nucleation and plastification of the Al matrix.
This process is not contingent on the presence of the CNT - in fact, the
cylindrical void surrounding the CNT without the embedded CNT acts in exactly
the same manner.

Figure 8: Deformation of Al–CNT composite under an applied tensile
displacement in $y$ direction where a straight short CNT blocks the predefined
edge crack. (a) $\epsilon_{yy}=0.058$, (b) $\epsilon_{yy}=0.062$, (c)
$\epsilon_{yy}=0.066$. Here only the atoms with perturbations of the fcc
environment (icosahedral arrangement of nearest neighbors) are visualized.
Green lines depict Shockley partial dislocations emitted from the crack tip.
Pink lines show stair-rod dislocations. Atoms located at stacking faults are
colored in red. Atoms with no crystalline structure are colored in light grey.

To understand the effect of CNT orientation, we placed a short CNT such that
it intersects the crack tip and then rotates the CNT axis clockwise by steps
of $\Delta\theta$ = 15$\degree$ around the direction of the crack front
(Figure 4 (e)). The effect of CNT inclination angle $\theta$ on crack blunting
is shown in Figure 7 (b). Stress concentrations at the Al/CNT interface and
the intersection of CNTs’ endings and Al facilitate dislocation nucleation,
resulting in a combination of crack growth and dislocation emission from the
Al/CNT interface. As a result of energy dissipation due to the ensuing
dislocation motions, the stress-strain curves show toughening of the
composite. This effect somewhat depends on the angle between the fracture
surface normal and the CNT longitudinal axis. For orientations close to
$\theta=45\degree$, the presence of the CNT can even have a slight weakening
effect. To understand this effect, we looked closer at the deformation process
of Al with a short CNT with the inclination angle of $\theta$ = 45$\degree$,
intersecting the crack tip as shown in Figure 9 from the side-view. In this
case, the CNT is aligned with a Burgers vector of the fcc lattice (and
accordingly with the intersection of two slip planes) and can, therefore, act
as a very efficient site for dislocation nucleation from the Al/CNT interface,
which happens before the critical fracture stress is reached. In Figure 9, we
can see that at a strain of 0.041, a pair of dislocation half loops of Burgers
vector $<110>$ nucleate on both sides of the CNT at the locations where the
CNT intersects the crack tip. These emergent loops split into Shockley
partials and expand on the two conjugate (1-11) and (1-1-1) slip planes such
that their endpoints travel on the crack surface and the Al/CNT interface.
Once they have reached the CNT endpoint, the two loops merge and detach from
the CNT and propagate through the matrix until they collide with the fixed
upper boundary layer or annihilate at the surface.

Figure 9: Crack activity in Al where a $\theta$ = 45$\degree$ inclined CNT is
placed at the crack-front, (a-c) $\epsilon_{yy}$ = 0.041, $\epsilon_{yy}$ =
0.043, and $\epsilon_{yy}$ = 0.046, respectively. The color scheme is same as
in Figure 8.

##  4 Conclusions

We studied the deformation mechanisms of aluminum nanocomposites reinforced
with carbon nanotubes of different sizes and geometries under tensile loading.
We found that the physical nature of the interactions between Al and CNT
significantly influences the composite’s mechanical behavior. Van der Waals
interactions between Al and $sp^{2}$ bonded carbon in CNT lead to weak bonding
and a ’flat’ interface energy landscape, which facilitates CNT pull-out from
the surrounding Al matrix. Thus, uniaxial tensile simulations of Al-CNT, where
CNTs are aligned parallel to the loading direction, showed that pristine CNTs
do not provide significant reinforcement as such CNTs can easily slide inside
the Al matrix. The same is true for CNT-crack interactions where crack
bridging was found to be inefficient.

A positive effect of CNTs is that their interfaces with the Al matrix may act
as sites for dislocation nucleation at crack tips and thus provide enhanced
toughness through crack blunting. However, this effect is not contingent on
the presence of CNTs as simulations with the surrounding cylindrical void
produce the same outcome.

In view of the negative verdict on CNT reinforcement as far as load sharing
and crack bridging are concerned, one may ask how the underlying root cause,
namely weak interfacial bonding, can be mitigated. To this end, three
different strategies are available: (i) The present study has demonstrated
that CNTs that are geometrically entangled inside the Al matrix can act as
efficient crack bridges and impede crack propagation. Such composites might be
fabricated, e.g., by infiltration of self-entangled CNT networks [60]. (ii) To
improve interfacial bonding, the chemical inertness of the CNT surface may be
modified by chemical functionalization that offers sites for chemical bonding.
In the context of Al-CNT composites, this might happen through carbide
formation [61], which, however, needs to be carefully controlled to preserve
the structural properties of the CNTs. (iii) An alternative which has been
studied in the context of both metal-CNT and polymer-CNT composites consists
in non-chemical functionalization of the CNT surface [62, 22] which in the
context of Al composites might be done by deposition of Ni nanoparticles or
coatings [28] or surface decoration by other nanoparticles such as TiC [63].

## Acknowledgments

SN and MZ acknowledge financial support from DFG under grant no. Za171/11-1.
The authors gratefully acknowledge the compute resources and support provided
by the Erlangen Regional Computing Center (RRZE).

## Conflict of Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## References

  * [1] Q. Zhao, M. B. Nardelli, J. Bernholc, Ultimate strength of carbon nanotubes: A theoretical study, Physical Review B 65 (14) (2002).  doi:10.1103/physrevb.65.144105.
  * [2] S. L. Mielke, D. Troya, S. Zhang, J.-L. Li, S. Xiao, R. Car, R. S. Ruoff, G. C. Schatz, T. Belytschko, The role of vacancy defects and holes in the fracture of carbon nanotubes, Chemical Physics Letters 390 (4-6) (2004) 413–420.  doi:10.1016/j.cplett.2004.04.054.
  * [3] M. Sammalkorpi, A. Krasheninnikov, A. Kuronen, K. Nordlund, K. Kaski, Mechanical properties of carbon nanotubes with vacancies and related defects, Physical Review B 70 (24) (2004).  doi:10.1103/physrevb.70.245416.
  * [4] R. Khare, S. L. Mielke, J. T. Paci, S. Zhang, R. Ballarini, G. C. Schatz, T. Belytschko, Coupled quantum mechanical/molecular mechanical modeling of the fracture of defective carbon nanotubes and graphene sheets, Physical Review B 75 (7) (2007).  doi:10.1103/physrevb.75.075412.
  * [5] C. Lee, X. Wei, J. W. Kysar, J. Hone, Measurement of the elastic properties and intrinsic strength of monolayer graphene, Science 321 (5887) (2008) 385–388.  doi:10.1126/science.1157996.
  * [6] M. Yu, Strength and breaking mechanism of multiwalled carbon nanotubes under tensile load, Science 287 (5453) (2000) 637–640.  doi:10.1126/science.287.5453.637.
  * [7] S. Nasiri, M. Zaiser, Rupture of graphene sheets with randomly distributed defects, AIMS materials science 3 (4) (2016) 1340–1349.  doi:10.3934/matersci.2016.4.1340.
  * [8] A. A. Balandin, Thermal properties of graphene and nanostructured carbon materials, Nature materials 10 (8) (2011) 569.  doi:10.1038/nmat3064.
  * [9] E. Pop, V. Varshney, A. K. Roy, Thermal properties of graphene: Fundamentals and applications, MRS bulletin 37 (12) (2012) 1273–1281.  doi:10.1557/mrs.2012.203.
  * [10] R. Zhang, Y. Zhang, Q. Zhang, H. Xie, W. Qian, F. Wei, Growth of half-meter long carbon nanotubes based on schulz–flory distribution, Acs Nano 7 (7) (2013) 6156–6161.  doi:10.1021/nn401995z.
  * [11] L. Zhang, F. Zhang, X. Yang, G. Long, Y. Wu, T. Zhang, K. Leng, Y. Huang, Y. Ma, A. Yu, et al., Porous 3d graphene-based bulk materials with exceptional high surface area and excellent conductivity for supercapacitors, Scientific reports 3 (2013) 1408.  doi:10.1038/srep01408.
  * [12] M. D. Stoller, S. Park, Y. Zhu, J. An, R. S. Ruoff, Graphene-based ultracapacitors, Nano letters 8 (10) (2008) 3498–3502.  doi:10.1021/nl802558y.
  * [13] H.-J. Choi, S.-M. Jung, J.-M. Seo, D. W. Chang, L. Dai, J.-B. Baek, Graphene for energy conversion and storage in fuel cells and supercapacitors, Nano Energy 1 (4) (2012) 534–551.  doi:10.1016/j.nanoen.2012.05.001.
  * [14] B. Chen, K. Kondoh, J. Umeda, S. Li, L. Jia, J. Li, Interfacial in-situ al2o3 nanoparticles enhance load transfer in carbon nanotube (CNT)-reinforced aluminum matrix composites, Journal of Alloys and Compounds 789 (2019) 25–29.  doi:10.1016/j.jallcom.2019.03.063.
  * [15] B. Chen, J. Shen, X. Ye, L. Jia, S. Li, J. Umeda, M. Takahashi, K. Kondoh, Length effect of carbon nanotubes on the strengthening mechanisms in metal matrix composites, Acta Materialia 140 (2017) 317–325.  doi:10.1016/j.actamat.2017.08.048.
  * [16] B. Chen, S. Li, H. Imai, L. Jia, J. Umeda, M. Takahashi, K. Kondoh, Load transfer strengthening in carbon nanotubes reinforced metal matrix composites via in-situ tensile tests, Composites Science and Technology 113 (2015) 1–8.  doi:10.1016/j.compscitech.2015.03.009.
  * [17] R. George, K. Kashyap, R. Rahul, S. Yamdagni, Strengthening in carbon nanotube/aluminium (CNT/al) composites, Scripta Materialia 53 (10) (2005) 1159–1163.  doi:10.1016/j.scriptamat.2005.07.022.
  * [18] B. Q. Li, M. L. Sui, B. Li, E. Ma, S. X. Mao, Reversible twinning in pure aluminum, Physical Review Letters 102 (20) (2009).  doi:10.1103/physrevlett.102.205504.
  * [19] S. R. Bakshi, A. Agarwal, An analysis of the factors affecting strengthening in carbon nanotube reinforced aluminum composites, carbon 49 (2) (2011) 533–544.  doi:10.1016/j.carbon.2010.09.054.
  * [20] Y. Kim, J. Lee, M. S. Yeom, J. W. Shin, H. Kim, Y. Cui, J. W. Kysar, J. Hone, Y. Jung, S. Jeon, et al., Strengthening effect of single-atomic-layer graphene in metal–graphene nanolayered composites, Nature communications 4 (2013) ncomms3114.  doi:10.1038/ncomms3114.
  * [21] Q. Li, A. Viereckl, C. A. Rottmair, R. F. Singer, Improved processing of carbon nanotube/magnesium alloy composites, Composites Science and Technology 69 (7-8) (2009) 1193–1199.  doi:10.1016/j.compscitech.2009.02.020.
  * [22] Q. Li, C. A. Rottmair, R. F. Singer, CNT reinforced light metal composites produced by melt stirring and by high pressure die casting, Composites Science and Technology 70 (16) (2010) 2242–2247.  doi:10.1016/j.compscitech.2010.05.024.
  * [23] S. Hippmann, Q. Li, R. Addinal, W. Volk, Carbon nanotubes–reinforced copper matrix composites produced by melt stirring, Proceedings of the Institution of Mechanical Engineers, Part N: Journal of Nanoengineering and Nanosystems 227 (2) (2013) 63–66.  doi:10.1177/1740349912467730.
  * [24] J. G. Park, D. H. Keum, Y. H. Lee, Strengthening mechanisms in carbon nanotube-reinforced aluminum composites, Carbon 95 (2015) 690–698.  doi:10.1016/j.carbon.2015.08.112.
  * [25] C. Balázsi, B. Fényi, N. Hegman, Z. Kövér, F. Wéber, Z. Vértesy, Z. Kónya, I. Kiricsi, L. P. Biró, P. Arató, Development of cnt/si3n4 composites with improved mechanical and electrical properties, Composites Part B: Engineering 37 (6) (2006) 418 – 424.  doi:10.1016/j.compositesb.2006.02.006.
  * [26] T. L. Anderson, Fracture mechanics: fundamentals and applications, CRC press, 2017\.  doi:10.1201/9781315370293.
  * [27] W. Zhou, G. Yamamoto, Y. Fan, H. Kwon, T. Hashida, A. Kawasaki, In-situ characterization of interfacial shear strength in multi-walled carbon nanotube reinforced aluminum matrix composites, Carbon 106 (2016) 37–47.  doi:10.1016/j.carbon.2016.05.015.
  * [28] S. Nasiri, K. Wang, M. Yang, Q. Li, M. Zaiser, Nickel coated carbon nanotubes in aluminum matrix composites: a multiscale simulation study, The European Physical Journal B 92 (8) (2019).  doi:10.1140/epjb/e2019-100243-6.
  * [29] M. Jagannatham, P. Chandran, S. Sankaran, P. Haridoss, N. Nayan, S. R. Bakshi, Tensile properties of carbon nanotubes reinforced aluminum matrix composites: A review, Carbon 160 (2020) 14–44.  doi:10.1016/j.carbon.2020.01.007.
  * [30] B. K. Choi, G. H. Yoon, S. Lee, Molecular dynamics studies of CNT-reinforced aluminum composites under uniaxial tensile loading, Composites Part B: Engineering 91 (2016) 119–125.  doi:10.1016/j.compositesb.2015.12.031.
  * [31] D. M. Park, J. H. Kim, S. J. Lee, G. H. Yoon, Analysis of geometrical characteristics of CNT-al composite using molecular dynamics and the modified rule of mixture (MROM), Journal of Mechanical Science and Technology 32 (12) (2018) 5845–5853.  doi:10.1007/s12206-018-1133-5.
  * [32] N. Silvestre, B. Faria, J. N. C. Lopes, Compressive behavior of CNT-reinforced aluminum composites using molecular dynamics, Composites Science and Technology 90 (2014) 16–24.  doi:10.1016/j.compscitech.2013.09.027.
  * [33] J. Xiang, L. Xie, S. A. Meguid, S. Pang, J. Yi, Y. Zhang, R. Liang, An atomic-level understanding of the strengthening mechanism of aluminum matrix composites reinforced by aligned carbon nanotubes, Computational Materials Science 128 (2017) 359–372.  doi:10.1016/j.commatsci.2016.11.032.
  * [34] M. E. Suk, Atomistic behavior of nanoporous carbon nanotube-aluminum composite under compressive loading, Materials Research Express 7 (1) (2020) 015028.  doi:10.1088/2053-1591/ab6092.
  * [35] S. J. Stuart, A. B. Tutein, J. A. Harrison, A reactive potential for hydrocarbons with intermolecular interactions, The Journal of chemical physics 112 (14) (2000) 6472–6486.  doi:10.1063/1.481208.
  * [36] M. Yang, V. Koutsos, M. Zaiser, Size effect in the tensile fracture of single-walled carbon nanotubes with defects, Nanotechnology 18 (15) (2007) 155708\.  doi:10.1088/0957-4484/18/15/155708.
  * [37] L. Pastewka, P. Pou, R. Pérez, P. Gumbsch, M. Moseler, Describing bond-breaking processes by reactive potentials: Importance of an environment-dependent interaction range, Physical Review B 78 (16) (2008).  doi:10.1103/physrevb.78.161402.
  * [38] D. Boda, D. Henderson, The effects of deviations from lorentz–berthelot rules on the properties of a simple mixture, Molecular Physics 106 (20) (2008) 2367–2370.  doi:10.1080/00268970802471137.
  * [39] J. DELHOMMELLE, P. MILLIÉ, Inadequacy of the lorentz-berthelot combining rules for accurate predictions of equilibrium properties by molecular simulation, Molecular Physics 99 (8) (2001) 619–625.  doi:10.1080/00268970010020041.
  * [40] C. Gong, G. Lee, B. Shan, E. M. Vogel, R. M. Wallace, K. Cho, First-principles study of metal–graphene interfaces, Journal of Applied Physics 108 (12) (2010) 123711.  doi:10.1063/1.3524232.
  * [41] P. Khomyakov, G. Giovannetti, P. Rusu, G. v. Brocks, J. Van den Brink, P. J. Kelly, First-principles study of the interaction and charge transfer between graphene and metals, Physical Review B 79 (19) (2009) 195425.  doi:10.1103/physrevb.79.195425.
  * [42] P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, et al., Quantum espresso: a modular and open-source software project for quantum simulations of materials, Journal of physics: Condensed matter 21 (39) (2009) 395502.  doi:10.1088/0953-8984/21/39/395502.
  * [43] P. E. Blöchl, Projector augmented-wave method, Physical review B 50 (24) (1994) 17953.  doi:10.1103/physrevb.50.17953.
  * [44] A. D. Becke, Density-functional exchange-energy approximation with correct asymptotic behavior, Physical review A 38 (6) (1988) 3098.  doi:10.1103/physreva.38.3098.
  * [45] G. Graziano, J. Klimeš, F. Fernandez-Alonso, A. Michaelides, Improved description of soft layered materials with van der waals density functional theory, Journal of Physics: Condensed Matter 24 (42) (2012) 424216.  doi:10.1088/0953-8984/24/42/424216.
  * [46] M. S. Christian, A. O. de-la Roza, E. R. Johnson, Adsorption of graphene to metal (111) surfaces using the exchange-hole dipole moment model, Carbon 124 (2017) 531 – 540.  doi:10.1016/j.carbon.2017.08.077.
  * [47] M. Vanin, J. J. Mortensen, A. K. Kelkkanen, J. M. Garcia-Lastra, K. S. Thygesen, K. W. Jacobsen, Graphene on metals: A van der waals density functional study, Physical Review B 81 (8) (2010).  doi:10.1103/physrevb.81.081408.
  * [48] S. Plimpton, Fast parallel algorithms for short-range molecular dynamics, Journal of computational physics 117 (1) (1995) 1–19.  doi:10.1006/jcph.1995.1039.
  * [49] P. Hirel, Atomsk: A tool for manipulating and converting atomic data files, Computer Physics Communications 197 (2015) 212–219.  doi:10.1016/j.cpc.2015.07.012.
  * [50] J. Schneider, J. Hamaekers, S. T. Chill, S. Smidstrup, J. Bulin, R. Thesen, A. Blom, K. Stokbro, Atk-forcefield: a new generation molecular dynamics software package, Modelling and Simulation in Materials Science and Engineering 25 (8) (2017) 085007.  doi:10.1088/1361-651x/aa8ff0.
  * [51] D. Stradi, L. Jelver, S. Smidstrup, K. Stokbro, Method for determining optimal supercell representation of interfaces, Journal of Physics: Condensed Matter 29 (18) (2017) 185901.  doi:10.1088/1361-648x/aa66f3.
  * [52] A. Stukowski, Visualization and analysis of atomistic simulation data with OVITO-the Open Visualization Tool, Modelling and Simulation in Materials Science and Engineering 18 (1) (2010).  doi:10.1088/0965-0393/18/1/015012.
  * [53] D. Faken, H. Jónsson, Systematic analysis of local atomic structure combined with 3d computer graphics, Computational Materials Science 2 (2) (1994) 279–286.  doi:10.1016/0927-0256(94)90109-0.
  * [54] J. D. Honeycutt, H. C. Andersen, Molecular dynamics study of melting and freezing of small lennard-jones clusters, The Journal of Physical Chemistry 91 (19) (1987) 4950–4963.  doi:10.1021/j100303a014.
  * [55] Y. Mishin, Atomistic modeling of the $\gamma$ and $\gamma\prime$-phases of the ni–al system, Acta Materialia 52 (6) (2004) 1451–1467.  doi:10.1016/j.actamat.2003.11.026.
  * [56] M. Moseler, F. Cervantes-Sodi, S. Hofmann, G. Csányi, A. C. Ferrari, Dynamic catalyst restructuring during carbon nanotube growth, ACS Nano 4 (12) (2010) 7587–7595.  doi:10.1021/nn102118y.
  * [57] A. Vaid, J. Guénolé, A. Prakash, S. Korte-Kerzel, E. Bitzek, Atomistic simulations of basal dislocations in mg interacting with mg17al12 precipitates, Materialia 7 (2019) 100355.  doi:https://doi.org/10.1016/j.mtla.2019.100355.
  * [58] C. Cui, H. Beom, Molecular dynamics simulations of edge cracks in copper and aluminum single crystals, Materials Science and Engineering: A 609 (2014) 102–109.  doi:10.1016/j.msea.2014.04.101.
  * [59] P. Gumbsch, An atomistic study of brittle fracture: Toward explicit failure criteria from atomistic modeling, Journal of Materials Research 10 (11) (1995) 2897–2907.  doi:10.1557/jmr.1995.2897.
  * [60] F. Schütt, S. Signetti, H. Krüger, S. Röder, D. Smazna, S. Kaps, S. N. Gorb, Y. K. Mishra, N. M. Pugno, R. Adelung, Hierarchical self-entangled carbon nanotube tube networks, Nature communications 8 (1) (2017) 1–10.  doi:10.1038/s41467-017-01324-7.
  * [61] L. Ci, Z. Ryu, N. Y. Jin-Phillipp, M. Rühle, Investigation of the interfacial reaction between multi-walled carbon nanotubes and aluminum, Acta Materialia 54 (20) (2006) 5367–5375.  doi:10.1016/j.actamat.2006.06.031.
  * [62] Q. Li, M. Zaiser, V. Koutsos, Carbon nanotube/epoxy resin composites using a block copolymer as a dispersing agent, physica status solidi (a) 201 (13) (2004) 89–91.  doi:10.1002/pssa.200409065.
  * [63] F. Saba, S. A. Sajjadi, M. Haddad-Sabzevar, F. Zhang, Formation mechanism of nano titanium carbide on multi-walled carbon nanotube and influence of the nanocarbides on the load-bearing contribution of the nanotubes inner-walls in aluminum-matrix composites, Carbon 115 (2017) 720–729.  doi:10.1016/j.carbon.2017.01.062.


# ASR-Generated Text for Language Model Pre-training
Applied to Speech Tasks

###### Abstract

We aim at improving spoken language modeling (LM) using very large amount of
automatically transcribed speech. We leverage the INA (French National
Audiovisual Institute111https://www.ina.fr) collection and obtain 19GB of text
after applying ASR on 350,000 hours of diverse TV shows. From this, spoken
language models are trained either by fine-tuning an existing LM
(FlauBERT222https://github.com/getalp/Flaubert) or through training a LM from
scratch. New models (FlauBERT-Oral) are shared with the community and
evaluated for 3 downstream tasks: spoken language understanding,
classification of TV shows and speech syntactic parsing. Results show that
FlauBERT-Oral can be beneficial compared to its initial FlauBERT version
demonstrating that, despite its inherent noisy nature, ASR-generated text can
be used to build spoken language models.

##  1 Introduction

Large pretrained language models (LMs) have been widely adopted as backbone
for performing natural language processing tasks, whether on text [1] or on
speech transcripts [2]. Those models are typically trained with massive
quantities of text, and do not reflect well the specifics of spoken language.
In this study, we investigate whether there is some benefit in better covering
spoken-language phenomena in pretrained LMs when applying them to speech use
cases.

Pretrained LMs, such as BERT [3], have been applied with success to a range of
tasks that involve speech: ASR hypothesis rescoring [4], Spoken Language
Understanding [5], Dialog State Tracking [6], Conversation Summarization [7],
Emotion Recognition [8], Speaker and Language recognition [9]. Yet, LM
training data is dominated by textual web content, which diverges from actual
spoken language. Even though the web is filled with conversation-like
interactions, such as forums and social media in which agrammaticality or
colloquiallity are frequent, it does not include some fundamental properties
that are inherent to spontaneous speech: floor holding and grabbing,
conversational discourse markers, disfluencies such as repairs, adverse
effects of noise and speaker overlap, among others. One may wonder if
addressing such phenomena without having seen them in training is a capability
of current techniques associated with LMs, and whether they matter in the
context of everyday spoken language processing tasks.

Building LMs aware of speech phenomena is a difficult task. Recent work
addresses the problem through direct learning of acoustic representations [10,
11], but have yet to show competitive performance compared to word-based
models for high-level linguistic tasks. Instead, we aim at retraining LMs from
speech using word transcripts, and are faced with the inherent challenge of
gathering enough training material to be on par with text-only models.
Therefore, we investigate the use of massive amounts of automatic speech
recognition (ASR) generated text for spoken language modeling. This could
bring diversity (oral/spontaneous style, different topics) to the LM training
data and might also be useful for languages with fewer text resources but high
availability of speech recordings. More precisely, we build and share oral LMs
trained from a large amount (350,000 hours) of French TV shows, and evaluate
them on three spoken language tasks: spoken language understanding, topic
classification of TV shows, and spoken language syntactic parsing. Results
suggest that ASR-generated text is a viable alternative to text not
originating from speech when training spoken language models.

To our knowledge, this is the first time a LM is trained on such a massive
amount of ASR transcripts. Indeed, [12] evaluated BERT based language models
(BERT, RoBERTa) trained on spoken transcripts to investigate their ability to
encode properties of spoken language, but most of their experiments are on
small quantities of manual transcripts, except one which is trained on 2000h
of ASR output (1k Librispeech + 1k proprietary dataset). We train LMs on
larger quantities of ASR output, analyse their benefits on speech-specific
tasks, and make them available for further research.

##  2 From FlauBERT to FlauBERT-Oral

###  2.1 Large speech corpus with ASR transcripts

The FlauBERT-Oral model is trained on 350,000 hours of speech collected from
various French TV and radio channels collected at Institut National
Audiovisuel (INA) from 2013 to 2020. TV news channels were continuously
captured between 6am and midnight each day (BFMTV, LCI, CNews, France 24,
France Info). For radio, the morning news were used (Europe1, RMC, RTL, France
Inter) and for generalist TV channels we collected evening news (TF1, France
2, France 3, M6).

Collected speech is automatically transcribed with an ASR system built from
Kaldi [13]. In that system, acoustic models are trained from French TV and
RADIO corpora (ESTER 1&2 [14], REPERE [15] VERA [16]). A regular backoff
n-gram model is estimated using manual transcripts augmented with several
French newspapers [17], resulting in a vocabulary of 160k most frequent words.
Automatic speech diarization of the collection was performed using
LIUMSpkDiarization [18]. The whole pipeline obtains the following ASR
performance on different reference test corpora: REPERE (12.1%WER), ESTER1
(8.8%WER) and ESTER2 (10.7%WER).

As the ASR system outputs raw text without punctuation nor capitalization,
speech turns from diarization serve as pseudo sentences for later processing.
This results in 51M unique speech segments for a total of 3.5G words (19GB of
data). It is important to note that this ’oral’ data is highly biased towards
radio and TV news type.

###  2.2 Fine-tuning or re-training FlauBERT

All the models generated in this study are built on the FlauBERT family of
models, one of the first pre-trained LMs for French [19]. FlauBERT is trained
with a masked language modeling (MLM) objective, and implements the RoBERTa
architecture [20]. It is trained on 71GB of natural texts collected from
various sources, among which a majority of crawled web pages. The baseline for
our experiments is flaubert-base-uncased
(FBU)333https://huggingface.co/flaubert/flaubert_base_uncased since we only
have lowercase transcripts. We compare it to several learning configurations
based on the same architecture in order to observe the impact of different
parameters on the quality of the obtained models.

The first configuration, FlauBERT-O-base_uncased (FT), consists in fine-tuning
flaubert-base-uncased for several epochs using the large corpus of ASR
transcripts.

The second configuration FlauBERT-O-mixed (MIX) is a full model re-trained
using a balanced mix of ASR transcripts and written text. Written texts come
from two main sources: a French Wikipedia dump and press articles captured by
the OTMedia research platform [21] (online press and AFP agency for the same
time period). Overall, this learning dataset is also strongly news-biased. The
written part of this training dataset is 94M randomly selected sentences
representing 13G of data stripped from punctuation and capitalization to make
it consistent with ASR data. For this mixed model, we also retrain the BPE
tokenizer (50K sub-word units).

The third configuration, FlauBERT-O-asr, consists in re-training two LMs from
scratch using ASR data only. For the first model (ORAL), we use the tokenizer
provided with flaubert-base-uncased and for the second one (ORAL_NB) we
retrain a BPE tokenizer (50K sub-word units). Both tokenizers share 52% of
their tokens only.

The four models were trained on a single server with two 12-core Xeon CPUs,
256 GB of RAM and 8 Nvidia GeForce RTX 2080 Ti GPUs with 11 GB of memory. It
lasted 15 days for 50 epochs of each model in the flaubert-base configuration
(137M parameters) using the original FlauBERT code. The models are made
available online.444https://huggingface.co/nherve The following sections
describe experiments on various downstream tasks.

##  3 Spoken Language Understanding

Spoken Language Understanding (SLU) systems extract semantic information from
speech utterances [22], which can be represented with slots or concepts tags
on the token sequence. Pipeline SLU systems typically generate transcripts
from spoken dialogues with an ASR system, and recognize slots with a Natural
Language Understanding (NLU) module that only uses the word sequence [23, 24].
For example, in the recent approach by [25], a first ASR model finetuned on
the domain obtains transcriptions, which is fed to a finetuned BERT-like model
for classification. In order to reduce the impact of error propagation in the
pipeline, alternative approaches have been proposed, such as end-to-end models
that directly predict concepts from speech [26, 27, 28]. Our assumption with
FlauBERT-Oral models is that the use of ASR-robust pretrained language models
for NLU can decrease the impact of ASR errors without resorting to end-to-end
training.

###  3.1 The MEDIA SLU benchmark

The French MEDIA dataset [29] contains $1,250$ telephone dialogues captured in
Wizard-of-Oz settings for hotel booking and information requests in French.
Transcribed segments are annotated with $76$ concept tags that denote both
domain-specific information (such as hotel-services or room-type) and generic
concepts (like logical connectors as connectprop). Normalized concept values
also have to be predicted. An example can be found in Table 1.

Table 1: Sample instance from MEDIA. (a) corresponds to the transcribed sentence; (b) with associated concepts; (c) with concepts and normalized values; (d) concepts in BIO format; utterances were translated to English. (a) |  i would like to book one double room in paris
---|---
(b) |  < cmd-task i would like to book > < nb-room one > < room-type double room > in  < loc-city paris >
(c) |  < cmd-task booking > < nb-room 1 > < room-type double > in  < loc-city paris >
(d) |  B-cmd-task I-cmd-task I-cmd-task I-cmd-task I-cmd-task B-nb-room B-room-type I-room-type O B-loc-city

We use the official train/dev/test split of the MEDIA corpus, containing
respectively $12.9$k, $1.3$k and $3.5$k user utterances. SLU performance is
evaluated with Concept Error Rate (CER), the rate of errors between force-
aligned reference and hypothesis concepts, and Concept-Value Error Rate
(CVER), the same for (concept, value) pairs. Values are extracted with the
same rule-based system as in other studies on MEDIA [25, 27, 30].

###  3.2 Experiments

We use the different pre-trained models presented in section 2.2 as pre-
trained Natural Language Understanding (NLU) components. We add a decision
layer at the representations generated by the models, in order to tag each
input token to its corresponding BIO concept. Each model is trained for $300$
epochs, and the best checkpoint is selected according to CER on the
development corpus. We release models and make them available
online.555https://huggingface.co/vpelloin/MEDIA_NLU_flaubert_uncased,
https://huggingface.co/vpelloin/MEDIA_NLU_flaubert_finetuned

Table 2 presents the test results for the different FlauBERT configurations.
The baseline State-Of-The-Art (SOTA) results [25] are obtained with a cascade
composed of wav2vec 2.0 [31] (finetuned on CommonVoice [32] and MEDIA),
followed by a finetuned CamemBERT [33], similar to our approach. For
transcription, we use a Kaldi system that reaches $9.1\%$ WER on MEDIA
(test)—ASR for the SOTA system has a WER of $8.5\%$.

Table 2: SLU Task: MEDIA test results according to Concept Error Rate and Concept-Value Error Rate (lower is better), using ASR transcriptions as input of NLU FlauBERT models. Confidence intervals are computed using the Student’s t-distribution at $95\%$. Model | %CER | %CVER
---|---|---
w2v 2.0+CamemBERT [25] | $11.2$ | $17.2$
FlauBERT cased | $13.20\pm 0.71$ | $18.25\pm 0.81$
FlauBERT uncased (FBU) | $12.40\pm 0.69$ | $17.49\pm 0.79$
FT | $\textbf{11.98}\pm 0.68$ | $\textbf{17.00}\pm 0.79$
MIX | $12.47\pm 0.69$ | $17.66\pm 0.80$
ORAL | $12.43\pm 0.69$ | $17.84\pm 0.80$
ORAL_NB | $12.24\pm 0.68$ | $17.67\pm 0.80$

We can see that all the FlauBERT-O uncased models (FT, MIX, ORAL and ORAL_NB)
achieve comparable results with the baseline FlauBERT models (cased and
uncased). It can be noticed that the FT model obtains an absolute improvement
of 0.42% CER compared to FlauBERT uncased. This suggests that pretraining LMs
on ASR transcripts can benefit downstream tasks even though the domain is
quite different (TV shows vs telephone conversations). Although we use ASR
transcripts with lower quality compared to [25] we obtain almost similar CER
using the FT model.

One limitation of our experiments is that, even though SLU is applied to
(noisy) ASR transcripts, during fine-tuning SLU models are updated using the
(error free) reference transcripts of MEDIA. We believe that fine-tuning them
on (noisy) ASR output could lead to even better results as it would match pre-
training and test conditions. This improvement is left for future work but the
next section will propose a classification task where ASR outputs are used
during both fine-tuning and evaluation.

##  4 Automatic Classification of TV Shows

This section describes the evaluation of the different LMs on news
classification. As part of the collection process, INA’s documentalists finely
segment newscasts of the main generalist TV/radio channels, and annotate them
with content descriptions. This very rich metadata is used in particular to
establish quantitative studies on the news in France. The InaStat
barometer666http://www.inatheque.fr/publications-evenements/ina-stat/ina-stat-
sommaire.html has set up a stable methodology over time to classify news items
into 14 categories (such as society, French politics, sport or environment).
For classification experiments, we use the news items of 4 channels (TF1,
France 2, France 3 and M6) for the years 2017, 2018 and 2019, which gives a
total of 47,867 short TV shows, running on average 92 seconds.

The objective is to assess classification accuracy into the 14 categories
solely on the basis of ASR transcripts. We compare results to a simple SVM
classifier (with a non-parametric triangular kernel) on TF-IDF vectors
baseline, with two vocabulary sizes of 5K and 20K words. A FlauBERT classifier
is created for each model by appending a classification layer after mean
pooling. Models are not selected and rather we report results for different
number of training epochs. Since the categories are not well-balanced, we use
weighted F1 to evaluate performance. The experiments are systematically
performed on 10 different random splits of the dataset, taking into account
the cardinality of the 14 categories, so as to have 38K examples for the
training set and 5K for the test set. Mean performance results and standard
deviation are shown in Table 3.

Table 3: TV news classification task: train 38K, test 5K, F1 weighted metric. Higher is better.  Model | Epoch 1 | Epoch 3 | Epoch 10
---|---|---|---
SVM-20K |  $0.777\pm 0.005$
FBU | $0.780\pm 0.012$ | $0.807\pm 0.010$ | $0.809\pm 0.006$
FT | $0.784\pm 0.006$ | $0.807\pm 0.006$ | $0.817\pm 0.004$
MIX | $0.791\pm 0.005$ | $0.812\pm 0.003$ | $0.820\pm 0.004$
ORAL | $\textbf{0.811}\pm 0.004$ | $\textbf{0.829}\pm 0.005$ | $\textbf{0.824}\pm 0.004$
ORAL_NB | $0.798\pm 0.004$ | $0.814\pm 0.005$ | $0.824\pm 0.004$

If we look at the performance at the first epoch, the FBU
(flaubert_base_uncased) model has almost equivalent performance to the SVM
baseline (0.777). It is only after a few iterations of fine-tuning that the
model fits the ASR data and reaches 0.809. On the other hand, the models that
have already seen ASR data during training have better performance from the
first epoch. The model trained only on ASR data is the best (ORAL). The
ORAL_NB model is slightly worse: in this model, only the tokenizer is
different, so it seems better to not adapt BPE to ASR transcripts, probably
because it suffers from the closed vocabulary of the underlying system. After
10 epochs, all FlauBERT-Oral models converge to the same performance and are
better than FBU for this task.

##  5 Syntactic Parsing of Spoken Dialogs

In this section, the downstream task consists in jointly predicting part of
speech tags (POS) and building a labelled dependency tree from speech
transcripts. We use our different LMs to generate contextual word
representations and use them for predicting syntactic parses of speech
transcripts. We contrast the obtained results to a baseline model trained
using FastText non-contextual representations,777https://fasttext.cc and a
model learning its own representations without any pretraining.

###  5.1 The ORFEO dataset

Our study relies on the annotated subset of the speech corpus of the ORFEO
project [34, 35], gathered with the goal of reflecting the contemporary usage
of the French language. The audio recordings include work meetings, family
dinners, narrations, political meetings, interviews, and goal-oriented
telephone conversations. Their duration varies from 4mn to 1h. The corpus is
annotated with part-of-speech (POS) tags, lemmas, labeled dependency trees and
sentences boundaries. Annotations include 20 possible POS tags and 12
syntactic functions. We randomly split the corpus into train/dev/test sets of
respective sizes 134,716/27,937/29,529 words; we sampled from each source so
that the various genres of speech are equally represented in each split.

###  5.2 Parsing Model

The syntax-predicting model is a transition based parser using the arc-eager
transition system, which has been extended for the joint prediction of POS
tags and parsing transitions [36]. A single classifier is trained to predict
transitions from features extracted from the current state of the analysis
(word embebdings, predicted tags, history of transitions), for stack elements
and a sliding window of size [-3;2] centered on the current word. The POS
sequence and dependency trees are greedily built from the sequence of
transitions predicted by the classifier. A dropout of 50% is applied to the
input vector, which passes through two hidden layers of size 3200 and 1600,
both with 40% of dropout and ReLU activations. The two decision layers are
linear with a softmax. Training runs for 40 epochs, using a dynamical oracle
[37], with scoring against the development set to select the best checkpoint.

###  5.3 Experiments

The first set of experiments compares input representations from the FlauBERT
variants (FBU, FT, MIX, ORAL) to non-contextual baselines. As with other
experiments, LMs are applied on uncased transcripts without punctuation, which
is not fair to the FBU model as it expects punctuation. Therefore we add a
contrastive system with periods added at the end of each utterance prior to
applying the LM (called FBU repunc), but added tokens are not passed to the
parser, they just inform other embeddings. Note that except for random
initialized embeddings, token representations are not fine-tuned during
training.

As pre-processing, we deanonymize the transcripts by replacing masked proper
name tokens with non-ambiguous names randomly chosen for each recording. In
the fasttext setting, representations are computed for unknown words from
their character n-gram factors. Token representations are computed at the
whole recording level in chunks of 512 tokens without overlap. The parser is
applied on the reference transcript and reference segmentation. We use mean
pooling for words that are split in multiple tokens by BPE.

Parsing performance is evaluated with Labeled Attachment Score (LAS), the
accuracy of predicting the governor of each word and its dependency label,
Unlabeled Attachment Score (UAS), which ignores dependency labels, and Part-
of-speech tagging accuracy (UPOS) (scoring script from CoNLL campaigns).

Results presented in Table 4 show that pre-training is valuable for syntactic
parsing in that setting and that pretraining on ASR-generated text (FT, MIX
and ORAL) leads to a substantial improvement in LAS over the text-only
FlauBERT model (FBU) even though there is no domain overlap between the TV
shows on which the LMs are trained and the data of the ORFEO corpus. As in the
previous task, we see no benefit in retraining BPEs (ORAL_NB). Performance in
the FBU repunc settings is on par with the best model, showing that absence of
punctuation is an important factor affecting performance when applying text-
only models, at least for predicting syntax.

Table 4: Syntax prediction task: metrics are Labeled Attachment Score (LAS), Unlabeled Attachment Score (UAS) and Part-of-speech tagging accuracy (UPOS). Higher is better. Model | LAS | UAS | UPOS
---|---|---|---
No pretrain | $84.92\pm 0.44$ | $88.48\pm 0.37$ | $94.51\pm 0.29$
Fasttext | $85.36\pm 0.10$ | $88.76\pm 0.05$ | $95.12\pm 0.03$
FBU | $85.55\pm 0.05$ | $89.02\pm 0.16$ | $93.36\pm 0.06$
FBU repunc | $87.48\pm 0.28$ | $90.69\pm 0.19$ | $95.03\pm 0.03$
FT | $86.81\pm 0.09$ | $90.22\pm 0.06$ | $94.99\pm 0.10$
MIX | $86.33\pm 0.15$ | $89.79\pm 0.05$ | $94.43\pm 0.29$
ORAL | $\textbf{87.65}\pm 0.11$ | $\textbf{90.92}\pm 0.10$ | $95.55\pm 0.04$
ORAL_NB | $87.54\pm 0.11$ | $90.73\pm 0.08$ | $\textbf{95.63}\pm 0.05$
Table 5: Differences in morphosyntactic performances, when parsing models of Table 4 are evaluated only on OOV words (OOV according to the automatic transcription system used to generate the text our LMs were trained on). Model |  $\Delta$LAS |  $\Delta$UAS |  $\Delta$UPOS
---|---|---|---
FBU | -11.45 | -6.82 | -14.36
MIX | -11.93 | -7.33 | -14.07
ORAL | -13.97 | -8.11 | -16.55

Finally, in Table 5 we compare the drop in parsing performance between three
models, when they are evaluated only against OOV words (according to the ASR
system). Revealing that the ORAL model variants are more affected than the
off-the-shelf FlauBERT baseline (FBU), despite their overall better
performance (as seen in Table 4). This suggests future improvements of LMs
trained on ASR-generated text if ASR output tokens were more consistent with
LM tokens, and also suggests applying open vocabulary ASR.

##  6 Conclusion and future work

We investigated spoken language modeling using a massive amount of ASR-
generated text (350,000 hours of diverse TV shows). Experiments suggest that,
for the tasks we experimented with, systems relying on speech-informed LMs
have similar and better performance than systems relying on text-only LMs. The
models resulting from this work for processing French speech are made
available to the community. Deeper analysis of these improvements, on when and
why models are better for spoken language are items that are still to be
addressed in the near future.

In this study, all texts are uncased as our ASR only generates lowercased,
unpunctuated transcripts. We believe that applying massively re-capitalisation
and restoring punctuation might be beneficial to train stronger LMs for spoken
language. We also believe that leveraging open vocabulary ASR and more diverse
speech sources might yield more versatile LMs. A more fine-grain analysis on
speech-only phenomena such as disfluencies is also needed. Finally, some of
the results obtained lead us to believe that it is important to further
evaluate the impact of BPE units for spoken language modeling, as well as the
consistency between ASR and LM tokens’ vocabulary.

##  7 Acknowledgements

Some co-authors were funded by the AISSPER project supported by the French
National Research Agency (ANR) under contract ANR-19-CE23-0004-01.

## References

  * [1] M. Koroteev, “Bert: A review of applications in natural language processing and understanding,” _arXiv_ , p. 2103, 2021.
  * [2] L. Qin, T. Xie, W. Che, and T. Liu, “A survey on spoken language understanding: Recent advances and new frontiers,” in _IJCAI_ , 2021.
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [4] J. Shin, Y. Lee, and K. Jung, “Effective sentence scoring method using bert for speech recognition,” in _Asian Conference on Machine Learning_. PMLR, 2019, pp. 1081–1093.
  * [5] C. Liu, S. Zhu, Z. Zhao, R. Cao, L. Chen, and K. Yu, “Jointly encoding word confusion network and dialogue context with bert for spoken language understanding,” _Proc. Interspeech 2020_ , pp. 871–875, 2020.
  * [6] T. M. Lai, Q. H. Tran, T. Bui, and D. Kihara, “A simple but effective bert model for dialog state tracking on resource-limited systems,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 8034–8038.
  * [7] B. Ma, H. Sun, J. Wang, Q. Qi, and J. Liao, “Extractive dialogue summarization without annotation based on distantly supervised machine reading comprehension in customer service,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 87–97, 2021.
  * [8] S. Siriwardhana, A. Reis, R. Weerasekera, and S. Nanayakkara, “Jointly fine-tuning” bert-like” self supervised models to improve multimodal speech emotion recognition,” _arXiv preprint arXiv:2008.06682_ , 2020.
  * [9] S. Ling, J. Salazar, Y. Liu, K. Kirchhoff, and A. Amazon, “Bertphone: Phonetically-aware encoder representations for utterance-level speaker and language recognition,” in _Proc. Odyssey 2020 the speaker and language recognition workshop_ , 2020, pp. 9–16.
  * [10] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [11] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [12] A. Kumar, M. Narayanan Sundararaman, and J. Vepa, “What BERT based language model learns in spoken transcripts: An empirical study,” in _BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_. ACL, Nov. 2021, pp. 322–336.
  * [13] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz _et al._ , “The kaldi speech recognition toolkit,” IEEE Signal Processing Society, Tech. Rep., 2011\.
  * [14] S. Galliano, G. Gravier, and L. Chaubard, “The ester 2 evaluation campaign for the rich transcription of french radio broadcasts,” in _Tenth Annual Conference of the International Speech Communication Association_ , 2009.
  * [15] A. Giraudel, M. Carré, V. Mapelli, J. Kahn, O. Galibert, and L. Quintard, “The repere corpus: a multimodal corpus for person recognition.” in _LREC_ , 2012, pp. 1102–1107.
  * [16] M. Goryainova, C. Grouin, S. Rosset, and I. Vasilescu, “Morpho-syntactic study of errors from speech recognition system.” in _LREC_ , vol. 14, 2014, pp. 3050–3056.
  * [17] P. Deléglise, Y. Esteve, S. Meignier, and T. Merlin, “Improvements to the lium french asr system based on cmu sphinx: what helps to significantly reduce the word error rate?” in _Tenth Annual Conference of the International Speech Communication Association_ , 2009.
  * [18] S. Meignier and T. Merlin, “Lium spkdiarization: an open source toolkit for diarization,” in _CMU SPUD Workshop_ , 2010.
  * [19] H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbé, L. Besacier, and D. Schwab, “FlauBERT: Unsupervised language model pre-training for French,” in _LREC_. Marseille, France: ELRA, May 2020, pp. 2479–2490.
  * [20] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019.
  * [21] N. Hervé, “OTMedia, the TransMedia news observatory,” in _FIAT/IFTA Media Management Seminar 2019_ , 05 2019.
  * [22] G. Tur and R. D. Mori, “Chapter 1: Spoken language understanding for human/machine interactions,” in _Spoken Language Understanding: Systems for Extracting Semantic Information from Speech_ , 2011.
  * [23] F. Morbini, K. Audhkhasi, R. Artstein, M. Van Segbroeck, K. Sagae, P. Georgiou, D. R. Traum, and S. Narayanan, “A reranking approach for recognition and classification of speech input in conversational dialogue systems,” in _2012 IEEE Spoken Language Technology Workshop (SLT)_ , 2012, pp. 49–54.
  * [24] G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, and G. Zweig, “Using recurrent neural networks for slot filling in spoken language understanding,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 3, pp. 530–539, 2015\.
  * [25] S. Ghannay, A. Caubrière, S. Mdhaffar, G. Laperrière, B. Jabaian, and Y. Estève, “Where Are We in Semantic Concept Extraction for Spoken Language Understanding?” _Lecture Notes in Computer Science_ , vol. 12997 LNAI, pp. 202–213, 2021.
  * [26] M. Dinarelli, N. Kapoor, B. Jabaian, and L. Besacier, “A data efficient end-to-end spoken language understanding architecture,” in _2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020_. IEEE, 2020, pp. 8519–8523.
  * [27] V. Pelloin, N. Camelin, A. Laurent, R. De Mori, A. Caubrière, Y. Estève, and S. Meignier, “End2end acoustic to semantic transduction,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [28] D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards End-to-end Spoken Language Understanding,” _ICASSP_ , 2018.
  * [29] H. Bonneau-Maynard, S. Rosset, C. Ayache, A. Kuhn, and D. Mostefa, “Semantic annotation of the french media dialog corpus,” in _INTERSPEECH_ , 2005.
  * [30] S. Hahn, M. Dinarelli, C. Raymond, F. Lefevre, P. Lehnen, R. De Mori, A. Moschitti, H. Ney, and G. Riccardi, “Comparing stochastic approaches to spoken language understanding in multiple languages,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 6, pp. 1569–1583, 2010.
  * [31] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460.
  * [32] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in _LREC_. Marseille, France: ELRA, May 2020, pp. 4218–4222.
  * [33] L. Martin, B. Muller, P. J. Ortiz Suárez, Y. Dupont, L. Romary, É. de la Clergerie, D. Seddah, and B. Sagot, “CamemBERT: a tasty French language model,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: ACL, Jul. 2020, pp. 7203–7219.
  * [34] C. Benzitoun, J.-M. Debaisieux, and H.-J. Deulofeu, “Le projet orféo: un corpus d’étude pour le français contemporain,” _Corpus_ , no. 15, 2016.
  * [35] A. Nasr, F. Dary, F. Bechet, and B. Favre, “Annotation syntaxique automatique de la partie orale du CÉFC,” _Langages_ , Sep. 2020.
  * [36] F. Dary and A. Nasr, “The reading machine: A versatile framework for studying incremental parsing strategies,” in _Proceedings of IWPT 2021_. Online: ACL, Aug. 2021, pp. 26–37.
  * [37] Y. Goldberg and J. Nivre, “A dynamic oracle for arc-eager dependency parsing,” in _Proceedings of COLING 2012_ , 2012, pp. 959–976.