# Harnessing $\mathcal{PT}$-symmetry in non-Hermitian stiffness-modulated
waveguides

Emanuele Riva  Department of Mechanical Engineering, Politecnico di Milano,
Milano 20156, Italy

###### Abstract

The recent progress in the context of elastic metamaterials and modulated
waveguides with digitally controllable properties has opened new pathways to
overcome the limitations dictated by Hermitian Hamiltonians in mechanics.
Among the possible implementations, non-Hermitian, $\mathcal{PT}$-symmetric
systems with balanced gain and loss have emerged as an elegant mechanism to
access novel functionalities by lifting the non-Hermitian degeneracies
(exceptional points). Motivated by this, the paper deals with a non-Hermitian
and $\mathcal{PT}$-symmetric elastic waveguide with complex stiffness-
modulation. The strength of the stiffness-modulations, tailored in the form of
a balanced gain/loss, delineates a transition from unbroken to broken
$\mathcal{PT}$-symmetric phases, where distinct Bloch-wave modes coalesce into
exceptional points. It is shown that, in the unbroken $\mathcal{PT}$-symmetric
regime, and due to the interplay between real and imaginary components of the
elasticity, the waveguide operates as a phononic filter. When the strength of
the gain/loss interactions increases, the frequency gap closes and the bulk
bands degenerate into an exceptional point, where the system operates as a
waveguide with asymmetric scattering capabilities. The paper provides a
connection between the distinct wave modes that populate the non-Hermitian
degeneracies and the directional reflection/transmission capabilities. The
asymmetric behavior is herein explained by combining the dispersion properties
of a $\mathcal{PT}$-symmetric rod, obtained through the plane wave expansion
method (PWEM), and the scattering matrix method (SMM) for a modulated slab
series-connected to semi infinite media.

Non-Hermitian, $\mathcal{PT}$-symmetry, acoustic metamaterial, phononic
crystal, asymmetric scattering.

††preprint: APS/123-QED

## Introduction

Modulated materials are key for many desired dynamic properties in the context
of phononic structures and can be regarded as building blocks for the
implementation of complex systems with unusual functionalities. Among the
relevant works, topological waveguides [1, 2, 3, 4, 5, 6, 7], cloaking [8, 9,
10, 11], rainbow devices [12, 13, 14, 15], lenses [16], should be mentioned.
These examples offer non-conventional energy transfer mechanisms and rely on
passive geometries, stiffness modulations, and symmetry breaks, which have
been sought in analogy to similar behaviors previously observed in different
realms of physics. However, the search of new functionalities, such as
nonreciprocity [17, 18, 19], digitally controllable waveguides [20, 21],
frequency conversion [22, 23], and parametric amplification [24], requires
more advanced space-time control of the material parameters, which justifies
the emergence of active times in phononics [25]. The experimental works done
in this direction have de facto opened new opportunities to engineer complex
media with exotic properties and have spurred additional research on this
matter. An emerging field, very active in other physical realms but less
explored especially in experimental mechanics, which can take advantage of
this new take on modulated systems, relies on non-Hermitian systems with
$\mathcal{PT}$-symmetry [26, 27, 28]. Despite non-Hermitian systems in many
cases are characterized by complex spectra and non-unitary evolution of the
states, there are special symmetries where a fine tuning of gain/loss
interactions yields phases with purely real spectra (unbroken
$\mathcal{PT}$-symmetric phase), and complex eigenstates (broken
$\mathcal{PT}$-symmetric phase), separated by spectral singularities, known as
exceptional points (EP) [29, 30, 31]. In other words, when non-Hermiticity
marries $\mathcal{PT}$-symmetry, the hypothesis of Hermiticity is replaced by
$\mathcal{PT}$-symmetric invariance, and additional functionalities can be
accessed. Motivated by this, the manuscript emulates asymmetric scattering in
the context of longitudinal waves in 1D solid, which is pursued through a
waveguide with complex sinusoidal elasticity that not only serves to induce
Bragg-scattering mechanisms, but also to activate balanced gain and loss
interactions. In contrast to prior works on the topic [32, 33, 34], the plane
wave expansion method (PWEM) [35, 36, 18] is used to show that there is a
synergistic interplay between the modulation strengths of the real and
imaginary parts of the stiffness profile, which coalesce the dispersion bands
and give birth to EPs. The modulation parameters are linked to the unbroken
and broken $\mathcal{PT}$-symmetric phases, to provide a quantitative analysis
of the phase transition. It is shown that, despite sharing the same frequency,
left and right propagating wave-modes are inherently different due to the
complex interplay between real and imaginary parts of the stiffness. The
asymmetric wave modes are combined to the scattering matrix method (SMM) [22]
to justify the emergence of asymmetric reflections induced by the modulation,
and to qualify the scattering properties of the PT-symmetric slab when series-
connected to two semi-infinite passive media.

##  I Dispersion analysis

Figure 1: (a-b) Schematic of the modulated slab and concept. The real and
imaginary components of the complex modulation are projected onto
perpendicular planes and are represented by colored surfaces. (a) A wave
incident from the left is reflected by the slab with amplification and
transmitted with unitary gain. (b) A wave incident from the right is
transmitted unaltered without reflection. (c-e) Dispersion relation (c)
$\mu\left(\Omega\right)$ for modulation parameters $\alpha=0.5$ and
$\gamma=0.25$, (d) $\alpha=\gamma=0.5$, and (e) $\Omega\left(\mu\right)$ for
$\alpha=0.5$ and $\gamma=0.75$. (f-g) Wavenumbers evaluated at constant
frequency ($\Omega=0.5$) upon varying $\gamma$. There is a complex wavenumber
split at $\gamma=0.5$ for negative variations of $\gamma$. (h-i) Frequency
transition evaluated at constant wavenumber ($\mu/\pi=1$) upon varying
$\gamma$. There is a complex frequency split at $\gamma=0.5$ for positive
variations of $\gamma$.

Consider the waveguide illustrated in Figure 1(a-b). As mentioned, aim of the
paper is to discuss the directional scattering capabilities of the elastic
slab and to provide a clear connection with the dispersion properties. The
concept is elucidated in the figure: a wave impinging on the modulation from
the left is reflected and, eventually, amplified. While a wave impinging from
the opposite side propagates with unitary amplitude and with no reflection.
The study is initially focused on the dispersion properties of the modulated
part, represented with the colored surfaces, which embodies a complex
elasticity
$E\left(x\right)=E_{0}\left[1+\alpha\cos{\left(\kappa_{m}x\right)}\right]+\rm{i}E_{0}\gamma\sin{\left(\kappa_{m}x\right)}$
having modulation wavenumber $\kappa_{m}=2\pi/\lambda_{m}$ and wavelength
$\lambda_{m}$. $\alpha$ and $\gamma$ are the modulation strengths. The former
is assumed fixed ($\alpha=0.5$) without loss of generality, while the role of
$\gamma$ is object of discussion. Note now that the function is even in its
real part and the imaginary part is odd. As such, $E\left(x\right)$ is
invariant under the combination of reflection $x\rightarrow-x$ and time
reversal $\rm{i}\rightarrow-\rm{i}$ operators. Longitudinal wave motion
$u\left(x,t\right)$ in such a system is governed by the following
elastodynamic equation:

$\left(E\left(x\right)u_{,x}\right)_{,x}=\left(\rho u_{,t}\right)_{,t}$ (1)

where $(\cdot)_{,\xi}$ stands for $\partial(\cdot)/\partial\xi$ and, without
loss of generality, the material density $\rho$ is assumed constant. Due to
the periodic nature of the underlying medium, i.e.
$E\left(x\right)=E\left(x+\lambda_{m}\right)$, the dispersion analysis is
encouraged and hereafter accomplished by following the plane wave expansion
method (PWEM) [18, 35]. As such, the elasticity is expanded in terms of $2N+1$
complex exponential functions
$E\left(x\right)=\sum_{n=-N}^{N}\hat{E}_{n}\rm{e}^{-\rm{i}n\kappa_{m}x}$,
where $\hat{E}_{n}$ are the Fourier coefficients and $N=10$ is found
sufficient to describe the dispersion properties with good approximation.
Ansatz $u\left(x,t\right)$ are in the same complex exponential form
$u\left(x,t\right)={\rm{u}}\left(x\right)\rm{e}^{\rm{i}\left(\omega t-\kappa
x\right)}$, with
${\rm{u}}\left(x\right)=\sum_{p=-P}^{P}\hat{u}_{p}\rm{e}^{-\rm{i}p\kappa_{m}x}$.
These are combined to Eq. 1 and the orthogonality of the exponential functions
is exploited to get to the dispersion relation in the form of a quadratic
eigenvalue problems for $\omega\left(\kappa\right)$ and
$\kappa\left(\omega\right)$:

$\sum_{p=-P}^{P}\hat{E}_{s-p}\left(\kappa+p\kappa_{m}\right)\left(\kappa+s\kappa_{m}\right)\hat{u}_{p}=\omega^{2}\hat{u}_{s}$
(2)

Now, the dispersion relation in dimensionless form is illustrated in Figures
1(b-d) for fixed $\alpha=0.5$ and for three distinct $\gamma$ values (i.e.
$\gamma=0.25$, $\gamma=0.5$, and $\gamma=0.75$), where the dimensionless
frequency and wavenumber are $\Omega=\omega\lambda_{m}/c_{0}$ and
$\mu=\kappa\lambda_{m}$ with $c_{0}=\sqrt{E_{0}/\rho}$. Interestingly, the
configuration with $\gamma<\alpha$ (see Figure 1(c)) displays real frequencies
and complex wavenumbers, induced by bandgap formation mechanisms at the
lattice points. Such a frequency gap is reduced in width until it coalesces to
a non-Hermitian degeneracy in case $\gamma=\alpha$. Under such a condition,
displayed in Figure 1(d), the frequency-wavenumber pairs are purely real. When
the modulation strength $\gamma$ is further increased to $\gamma>\alpha$,
there is a complex frequency split and a $\kappa$-gap is induced in the
neighborhood of the lattice points, see Figure 1(e). To gain insight on the
physics of phase transition, Eq. 2 is truncated to the first term, i.e. $N=1$
and the analytical form of the Fourier coefficients is used. The eigenvalue
problem is conveniently rewritten in matrix form
$H\left(\kappa\right)\bm{u}=\rho\omega^{2}I\bm{u}$, where
$\bm{u}=\left[u_{-1},u_{0},u_{1}\right]^{T}$ is the eigenvector that
accommodates the $\hat{u}_{p}$ Bloch-wave coefficients of the expansion, and
the matrix $H$ reads:

$\begin{split}H=&\;E_{0}\begin{bmatrix}\left(\kappa-\kappa_{m}\right)^{2}&\displaystyle\frac{\kappa}{2}\alpha\left(\kappa-\kappa_{m}\right)&0\\\\[5.0pt]
\displaystyle\frac{\kappa}{2}\alpha\left(\kappa-\kappa_{m}\right)&\kappa^{2}&\displaystyle\frac{\kappa}{2}\alpha\left(\kappa+\kappa_{m}\right)\\\\[5.0pt]
0&\displaystyle\frac{\kappa}{2}\alpha\left(\kappa+\kappa_{m}\right)&\left(\kappa+\kappa_{m}\right)^{2}\end{bmatrix}+\\\\[5.0pt]
&\;E_{0}\begin{bmatrix}0&\displaystyle\frac{\kappa}{2}\gamma\left(\kappa-\kappa_{m}\right)&0\\\\[5.0pt]
-\displaystyle\frac{\kappa}{2}\gamma\left(\kappa-\kappa_{m}\right)&0&\displaystyle\frac{\kappa}{2}\gamma\left(\kappa+\kappa_{m}\right)\\\\[5.0pt]
0&-\displaystyle\frac{\kappa}{2}\gamma\left(\kappa+\kappa_{m}\right)&0\end{bmatrix}\end{split}$
(3)

here, the contributions due to $\alpha$ and $\gamma$ are separated. It is
observed that $\alpha$ is associated with the Hermitian part of the matrix
$H$, while $\gamma$ induces non-Hermiticity. Also, due to the aforementioned
considerations for $E\left(x\right)$, $H$ commutes with the $\mathcal{PT}$
operator, i.e., $\left(H,\mathcal{PT}\right)=0$, which suggests the emergence
of a phase transition between unbroken and broken $\mathcal{PT}$-symmetric
phases. This is further confirmed by spanning the parameter space $\gamma$ for
the fixed $\alpha$ value, as shown in figure 1(f-i) and object of discussion.
Figure 1(f-g) illustrate a transition for a fixed frequency $\Omega=0.5$, in
which complex-conjugate wavenumber pairs split for negative variations of
$\gamma$. Hence, the dispersion relation is sought in the form
$\mu\left(\Omega\right)$ to show that $\gamma<\alpha$ yields the formation of
a frequency gap, while in case $\gamma>\alpha$ the wavenumbers are purely real
and the dispersion is to be sought in the form $\Omega\left(\mu\right)$. In
contrast, the behavior for $\mu/\pi=1$, shown in figure 1(h-i), is
characterized by frequencies initially real which coalesce if $\gamma=\alpha$
and, due to a further increase of $\gamma$ split into complex conjugate pairs.
To conclude the dispersion analysis of the slab, the discussion is now focused
on the behavior at the non-Hermitian degeneracy, which is preparatory to the
following. To that end, $\gamma=\alpha$ is imposed into Eq. 2 with $N=1$
harmonics and the characteristic equation relative to the eigenvalue problem
in dimensionless form $\left(\Omega^{2}I-H\right)\bm{u}=0$ is:

$\begin{split}det\;\left(\Omega^{2}I-H\right)=\left[\Omega^{2}-\left(\frac{\mu^{2}}{4\pi^{2}}-\frac{\mu}{\pi}+1\right)\right]\left[\Omega^{2}+\right.\\\\[5.0pt]
-\left.\frac{\mu^{2}}{4\pi^{2}}\right]\left[\Omega^{2}-\left(\frac{\mu^{2}}{4\pi^{2}}+\frac{\mu}{\pi}+1\right)\right]\end{split}$
(4)

by looking for coincident solutions of Eq. 4 it is here demonstrated that,
independently of frequency $\Omega$, the eigenvalues coalesce in the origin
$\mu=0$ and, more interestingly, for $\mu=\pm\pi$. Now, plugging $\mu=\pi$
(i.e. forward-traveling waves) into Eq. 2 gives the frequencies of the non-
Hermitian degeneracy $\Omega=\pm\frac{1}{2}$ both with algebraic multiplicity
2 and $\Omega=\pm\frac{3}{2}$ with algebraic multiplicity 1. This latter is
not relevant for the scope of the present paper and corresponds to a higher
frequency crossing. Notice that $\mu=-\pi$ also gives $\Omega=\pm\frac{1}{2}$
and $\Omega=\pm\frac{3}{2}$. The study is focused on counter-propagating waves
at EPs, i.e. when $\Omega=\frac{1}{2}$ and $\mu=\pm\pi$ and the matrix $H$ is
rank deficient, since here the geometric multiplicity is 1. Interestingly,
even though opposite wavenumbers share the same frequency, the eigenvectors
are different, which suggests that directional wave motion supported by the
slab:

$\begin{split}\bm{u}^{+}&=\begin{pmatrix}1\\\\[5.0pt] 0\\\\[5.0pt]
0\end{pmatrix}\;\;\;for\;\;\mu^{+}=\pi;\\\\[5.0pt]
\bm{u}^{-}&=\begin{pmatrix}-\displaystyle\frac{3}{8}\alpha\\\\[7.0pt]
1\\\\[3.0pt] 0\end{pmatrix}\;\;\;for\;\;\mu^{-}=-\pi;\end{split}$ (5)

This is key for the following discussion.

##  II Scattering analysis

Figure 2: (a) Schematic of the modulated slab (colored region) connected to
the left and to the right with homogeneous semi-infinite elements (red
domains). In the figure, the incident and scattered waves are represented with
arrows. Scattering coefficients upon varying frequency $\Omega$ for (b)
$\alpha=0.5$ and $\gamma=0$, and for (c) $\alpha=\gamma=0.5$.

Consider now a slab made of $Q=10$ unit cells, which is connected to two semi-
infinite and homogeneous beam elements with stiffness
$E_{0}\left(1+\alpha\right)$,
$c_{0,\alpha}=\sqrt{E_{0}\left(1+\alpha\right)/\rho}$, and length
$L_{e}=20\lambda_{m}$. Note that an integer number of unit cells is required
for the modulated rod to embody balanced gain/loss interactions. The reference
system is arbitrarily set according to the schematic in Figure 2(a) and, in
the following, a number of incident and scattered waves are considered. That
is, a plane wave $u_{i}^{+}$ that impinges on the modulated slab from the
left, and a plane wave $u_{i}^{-}$ that impinges from the right are input of
the problem:

$u_{i}^{+}=A_{0}{\rm e}^{{\rm
i}\displaystyle\omega\left(-\displaystyle\frac{x}{c_{0,\alpha}}+t\right)}\hskip
14.22636ptu_{i}^{-}=G_{0}{\rm e}^{{\rm
i}\displaystyle\omega\left(\displaystyle\frac{x}{c_{0,\alpha}}+t\right)}$ (6)

due to the interface conditions between the homogeneous and the modulated
media, a pair of scattered waves $u_{s}^{+}$ and $u_{s}^{-}$ populate the
region prior to and after the slab:

$u_{s}^{+}=F_{0}{\rm e}^{{\rm
i}\displaystyle\omega\left(-\displaystyle\frac{x}{c_{0,\alpha}}+t\right)}\hskip
14.22636ptu_{s}^{-}=B_{0}{\rm e}^{{\rm
i}\displaystyle\omega\left(\displaystyle\frac{x}{c_{0,\alpha}}+t\right)}$ (7)

finally, the modulated medium is populated by left and right propagating terms
$u_{m}^{+}$ and $u_{m}^{-}$:

$\begin{split}u_{m}^{+}=C_{0}\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}{\rm
e}^{\displaystyle{\rm-i}\left(\kappa_{0}^{+}+p\kappa_{m}\right)x}{\rm
e}^{\displaystyle{\rm i}\omega t}\\\\[5.0pt]
u_{m}^{-}=D_{0}\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}{\rm
e}^{\displaystyle{\rm-i}\left(\kappa_{0}^{-}+p\kappa_{m}\right)x}{\rm
e}^{\displaystyle{\rm i}\omega t}\end{split}$ (8)

where the subscript $0$ stands for central branch of the dispersion in the
form $\kappa\left(\omega\right)$ and the superscripts $\pm$ stands for right
and left right traveling waves with wavenumbers $\kappa_{0}^{\pm}$, which are
identified and separated by way of the group velocity
$c_{g}=\omega/\kappa_{0}$ (see the supplementary material for more details on
this procedure [37]). The analysis is continued by enforcing compatibility of
the displacements and force equilibrium at the interface coordinates $x_{1}$
and $x_{2}$. These conditions are hereafter employed to find a relationship
between the amplitude coefficients defined in Eq. 6-8. For the boundary at
$x=x_{1}$:

$\begin{split}&u_{i}^{+}\left(x_{1}\right)+u_{s}^{-}\left(x_{1}\right)=u_{m}^{+}\left(x_{1}\right)+u_{m}^{-}\left(x_{1}\right)\\\\[5.0pt]
&E_{0}\left(1+\alpha\right)\left(u_{i}^{+}\left(x_{1}\right)+u_{s}^{-}\left(x_{1}\right)\right)_{,x}=\\\
&\hskip
71.13188ptE\left(x_{1}\right)\left(u_{m}^{+}\left(x_{1}\right)+u_{m}^{-}\left(x_{1}\right)\right)_{,x}\end{split}$
(9)

and for $x=x_{2}$:

$\begin{split}&u_{m}^{+}\left(x_{2}\right)+u_{m}^{-}\left(x_{2}\right)=u_{i}^{-}\left(x_{2}\right)+u_{s}^{+}\left(x_{2}\right)\\\\[5.0pt]
&E\left(x_{2}\right)\left(u_{m}^{+}\left(x_{2}\right)+u_{m}^{-}\left(x_{2}\right)\right)_{,x}=\\\
&\hskip
71.13188ptE_{0}\left(1+\alpha\right)\left(u_{i}^{-}\left(x_{2}\right)+u_{s}^{+}\left(x_{2}\right)\right)_{,x}\end{split}$
(10)

notice that the wave modes, reported into Eq. 5, are asymmetric at the
exceptional points and take part in the scattering process described in Eq.
9-10. It is therefore expected that asymmetric scattering emerges for
frequencies where the wave modes are asymmetric. The scattering matrix $S$,
which links the incident and the scattered wave amplitudes, is obtained after
a few manipulations of Eq. 9-10 (see the supplementary material [37] for
additional details on the derivation):

$\begin{pmatrix}B_{0}\\\\[5.0pt]
F_{0}\end{pmatrix}=\begin{bmatrix}R_{11}&T_{21}\\\\[5.0pt]
T_{12}&R_{22}\end{bmatrix}\begin{pmatrix}A_{0}\\\\[5.0pt]
G_{0}\end{pmatrix}=S\begin{pmatrix}A_{0}\\\\[5.0pt] G_{0}\end{pmatrix}$ (11)

The coefficients $R_{11}$, $T_{12}$, $T_{21}$, and $R_{22}$ are displayed in
figures 2(b,c) for $\gamma=0$ and $\gamma=\alpha$, respectively.

Figure 3: Displacement field $|u\left(x,\Omega\right)|$ upon varying frequency
$\Omega$ and with $\gamma=0$ for a wave impinging (a) from the left and (b)
from the right. (c,d) Same displacement fields for $\Omega=0.5$. (e)
Displacement field $|u\left(x,\Omega\right)|$ upon varying frequency $\Omega$
and with $\gamma=0.5$ for a wave impinging (e) from the left and (f) from the
right. The response is clearly asymmetric. (g-h) Same response but limited to
$\Omega=0.5$.

Some considerations follow: (i) when the gain/loss interactions are not
active, the reflections are almost unitary and the transmissions drop to zero.
Furthermore, $T_{12}=T_{21}$, $R_{11}=R_{22}$ and, hence, the medium is
reciprocal. (ii) It is observed that, in correspondence of the exceptional
points, the reflection from the right side of the slab drops to zero
($R_{22}=0$), while the transmission is unitary ($T_{21}=1$). In contrast,
wave propagation from the left to the right exhibits reflection with
amplification ($R_{11}>>1$), which is accompanied by unitary transmission
($T_{12}=1$) as per right-to-left propagating waves and similarly to prior
works on this matter [34, 32]. This confirms the asymmetric scattering
capabilities of the slab that, as mentioned, are justified by the emergence of
asymmetric wave modes at the non-Hermitian degeneracy. The analysis of the
scattering matrix is concluded in the supplementary material [37], where the
scattering coefficients are reported upon varying $\gamma$.
Now, given the relationship between the amplitude coefficients, the
transmitted and reflected wave fields generated by left and right impinging
waves are hereafter studied, with emphasis on the behavior at the EP. It is
initially assumed that a wave with unitary amplitude propagates from left to
right, i.e. $A_{0}=1$ and $G_{0}=0$. As such, the scattered wave amplitudes
$B_{0}$ and $F_{0}$ are due to Eq. 11 which, when combined to the waves in Eq.
6 and Eq. 7, give the displacement field outside the modulated slab. $C_{0}$
and $D_{0}$ are instead evaluated through Eq. 9 and are employed to
reconstruct the response between $x_{1}$ and $x_{2}$. The same procedure is
applied to right impinging waves, i.e. $A_{0}=0$ and $G_{0}=1$. The response
of the slab for the two excitation conditions is illustrated in Figure 3(a-d)
in terms of wave amplitude $|u\left(x,\Omega\right)|$ and, initially, for
$\gamma=0$. Figure 3(a) is relative to a wave that propagates from the left to
the right and, to ease visualization, the response for $\Omega=0.5$ is
reported in Figure 3(c). Vice-versa Figure 3(b,d) illustrate the behavior for
a right-impinging wave.

Figure 4: Schematic of the interface between a semi-infinite homogeneous
domain (red) and a modulated semi-infinite medium (colored part). Modulation
is on the (a) left or on the (b) right side of the interface. The incident and
scattered waves are displayed with arrows. (b) Scattering coefficients of the
interface for (c) $\gamma=0$ and (d) $\gamma=\alpha=0.5$.

It is straightforward to conclude that, since the diagrams for the mirror-
symmetric inputs are mirror-symmetric about the center of the slab, the medium
is reciprocal. In addition to that, the slab exhibits exponential attenuation
for bandgap frequencies, regardless the direction of the impinging wave.
Consider now the case $\gamma=\alpha$, which is illustrated in Figure 3(e-h).
When a plane wave with frequency in the neighborhood of $\Omega=0.5$ impinges
from the left, there is reflection with a strong amplification factor, see
Figure 3(e,g). Such a strong selective amplification is accompanied by unitary
transmission. In contrast, when a wave impinges from the right, see Figure
3(f,h) there is no reflection and the wave is transmitted unaltered. This
blueprint of nonreciprocity is justified by the directional wave modes
$\hat{u}_{0,p}^{\pm}$ which take part on the scattering process described in
Eq. 11 and yield directional scattering coefficients. Similar results are
obtained through a modulated slab with a single unit of dimension
$\lambda_{m}$, and reported in the supplementary material [37], where the
benefits of periodic configurations are also discussed.
The paper is concluded with the analysis of the two interfaces, to provide
additional insight on the role of the wave modes on the directional scattering
capabilities. Consider the lone interfaces at $x_{1}$ and $x_{2}$, which now
consists of two semi-infinite media and, for simplicity, a new reference
system is added in correspondence of the interface, see the schematic in
Figure 4(a-b). The homogeneous medium is located at the left (right) of the
interface and rigidly connected to the modulated slab. Since the media are now
semi-infinite, a left or right incident waves of amplitude $A_{0}$ or $G_{0}$
are capable of exciting either the $u_{m}^{+}$ or the $u_{m}^{-}$ wave modes,
and the reflections are dictated by new equilibrium conditions. For the left
interface:

$\begin{split}&u_{i}^{+}\left(0\right)+u_{s}^{-}\left(0\right)=u_{m}^{+}\left(0\right)\\\\[5.0pt]
&E_{0}\left(1+\alpha\right)\left(u_{i}^{+}\left(0\right)+u_{s}^{-}\left(0\right)\right)_{,x}=E\left(0\right)\left(u_{m}^{+}\left(0\right)\right)_{,x}\end{split}$
(12)

for the right interface:

$\begin{split}&u_{m}^{-}\left(0\right)=u_{i}^{-}\left(0\right)+u_{s}^{+}\left(0\right)\\\\[5.0pt]
&E\left(0\right)\left(u_{m}^{+}\left(0\right)\right)_{,x}=\\\ &\hskip
71.13188ptE_{0}\left(1+\alpha\right)\left(u_{i}^{-}\left(0\right)+u_{s}^{+}\left(0\right)\right)_{,x}\end{split}$
(13)

These equations are combined to get to an expression for the reflected and
transmitted wave amplitudes. For the left interface:

$\begin{split}T_{L}=\frac{C_{0}}{A_{0}}=\frac{2}{\displaystyle\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}+\displaystyle\frac{E_{0}(0)}{E_{0}\left(1+\alpha\right)}\frac{c_{0,\alpha}}{\omega}\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}\left(\kappa_{0}^{+}+p\kappa_{m}\right)}\\\\[5.0pt]
R_{L}=\frac{B_{0}}{A_{0}}=\frac{\displaystyle\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}-\displaystyle\frac{E_{0}(0)}{E_{0}\left(1+\alpha\right)}\frac{c_{0,\alpha}}{\omega}\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}\left(\kappa_{0}^{+}+p\kappa_{m}\right)}{\displaystyle\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}+\displaystyle\frac{E_{0}(0)}{E_{0}\left(1+\alpha\right)}\frac{c_{0,\alpha}}{\omega}\sum_{p=-P}^{P}\hat{u}_{p,0}^{+}\left(\kappa_{0}^{+}+p\kappa_{m}\right)}\end{split}$
(14)

and for a right interface, when a wave impinges from the right:

$\begin{split}T_{R}=\frac{D_{0}}{G_{0}}=\frac{2}{\displaystyle\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}-\displaystyle\frac{E_{0}(0)}{E_{0}\left(1+\alpha\right)}\frac{c_{0,\alpha}}{\omega}\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}\left(\kappa_{0}^{-}+p\kappa_{m}\right)}\\\\[5.0pt]
R_{R}=\frac{F_{0}}{G_{0}}=\frac{\displaystyle\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}+\displaystyle\frac{E_{0}(0)}{E_{0}\left(1+\alpha\right)}\frac{c_{0,\alpha}}{\omega}\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}\left(\kappa_{0}^{-}+p\kappa_{m}\right)}{\displaystyle\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}-\displaystyle\frac{E_{0}(0)}{E_{0}\left(1+\alpha\right)}\frac{c_{0,\alpha}}{\omega}\sum_{p=-P}^{P}\hat{u}_{p,0}^{-}\left(\kappa_{0}^{-}+p\kappa_{m}\right)}\end{split}$
(15)

it is straightforward to conclude that waves impinging from different
directions (left $(\cdot)_{L}$ or right $(\cdot)_{R}$), capable of exciting
different wave modes $\hat{u}_{p,0}^{+}\neq\hat{u}_{p,0}^{-}$, generate
different reflection ($R_{L,R}$) and transmission ($T_{L,R}$) coefficients at
the interface. This is evident in Eq. 14-15 and illustrated in Figure 4(c-d)
where $T_{L}$, $R_{L}$, $T_{R}$, and $R_{R}$ are displayed upon varying
frequency. The results for $\gamma=0$ and $\gamma=\alpha$ are shown in Figure
4(c) and Figure 4(d), respectively. While for completeness, different $\gamma$
values are provided in the supplementary material [37]. As expected, the
curves for $\gamma=0$ are identical for waves impinging from the left or from
the right. In contrast, complex modulations with $\gamma=\alpha$ display a
sharp peak if waves impinge from the left. Waves impinging from the right are
not reflected at all, which further confirms the asymmetric scattering
capabilities of the modulated slab.

## Conclusions

In this manuscript, the plane wave expansion method (PWEM) and the scattering
matrix method (SMM) are employed to study asymmetric scattering of
longitudinal waves in a $\mathcal{PT}$-symmetric modulated beam with complex
elasticity. It is shown that such a system support either frequency gaps and
$\kappa$-gaps depending on the amount of complex stiffness modulation. The
transition among them is characterized by dispersion branches that coalesce
into an exceptional point, which separates the two phases. This condition
triggers a rich scattering behavior, which is studied in the paper with
emphasis on the role of the wave modes. In analogy with prior works, it is
shown that waves impinging from opposite directions can be either transmitted
with no reflection (right to left), or reflected with strong amplification and
transmitted with unitary amplitude (left to right). The directional behavior
of the slab is justified by the asymmetric wave modes supported by the slab,
which are selectively excited at frequencies in the neighborhood of the EP by
left or right impinging waves. This work can be extended to elastic waveguides
supporting different wave modes (e.g. torsional, flexural) and the
observations made in the paper are key for the design of elastic systems with
asymmetric capabilities that can be functional for signal processing,
vibration control in structures, and selective amplification.

###### Acknowledgements.

E.R. wishes to thank M. Rosa for useful discussion.

## References

  * Miniaci  _et al._ [2018] M. Miniaci, R. Pal, B. Morvan, and M. Ruzzene, Experimental observation of topologically protected helical edge modes in patterned elastic plates, Physical Review X 8, 031074 (2018).
  * Süsstrunk and Huber [2015] R. Süsstrunk and S. D. Huber, Observation of phononic helical edge states in a mechanical topological insulator, Science 349, 47 (2015).
  * Vila  _et al._ [2017] J. Vila, R. K. Pal, and M. Ruzzene, Observation of topological valley modes in an elastic hexagonal lattice, Physical Review B 96, 134307 (2017).
  * Riva  _et al._ [2018] E. Riva, D. Quadrelli, G. Cazzulani, and F. Braghin, Tunable in-plane topologically protected edge waves in continuum kagome lattices, Journal of Applied Physics 124, 164903 (2018).
  * Liu and Semperlotti [2018] T.-W. Liu and F. Semperlotti, Tunable acoustic valley–hall edge states in reconfigurable phononic elastic waveguides, Physical Review Applied 9, 014001 (2018).
  * Rosa  _et al._ [2019] M. I. Rosa, R. K. Pal, J. R. Arruda, and M. Ruzzene, Edge states and topological pumping in spatially modulated elastic lattices, Physical review letters 123, 034301 (2019).
  * Riva  _et al._ [2020] E. Riva, M. I. Rosa, and M. Ruzzene, Edge states and topological pumping in stiffness-modulated elastic plates, Physical Review B 101, 094307 (2020).
  * Norris [2008] A. N. Norris, Acoustic cloaking theory, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 464, 2411 (2008).
  * Quadrelli  _et al._ [2021a] D. E. Quadrelli, R. Craster, M. Kadic, and F. Braghin, Elastic wave near-cloaking, Extreme Mechanics Letters 44, 101262 (2021a).
  * Quadrelli  _et al._ [2021b] D. E. Quadrelli, G. Cazzulani, S. La Riviera, and F. Braghin, Acoustic scattering reduction of elliptical targets via pentamode near-cloaking based on transformation acoustics in elliptic coordinates, Journal of Sound and Vibration 512, 116396 (2021b).
  * Chen  _et al._ [2015] Y. Chen, X. Liu, and G. Hu, Latticed pentamode acoustic cloak, Scientific reports 5, 1 (2015).
  * De Ponti  _et al._ [2021] J. M. De Ponti, L. Iorio, E. Riva, R. Ardito, F. Braghin, and A. Corigliano, Selective mode conversion and rainbow trapping via graded elastic waveguides, Phys. Rev. Applied 16, 034028 (2021).
  * De Ponti  _et al._ [2020] J. M. De Ponti, A. Colombi, E. Riva, R. Ardito, F. Braghin, A. Corigliano, and R. V. Craster, Experimental investigation of amplification, via a mechanical delay-line, in a rainbow-based metamaterial for energy harvesting, Applied Physics Letters 117, 143902 (2020).
  * Alshaqaq and Erturk [2020] M. Alshaqaq and A. Erturk, Graded multifunctional piezoelectric metastructures for wideband vibration attenuation and energy harvesting, Smart Materials and Structures 30, 015029 (2020).
  * Chaplain  _et al._ [2020] G. J. Chaplain, J. M. De Ponti, G. Aguzzi, A. Colombi, and R. V. Craster, Topological rainbow trapping for elastic energy harvesting in graded su-schrieffer-heeger systems, Physical Review Applied 14, 054035 (2020).
  * Tol  _et al._ [2017] S. Tol, F. Degertekin, and A. Erturk, Phononic crystal luneburg lens for omnidirectional elastic wave focusing and energy harvesting, Applied Physics Letters 111, 013503 (2017).
  * Marconi  _et al._ [2020] J. Marconi, E. Riva, M. Di Ronco, G. Cazzulani, F. Braghin, and M. Ruzzene, Experimental observation of nonreciprocal band gaps in a space-time-modulated beam using a shunted piezoelectric array, Physical Review Applied 13, 031001 (2020).
  * Trainiti and Ruzzene [2016] G. Trainiti and M. Ruzzene, Non-reciprocal elastic wave propagation in spatiotemporal periodic structures, New Journal of Physics 18, 083047 (2016).
  * Attarzadeh  _et al._ [2019] M. Attarzadeh, S. Maleki, J. Crassidis, and M. Nouh, Non-reciprocal wave phenomena in energy self-reliant gyric metamaterials, The Journal of the Acoustical Society of America 146, 789 (2019).
  * Xia  _et al._ [2021] Y. Xia, E. Riva, M. I. Rosa, G. Cazzulani, A. Erturk, F. Braghin, and M. Ruzzene, Experimental observation of temporal pumping in electromechanical waveguides, Physical Review Letters 126, 095501 (2021).
  * Riva  _et al._ [2021] E. Riva, G. Castaldini, and F. Braghin, Adiabatic edge-to-edge transformations in time-modulated elastic lattices and non-hermitian shortcuts, New Journal of Physics 23, 093008 (2021).
  * Yi  _et al._ [2018] K. Yi, S. Karkar, and M. Collet, One-way energy insulation using time-space modulated structures, Journal of Sound and Vibration 429, 162 (2018).
  * Yi  _et al._ [2017] K. Yi, M. Collet, and S. Karkar, Frequency conversion induced by time-space modulated media, Physical review B 96, 104110 (2017).
  * Trainiti  _et al._ [2019] G. Trainiti, Y. Xia, J. Marconi, G. Cazzulani, A. Erturk, and M. Ruzzene, Time-periodic stiffness modulation in elastic metamaterials for selective wave filtering: Theory and experiment, Physical review letters 122, 124301 (2019).
  * Zangeneh-Nejad and Fleury [2019] F. Zangeneh-Nejad and R. Fleury, Active times for acoustic metamaterials, Reviews in Physics 4, 100031 (2019).
  * Longhi [2018] S. Longhi, Parity-time symmetry meets photonics: A new twist in non-hermitian optics, EPL (Europhysics Letters) 120, 64001 (2018).
  * Feng  _et al._ [2017] L. Feng, R. El-Ganainy, and L. Ge, Non-hermitian photonics based on parity–time symmetry, Nature Photonics 11, 752 (2017).
  * Longhi [2009] S. Longhi, Bloch oscillations in complex crystals with p t symmetry, Physical review letters 103, 123601 (2009).
  * Bender and Boettcher [1998] C. M. Bender and S. Boettcher, Real spectra in non-hermitian hamiltonians having p t symmetry, Physical Review Letters 80, 5243 (1998).
  * Bender [2019] C. M. Bender,  _PT symmetry: In quantum and classical physics_ (World Scientific, 2019).
  * Longhi [2010] S. Longhi, Spectral singularities and bragg scattering in complex crystals, Physical Review A 81, 022102 (2010).
  * Fleury  _et al._ [2015] R. Fleury, D. Sounas, and A. Alu, An invisible acoustic sensor based on parity-time symmetry, Nature communications 6, 1 (2015).
  * Achilleos  _et al._ [2017] V. Achilleos, Y. Aurégan, and V. Pagneux, Scattering by finite periodic $\mathcal{P}\mathcal{T}$-symmetric structures, Phys. Rev. Lett. 119, 243904 (2017).
  * Wu  _et al._ [2019] Q. Wu, Y. Chen, and G. Huang, Asymmetric scattering of flexural waves in a parity-time symmetric metamaterial beam, The Journal of the Acoustical Society of America 146, 850 (2019).
  * Riva  _et al._ [2019] E. Riva, J. Marconi, G. Cazzulani, and F. Braghin, Generalized plane wave expansion method for non-reciprocal discretely modulated waveguides, Journal of Sound and Vibration 449, 172 (2019).
  * Enrico Quadrelli  _et al._ [2020] D. Enrico Quadrelli, E. Riva, G. Cazzulani, and F. Braghin, Omindirectional non-reciprocity via 2d modulated radial sonic crystals, Crystals 10, 624 (2020).
  * [37] See supplemental material at xxxx for more details on the simulation procedures and for additional results., .


Marjorie |  Spatial Language for Human-Robot Dialogs[70] |  Speech, Gesture |  Y |  Personal Digital Assistant, wireless microphone, interacts with a robot via a touch screen and speech. |  X |  No Empirical Data |  The robot will provide detailed spatial descriptions. |  Does not facilitate commands concerning objects in the environment, e.g., “Move forward until the pillar is behind you.”
Markku |  Interface Based on Speech, Gestures, and Haptic Feedback[71] |  Speech, Gestures, Haptic |  Y |  PC, the Athlon X2 3800, Nokia N95, 40” high definition television, and a wireless connector. The application was developed using # and ran under Windows XP. |  X |  Perceived quality of the speech input surpassed the upper limit of user expectations |  Accepts speech input and the response was good. |  The interface would work for restricted input.
Stéphane |  Audio-Visual Speech Modeling [73] |  Audio-Visual, Speech |  Y |  M2VTS audio-visual database |  X |  95.80% |  The database used is extensive with samples collected from 37 different users |  The experiment deals with only Speech input. No other input from the user is accepted.
Chieh |  Encoder-Camera-Ground Penetrating Radar Tri-Sensor Mapping[54] |  Images and GPR/encoder data which are spatially evenly-spaced |  Y |  Camera, a GPR module which includes control unit, wheel encoder, and GPR antenna. |  Y |  98% |  Developed a encoder-camera-GPR tri-sensor transportation infrastructure inspection sensing suite. |  Need to improve further speed and accuracy of the proposed algorithm.
Tom |  Road Marking Segmentation via Weakly-Supervised Annotations[55] |  Camera captures Images with annotated images. |  Y |  Camera captured images with annotated images in a weakly-supervised way. Experiment performed on Oxford RobotCar dataset. |  Y |  75.04% |  It has inexpensive manual labelling by exploiting sensor modalities. Useful in creating maps for autonomous vehicles. |  The experiment does not include semantic classification of the road markings to retrieve the rules of the road.
Shan |  Vision and Tactile Sensing for Cloth Texture Recognition[56] |  Tactile images and vision |  Y |  GelSight sensor used for capturing camera images and tactile data. Deep Maximum Covariance Analysis (DMCA) algorithm is implemented |  Y |  90% |  Calculated perception performance of either vision or tactile sensing. |  Temporal information is not included during the experiments.
Tian |  Prediction with Spiking Neural Networks for Human-Robot Collaboration[56] |  Gesture, EEG Signals, Speech |  Y |  MYO armband, Emotiv EPOC, Microsoft Kinect |  Y |  88% |  The experiment exploits unique implementation of Myo armband, Kinect and EPOC Emotiv devices in surgery. |  Does not include contextual information to improve early prediction capability e.g., the current status of task progress
Zhe |  Manipulation Graphs from Demonstrations Using Multimodal Sensory Signals[75] |  Multimodal Sensory Signals |  Y |  Barrett arm and hand equipped with two BioTacs |  X |  80 - 90% |  Able to perform grasping, unscrewing, and insertion tasks on a Barrett’s arm. |  The robotic arm didn’t work in those trajectories which are demonstrated earlier.
Jonas |  Multi-Modal Sensor Data for Lower Limb Exoskeletons[76] |  Multi-modal sensor data |  Y |  Hidden Markov Models (HMMs) |  X |  92.80% |  Used for classification of motion patterns at each time step while climbing stairs. |  Unable to conduct a deeper analysis of the latencies for different motion transitions.
Castro |  Tracking-based Depth Estimation of Metallic Pieces for Robotic Guidance [58] |  Image, Lasers |  Y |  Kinect, Lasers, mono-camera |  Y |  95% |  Performs object recognition and tracking system in real time |  The model developed is not used in any application; it just a prototype. To increase the usability, new utilities needs to be incorporated.
Dennis |  Multimodal Heading and Pointing Gestures for Co-Located Mixed Reality Human-Robot Interaction [69] |  Speech and Gesture |  Y |  Mixed reality interface implemented using Microsoft HoloLens |  X |  93.92% |  The interface is capable of guiding a robotic arm to picks things |  Simple operation are performed, unable to investigate if more complex pick poses are requested.
Zhenzhou |  Unsupervised Trajectory Segmentation and Promoting of Multi-Modal Surgical Demonstrations [59] |  Video and Kinematic data |  Y |  Unsupervised deep learning network, stacking convolutional auto-encoder is used. |  Y |  TSC-K is the biggest beneficiary with the improvement of 15.2% on average |  The experiment was designed to accomplish tasks like needle passing and suturing during the surgery. |  X
Luís |  Extended Bayesian User Model (BUM) for Capturing Cultural Attributes [60] |  Sensor simulator and Signal generator |  Y |  Experiment capture a unified representation of cultural attributes from heterogeneous information. |  Y |  The framework has a significant impact on classifier precision over time, with an overall improvement of 27.88%. |  Ability to learn user attributes from a distributed data stream, with increasing performance over time. |  Unable to find information on the user’s current state, such as their mood, satisfaction level, etc.

###  IV-D Future Idea for Human-Machine Interaction

As the field is evolving day by day, the researchers are working hard to
implement a robust and foolproof solution for daily use. In the era of the
Internet of Things (IoT), most of the devices are connected with the network,
including household devices such as refrigerators, fans, lights, and even our
garages,which those simple things usually be controlled by remote device in
Figure 37. Following that model, the remote device connected to the Host PC
300 through the media capabilities. However, this is a elementary mechanise
network since it shows the household devices transmission. We believe that
User Interface will fundamentally change. The design of an operate system is
not enable to accomplish yet because building high precision devices are
expensive and the mechanise network are more complex[87]. Moreover even the
expensive components have probability of error percentage in every device. The
technology scientists are tried to create more stability and accuracy with IoT
based on the basic diagram below.

Figure 36: Remote device media capabilities Figure 37: Remote device media
capabilities

  1. 1.

Hardware equipment is easier to develop and manufacture thanks to innovations
like 3-D printers or Arduino.

  2. 2.

The cost of equipment is dropping significantly on account of the mass
adoption of consumer gadgets like cell phones.

  3. 3.

Technologies like sensors or WiFi chips turn into a generally accessible and
cheap commodity and are easy to integrate.

  4. 4.

As the equipment is becoming easier to develop and manufacture, the focus will
move far from innovation and will concentrate more on design or problem
solving, the same way it happened in software thanks to the growth of APIs.

  5. 5.

Software today is built with fifty years of oblivious assumptions of a work
tool as a primary concern. There are innovators, particularly from the design
world, who get through this presumption and make new UIs [88].

The ideal system designed should consist of the following characteristics:

  1. 1.

Decentralized: The user interfaces like the light switch shifted onto the
smart phone and will now shift away again into smart light switches, speech,
or completely new forms like eye tracking.

  2. 2.

Specific: Interfaces will move far from a nonexclusive screen towards
increasingly explicit interfaces that complete only a few things and that are
explicitly intended for that utilization case. This implies explicit
interfaces for designers that have attention on haptics, interfaces for
elderly people that have an emphasis on straightforwardness and unambiguity,
or interfaces for children will have an emphasis on playfulness.

  3. 3.

Human-centered: Graphical UIs have numerous limitations. They are not
accessible to visually impaired or disabled people. They utilize the visual
sense and a reduced version of haptics. There can be straining to affect our
hand, neck or eyes. Future interfaces will be designed with human science and
psychology in mind. It will incorporate more of our human senses.

  4. 4.

Instant: Putting numerous applications on one device implies that the user
will need to deal with menus. With decentralized, explicit interfaces, this
will be obsolete. Things will be instant again; the question is not whether an
action takes 1, 3, or 5 stages. The question will be if an activity should be
possible in a split second or not. This additionally diminishes our cognitive
load, which enables us to concentrate on the task at hand or the person in
front of us.

  5. 5.

Simple: Future interfaces will disregard the assumed integration with
graphical UI and will concentrate on making things less difficult than
existing arrangements.

  6. 6.

Augmented and virtual: The digital and physical will mix together. Whether
through augmented reality glasses or not, the user should have the capacity to
peruse setting data about a broken device, not through a cell phone but rather
specifically in the surrounding ”space” of the object.

  7. 7.

Passive: An action of a device should be fed as an input to the other. Passive
devices are already a major trend in HCI. The classic example of such an
application would be turning on the AC of your home when the headlights are
approaching the garage [88].

These are few features that a multimodal HCI device must comprise to become
successful in the market and accepted by the masses. In the recent past, there
are several attempts by companies like which failed deliberately. The Google
Glass is one such fine example which became a burden on the user and failed
miserably, and the users would not able to leverage the functionalities
despite investing a huge chunk of money. Another big factor that also plays a
vital role in a device being accepted by a wide range of people is its ”cost”.

Figure 38: Bolt’s “Put That There” System (Bolt, 1980).

##  V Conclusion and Future Work

With the classification of multimodal systems, there is an ample amount of
space for improvement regarding precision, quality, and robustness. The
implementation of the multimodal system using supervised learning has high
precision and can capture various modalities. The system, designed recently in
2018 and 2019 has advanced in exploring different modalities and its
applications, including their application from surgery to find the texture of
the cloth and fixing the screw using the robotic arm. This study enables us to
develop a system that can implement multimodal fusion using voice, gesture and
peripheral device as an input. In most of the systems with fusion, we have
analyzed has a huge scope of improvement. In our future work, we plan to
develop a system that captures raw EMG data using human limbs along with
voice, including input from a third modality, which could be a lever or a
keyboard input, that allows to generate input by combination of modalities if
the robot misses a gesture or a speech. The initial phase has been implemented
and incorporates two modalities, speech and gesture [89]. The results show
significant improvement in comparison with individual modality, with the
average error rate reduced to 5.2 percent. Age plays an important factor when
using multimodality. Younger people are more comfortable with MYO arm band and
other peripheral devices devices , whereas older adults prefer devices with
key input in comparison to haptic and gestures.

Moreover, the voice input modality is still preferable, but it brings another
challenge in that it does not accept all the accents of various ethnicities,
and the shaky voices of older adults. An incorrect input of voice commands may
lead to disaster in situations involving driverless cars. These error can be
avoided by training the Artificial Intelligence model using some balanced
precentage of data from these subjects or users. A multimodal autonomous car
should accept canned pre-defined input and discard others to function properly
and seek minimum human intervention. Otherwise, if a car analyzes human speech
and receives wrong input, it could be fatal for the users. Likewise, giant
robots used in automation or manufacturing units, if developed with multimodal
functionality, must understand canned inputs and discard others to function
properly. The assisted robot for the blind project is one of the cutting-edge
projects in the field of multimodal inputs, but if the system is not efficient
enough to adapt, the environment will fail during the evacuation of a building
when developing systems for differently abled people, efficiency and accuracy
would be considered the most important criteria. Otherwise, this robotic
system might prove fatal to humans. There is a need to develop a device of
multimodality with fusion that can be used in myriad industries. During the
review, we did not come across a system that can accept multimodal inputs with
fusion and is efficient enough to perform tasks in an industrial or a health
care sector. So far, the systems designed are either multimodal without
fusion, or they accept pre-defined inputs that work under certain conditions.

Moreover, in an industrial environment, it is necessary to move a robot
dynamically in any direction to move heavy objects from one place to another.
Usually, assembly lines can perform predefined static tasks. Along the similar
lines, a speech assistant with an intelligent robotic arm is needed to
understand human speech irrespective of ethnicity and pronunciation and that
can guide a differently abled person to move in and around a city without the
help of other human beings. Google and Microsoft developed speech APIs, which
are paid and too expensive to be used by everyday people, and they also have
the challenge of understanding of human speech with high accuracy. An error in
understanding speech could lead to disaster and may endanger human lives. The
ECOMODE device discussed earlier brings a lot of challenges to be used by
senior citizens. Emotions and gestures are an easy way to capture input by
avoiding speech.

According to our study, there are still avenues open for research in
multimodal fusion while exploring different combinations of input modalities.
If a system is designed along similar lines, it not only improves the
capability of handling industrial robots but also makes life easier for the
differently abled.

## References

  * [1] P. Adkar, “Unimodal and multimodal human computer interaction: a modern overview,” Pratibha-International J. Comput. Sci. Inf. Engg, vol. 2, pp. 2277–4408, 2013.
  * [2] S. Oviatt, “Ten myths of multimodal interaction,” Communications of the ACM, vol. 42, no. 11, pp. 74–81, 1999.
  * [3] D. Lalanne, L. Nigay, P. Robinson, J. Vanderdonckt, J.-F. Ladry, et al., “Fusion engines for multimodal input: a survey,” in Proceedings of the 2009 international conference on Multimodal interfaces, pp. 153–160, ACM, 2009\.
  * [4] S. Keates and P. Robinson, “The use of gestures in multimodal input,” 1998.
  * [5] L. Zhang, J.-Q. Li, K. Zhou, S. D. Gupta, M. Li, W.-B. Zhang, M. A. Miller, and J. A. Misener, “Traveler information tool with integrated real-time transit information and multimodal trip planning: Design and implementation,” Transportation Research Record, vol. 2215, no. 1, pp. 1–10, 2011.
  * [6] F. Karray, M. Alemzadeh, J. A. Saleh, and M. N. Arab, “Human-computer interaction: Overview on state of the art,” 2008.
  * [7] N. Mavridis, “A review of verbal and non-verbal human–robot interactive communication,” Robotics and Autonomous Systems, vol. 63, pp. 22–35, 2015\.
  * [8] J. C. Augusto, H. Nakashima, and H. Aghajan, “Ambient intelligence and smart environments: A state of the art,” in Handbook of ambient intelligence and smart environments, pp. 3–31, Springer, 2010.
  * [9] T. Carter, S. A. Seah, B. Long, B. Drinkwater, and S. Subramanian, “Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces,” in Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 505–514, ACM, 2013.
  * [10] M. Ferron, N. Mana, and O. Mich, “Mobile for older adults: towards designing multimodal interaction,” in Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia, pp. 373–378, ACM, 2015.
  * [11] “Bosch’s view of the future car: truly keyless entry, haptic feedback, smart parking, cybersecurity.” https://www.extremetech.com/extreme/241815-boschs-view-future-car-truly-keyless-entry-haptic-feedback-smart-parkingcybersecurity.  (Accessed on 06/06/2019).
  * [12] S. Mitra and T. Acharya, “Gesture recognition: A survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 37, no. 3, pp. 311–324, 2007.
  * [13] I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” The International Journal of Robotics Research, vol. 34, no. 4-5, pp. 705–724, 2015.
  * [14] N. Neverova, C. Wolf, G. Taylor, and F. Nebout, “Moddrop: adaptive multi-modal gesture recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 8, pp. 1692–1706, 2015.
  * [15] H. A. Vasquez, H. S. Vargas, and L. E. Sucar, “Using gestures to interact with a service robot using kinect 2,” Advances in Pattern Recognition, vol. 85, 2015.
  * [16] “How does the leap motion controller work?.” http://blog.leapmotion.com/hardware-to-software-how-does-the-leap-motion-controller-work/.  (Accessed on 06/07/2019).
  * [17] “How does the leap motion controller work?.” http://blog.leapmotion.com/.  (Accessed on 06/11/2019).
  * [18] “Getting started with your myo armband.” https://support.getmyo.com/hc/en-us/articles/203398347-Getting-started-with-your-Myo-armband.  (Accessed on 06/12/2019).
  * [19] C. M. Rebman Jr, M. W. Aiken, and C. G. Cegielski, “Speech recognition in the human–computer interface,” Information & Management, vol. 40, no. 6, pp. 509–519, 2003.
  * [20] F. Doshi and N. Roy, “Spoken language interaction with model uncertainty: an adaptive human–robot interaction system,” Connection Science, vol. 20, no. 4, pp. 299–318, 2008.
  * [21] “Human-computer interaction: Why voice changes everything.” https://medium.com/@joshdotai/human-computer-interaction-why-voice-changes-everything-f16cdc9df39.  (Accessed on 06/09/2019).
  * [22] “Siri.” https://en.wikipedia.org/wiki/Siri.  (Accessed on 06/06/2019).
  * [23] “Cortana. your intelligent assistant across your life..” https://www.microsoft.com/en-us/cortana.  (Accessed on 06/07/2019).
  * [24] “The eyegaze edge.” https://eyegaze.com/products/eyegaze-edge/.  (Accessed on 06/06/2019).
  * [25] “Okay, but how do touch screens actually work?.” https://scienceline.org/2012/01/okay-but-how-do-touch-screens-actually-work/.  (Accessed on 06/08/2019).
  * [26] “Touchscreen.” https://en.wikipedia.org/wiki/Touchscreen.  (Accessed on 06/06/2019).
  * [27] “Galaxy fold.” https://www.samsung.com/global/galaxy/galaxy-fold/design/.  (Accessed on 06/07/2019).
  * [28] “Meet nintendo switch.” https://www.nintendo.com/switch/features/.  (Accessed on 06/07/2019).
  * [29] “Digital pens.” https://www.explainthatstuff.com/digitalpens.html.  (Accessed on 06/07/2019).
  * [30] “Field mapping for agriculture.” https://www.skyswoop.com/services/field-mapping-for-agriculture/.  (Accessed on 06/07/2019).
  * [31] “Epoc user manual.” https://emotiv.gitbook.io/epoc-user-manual/introduction-1/about.  (Accessed on 06/12/2019).
  * [32] S. Oviatt, “Multimodal interfaces,” The human-computer interaction handbook: Fundamentals, evolving technologies and emerging applications, vol. 14, pp. 286–304, 2003.
  * [33] R. A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, vol. 14.  ACM, 1980.
  * [34] T. Zaheer, “Multimodal user interfaces for newspaper distribution transcription on mobile devices with accelerometer sensors,” Master’s thesis, 2009.
  * [35] P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, “Multimodal fusion for multimedia analysis: a survey,” Multimedia systems, vol. 16, no. 6, pp. 345–379, 2010.
  * [36] D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: an overview of methods, challenges, and prospects,” Proceedings of the IEEE, vol. 103, no. 9, pp. 1449–1477, 2015.
  * [37] Y. Vershinin, “A data fusion algorithm for multisensor systems,” in Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002.(IEEE Cat. No. 02EX5997), vol. 1, pp. 341–345, IEEE, 2002.
  * [38] V. Pitsikalis, A. Katsamanis, S. Theodorakis, and P. Maragos, “Multimodal gesture recognition via multiple hypotheses rescoring,” The Journal of Machine Learning Research, vol. 16, no. 1, pp. 255–284, 2015.
  * [39] J. Wagner, F. Lingenfelser, T. Baur, I. Damian, F. Kistler, and E. André, “The social signal interpretation (ssi) framework: multimodal signal processing and recognition in real-time,” in Proceedings of the 21st ACM international conference on Multimedia, pp. 831–834, ACM, 2013.
  * [40] M. Turk, “Multimodal interaction: A review,” Pattern Recognition Letters, vol. 36, pp. 189–195, 2014.
  * [41] D. M. Krum, O. Omoteso, W. Ribarsky, T. Starner, and L. F. Hodges, “Speech and gesture multimodal control of a whole earth 3d visualization environment,” tech. rep., Georgia Institute of Technology, 2002.
  * [42] M. Vallee, B. Burger, D. Ertl, F. Lerasle, and J. Falb, “Improving user interfaces of interactive robots with multimodality,” in 2009 International Conference on Advanced Robotics, pp. 1–6, IEEE, 2009.
  * [43] C. Bogdan, H. Kaindl, J. Falb, and R. Popp, “Modeling of interaction design by end users through discourse modeling,” in Proceedings of the 13th international conference on Intelligent user interfaces, pp. 305–308, ACM, 2008\.
  * [44] A. Kendon, Gesture: Visible action as utterance.  Cambridge University Press, 2004.
  * [45] H. Chizari, D. Lalanne, and M. Schwaller, “Combining voice and gesture for human computer interaction,” Master’s thesis, Freiburg University, 2013\.
  * [46] N. Vidakis, M. Syntychakis, G. Triantafyllidis, and D. Akoumianakis, “Multimodal natural user interaction for multiple applications: The gesture—voice example,” in 2012 International Conference on Telecommunications and Multimedia (TEMU), pp. 208–213, IEEE, 2012.
  * [47] E. Ferreira and F. Lefevre, “Reinforcement-learning based dialogue system for human–robot interactions with socially-inspired rewards,” Computer Speech & Language, vol. 34, no. 1, pp. 256–274, 2015.
  * [48] V. Vielzeuf, A. Lechervy, S. Pateux, and F. Jurie, “Multilevel sensor fusion with deep learning,” IEEE sensors letters, vol. 3, no. 1, pp. 1–4, 2019\.
  * [49] M. S. Hossain and G. Muhammad, “Emotion recognition using deep learning approach from audio–visual emotional big data,” Information Fusion, vol. 49, pp. 69–78, 2019.
  * [50] V. Echeverria, R. Martinez-Maldonado, and S. Buckingham Shum, “Towards collaboration translucence: Giving meaning to multimodal group data,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, p. 39, ACM, 2019.
  * [51] M. N. Giannakos, K. Sharma, I. O. Pappas, V. Kostakos, and E. Velloso, “Multimodal data as a means to understand the learning experience,” International Journal of Information Management, vol. 48, pp. 108–119, 2019\.
  * [52] S. Kumar, M. Yadava, and P. P. Roy, “Fusion of eeg response and sentiment analysis of products review to predict customer satisfaction,” Information Fusion, vol. 52, pp. 41–52, 2019.
  * [53] C. Park, P. Moghadam, S. Kim, A. Elfes, C. Fookes, and S. Sridharan, “Elastic lidar fusion: Dense map-centric continuous-time slam,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1206–1213, IEEE, 2018.
  * [54] C. Chou, A. Kingery, D. Wang, H. Li, and D. Song, “Encoder-camera-ground penetrating radar tri-sensor mapping for surface and subsurface transportation infrastructure inspection,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1452–1457, IEEE, 2018.
  * [55] T. Bruls, W. Maddern, A. A. Morye, and P. Newman, “Mark yourself: Road marking segmentation via weakly-supervised annotations from multimodal data,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1863–1870, IEEE, 2018.
  * [56] S. Luo, W. Yuan, E. Adelson, A. G. Cohn, and R. Fuentes, “Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2722–2727, IEEE, 2018.
  * [57] T. Zhou and J. P. Wachs, “Early turn-taking prediction with spiking neural networks for human robot collaboration,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–9, IEEE, 2018.
  * [58] M. Di Castro, C. V. Almagro, G. Lunghi, R. Marin, M. Ferre, and A. Masi, “Tracking-based depth estimation of metallic pieces for robotic guidance,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5503–5508, IEEE, 2018.
  * [59] Z. Shao, H. Zhao, J. Xie, Y. Qu, Y. Guan, and J. Tan, “Unsupervised trajectory segmentation and promoting of multi-modal surgical demonstrations,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 777–782, IEEE, 2018.
  * [60] L. Santos, G. S. Martins, and J. Dias, “An extended bayesian user model (bum) for capturing cultural attributes with a social robot,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 320–325, IEEE, 2018.
  * [61] “Human-computer interaction: Why voice changes everything.” https://www.oculus.com/rift-s/features/.  (Accessed on 06/11/2019).
  * [62] A. Lim and H. G. Okuno, “The mei robot: towards using motherese to develop multimodal emotional intelligence,” IEEE Transactions on Autonomous Mental Development, vol. 6, no. 2, pp. 126–138, 2014.
  * [63] T. Oka and K. Matsushima, “Multimodal manipulator control interface using speech and multi-touch gesture recognition,” in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts, pp. 21–22, ACM, 2015.
  * [64] E. Bozkurt, Y. Yemez, and E. Erzin, “Multimodal analysis of speech and arm motion for prosody-driven synthesis of beat gestures,” Speech Communication, vol. 85, pp. 29–42, 2016.
  * [65] J. Rico and S. Brewster, “Gesture and voice prototyping for early evaluations of social acceptability in multimodal interfaces,” in International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, p. 16, ACM, 2010.
  * [66] N. Vidakis, A. Vlasopoulos, T. Kounalakis, P. Varchalamas, M. Dimitriou, G. Kalliatakis, E. Syntychakis, J. Christofakis, and G. Triantafyllidis, “Multimodal desktop interaction: The face-object-gesture-voice example,” in 2013 18th International Conference on Digital Signal Processing (DSP), pp. 1–8, IEEE, 2013.
  * [67] O. Rogalla, M. Ehrenmann, R. Zollner, R. Becher, and R. Dillmann, “Using gesture and speech control for commanding a robot assistant,” in Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication, pp. 454–459, IEEE, 2002.
  * [68] S. Kang, K. Kim, and S. Chi, “Multimodal data fusion and intention recognition for horse riding simulators,” in Proceedings of the 2015 International Conference on Big Data Applications and Services, pp. 296–297, ACM, 2015.
  * [69] D. Perzanowski, A. C. Schultz, W. Adams, E. Marsh, and M. Bugajska, “Building a multimodal human-robot interface,” IEEE intelligent systems, vol. 16, no. 1, pp. 16–21, 2001.
  * [70] M. Skubic, D. Perzanowski, S. Blisard, A. Schultz, W. Adams, M. Bugajska, and D. Brock, “Spatial language for human-robot dialogs,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 34, no. 2, pp. 154–167, 2004.
  * [71] M. Turunen, J. Hakulinen, J. Hella, J.-P. Rajaniemi, A. Melto, E. Mäkinen, J. Rantala, T. Heimonen, T. Laivo, H. Soronen, et al., “Multimodal media center interface based on speech, gestures and haptic feedback,” in IFIP Conference on Human-Computer Interaction, pp. 54–57, Springer, 2009\.
  * [72] T. Buffomante, “Rethink robotics.”  (2017).
  * [73] S. Dupont and J. Luettin, “Audio-visual speech modeling for continuous speech recognition,” IEEE transactions on multimedia, vol. 2, no. 3, pp. 141–151, 2000.
  * [74] J. F. Gorostiza, R. Barber, A. M. Khamis, M. Malfaz, R. Pacheco, R. Rivas, A. Corrales, E. Delgado, and M. A. Salichs, “Multimodal human-robot interaction framework for a personal robot,” in ROMAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive Communication, pp. 39–44, IEEE, 2006.
  * [75] Z. Su, O. Kroemer, G. E. Loeb, G. S. Sukhatme, and S. Schaal, “Learning manipulation graphs from demonstrations using multimodal sensory signals,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2758–2765, IEEE, 2018.
  * [76] J. Beil, I. Ehrenberger, C. Scherer, C. Mandery, and T. Asfour, “Human motion classification based on multi-modal sensor data for lower limb exoskeletons,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5431–5436, IEEE, 2018.
  * [77] D. Krupke, F. Steinicke, P. Lubos, Y. Jonetzko, M. Görner, and J. Zhang, “Comparison of multimodal heading and pointing gestures for co-located mixed reality human-robot interaction,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–9, IEEE, 2018.
  * [78] “Who invented the computer?.” https://science.howstuffworks.com/innovation/inventions/who-invented-the-computer.htm.  (Accessed on 06/11/2019).
  * [79] “History of mobile cell phones.” https://bebusinessed.com/history/history-cell-phones/.  (Accessed on 06/12/2019).
  * [80] “Apple smart watch.” https://www.apple.com/apple-watch-series-4/.  (Accessed on 06/07/2019).
  * [81] “Microsoft modern keyboard with fingerprint id review.” https://www.techradar.com/reviews/microsoft-modern-keyboard-with-fingerprint-id.  (Accessed on 06/09/2019).
  * [82] “Sit back, relax, and enjoy a ride through the history of self-driving cars.” https://www.digitaltrends.com/cars/history-of-self-driving-cars-milestones/.  (Accessed on 06/12/2019).
  * [83] M. Menze and A. Geiger, “Object scene flow for autonomous vehicles,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3061–3070, 2015.
  * [84] “These are the 11 coolest features of the tesla model s.” https://www.cnbc.com/2017/11/17/tesla-model-s-best-features.html.  (Accessed on 06/08/2019).
  * [85] “The pal-v liberty claims to be the first commercially available flying car.” https://www.theverge.com/2018/3/7/17091136/flying-car-pal-v-liberty-geneva-motor-show-2018.  (Accessed on 06/08/2019).
  * [86] L. M. Reeves, J. Lai, J. A. Larson, S. Oviatt, T. Balaji, S. Buisine, P. Collings, P. Cohen, B. Kraal, J. C. Martin, et al., “Guidelines for multimodal user interface design,” Communications of the ACM, vol. 47, no. 1, pp. 57–59, 2004.
  * [87] C. H. Shoemaker, C.-C. Teng, H. C. Vidos, and J. C. Gibson, “Systems and methods for determining remote device media capabilities,” Aug. 26 2008.  US Patent 7,418,472.
  * [88] Senic, “The future of human computer interaction – hardware startup knowledge – medium,” Jan 2015.
  * [89] T. K. Mohd, J. Carvalho, and A. Y. Javaid, “Multi-modal data fusion of voice and emg data for robotic control,” in 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), pp. 329–333, IEEE, 2017.