# Effect of Pt vacancies on magnetotransport of Weyl semimetal candidate
GdPtSb epitaxial films

Dongxue Du  Materials Science and Engineering, University of Wisconsin-
Madison, Madison, WI 53706  Laxman Raju Thoutam  Amrita School of Nanosciences
and Molecular Medicine, Amrita Vishwa Vidyapeetham, Ponekkara, Kochi 682041,
India  Department of Chemical Engineering and Materials Science, University of
Minnesota – Twin Cities  Konrad T. Genser  Department of Physics and
Astronomy, Rutgers University  Chenyu Zhang  Materials Science and
Engineering, University of Wisconsin-Madison, Madison, WI 53706  Karin M. Rabe
Department of Physics and Astronomy, Rutgers University  Bharat Jalan
Department of Chemical Engineering and Materials Science, University of
Minnesota – Twin Cities  Paul M. Voyles  Materials Science and Engineering,
University of Wisconsin-Madison, Madison, WI 53706  Jason K. Kawasaki
Materials Science and Engineering, University of Wisconsin-Madison, Madison,
WI 53706<EMAIL_ADDRESS>
###### Abstract

We examine the effects of Pt vacancies on the magnetotransport properties of
Weyl semimetal candidate GdPtSb films, grown by molecular beam epitaxy on
c-plane sapphire. Rutherford backscattering spectrometry (RBS) and x-ray
diffraction measurements suggest that phase pure GdPtxSb films can accommodate
up to $15\%$ Pt vacancies ($x=0.85$), which act as acceptors as measured by
Hall effect. Two classes of electrical transport behavior are observed. Pt-
deficient films display a metallic temperature dependent resistivity
(d$\rho$/dT$>$0). The longitudinal magnetoresistance (LMR, magnetic field
$\mathbf{B}$ parallel to electric field $\mathbf{E}$) is more negative than
transverse magnetoresistance (TMR, $\mathbf{B}\perp\mathbf{E}$), consistent
with the expected chiral anomaly for a Weyl semimetal. The combination of Pt-
vacancy disorder and doping away from the expected Weyl nodes; however,
suggests conductivity fluctuations may explain the negative LMR rather than
chiral anomaly. Samples closer to stoichiometry display the opposite behavior:
semiconductor-like resistivity (d$\rho$/dT$<$0) and more negative transverse
magnetoresistance than longitudinal magnetoresistance. Hysteresis and other
nonlinearities in the low field Hall effect and magnetoresistance suggest that
spin disorder scattering, and possible topological Hall effect, may dominate
the near stoichiometric samples. Our findings highlight the complications of
transport-based identification of Weyl nodes, but point to possible
topological spin textures in GdPtSb.

##  I Introduction

The lanthanide half Heusler compounds $Ln$PtBi and $Ln$PtSb are attractive due
to their tunable topological and magnetic properties as functions of
lanthanide substitution Nakajima et al. (2015); Lin et al. (2010); Gofryk et
al. (2011), strain Lin et al. (2010), and strain gradients Du et al. (2021).
Compounds in this family of materials were among the first identified as zero
bandgap topological semimetals via density functional theory (DFT) Lin et al.
(2010) with confirmation for LuPtSb Logan et al. (2016), LuPtBi Liu et al.
(2016) and YPtBi Liu et al. (2016) by angle-resolved photoemission
spectroscopy (ARPES) measurements Liu et al. (2016); Logan et al. (2016).

More recently, bandstructure calculations and magnetotransport measurements
suggest that GdPtBi Hirschberger et al. (2016); Shekhar et al. (2018), TbPtBi
Chen et al. (2020), HoPtBi Chen et al. (2020), and ErPtBi Chen et al. (2020)
compounds are magnetic-field-induced Weyl semimetals. In these materials,
magnetic field, either directly through Zeeman splitting or indirectly through
exchange splitting from field-induced magnetization, is expected to lift the
degeneracy of quadratic bands that touch at $\Gamma$, to create pairs of Weyl
nodes Hirschberger et al. (2016); Shekhar et al. (2018). One experimental
signature of these Weyl nodes is the chiral anomaly: charge pumping between
Weyl nodes of opposite chirality when an applied magnetic field B is parallel
to the electric field E Adler (1969); Liang et al. (2018). This appears as a
negative longitudinal magnetoresistance (LMR) with a characteristic angle
dependence $\mathbf{E}\cdot\mathbf{B}$, and has been observed in several
$Ln$PtBi compounds Chen et al. (2020); Hirschberger et al. (2016); Shekhar et
al. (2018); Liang et al. (2018).

A fundamental challenge, however, is that negative LMR is not unique to the
chiral anomaly. Other mechanisms for negative LMR include current jetting
Liang et al. (2018); Dos Reis et al. (2016), conductivity fluctuations Hu et
al. (2005); Dos Reis et al. (2016); Steele (1955); Schumann et al. (2017), and
spin-disorder scattering in magnetically ordered materials Usami (1978). In
half Heusler compounds, conductivity fluctuations may contribute because these
materials are highly susceptible to natural nonstoichiometry Yonggang et al.
(2017) and variations in atomic site ordering Öğüt and Rabe (1995), due to the
low formation energies for point defects. Moreover, since $Ln$PtBi and
$Ln$PtSb are typically antiferromagnetic below $T_{N}\sim 10$ K, spin disorder
scattering is likely to play a role.

Here we explore the magnetotransport properties of GdPtSb, which like GdPtBi,
is seen in bandstructure calculations to have quadratic bands that touch at
$Gamma$, taking into account the effects of naturally occurring
nonstoichometry. Specifically, we examine effects of Pt vacancies on
magnetotransport of GdPtSb films, grown by molecular beam epitaxy on c-plane
sapphire substrates. We find that the angle-dependent magnetoresistance
depends strongly on Pt stoichiometry, and comment on the relative roles of
chiral anomaly, conductivity fluctuations, spin-disorder scattering, and
topological hall effect in GdPtxSb. In particular, our observation of a
plateau in the Hall resistivity in near stoichiometric GdPtSb suggests a
topological Hall effect which may arise from topological spin textures that
have not previously been observed in $Ln$PtSb or $Ln$PtBi systems.

##  II Results

Figure 1: DFT calculations. (a) DFT-GGA band structures of GdPtSb and (b)
GdPtBi. The bands near the Fermi energy at $\Gamma$ have strong Sb $5p_{3/2}$
or Bi $6p_{3/2}$ character. The red color denote the $m_{j}=\pm 1/2$ states
and the blue color denote the $m_{j}=\pm 3/2$ states. (c) Cartoon showing how
the quadratic band touching at $\Gamma$ splits due to exchange or Zeeman
splitting to form pairs of Weyl nodes with opposite chirality.

We first establish the possibility of Weyl nodes for GdPtSb using density
functional theory (DFT) calculations. In the well studied material GdPtBi, the
essential feature is a quadratic band touching of four Bi $6p$ $j=3/2$ states
near the Fermi energy (Fig. 1(b)). The Zeeman energy or exchange energy splits
the $m_{j}=\pm 3/2$ and $m_{j}=\pm 1/2$ to create Weyl nodes near the Fermi
energy (Fig. 1(c)) Hirschberger et al. (2016); Shekhar et al. (2018) .

Our nonmagnetic DFT calculations suggest that GdPtSb replicates this essential
feature, with a quadratic touching of Sb $5p$ $j=3/2$ states at the Fermi
energy at $\Gamma$ (Fig. 1(a)). However, we note that for GdPtSb, there is an
additional hole band approximately $60$ meV below the charge neutrality point
(0 eV) at $\Gamma$ that may complicate the transport by providing an
additional conduction channel. For GdPtBi, this hole band is pushed further
down to $\sim 700$ meV, presumably due to the larger spin-orbit coupling for
Bi compared with Sb.

Figure 2: Structures and basic transport properties of stoichiometric and Pt-
deficient GdPtSb epitaxial films. (a) X-ray diffraction (Cu $K\alpha$ of
(111)-oriented GdPtxSb thin films grown on Al2O3 (0001). The “+” sign for
$x=0.80$ denotes a GdSb impurity phase. (b) Corresponding reflection high
energy electron diffraction (RHEED) patterns. (c) Out-of-plane ($d_{111}$,
red) and in-plane ($d_{110}$, blue) lattice spacings extracted from on axis
$222$ and off axis $202$ reflections (Supplemental). (d) High angle annular
dark field scanning transmission electron microscopy (STEM) image of GdPtSb,
measured along a $\langle 110\rangle$ zone axis.

We synthesize GdPtSb films by molecular beam epitaxy on (0001)-oriented Al2O3
substrates, using conditions similar to Ref. Du et al. (2019). The growth
temperature is 600 $\degree$C. The Gd flux was supplied by a thermal effusion
cell. A Sb2/Sb1 mixture was supplied by a thermal cracker cell with a cracker
zone operated at 1200 ∘C. The Pt flux was supplied by an electron beam
evaporator. Fluxes were measured in-situ using a quartz crystal microbalance
(QCM) immediately prior to growth. Absolute compositions were measured by
Rutherford Backscattering Spectrometry (RBS). Due to the high relative
volatility of Sb, GdPtSb films were grown in an Sb adsorption-controlled
regime with a 30% excess Sb flux, such that the Sb stoichiometry is self
regulated Shourov et al. (2020).

X-ray diffraction (XRD) and reflection high energy electron diffraction
(RHEED) measurements reveal that that epitaxial (111)-oriented GdPtxSb films
with half Heusler structure are readily stabilized under Pt-deficient
conditions (Fig. 2(a)). For $x=0.85$ to 1, only the anticipated $111$-type
reflections are observed by XRD indicating phase pure half Heusler growth. The
corresponding streaky RHEED patterns indicate smooth epitaxial films. Samples
closer to stoichiometry show an enhanced intensity of the higher order $333$
and $444$ XRD reflections and a well ordered $3\times$ surface reconstruction
by RHEED, compared to the $1\times$ periodicity for Pt deficient films
($x=0.85$). For highly Pt deficient conditions, $x\leq 0.8$, we observe
precipitation of a secondary GdSb phase by XRD and rough three-dimensional
growth by RHEED. For Pt-rich conditions $x>1.01$ the higher order $222$,
$333$, and $444$ half Heusler XRD reflections disappear and we observe faint
polycrystalline rings in the RHEED pattern.

Focusing on the phase pure GdPtxSb samples with $x=0.85-1$, we observe a
systematic increase for the out-of-plane $d_{111}$ and in-plane $d_{110}$
lattice spacings (Fig. 2(c)). The in-plane $d_{110}$ spacings are calculated
from measurements of an off-axis $202$ reflection (Appendix Fig. 8). The
observed Pt deficient samples are consistent with DFT calculations that
predict Pt vacancies are the lowest energy defects for the related compound
LuPtSb Khalid et al. (2022). High angle annual dark field (HAADF) scanning
transmission electron microscopy (STEM) measurements of the $x=0.85$ sample
shown in Fig. 2(d) are in agreement with the expected site ordering for GdPtSb
in cubic half Heusler structure (space group F$\bar{4}$3m). Here, the
brightest atomic columns correspond to columns of Pt atoms (which have the
largest atomic mass), followed by columns of Gd and columns of Sb.

Figure 3: (a) Zero-field resistivity for GdPtxSb with different Pt
concentration. Insert shows the magnetic susceptibility versus temperature for
the $x=0.84$ (green) and $x=0.96$ (purple) samples measured by SQUID, showing
the Neel transition. Kinks in the susceptibility coincide with kinks in the
resistivity. (b) Transverse (Hall) resistivity for the $x=0.96$ sample,
measured at 1.8 K. At low field $|B|<1$ T we observe nonlinearities suggestive
of the topological Hall effect. Higher field ($|B|>2$ T) nonlinearities are
associated with ordinary Hall effect with multiple carriers. The red line is
an ordinary Hall effect two band model fit. (c) Hall effect at 1.8 K for the
$x=0.85$ sample showing near linear behavior for $|B|<6$ T.

Zero-field resistivity measurements of near stoichiometric GdPtSb films
($x>0.9$) display an insulator-like temperature dependence ($d\rho/dT<0$, Fig.
3(a) purple). We attribute the insulator-like resistivity to the Fermi energy
being near the quadratic band touching (Fig. 1). In contrast, for heavily Pt-
deficient samples ($x<0.9$) we observe metallic transport ($d\rho/dT>0$, Fig.
3(a), green). We attribute the more metallic transport to doping induced by Pt
vacancies.

We observe kinks in the temperature dependent resistivity that correspond to
kinks at the same temperature in magnetic susceptibility, as measured by
superconducting quantum interference device (SQUID) magnetometry (Fig. 3a
insert). We attribute these kinks to the antiferromagnetic Neel transition
$T_{N}$. We find that $T_{N}$ varies with Pt concentration $x$: the insulator-
like samples ($x>0.9$) have a $T_{N}\approx 9$ K, whereas samples with a more
metallic resistivity ($x<0.9$) have $T_{N}\approx 14$ K (Fig. 4(a)). We
speculate this jump in $T_{N}$ may arise from an enhanced
Ruderman–Kittel–Kasuya–Yosida (RKKY) coupling between Gd moments and the Fermi
sea with increasing carrier density.

Hall effect measurements reveal that Pt vacancies in GdPtSb are acceptors. The
heavily Pt-deficient samples ($x<0.9$) show a positive and near linear
dependence of the Hall resistivity $\rho_{xy}$ on magnetic field (Fig. 3(c)),
indicating dominant hole carriers. Closer to stoichiometry ($x>0.9$) the
samples show stronger nonlinearities that are well fit by a two band model
with one hole and one electron for $|B|>2$ T (Fig. 3c). Fig. 4(b) summarizes
the effective electron and hole densities versus Pt concentration $x$,
extracted from fitting to the two band model (Methods). Note that in reality
there are 2-3 hole bands near the Fermi energy; however, since we are not able
to distinguish these bands from a simple Hall effect fit we emphasize that
these are “effective” carrier densities. We find that charge neutrality $n=p$
appears as $x$ approaches 1, and the effect of Pt vacancies is to increase the
hole density and decrease the electron density. Based on a three-dimensional
parabolic band model (Methods), we estimate that the Fermi energy for the
$x=0.85$ sample, which has effective hole density $p=2.72\times 10^{20}$ cm-3,
lies 170 meV below the charge neutrality point. For the $x=0.96$ sample
($p=1.75\times 10^{19}$ cm-3, $n=2.5\times 10^{17}$ cm-3), we estimate the
Fermi energy is approximately 40 meV below the charge neutrality point. We
caution that while we call this $x=0.96$ sample “near stoichiometric,” the
hole density is still nearly two orders of magnitude larger than the electron
density ($p>>n$).

Interestingly, the $x=0.96$ sample displays low field ($|B|<1$ T) hysteresis
and nonlinearities consistent with the topological Hall effect (Fig. 3(b),
insert), suggesting a nontrivial Berry phase. This topological Hall effect
often indicates topological spin textures like skyrmions, e.g., in chiral
$B20$ compounds Neubauer et al. (2009) and tetragonal Heusler compounds Kumar
et al. (2020), but to our knowledge has not yet been reported in the $R$PtBi
or $R$PtSb family.

Figure 4: (a) Neel temperature ($T_{N}$) extracted from kinks in the
temperature dependent magnetic susceptibility and resistivity. (b) Hall
carrier density versus Pt concentration at 1.8 K, extracted from fitting to a
model with one electron and one hole. The insert shows the approximate
position of the Fermi energy for $x=0.85$ in a two band model, which is
clearly an over simplification. (c) Longitudinal
($\mathbf{B}\parallel\mathbf{E}$) and transverse ($\mathbf{B}\perp\mathbf{E}$)
magnetoresistance at 1 T and 1.8 K for samples with varying Pt concentration.

We now analyze the magnetoresistance as a function of the angle $\theta$
between $\mathbf{B}$ and $\mathbf{E}$. The chiral anomaly, i.e. charge pumping
between Weyl nodes, is expected to produce an additional current that is
proportional to $\mathbf{B}\cdot\mathbf{E}$. Therefor, for Weyl semimetals the
magnetoresistance $\Delta\rho/\rho_{0}$ should be negative for $\theta=0$ and
become more positive with increasing $\theta$ Xiong et al. (2015);
Parameswaran et al. (2014); Hirschberger et al. (2016); Nielsen and Ninomiya
(1983). Fig. 5(a) shows the angle dependent magnetoresistance of the Pt
deficient $x=0.85$ sample, measured at 1.8 K using voltage contacts along the
edge of the sample and current sourced along the center of the sample (Fig.
5(c)). We find that the longitudinal magnetoresistance (LMR,
$\theta=0^{\circ}$) is negative and the transverse magnetoresistance (TMR,
$\theta=90^{\circ}$) is positive, as expected for the chiral anomaly. A
continuously varying angular dependence, in a van der Pauw geometry, is shown
in Appendix Fig. 9. The general angular dependence is qualitatively similar to
previous studies of GdPtBi single crystals Hirschberger et al. (2016); Liang
et al. (2018). However, magnitude of change for our epitaxial GdPtSb films is
only a few percent, whereas the magnetoresistance change for GdPtBi crystals
is $\sim 80\%$.

Figure 5: Chiral anomaly tests. (a,b) Magnetoresistance for a Pt deficient
x=0.85 sample, as function of angle $\theta$ between $\mathbf{B}$ and
$\mathbf{E}$. Comparison of voltage contacts along the edge (a) with voltage
contacts along the center (b) constitutes the “squeeze test” for analyzing the
contribution of extrinsic current jetting. (c) Measurement geometry. (d) Angle
dependent magnetoresistance for a near stoichiometric $x=0.96$ sample. Insert
shows hysteresis for $\theta=90\degree$.

The $x=0.85$ sample passes the “squeeze test” Liang et al. (2018), suggesting
that current jetting is not the primary origin of the negative LMR. Here, we
find that measurements with contacts along the center of the sample (Fig.
5(b)) produce the same qualitative behavior as edge contacts (Fig. 5(a)),
namely, negative LMR and positive TMR at modest magnetic field ($|B|<6$ T).
2-point resistance measurements also produce the negative magnetoresistance
for $\theta=0$ that increases with $\theta$ (Appendix Fig. 10), confirming
that current jetting is not a dominant factor. Furthermore, current jetting
effects are expected to be strongest for materials with high carrier mobility
and anisotropic conduction Liang et al. (2018). Our samples have more modest
Hall effect mobility ($\mu\sim 50$ cm2/Vs) and are expected to be isotropic in
the (111) plane, compared to reports of single crystals that have higher
mobility of 1500 cm2/Vs at 6 K Hirschberger et al. (2016).

Near stoichiometric samples display the opposite angular dependence. Fig. 5(d)
shows the angle-dependent magnetoresistance for a sample with composition
$x=0.96$, measured at 1.8 K. Besides a weak positive magnetoresistance at low
field ($B<4$ T) and low $\theta$, the magnetoresistance is generally negative
and becomes more negative with increasing $\theta$. This is opposite the
expected dependence for chiral anomaly, which is expected to produce the most
negative magnetoresistance for $\theta=0$. Additionally, the transverse
magnetoresistance shows a weak hysteresis within the range $|B|<1$ T (Fig.
5(d) insert), the same range as observed for the Hall resistance (Fig. 3(b)
insert). Minimal hysteresis is observed in the magnetoresistance of the
$x=0.85$ sample.

We summarize the TMR and LMR at field 1 T and temperature 1.8 K for several
samples with varying Pt concentration in Fig. 4(c). For Pt deficient samples
($x<0.9$) the LMR ($\mathbf{B}\parallel\mathbf{E}$) is generally more negative
than the TMR ($\mathbf{B}\perp\mathbf{E}$), as expected for chiral anomaly.
For samples closer to stoichiometry ($x>0.9$) the LMR is generally more
positive than TMR. This dependence on Pt concentration is opposite to the
expectation for chiral anomaly, which we expect to be strongest for
stoichiometric samples in which the Fermi energy is closer to the Weyl nodes
($x\approx 1$). Previous experiments on doped GdPtBi show that the negative
LMR is maximized near the charge neutrality point Hirschberger et al. (2016).

Figure 6: Temperature dependence of the LMR and TMR. (a) $x=0.85$ sample at 1
T. (b) $x=0.95$ sample at 1 T. (c) $x=0.85$ sample at 1 9 T. (e) $x=0.95$
sample at 9 T. These measurements were performed in a van der Pauw geometry.
Note the magnitudes are different than in Fig 5 due to the different contact
geometry.

##  III Discussion

What explains the Pt stoichiometry dependence? For Pt deficient samples
($x<0.9$), the Pt vacancy disorder and position of the Fermi energy away from
the Weyl nodes suggests that conductivity fluctuations rather than chiral
anomaly may dominate. For inhomogeneous or disordered materials, spatial
fluctuations in the conductivity can lead to a component of the carrier
velocity that is perpendicular to $\mathbf{B}$, even when the global
$\mathbf{E}$ is parallel to $\mathbf{B}$ Hu et al. (2005, 2007); Schumann et
al. (2017); Dos Reis et al. (2016); Xu et al. (2019). The resulting Lorentz
force leads to a decrease in the LMR with increasing $\mathbf{B}$. We expect
Pt-deficient samples to show these fluctuations more strongly than
stoichiometric samples. Moreover, the very Pt deficient samples have Fermi
energy furthest away from the expected Weyl nodes, and thus are not
anticipated to show strong effects from chiral anomaly compared to samples
closer to stoichiometry ($x\approx 1$).

For samples closer to stoichiometry, trivial bands near the Fermi energy and
spin disorder scattering may explain the negative LMR. We first note that
unlike the well studied Weyl semimetal GdPtBi, GdPtSb has an additional hole
band approximately 60 meV below the charge neutrality point (Fig. 1(a)) that
may contribute to the transport and obscure the chiral anomaly.

Additionally, in magnetically ordered materials like GdPtSb (which is
antiferromagnetic), field alignment of spins or other forms of spin-disorder
scattering can also cause a negative LMR Usami (1978); Ritchie et al. (2003);
Borca et al. (2001), and the angular dependence can arise from
magnetocrystalline anisotropy or shape anisotropy. For reasons that are still
unclear, the effects of magnetic ordering are more prominent for our near
stoichiometric GdPtSb samples than for heavily Pt-deficient samples. First,
the TMR of the $x=0.96$ sample shows magnetic hysteresis that does not appear
for the $x=0.85$ sample (Fig. 5(b,d) inserts). Second, the temperature-
dependent magnetoresistance for the $x=0.96$ sample at low field diverges
below $T_{N}\sim 9$ K (Fig. 6(b)), where the LMR increases and the TMR
decreases with decreasing temperature. In contrast, the LMR and TMR for the
$x=0.85$ sample do not diverge below $T_{N}$ (Fig. 6(a)). Third, $T_{N}$
abruptly jumps from $\sim 14$ K for Pt deficient samples to $T_{N}\sim 9$ K
for near stoichiometric samples (Fig. 4(a)). This suggests an abrupt change in
the exchange, and possibly magnetic ordering, as a function of $x$. Further
magnetization studies, as a function of magnetic field orientation, are
required to understand the mechanisms for the angle-dependent
magnetoresistance in near stoichiometric samples.

Figure 7: (a) Low field Hall effect, showing possible topological hall effect.
No background has been subtracted. (b) Low field magnetoresistance showing
hysteresis over the same range as the possible topological component of the
Hall effect ($|B|<1$ T).

More intriguingly, the near stoichiometric samples display indications of
topological Hall effect (Fig. 3b insert) that do not appear for the very Pt
deficient samples (Fig. 3(a,b)). We reproduce the low field transverse (Hall)
and longitudinal magnetroresistance of the $x=0.96$ sample in Fig. 7. In the
transverse resistivity $\rho_{xy}$ versus magnetic field, we observe low field
nonlinearities and hysteresis that is reminiscent of the topological Hall
effect. Topological Hall effect indicates a nontrivial Berry phase and is
often an indication of chiral spin textures like skyrmions; however, a more
detailed analysis is required to clearly identify this nonlinearity as
topological Hall effect Gerber (2018). The low field magnetoresistance (Fig.
5(d) insert and Fig. 7(b)) shows the same width of hysteresis ($\pm 1$ T).
Systems that display topological Hall effect also display negative
magnetoresistance Ahadi et al. (2017); Huang et al. (2022).

##  IV Conclusions

We demonstrate that half Heusler GdPtxSb epitaxial films grown on sapphire
accommodate a large concentration of Pt vacancies, which act as acceptors.
Samples with high Pt vacancy concentration have a metallic resistivity vs
temperature and display a negative longitudinal magnetoresistance that becomes
more positive as the magnetic field tilts away from the electric field,
consistent with chiral anomaly. However, the large concentration of Pt
vacancies and position of the Fermi energy away from the Weyl nodes suggests
that conductivity fluctuations, rather than chiral anomaly, dominates. Samples
closer to stoichiometry show an insulator-like resistivity versus temperature
and TMR that is more negative than LMR, opposite the expected behavior of
chiral anomaly. The low field Hall effect shows nonlinearities similar to
topological Hall effect. Further detailed magnetotransport studies are
required to understand the possible balance of Weyl nodes and topological spin
textures in GdPtSb.

##  V Methods

First-principles calculations. Calculations were done with ABINIT using the
PBE GGA exchange correlation potential and norm-conserving pseudopotentials
from ONCVPSP-3.3.0, except for the Gd pseudopotential, which was constructed
to have the f-orbitals in the core. The k point mesh used was $18\times
18\times 18$ and the energy cutoff was 1400 eV (50 Hartree). Computed lattice
parameters for the half Heusler structure were 6.647 Å for GdPtSb and 6.777 Å
for GdPtBi in the conventional fcc unit cell.

Transport measurements and fitting. Magnetotransport measurements for GdPtxSb
samples were performed using a Quantum Design Dynacool Physical Property
Measurement System. Hall effect measurements were generally performed in a
Hall bar geometry with typical dimensions 5 mm by 1 mm. For the $x=0.85$
sample the Hall effect was measured in a van der Pauw geometry. Angle-
dependent magnetoresistance measurements were performed using a horizontal
rotator probe and contacts in a “squeeze test” geometry, as shown in Fig.
5(c). The “center” channel is a standard linear four point geometry with
contacts in a line down the center of the sample, where the outer contacts are
current and the inner contacts are voltage. The “edge” channel uses the same
current contacts, but places the voltage contacts at the edge of the sample to
test the effects of current jetting down the center of the sample.

Nonlinear Hall data are fit with a two-band model of the following form

$\rho_{xy}(B)=\frac{B}{e}\frac{(n_{h}\mu_{h}^{2}-n_{e}\mu_{e}^{2})+(n_{h}-n_{e})\mu_{h}^{2}\mu_{e}^{2}B^{2}}{(n_{h}\mu_{h}+n_{e}\mu_{e})^{2}+(n_{h}-n_{e})^{2}\mu_{h}^{2}\mu_{e}^{2}B^{2}}$

where $n_{h}$ ($n_{e}$) and $\mu_{h}$ ($\mu_{e}$) are concentration and
mobility of holes (electrons). We constrain the two-band Hall fit by also
fitting to the zero magnetic field longitudinal resistivity

$\rho_{xx}(0)=\frac{1}{e(n_{h}\mu_{h}+n_{e}\mu_{e})}.$

We use the carrier concentration to estimate the Fermi level positions. For a
rough estimation, we used 3D parabolic band density of states.

$D(E)=\frac{\sqrt{2}}{\pi^{2}}\frac{(m^{*})^{\frac{3}{2}}}{\hbar^{3}}\sqrt{E}$

where m∗ is the effective mass.

$m^{*}=\frac{\hbar^{2}}{d^{2}E/dk^{2}}$

We calculated the effective mass for the 3 valence bands near E=0 eV at
$\Gamma$ point, based on our band structure calculation in Fig. 1. The Fermi
level is extracted from the following integral,

$\sum_{n=1}^{2or3}\int_{0}^{E_{F}}D(E)\,dE=\frac{N}{V}$

From the above calculation, the EF for the $x=0.84$ sample with hole
concentration $2.72\times 10^{20}$ cm-3 is about -170 meV and that for the
$x=0.96$ sample with hole concentration $1.75\times 10^{19}$ cm-3 is about -40
meV.

Magnetization measurements. Magnetic properties were measured using a Quantum
Design MPMS SQUID (Superconducting Quantum Interference Device) Magnetometer.
The magnetic field is applied perpendicular to the sample surface. The net
magnetization data for the GdPtxSb thin films is extracted by subtracting a
background measurement of the Al2O3 substrate from the total magnetization
signal.

Transmission electron microscopy. GdPtSb cross-section samples were prepared
with a Zeiss Ga-focused ion beam, followed by final thinning in a Fishione
Model 1040 Nanomill using Ar ions at 900V. Samples were stored in vacuum and
cleaned in a GV10x DS Asher cleaner at 20 W for 10 min to remove contamination
and minimize the oxidization on the sample before being transferred into the
STEM column. A probe corrected Thermo-Fisher Titan STEM equipped with CEOS
aberration corrector operated at 200 kV was used to collect the atomic
resolution STEM images. A 24.5 mrad probe semi-angle, 18.9 pA probe current
was used to collect HAADF image series with a Fishione 3000 annular detector
covering collection angle ranging from 53.9 to 269.5 mrad. Each frame in the
image series took about 0.6 second to acquire with 10 $\mu$s on each STEM scan
position and 256-by-256 scan grid. Non-rigid registration Yankovich et al.
(2014) was used to compensate for the drift and distortions during image
series acquisition, before the series was averaged to get one single frame
with high signal-to-noise ratio as shown in Fig. 2(e).

X-ray diffraction. X-ray diffraction measurements were performed using a
Malvern Empyrean diffractometer with Cu $K\alpha$ radiation.

Rutherford Backscattering Spectrometry (RBS). RBS measurements were performed
at the University of Minnesota Characterization Facility.

##  VI Acknowledgment

We thank Max Hirschberger for helpful discussions on chiral anomaly tests. We
thank Greg Haugstad for performing RBS measurements. Special thanks to D. R.
Hamman for providing the Gd pseudopotential.

Heusler epitaxial film growth and magnetotransport at the University of
Wisconsin were supported by the Air Force Office of Scientific Research
(FA9550-21-0127). Preliminary synthesis was supported by the Army Research
Office (ARO Award number W911NF-17-1-0254) and the National Science Foundation
(DMR-1752797). Calculations by KR and KG were supported by the Office of Naval
Research N00014-21-1-2107. Transport measurements at the University of
Minnesota by L.R.T. and B.J. were supporteb by the National Science Foundation
through the University of Minnesota MRSEC under Award No. DMR-2011401. TEM
experiments by CZ and PMV were supported by the US Department of Energy, Basic
Energy Sciences (DE-FG02-08ER46547) and used facilities are supported by the
Wisconsin MRSEC (DMR-1720415). We gratefully acknowledge the use of x-ray
diffraction facilities supported by the NSF through the University of
Wisconsin Materials Research Science and Engineering Center under Grant No.
DMR-1720415. We acknowledge PARADIM for annealed sapphire substrates.

## References

  * Nakajima et al. (2015) Y. Nakajima, R. Hu, K. Kirshenbaum, A. Hughes, P. Syers, X. Wang, K. Wang, R. Wang, S. R. Saha, D. Pratt, et al., Science advances 1, e1500242 (2015).
  * Lin et al. (2010) H. Lin, L. A. Wray, Y. Xia, S. Xu, S. Jia, R. J. Cava, A. Bansil, and M. Z. Hasan, Nature materials 9, 546 (2010).
  * Gofryk et al. (2011) K. Gofryk, D. Kaczorowski, T. Plackowski, A. Leithe-Jasper, and Y. Grin, Physical Review B 84, 035208 (2011).
  * Du et al. (2021) D. Du, S. Manzo, C. Zhang, V. Saraswat, K. T. Genser, K. M. Rabe, P. M. Voyles, M. S. Arnold, and J. K. Kawasaki, Nature communications 12, 1 (2021).
  * Logan et al. (2016) J. A. Logan, S. Patel, S. D. Harrington, C. Polley, B. D. Schultz, T. Balasubramanian, A. Janotti, A. Mikkelsen, and C. J. Palmstrøm, Nature communications 7, 1 (2016).
  * Liu et al. (2016) Z. Liu, L. Yang, S.-C. Wu, C. Shekhar, J. Jiang, H. Yang, Y. Zhang, S.-K. Mo, Z. Hussain, B. Yan, et al., Nature communications 7, 12924 (2016).
  * Hirschberger et al. (2016) M. Hirschberger, S. Kushwaha, Z. Wang, Q. Gibson, S. Liang, C. A. Belvin, B. A. Bernevig, R. J. Cava, and N. P. Ong, Nature materials 15, 1161 (2016).
  * Shekhar et al. (2018) C. Shekhar, N. Kumar, V. Grinenko, S. Singh, R. Sarkar, H. Luetkens, S.-C. Wu, Y. Zhang, A. C. Komarek, E. Kampert, et al., Proceedings of the National Academy of Sciences 115, 9140 (2018).
  * Chen et al. (2020) J. Chen, H. Li, B. Ding, E. Liu, Y. Yao, G. Wu, and W. Wang, Applied Physics Letters 116, 222403 (2020).
  * Adler (1969) S. L. Adler, Physical Review 177, 2426 (1969).
  * Liang et al. (2018) S. Liang, J. Lin, S. Kushwaha, J. Xing, N. Ni, R. J. Cava, and N. P. Ong, Physical Review X 8, 031002 (2018).
  * Dos Reis et al. (2016) R. Dos Reis, M. Ajeesh, N. Kumar, F. Arnold, C. Shekhar, M. Naumann, M. Schmidt, M. Nicklas, and E. Hassinger, New Journal of Physics 18, 085006 (2016).
  * Hu et al. (2005) J. Hu, T. Rosenbaum, and J. Betts, Physical review letters 95, 186603 (2005).
  * Steele (1955) M. Steele, Physical Review 97, 1720 (1955).
  * Schumann et al. (2017) T. Schumann, M. Goyal, D. A. Kealhofer, and S. Stemmer, Physical Review B 95, 241113 (2017).
  * Usami (1978) K. Usami, Journal of the Physical Society of Japan 45, 466 (1978).
  * Yonggang et al. (2017) G. Y. Yonggang, X. Zhang, and A. Zunger, Physical Review B 95, 085201 (2017).
  * Öğüt and Rabe (1995) S. Öğüt and K. M. Rabe, Physical Review B 51, 10443 (1995).
  * Du et al. (2019) D. Du, A. Lim, C. Zhang, P. J. Strohbeen, E. H. Shourov, F. Rodolakis, J. L. McChesney, P. Voyles, D. C. Fredrickson, and J. K. Kawasaki, APL Materials 7, 121107 (2019).
  * Shourov et al. (2020) E. H. Shourov, R. Jacobs, W. A. Behn, Z. J. Krebs, C. Zhang, P. J. Strohbeen, D. Du, P. M. Voyles, V. W. Brar, D. D. Morgan, et al., Physical Review Materials 4, 073401 (2020).
  * Khalid et al. (2022) S. Khalid, H. S. Inbar, S. Chatterjee, C. J. Palmstrom, and A. Janotti, arXiv preprint arXiv:2208.05415 (2022).
  * Neubauer et al. (2009) A. Neubauer, C. Pfleiderer, B. Binz, A. Rosch, R. Ritz, P. Niklowitz, and P. Böni, Physical review letters 102, 186602 (2009).
  * Kumar et al. (2020) V. Kumar, N. Kumar, M. Reehuis, J. Gayles, A. Sukhanov, A. Hoser, F. Damay, C. Shekhar, P. Adler, and C. Felser, Physical Review B 101, 014424 (2020).
  * Xiong et al. (2015) J. Xiong, S. K. Kushwaha, T. Liang, J. W. Krizan, M. Hirschberger, W. Wang, R. J. Cava, and N. P. Ong, Science 350, 413 (2015).
  * Parameswaran et al. (2014) S. Parameswaran, T. Grover, D. Abanin, D. Pesin, and A. Vishwanath, Physical Review X 4, 031035 (2014).
  * Nielsen and Ninomiya (1983) H. B. Nielsen and M. Ninomiya, Physics Letters B 130, 389 (1983).
  * Hu et al. (2007) J. Hu, M. M. Parish, and T. Rosenbaum, Physical Review B 75, 214203 (2007).
  * Xu et al. (2019) J. Xu, M. K. Ma, M. Sultanov, Z.-L. Xiao, Y.-L. Wang, D. Jin, Y.-Y. Lyu, W. Zhang, L. N. Pfeiffer, K. W. West, et al., Nature communications 10, 1 (2019).
  * Ritchie et al. (2003) L. Ritchie, G. Xiao, Y. Ji, T. Chen, C. Chien, M. Zhang, J. Chen, Z. Liu, G. Wu, and X. Zhang, Physical Review B 68, 104430 (2003).
  * Borca et al. (2001) C. Borca, T. Komesu, H.-K. Jeong, P. A. Dowben, D. Ristoiu, C. Hordequin, J. Nozieres, J. Pierre, S. Stadler, and Y. Idzerda, Physical Review B 64, 052409 (2001).
  * Gerber (2018) A. Gerber, Physical Review B 98, 214440 (2018).
  * Ahadi et al. (2017) K. Ahadi, L. Galletti, and S. Stemmer, Applied Physics Letters 111, 172403 (2017).
  * Huang et al. (2022) D. Huang, H. Li, B. Ding, X. Xi, J. Gao, Y.-C. Lau, and W. Wang, Applied Physics Letters 121, 232404 (2022).
  * Yankovich et al. (2014) A. B. Yankovich, B. Berkels, W. Dahmen, P. Binev, S. I. Sanchez, S. A. Bradley, A. Li, I. Szlufarska, and P. M. Voyles, Nature communications 5, 4155 (2014).

##  VII Appendix

Figure 8: XRD $222$ and $202$ reflections used to extract the in plane and out
of plane lattice spacings, respectively. Figure 9: (a) Magnetoresistance
versus $\theta$ for the $x=0.85$ sample. (b) This measurement was performed in
a van der Pauw geometry. Figure 10: Two point versus four point resistivity
test to analyze the contribution of current jetting.


# Deep Reinforcement Learning for Intelligent Reflecting Surface-assisted D2D
Communications

Khoi Khac Nguyen, Antonino Masaracchia, Cheng Yin, Long D. Nguyen, Octavia A.
Dobre, and Trung Q. Duong K. K. Nguyen, A. Masaracchia, C. Yin, and T. Q.
Duong are Queen’s University Belfast, UK (e-mail:
{knguyen02,a.masaracchia,cyin01,trung.q.duong}@qub.ac.uk). L. D. Nguyen is
with Duy Tan University, Vietnam (email: dinhlonghcmut@gmail.com). O. A. Dobre
is with Memorial University, Canada (e-mail<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a deep reinforcement learning (DRL) approach for
solving the optimisation problem of the network’s sum-rate in device-to-device
(D2D) communications supported by an intelligent reflecting surface (IRS). The
IRS is deployed to mitigate the interference and enhance the signal between
the D2D transmitter and the associated D2D receiver. Our objective is to
jointly optimise the transmit power at the D2D transmitter and the phase shift
matrix at the IRS to maximise the network sum-rate. We formulate a Markov
decision process and then propose the proximal policy optimisation for solving
the maximisation game. Simulation results show impressive performance in terms
of the achievable rate and processing time.

###### Index Terms:

Intelligent reflecting surface (IRS), D2D communications, deep reinforcement
learning.

##  I Introduction

Device-to-device (D2D) communications play a critical role in 5G networks by
allowing users to communicate directly without the involvement of base
stations. It helps reduce the latency and improve the information transmission
efficiency [1, 2]. In [1], the optimised power allocation at the D2D
transmitters was proposed to maximise the energy efficiency (EE) performance,
by following a machine learning-based approach. In [2], the D2D transmitters
harvest energy through the simultaneous wireless information and power
transfer protocol (SWIPT). Then, a game theory approach was proposed to solve
the power allocation and power splitting at SWIPT with pricing strategies for
maximising the network performance.

Intelligent reflecting surface (IRS), referring to the technology of massive
elements of flexible reflection capability that are controlled by an
intelligent unit, has recently attracted great attention from the research
community as an efficient means to expand wireless coverage. The IRS can
manage the incoming signal by a controller, which allows to efficiently adapt
the angle of passive reflection from the transmitters toward the receivers [3,
4, 5, 6]. In [4], the IRS harvests energy from the access point (AP) and uses
it for reflecting the signal in two phases. The AP beamforming vector, the
IRS’s phase scheduling, and the passive beamforming were optimised to maximise
the information rate. In [5], a channel estimation scheme for a multi-user
multiple-input multiple-output (MIMO) system has been designed with the
support of double IRS panels.

Some research works have investigated the efficiency of the IRS in assisting
the D2D communications [7, 8]. In [7] and [8], two sub-problems with fixed
passive beamforming vector and fixed phase shift matrix were considered. To
solve the power allocation optimisation with the fixed phase shift matrix, the
authors in [7] used the gradient descent method while the authors in [8]
employed the Dinkelbach method. For the phase shift optimisation, a local
search algorithm was proposed in [7] while fractional programming was utilised
in [8]. However, these approaches assume a discrete phase shift and only reach
a sub-optimal solution. Moreover, these works only consider perfect
conditions, e.g., channel state information (CSI). In addition, these
algorithms cause large delays due to high computational complexity.

Very recently, deep reinforcement learning (DRL) has been applied as an
effective solution for solving complicated problems in wireless networks [9,
10, 11, 12, 13, 14]. In [9], we defined the discrete power level and used the
DRL algorithm to choose the transmit power at the D2D transmitter for
maximising the EE. In [11], discrete and continuous action spaces were
considered for the beamforming vector and the IRS phase shift in multiple-
input single-output (MISO) communications. Then, two DRL algorithms were used
to maximise the total throughput. In [12], a method based on the DRL was used
for optimising the unmanned aerial vehicle (UAV)’s altitude and the IRS
diagonal matrix to minimise the sum age-of-information. In [13], the authors
used the DRL technique to maximise the signal-to-noise ratio.

In this paper, we propose a DRL algorithm for solving the joint power
allocation and phase shift matrix optimisation in the IRS-assisted D2D
communications. Firstly, we conceive a D2D communication system with the
support of the IRS. The D2D channel is a combination of the direct link and
the reflective link. The IRS is used for mitigating the interference and
enhancing the information transmission channel. Secondly, we formulate a
Markov decision process (MDP) [15] for the network throughput maximisation in
the IRS-assisted D2D communications, in which the optimisation variables are
the power at the D2D users and the phase shifts at the IRS. Then, a DRL
algorithm is used to search for an optimal policy for maximising the network
sum-rate. Finally, we compare the efficiency of our proposed methods with
other schemes in terms of the achievable network sum-rate.

##  II System Model and Problem Formulation

We consider an IRS-assisted wireless network with $N$ pairs of D2D users
distributed randomly and an IRS panel, as shown in Fig. 1. Each pair of D2D
users comprises of a single-antenna D2D transmitter (D2D-Tx) and a single-
antenna D2D receiver (D2D-Rx). An IRS panel with $K$ reflective elements is
deployed to enhance the signal from the D2D-Tx to the associated D2D-Rx and
mitigate the interference from other D2D-Txs. The IRS with reflective elements
maps the receiver’s signal by the value of the phase shift matrix controlled
by an intelligent unit. The received signal at the D2D-Rx is composed of a
direct signal and a reflective one.

We denote the position of the $n$th D2D-Tx at time step $t$ as
$X^{t}_{n}(\text{Tx})=\big{(}x^{t}_{n}(\text{Tx}),y^{t}_{n}(\text{Tx})\big{)},n=1,\dots,N$
and that of the $m$th D2D-Rx as
$X^{t}_{m}(\text{Rx})=\big{(}x^{t}_{m}(\text{Rx})),y^{t}_{m}(\text{Rx})\big{)},m=1,\dots,N$.
The IRS is fixed at the position
$(x^{t}_{\text{IRS}},y^{t}_{\text{IRS}},z^{t}_{\text{IRS}})$. The phase shift
value of each element in the IRS belongs to $[0,2\pi]$.

Figure 1: System model of the IRS-assisted D2D communications.

We denote the direct channel from the $n$th D2D-Tx to the $m$th D2D-Rx at time
step $t$ by $h_{nm}^{t}$, and the reflective channel by $H^{t}_{nm}$. The
phase shift matrix at the IRS at time step $t$ is defined by
$\Phi^{t}=\operatorname{\mathrm{diag}}(\eta_{2}^{t}\theta_{1}^{t},\eta_{2}^{t}\theta_{2}^{t},\dots,\eta_{K}^{t}\theta_{K}^{t})$,
where $\eta_{k}^{t}\in[0,1]$ and $\theta_{k}^{t}\in[0,2\pi]$ represent the
amplitude and the phase shift value, respectively. In this paper, we assume
that the amplitudes of all elements are set to $\eta_{k}^{t}=1$.

The distance between the $n$th D2D-Tx and the $m$th D2D-Rx at time step $t$ is
defined as

$d_{nm}^{t}=\sqrt{\big{(}x^{t}_{n}(\text{Tx})-x^{t}_{m}(\text{Rx})\big{)}^{2}+\big{(}y^{t}_{n}(\text{Tx})-y^{t}_{m}(\text{Rx})\big{)}^{2}}.$
(1)

Similarly, the distance between the $n$th D2D-Tx and the IRS is
$d_{n,IRS}^{t}$ and the distance between the IRS and the $m$th D2D-Rx is
$d_{IRS,m}^{t}$ at time step $t$. The direct channel is formulated as

$h_{nm}^{t}=\hat{h}_{n}\sqrt{\beta_{0}(d^{t}_{nm})^{-\kappa_{0}}},$ (2)

where $\beta_{0}$ and $\hat{h}_{n}$ are the channel power gain at the
reference distance $d_{0}=1$ m and $\kappa_{0}$ is the path-loss exponent in
the D2D link. Here, we assume that the small-scale fading follows the
Nakagami-$m$ distribution with $m$ as the fading severity parameter.

The reflective channel via the IRS from the $n$th D2D-Tx toward the $m$th
D2D-Rx is considered as a Rician fading channel at time step $t$ described by

$H_{nm}^{t}=\sqrt{\frac{\beta_{1}}{1+\beta_{1}}}\tilde{h}^{LoS}_{nm}+\sqrt{\frac{1}{\beta+1}}\tilde{h}^{NLoS}_{nm},$
(3)

where $\beta_{1}$ is the Rician factor, and $\tilde{h}^{LoS}_{nm}$,
$\tilde{h}^{NLoS}_{nm}$ are the line-of-sight (LoS) and the non-line-of-sight
(NLoS) components for the reflected channel, respectively. Specifically, the
LoS component is defined as [7]

$\tilde{h}^{LoS}_{nm}=\sqrt{\beta_{0}(d_{n,IRS}^{t}d_{IRS,m}^{t})^{-\kappa_{0}}}e^{-j\theta^{\prime}},$
(4)

where $\theta^{\prime}\in[0,2\pi]$ is the random phase. The NLoS component is
defined as

$\tilde{h}^{NLoS}_{nm}=\sqrt{\beta_{0}(d_{n,IRS}^{t}d_{IRS,m}^{t})^{-\kappa_{1}}}\hat{h}^{NLoS}_{nm},$
(5)

where $\kappa_{1}$ is the path loss exponent for the NLoS component and the
small-scale fading $\hat{h}^{NLoS}_{nm}\sim\mathcal{C}\mathcal{N}(0,1)$ is
i.i.d. complex Gaussian distribution with zero mean and unit variance.

The received signal at the $n$th D2D-Rx at time step $t$ can be written as

$\begin{split}s_{n}^{t}=\Big{(}h_{nn}^{t}&+\sum^{K}_{k=1}H_{nn}^{t}\Phi^{t}\Big{)}\sqrt{p_{n}^{t}}u_{n}^{t}\\\
&+\sum_{m\neq
n}^{N}\Big{(}h_{mn}^{t}+\sum^{K}_{k=1}H_{mn}^{t}\Phi^{t}\Big{)}\sqrt{p_{m}^{t}}u_{m}^{t}+\varpi,\end{split}$
(6)

where $p_{n}^{t}$ is the transmit power at the $n$th D2D-Tx at time step $t$,
$u_{n}^{t}$ is the transmitted symbol from the $n$th D2D-Tx, and
$\varpi\sim\mathcal{N}(0,\alpha^{2})$ is the complex additive white Gaussian
noise.

Accordingly, the received signal-to-interference-plus-noise ratio (SINR) at
the $n$th D2D-Rx can be represented as

$\gamma_{n}^{t}=\frac{|h_{nn}^{t}+\sum^{K}_{k=1}H_{nn}^{t}\Phi^{t}|^{2}p_{n}^{t}}{\sum_{m\neq
n,m\in
N}|h_{mn}^{t}+\sum^{K}_{k=1}H_{mn}^{t}\Phi^{t}|^{2}p_{m}^{t}+\alpha^{2}}.$ (7)

The achievable sum-rate at the $n$th D2D pair during time step $t$ is defined
as

$R_{n}^{t}=B\log_{2}(1+\gamma_{n}^{t}),$ (8)

where $B$ is the bandwidth.

In this paper, we aim at optimising the power allocation of all $N$ pairs of
D2D users $P=\\{p_{1},p_{2},\dots,p_{N}\\}$ and the phase shift matrix $\Phi$
of the IRS to maximise the network sum-rate while satisfying all the
constraints. The considered network optimisation can be formulated as follows:

$\begin{split}\max_{P,\Phi}\quad&R_{total}^{t}=\sum_{n=1}^{N}R_{n}^{t}\\\
s.t.\quad&0<p_{n}<P_{\max},\forall n\in N\\\ &R_{n}^{t}\geq r_{\min},\forall
n\in N\\\ &\theta_{k}\in[0,2\pi],\forall k\in K,\\\ \end{split}$ (9)

where $P_{\max}$ is the maximum transmit power at the D2D-Tx and the
constraint $R_{n}^{t}\geq r_{\min},\forall n\in N$ indicates the quality-of-
service (QoS) of the D2D communications.

##  III Joint Optimisation of Power Allocation and Phase Shift Matrix

Given the optimisation problem (9), we formulate the MDP with the agent, the
state space $\mathcal{S}$, the action space $\mathcal{A}$, the transition
probability $\mathcal{P}$, the reward function $\mathcal{R}$ and the discount
factor $\zeta$. Let us denote $\mathcal{P}_{ss^{\prime}}(a)$ as the
probability when the agent takes action $a^{t}\in\mathcal{A}$ at the state
$s=s^{t}\in\mathcal{S}$ and transfers to the next state
$s^{\prime}=s^{t+1}\in\mathcal{S}$. In particular, we formulate the MDP game
as follows:

  * •

_State space_ : The channel gain of the D2D users forms the state space as

$\displaystyle\mathcal{S}$
$\displaystyle=\Big{\\{}h_{11}+\sum_{k=1}^{K}H_{11}\Phi,\dots,h_{1N}+\sum_{k=1}^{K}H_{1N}\Phi,\dots,$
$\displaystyle
h_{nm}+\sum_{k=1}^{K}H_{nm}\Phi,\dots,h_{nN}+\sum_{k=1}^{K}H_{nN}\Phi,\dots,$
$\displaystyle
h_{N1}+\sum_{k=1}^{K}H_{N1}\Phi,\dots,h_{NN}+\sum_{k=1}^{K}H_{NN}\Phi\Big{\\}}.$
(10)

  * •

_Action space_ : The D2D-Txs adjust the transmit power and the IRS changes the
phase shift for maximising the expected reward. Thus, The action space for the
D2D users and the IRS is considered as follows:

$\mathcal{A}=\\{p_{1},p_{2},\dots,p_{N},\theta_{1},\theta_{2},\dots,\theta_{K}\\}.$
(11)

  * •

_Reward function_ : The agent needs to find an optimal policy for maximising
the reward. In our problem, our objective is to maximise the network sum-rate;
thus, the reward function is defined as $\mathcal{R}=$

$\sum_{n=1}^{N}B\log_{2}\Bigg{(}1+\frac{|h_{nn}+\sum^{K}_{k=1}H_{nn}\Phi|^{2}p_{n}}{\sum_{m\neq
n}^{N}|h_{mn}+\sum^{K}_{k=1}H_{mn}\Phi|^{2}p_{m}+\alpha^{2}}\Bigg{)}.$ (12)

By following the MDP, the agent interacts with the environment and receives
the response to achieve the best expected reward. Particularly, the state of
the agent at time step $t$ is $s^{t}$. The agent chooses and executes the
action $a^{t}$ under the policy $\pi$. The environment responds with the
reward $r^{t}$. After taking the action $a^{t}$, the agent moves to the new
state $s^{t+1}$ with probability $P_{ss^{\prime}}(a)$. The interactions are
iteratively executed and the policy is updated for the optimal reward.

In this paper, we propose a DRL approach to search for an optimal policy for
maximising the reward value in (12). The optimal policy can be obtained by
modifying the estimation of the value function or directly by the objective.
We use an on-policy algorithm for our work, namely proximal policy
optimisation (PPO) with the clipping surrogate technique [16]. Consider the
probability ratio of the current policy and obtained policy
$p^{t}_{\theta}=\frac{\pi(s,a;\theta)}{\pi(s,a;\theta_{old})}$, we need to
find the optimal policy to maximise the total expected reward as follows:

$\mathcal{L}(s,a;\theta)\\!=\\!\mathbb{E}\\!\Bigg{[}\\!\frac{\pi(s,a;\theta)}{\pi(s,a;\theta_{old})}A^{\pi}(s,a)\Bigg{]}\\!=\\!\mathbb{E}\\!\Bigg{[}\\!p^{t}_{\theta}A^{\pi}(s,a)\\!\Bigg{]},$
(13)

where $\mathbb{E}[\cdot]$ is the expectation operation and
$A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s)$ denotes the advantage function [17];
$V^{\pi}(s)$ denotes the state-value function while $Q^{\pi}(s,a)$ is the
action-value function.

In the PPO method, we limit the current policy such that it does not go far
from the obtained policy by using different techniques, e.g., the clipping
technique and Kullback-Leiber [17]. In this work, we use the clipping
surrogate method to prevent the excessive modification of the objective value,
as follows:

$\begin{split}\mathcal{L}^{\operatorname*{\mathrm{clip}}}(s,a;\theta)=\mathbb{E}\Bigg{[}&\min\Big{(}p^{t}_{\theta}A^{\pi}(s,a),\\\
&\operatorname*{\mathrm{clip}}(p^{t}_{\theta},1-\epsilon,1+\epsilon)A^{\pi}(s,a)\Big{)}\Bigg{]},\end{split}$
(14)

where $\epsilon$ is a hyperparameter.

When the advantage $A^{\pi}(s,a)$ is positive, the term $(1+\epsilon)$ takes
action. Meanwhile, for the negative case of the advantage $A^{\pi}(s,a)$, the
term $(1-\epsilon)$ sets a ceiling to limit the objective value. Moreover, for
the advantage function $A^{\pi}(s,a)$, we use [18]:

$A^{\pi}(s,a)=r^{t}+\zeta V^{\pi}(s^{t+1})-V^{\pi}(s^{t}),$ (15)

where the state-value function $V^{\pi}(s)$ is obtained at the state $s$ under
the policy $\pi$ as follows:

$V^{\pi}=\mathbb{E}\Big{\\{}\mathcal{R}|s,\pi\Big{\\}}.$ (16)

To train the policy network, we store the transition into a mini-batch memory
$D$ and then use the stochastic policy gradient (SGD) method to maximise the
objective. By denoting the policy parameter by $\theta$, it is updated as

$\theta^{d+1}=\operatorname*{arg\,max}\mathbb{E}\Big{[}\mathcal{L}(s,a;\theta^{d})\Big{]}.$
(17)

The PPO algorithm for joint optimisation of the transmit power and the phase
shift matrix in the IRS-aided D2D communications is presented in Algorithm 1,
where $M$ denotes the maximum number of episodes and $T$ is the number of
iterations during a period of time.

Algorithm 1 Proposed approach based on the PPO algorithm for the IRS-assisted
D2D communications.

1: Initialise the policy $\pi$ with the parameter $\theta_{\pi}$

2: Initialise other parameters

3: for episode = $1,\dots,M$ do

4: Receive initial observation state $s^{0}$

5: for iteration = $1,\dots,T$ do

6: Obtain the action $a^{t}$ at state $s^{t}$ by following the current policy

7: Execute the action $a^{t}$

8: Receive the reward $r^{t}$ according to (12)

9: Observe the new state $s^{t+1}$

10: Update the state $s^{t}=s^{t+1}$

11: Collect set of partial trajectories with $D$ transitions

12: Estimate the advantage function according to (15)

13: end for

14: Update policy parameters using SGD with mini-batch $D$
$\theta^{t+1}=\operatorname*{arg\,max}\frac{1}{D}\sum^{D}\mathcal{L}^{\operatorname*{\mathrm{clip}}}(s,a;\theta^{t})$
(18)

15: end for

##  IV Simulation Results

For numerical results, we use Tensorflow 1.13.1 [19]. The IRS is deployed at
$(0,0,0)$, while the D2D devices are randomly distributed within a circle of
$100$ m from the center. The maximum distance between the D2D-Tx and the
associated D2D-Rx is set to $10$ m. We assume $d/\lambda=1/2$, and set the
learning rate for the PPO algorithm to $0.0001$. For the neural networks, we
initialise two hidden layers with $128$ and $64$ units, respectively. All
other parameters are provided in Table I. We consider the following algorithms
in the numerical results.

  * •

The proposed algorithm: We use the PPO algorithm with the clipping surrogate
technique to solve the joint optimisation of the power allocation and the
phase shift matrix of the IRS.

  * •

Maximum power transmission (MPT): The D2D-Tx transmits information with
maximum power, $P_{\max}$. We use the PPO algorithm to optimise the phase
shift matrix of the IRS panel.

  * •

Random phase shift matrix selection (RPS): We optimise the power allocation at
the D2D-Tx with random selection of the phase shift matrix $\Phi$.

  * •

Without IRS: The D2D-Tx transmits information without the support of the IRS.
We optimise the power allocation by using the PPO algorithm.

TABLE I: SIMULATION PARAMETERS. Parameters | Value
---|---
Bandwidth ($W$) |  $1$ MHz
Path-loss parameter | $\kappa_{0}=2.5,\kappa_{1}=3.6$
Channel power gain |  $-30$ dB
Rician factor | $\beta_{1}=4$
Noise power |  $\alpha^{2}=-80$ dBm
Clipping parameter | $\epsilon=0.2$
Discounting factor | $\zeta=0.9$
Max number of D2D pairs | $10$
Initial batch size | $K=128$

Firstly, we compare the achievable network sum-rate provided by our proposed
algorithm with that of other schemes. Fig. 2 plots the sum-rate versus
different numbers of the IRS elements, $K$, where the number of D2D pairs is
set to $N=5$. As can be observed from this figure, the PPO algorithm-based
technique outperforms other schemes and is followed by the MPT technique. The
RP and WithoutIRS schemes show poorer performance in terms of the network sum-
rate. The achievable network sum-rate using our proposed algorithm and MPT
improves with increasing the number of IRS elements. The results show that
with the monotonic increase in the value of $K$, the communication quality
between the D2D-Tx and associated D2D-Rx is enhanced, while the interference
from other D2D-Txs is suppressed.

Figure 2: The network sum-rate versus the number of IRS elements, $K$.

Next, the performance of the previously mentioned four schemes is compared
while varying the number of D2D pairs, $N$, in Fig. 3. We set the number of
IRS element to $K=20$ and take the average over $500$ episodes to obtain the
results. Our proposed algorithm shows better performance, followed by MPT.
With higher number of D2D users, $N\geq 6$, the performance attained by the
proposed algorithm still increases while it decreases for the other schemes.
The RPS and WithoutIRS models show the worse performance.

Figure 3: The network sum-rate versus the number of D2D pairs, $N$.

Further, we set $N=5$, $K=20$ and compare the performance results of the four
schemes while changing the value of the threshold, $r_{\min}$, in Fig. 4. When
the value of $r_{\min}$ increases towards infinity, the number of D2D pairs
that satisfies the QoS constraints decreases and the sum-rate of all schemes
tends to $0$. The proposed algorithm outperforms the other schemes for all
values of $r_{\min}$. The gap between our algorithm and others increases
following the increase in $r_{\min}$ when $r_{\min}\geq 5$. The MPT algorithm
exhibits the worst performance when $r_{\min}\geq 7$. This suggests that the
optimisation of power allocation is important for efficient D2D
communications.

Figure 4: The network sum-rate versus the QoS threshold, $r_{\min}$.

Next, we compare the total sum-rate of the four schemes by setting different
maximum transmission powers at the D2D-Tx, $P_{\max}$, in Fig. 5, with $N=5$,
$K=20$. As $P_{\max}$ varies from $100$ mW to $400$ mW, the performance of the
four schemes increases in the same upward trend. The gap between our proposed
algorithm and the other schemes increases with the increase value of
$P_{\max}$ as we jointly optimise both power allocation at the D2D-Tx and the
IRS’s phase shift matrix. It is clear that the proposed algorithm is more
effective for mitigating interference and providing a better communication
quality.

Furthermore, we use neural networks for establishing the DRL algorithm. Thus,
after iterative interactions with the environment, the neural networks are
trained for achieving an optimal solution. After training offline, the neural
network can be deployed to the system for online execution. The online neural
networks can determine the proper action for the IRS phase shift value and the
D2D-Tx power allocation for maximising the network sum-rate in real-time.

Figure 5: The network sum-rate versus the maximum transmit power, $P_{\max}$.

##  V Conclusion

In this paper, we have presented a DRL-based optimal resource allocation
scheme for IRS-assisted D2D communications. The PPO algorithm with the
clipping surrogate technique has been proposed for joint optimisation of the
D2D-Tx power and the IRS’s phase shift matrix. Numerical results have showed a
significant improvement in the achievable network sum-rate performance
compared with the benchmark schemes. Our proposed scheme demonstrates the
superiority of using IRS in mitigating the interference in the D2D
communications when compared with other existing schemes.

## References

  * [1] K. K. Nguyen, T. Q. Duong, N. A. Vien, N.-A. Le-Khac, and N. M. Nguyen, “Non-cooperative energy efficient power allocation game in D2D communication: A multi-agent deep reinforcement learning approach,” _IEEE Access_ , vol. 7, pp. 100 480–100 490, Jul. 2019.
  * [2] J. Huang, C.-C. Xing, and M. Guizani, “Power allocation for D2D communications with SWIPT,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 4, pp. 2308–2320, Apr. 2020.
  * [3] H. Yu, H. D. Tuan, A. A. Nasir, T. Q. Duong, and H. V. Poor, “Joint design of reconfigurable intelligent surfaces and transmit beamforming under proper and improper Gaussian signaling,” _IEEE J. Select. Areas Commun._ , vol. 38, no. 11, pp. 2589–2603, Nov. 2020.
  * [4] Y. Zou, S. Gong, J. Xu, W. Cheng, D. T. Hoang, and D. Niyato, “Wireless powered intelligent reflecting surfaces for enhancing wireless communications,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 10, pp. 12 369–12 373, Oct. 2020.
  * [5] B. Zheng, C. You, and R. Zhang, “Efficient channel estimation for double-IRS aided multi-user MIMO system,” _IEEE Trans. Commun._ , vol. 69, no. 6, pp. 3818–3832, Jun. 2021.
  * [6] K. K. Nguyen, S. Khosravirad, L. D. Nguyen, T. T. Nguyen, and T. Q. Duong, “Intelligent reconfigurable surface-assisted multi-UAV networks: Efficient resource allocation with deep reinforcement learning,” 2021. [Online]. Available: https://arxiv.org/abs/2105.14142
  * [7] Y. Chen, B. Ai, H. Zhang, Y. Niu, L. Song, Z. Han, and H. V. Poor, “Reconfigurable intelligent surface assisted device-to-device communications,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 5, pp. 2792–2804, May 2021.
  * [8] S. Jia, X. Yuan, and Y.-C. Liang, “Reconfigurable intelligent surfaces for energy efficiency in D2D communication network,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 3, pp. 683–687, Mar. 2021.
  * [9] K. K. Nguyen, T. Q. Duong, N. A. Vien, N.-A. Le-Khac, and L. D. Nguyen, “Distributed deep deterministic policy gradient for power allocation control in D2D-based V2V communications,” _IEEE Access_ , vol. 7, pp. 164 533–164 543, Nov. 2019.
  * [10] K. K. Nguyen, N. A. Vien, L. D. Nguyen, M.-T. Le, L. Hanzo, and T. Q. Duong, “Real-time energy harvesting aided scheduling in UAV-assisted D2D networks relying on deep reinforcement learning,” _IEEE Access_ , vol. 9, pp. 3638–3648, Dec. 2021.
  * [11] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Select. Areas Commun._ , vol. 38, no. 8, pp. 1839–1850, Aug. 2020.
  * [12] M. Shokry, M. Elhattab, C. Assi, S. Sharafeddine, and A. Ghrayeb, “Optimizing age of information through aerial reconfigurable intelligent surfaces: A deep reinforcement learning approach,” _IEEE Trans. Veh. Technol._ , vol. 70, no. 4, pp. 3978–3983, Apr. 2021.
  * [13] K. Feng, Q. Wang, X. Li, and C.-K. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 745–749, May 2020.
  * [14] K. K. Nguyen, T. Q. Duong, T. Do-Duy, H. Claussen, and L. Hanzo, “3D UAV trajectory and data collection optimisation via deep reinforcement learning,” 2021. [Online]. Available: https://arxiv.org/abs/2106.03129
  * [15] D. P. Bertsekas, _Dynamic Programming and Optimal Control_. Athena Scientific Belmont, MA, 1995, vol. 1, no. 2.
  * [16] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
  * [17] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” in _Proc. 4th International Conf. Learning Representations (ICLR)_ , 2016.
  * [18] V. Mnih _et al._ , “Asynchronous methods for deep reinforcement learning,” in _Proc. Int. Conf. Mach. Learn._ PMLR, 2016, pp. 1928–1937.
  * [19] M. Abadi _et al._ , “Tensorflow: A system for large-scale machine learning,” in _Proc. 12th USENIX Sym. Opr. Syst. Design and Imp. (OSDI 16)_ , Nov. 2016, pp. 265–283.