IR-Improved Amplitude-Based Resummation in Quantum Field Theory: New Results
and New Issues

B.F.L. Ward1$\star$, S. Jadach2, W. Placzek3, M. Skrzypek2, Z. Was2 and S.A.
Yost4

1 Baylor University, Waco, TX, USA

2 Institute of Nuclear Physics, Krakow, Poland

3 Institute of Applied Computer Science, Jagiellonian University, Krakow,
Poland

4 The Citadel, Charleston, SC, USA

*<EMAIL_ADDRESS>
BU-HEPP-21-05

15th International Symposium on Radiative Corrections:
Applications of Quantum Field Theory to Phenomenology,
FSU, Tallahasse, FL, USA, 17-21 May 2021 10.21468/SciPostPhysProc.?

## Abstract

With the advancement of strategies for the precision physics programs for the
HL-LHC, FCC-ee, FCC-hh, ILC, CLIC, CEPC, and CPPC, the need for proper control
of the attendant theoretical precision tags is manifest. We discuss the role
that amplitude-based resummation may play in this regard with examples from
the LHC, the proposed new colliders and quantum gravity.

###### Contents

  1. 1 Introduction
  2. 2 Review of Exact Amplitude-Based Resummation Theory
  3. 3 Precision LHC Physics: New Results and New Issues
  4. 4 Precision FCC Physics: New Results and New Issues
  5. 5 Quantum Gravity: New Results and New Issues
  6. 6 Summary

##  1 Introduction

The subject of resummation in quantum field theory has applicability in the
infrared (IR), collinear (CL), and ultraviolet (UV) limits. In what follows,
we focus on the IR limit. Independent of the limit in which one works, one can
distinguish two general types of resummation: exact resummation and
approximate resummation. Inside these two general classes, one has further
sub-classes as we will illustrate. We start by defining the two general
classes.

Toward this end, we recall that, even in elementary examples of summation,
such as $\sum_{n=0}^{\infty}x^{n}=1/(1-x)$, infinite order summation can
greatly improve the behavior of the series relative to what one concludes from
an examination of the series term-by-term. With this motivational background,
we note that, in resummation, where we resum existing infinite order sums in
quantum field theory, we can work either at the level of the Feynman
amplitudes or at the level of observable cross sections. In either case, we
can distinguish two types of resummation of the attendant Feynman series as
follows:

$\sum_{n=0}^{\infty}C_{n}(Q,m)\tilde{\alpha}^{n}=\begin{cases}F_{\text{
RES}}(Q,m,\tilde{\alpha})\sum_{n=0}^{\infty}B_{n}(Q,m)\tilde{\alpha}^{n},{\text{EXACT}}\\\
G_{\text{
RES}}(Q,m,\tilde{\alpha})\sum_{n=0}^{N}\tilde{B}_{n}(Q,m)\tilde{\alpha}^{n},{\text{APPROX}},\end{cases}$
(1)

where $\tilde{\alpha}$ is the coupling constant expansion parameter (it is
$\alpha_{s}$ in QCD, for example) and $N$ characterizes the order of the
exactness of the approximate case labeled as ’APPROX’ here while the first
result on the right-hand side of the last equation defines what we call
’EXACT’ resummation. $Q$ and $m$ are generic representatives of momentum-
dependent variables and masses that may enter the respective amplitudes or
cross sections. As we have described in Ref. [1], there is a long history of
the comparison and competition between these two classes of resummation in
quantum field theory. Generally, there is a limit to the theoretical precision
tag associated with the APPROX result determined by $N$ while, in principle,
the EXACT result has no such limit. In what follows, we use the EXACT
representation of resummation in new paradigms to probe new issues in
precision LHC/FCC physics and in quantum gravity.

As it has been noted in Refs. [2, 3], we have recently passed 50 years of the
Standard Theory[4, 5, 6, 7, 8, 9]111We follow Prof. D.J. Gross [10] and
henceforth refer to the Standard Model as the Standard Theory of elementary
particles. of elementary particles. Given the current status of the
observations at the LHC [11, 12], one has to keep a proper historical
perspective with an eye toward the future. In Ref. [13], one can see a path
forward led by experiments at future energy frontier colliders (The interplay
between experiment and theory is crucial to the progress of physics.). This
allows us to look to the future with optimism.

The discussion is organized as follows. In the next Section, we review the
representation of EXACT resummation which we will use. The discussion of its
application to obtain new results with new issues in precision LHC physics is
given in Section 3. The analogous discussion for the FCC is given in Section
4. Section 5 discusses quantum gravity in this context. Section 6 contains our
concluding remarks.

##  2 Review of Exact Amplitude-Based Resummation Theory

We include here a brief review of exact amplitude-based resummation theory, as
it is still not generally familiar. The theory is encoded by the following
master formula:

$\displaystyle d\bar{\sigma}_{\rm res}=e^{\rm
SUM_{IR}(QCED)}\sum_{{n,m}=0}^{\infty}\frac{1}{n!m!}\int\prod_{j_{1}=1}^{n}\frac{d^{3}k_{j_{1}}}{k_{j_{1}}}$
(2)
$\displaystyle\prod_{j_{2}=1}^{m}\frac{d^{3}{k^{\prime}}_{j_{2}}}{{k^{\prime}}_{j_{2}}}\int\frac{d^{4}y}{(2\pi)^{4}}e^{iy\cdot(p_{1}+q_{1}-p_{2}-q_{2}-\sum
k_{j_{1}}-\sum{k^{\prime}}_{j_{2}})+D_{\rm QCED}}$ (3)
$\displaystyle{\tilde{\bar{\beta}}_{n,m}(k_{1},\ldots,k_{n};k^{\prime}_{1},\ldots,k^{\prime}_{m})}\frac{d^{3}p_{2}}{p_{2}^{\,0}}\frac{d^{3}q_{2}}{q_{2}^{\,0}},$
(4)

where the new222The non-Abelian nature of QCD requires a new treatment of the
corresponding part of the IR limit.(YFS-style) residuals
$\tilde{\bar{\beta}}_{n,m}(k_{1},\ldots,k_{n};k^{\prime}_{1},\ldots,k^{\prime}_{m})$
have $n$ hard gluons and $m$ hard photons. The new residuals and the infrared
functions ${\rm SUM_{IR}(QCED)}$ and ${D_{\rm QCED}}$ are defined in Ref. [14,
15]. Parton shower/ME matching leads to the replacements
$\tilde{\bar{\beta}}_{n,m}\rightarrow\hat{\tilde{\bar{\beta}}}_{n,m}$, which
allow us to connect with MC@NLO [16, 17], via the basic formula

${d\sigma}=\sum_{i,j}\int
dx_{1}dx_{2}{F_{i}(x_{1})F_{j}(x_{2})}d\hat{\sigma}_{\rm res}(x_{1}x_{2}s),$
(5)

as explained in Ref. [14, 15].

We have used Eq.(4) to obtain results in precision LHC and FCC physics and we
have extended it to general relativity as an approach to quantum gravity. In
each respective application, our new results are accompanied with new issues.
We discuss such new results and new issues in precision LHC physics in the
next Section.

##  3 Precision LHC Physics: New Results and New Issues

The large data sample of Z’s and W’s at the LHC affords the opportunity for
precision EW studies as evidence by the ATLAS state-of-the-art measurement of
$M_{W}$ in Ref. [18]. The effort to make an analogous state-of-the-art
measurement of the weak mixing angle via $\sin^{2}\theta_{W}^{eff}$ is in
progress in Ref. [19]. In this context, four of us (SJ, BFLW, SAY, ZW) have
introduced the MC ${\cal KK}$MC-hh [20, 21, 22] which realizes exact ${\cal
O}(\alpha^{2}L)$ CEEX [23, 24] EW corrections for hadron-hadron scattering
processes with built-in Herwig6.5 and Herwiri1.031 showers as well as with an
LHE[25] format for interfacing to other parton shower MC’s. In the studies
aimed at extracting observables such as $\sin^{2}\theta_{W}^{eff}$ from the
precision analysis of the Z decays to lepton pairs at the LHC, the IR-
improvement in ${\cal KK}$MC-hh opens the way for quantifying the effects of
the ISR on such observables.

We focus here on angular variables in $pp\rightarrow
Z/\gamma^{*}\rightarrow\ell\bar{\ell},\ell=e^{-},\;\mu^{-},$ augmented with
the dilepton mass, $M_{\ell\ell}$, and rapidity, $Y_{\ell\ell},$ and consider
distributions of the angle $\theta_{CS}$ of $\ell$ defined in the Collins-
Soper [26] frame – the CM frame of $\ell\bar{\ell}$, referenced to a $z$-axis
oriented as shown in Fig. 1.

Figure 1: Angular variables in the Collins-Soper frame.

For $P=p_{\ell}+p_{\bar{\ell}},\;p^{\pm}=p^{0}\pm p^{z}$ in the Lab, we have

$\cos(\theta_{CS})=sgn(P^{z})\frac{p_{\ell}^{+}p_{\bar{\ell}}^{-}-p_{\ell}^{-}p_{\bar{\ell}}^{+}}{\sqrt{P^{2}P^{+}P^{-}}}.$
(6)

Using the angular distribution for $\theta_{CS}$, with an eye toward
$\sin^{2}\theta_{W}^{eff}$, we may study, as we have done in Ref. [22], the
observables $A_{4}$ and $A_{FB}$, which we define as

$A_{4}=\frac{4}{\sigma}\int\cos\theta_{CS}d\sigma=4<\cos\theta_{CS}>$

,

$A_{FB}=\frac{1}{\sigma}\int\operatorname{sgn}(\cos\theta_{CS})d\sigma=<\operatorname{sgn}(\cos\theta_{CS})>$

where we follow the notation of Ref. [27] for the angular coefficients
$A_{i},\;i=0,\ldots,7$ in the respective differential cross section
$d\sigma(\theta_{CS},\phi_{CS})$ for the lepton in Fig. 1. To illustrate our
results, we start by showing in Fig. 2, as a cross-check, the dependence of
the size of the ISR and IFI effects on $A_{4}$, plotted as a function of
$M_{\ell\ell}$, on the choice of PDF set when we feature results based on the
NNPDF 3.1 NLO PDF set [28] and on the MMHT2014 NLO set [29]. We see that the
${\cal KK}$MC-hh results with the two sets are consistent within the errors on
the respective sets.

Figure 2: PDF comparison for $A_{4}(\frac{8}{3}A_{FB})$

as a function of $M_{\ell\ell}$ with no lepton cuts: (a) shows results
calculated with NNPDF3.1 NLO, (b) shows results calculated with MMHT2014 NLO.
LUXQED denotes the QED-PDF realized by NNPDF3.1 and MMHT2014 following the
approach of Ref. [30].

In both cases, we see that the effects of the ISR are significant at the level
of precision relevant for the expected uncertainty for the current LHC data
analysis [19]. We also see from the difference between the ISR from ${\cal
KK}$MChh and that from LUXQED that the transverse degrees of freedom in the
photon radiation in ${\cal KK}$MChh are significant.

A new issue that our approach raises is the role of quark masses in precision
LHC EW physics. In this context, we show in Fig. 3, using results from ${\cal
KK}$MChh,

Figure 3: Mass comparison for $A_{4}$

as a function of $M_{\ell\ell}$ with no lepton cuts: (a) shows results
calculated with $m_{u}=2.2MeV,\;m_{d}=4.7MeV$, (b) shows results with these
masses increased by a factor of 10.

the effect of changing the quark masses by a factor 10 on the size of the ISR
and the IFI in $A_{4}(\frac{8}{3}A_{FB})$, where the word ’heavy’ in the
figure refers to this factor of 10. The results, which again feature $A_{4}$
plotted as a function of $M_{\ell\ell}$ and are generated with the NNPDF3.1
NLO PDF set, show that the ISR is not very much affected by the change but
that there is some effect on the IFI.

The role of the quark masses as observable parameters obtains in Figs. 3 –
they are not just a collinear regulators. This brings us the the issue of the
input data at $Q_{0}\sim 1\;\text{GeV}$ for the non-QED PDFs used in ${\cal
KK}$MChh. Is it a double counting if the collinearly singular quark mass terms
are not removed? In local, relativistic quantum field theory, processes at
different space-time regimes cannot double count one another. Thus, the quark
mass singular terms in the input data at $Q_{0}\sim 1\;\text{GeV}$ do not
double the quark mass singular terms at $Q\sim M_{Z}$; rather, the former
produce an error in the probability to find the quark at the scale $Q\sim
M_{Z}$, an error well below the error on the PDF itself. Since it is possible
[31] to remove the quark mass singular terms from the input data at $Q_{0}\sim
1\;\text{GeV}$, it will soon be possible to remove this small error.

##  4 Precision FCC Physics: New Results and New Issues

The FCC project received a strong endorsement from the CERN Council [13]. One
possible scenario invovles the first stage with the FCC-ee for precision
physics studies with more than 5 Tera Z’s. For such studies, the $e^{+}e^{-}$
luminosity theory error needs to be controlled at the precision of 0.01% and
five of us (SJ,WP,MS,BFLW,SAY) have shown [32] that, with sufficient
resources, this theoretical precision tag can be realized by upgrading the LEP
era state-of-the-art MC BHLUMI4.04 [33] from its current precision tag, which
was recently shown to be $0.37\%$ in Ref. [34], to the desired 0.01%. The
steps required for this upgrade are discussed in detail Ref. [32]. For
completeness, we show the resultant error budget for the upgraded BHLUMI in
Table 1.

Type of correction / Error | Update 2019 | FCC-ee forecast
---|---|---
(a) Photonic $[{\cal O}(L_{e}\alpha^{2})]\;{\cal O}(L_{e}^{2}\alpha^{3})$ | 0.027% | $0.1\times 10^{-4}$
(b) Photonic $[{\cal O}(L_{e}^{3}\alpha^{3})]\;{\cal O}(L_{e}^{4}\alpha^{4})$ | 0.015% | $0.6\times 10^{-5}$
(c) Vacuum polariz. | 0.013% [35, 36] | $0.6\times 10^{-4}$
(d) Light pairs | 0.010% [37, 38] | $0.5\times 10^{-4}$
(e) $Z$ and $s$-channel $\gamma$ exchange | 0.090% [39] | $0.1\times 10^{-4}$
(f) Up-down interference | 0.009% [40] | $0.1\times 10^{-4}$
(f) Technical Precision | (0.027)% | $0.1\times 10^{-4}$
Total | 0.097% | $1.0\times 10^{-4}$

Table 1: Anticipated total (physical+technical) theoretical uncertainty for a
FCC-ee luminosity calorimetric detector with the angular range being
$64$–$86\,$mrad (narrow), near the $Z$ peak. Description of photonic
corrections in square brackets is related to the 2nd column. The total error
is summed in quadrature.

The results in the Table 1 and the discussion in Ref. [32] raise new issues
and illustrate the synergies between the effort to realize the FCC-ee 0.01%
theoretical precision tag and other precision theory paradigms. For example,
the need to realize the technical precision of $0.1\times 10^{-4}$ means we
will need two independent MC realizations that can realize the required
precision. We can do this using the CEEX and EEX realizations of our YFS
resummed MC methods or we can use an upgraded version of BabaYaga [41] or the
further development of the results in Refs. [42, 43, 44, 45] to achieve the
required cross check. The upgrade of BHLUMI for each of the items in Table 1
is synergistic. Preparation for the item (a) ${\cal O}(L_{e}\alpha^{2})$
upgrade allowed, via crossing, the upgrade of the CEEX 2f production in ${\cal
KK}$MC[46, 47] which, combined with Herwig6.5 [48], has now been extended to Z
production in hadron-hadron collisions in the MC ${\cal KK}$MChh [20]. Indeed,
the need to extend CEEX to BHLUMI leads naturally to its extension to the
other LEP era MC’s BHWIDE [49], YFSWW3 [50, 51] together with KORALW&YFSWW3
[52], and YFSZZ [53],all but the last of which will be needed for the various
precision measurements near the Z pole and in the WW production and
reconstruction as discussed in Ref. [54]. Indeed, the first step toward the
extension of CEEX to WW production has been given in Ref. [55], where we note
that the contact with the usual Kleiss-Stirling [56] spinor-product-based
photon helicity infrared factors in CEEX obtains via

$ej^{\mu}_{X}(k_{I})=eQ_{X}\theta_{X}\frac{2p^{\mu}_{X}}{2p_{X}k_{i}}\rightarrow\mathfrak{s}_{\sigma_{i}}(k_{i})=eQ_{X}\theta_{X}\frac{b_{\sigma_{i}}(k_{i},p_{X})}{2p_{X}k_{i}},$
(7)

with

$b_{\sigma}(k,p_{X})=\sqrt{2}\frac{\bar{u}_{\sigma}(k)\not
p\mathfrak{u}_{\sigma}(\zeta)}{\bar{u}_{-\sigma}(k)\mathfrak{u}_{\sigma}(\zeta)},$
(8)

where the $\mathfrak{u}_{\sigma}(\zeta)$ are defined in Ref. [24]. The way
forward is an open one.

##  5 Quantum Gravity: New Results and New Issues

With an eye toward the question of whether or not quantum gravity is
calculable in relativistic quantum field theory,which is still open in the
literature333See Refs. [57, 58] for further discussion on this point., we turn
to the role of IR-improvement in quantum gravity. In this context, one of us
(BFLW) argues that the attendant calculability holds if he extends the YFS
[59, 60, 24] version444YFS-type soft resummation and its extension to quantum
gravity was also worked-out by Weinberg in Ref. [61]. of the exact resummation
example to resum the Feynman series for the Einstein-Hilbert Lagrangian for
quantum gravity. Indeed, very much in analogy with what we see in the
elementary example discussed just before Eq.(1), the resultant resummed
theory, resummed quantum gravity (RQG), is very much better behaved in the UV
compared to what one would estimate from that Feynman series.

One of us (BFLW) discussed many of the interesting consequences of RQG in
Refs. [62, 63, 64, 65, 66]. Among these results, here he calls attention to
the prediction for the cosmological constant $\Lambda$ from RQG. Specifically,
in Ref. [66], he has shown that the RQG theory, taken together with the Planck
scale inflationary [67, 68] cosmology formulation in Refs. [69, 70]555The
authors in Ref. [71] also proposed the attendant choice of the scale $k\sim
1/t$ used in Refs. [69, 70]. from the asymptotic safety approach to quantum
gravity in Refs. [72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83], allows him
to predict, employing the arguments in Refs. [84], the cosmological constant
$\Lambda$ via the result

$\begin{split}\rho_{\Lambda}(t_{0})&\cong\frac{-M_{Pl}^{4}(1+c_{2,eff}k_{tr}^{2}/(360\pi
M_{Pl}^{2}))^{2}}{64}\sum_{j}\frac{(-1)^{F}n_{j}}{\rho_{j}^{2}}\times\frac{t_{tr}^{2}}{t_{eq}^{2}}\times(\frac{t_{eq}^{2/3}}{t_{0}^{2/3}})^{3}\cr&\cong\frac{-M_{Pl}^{2}(1.0362)^{2}(-9.194\times
10^{-3})}{64}\frac{(25)^{2}}{t_{0}^{2}}\cong(2.4\times
10^{-3}eV)^{4}.\cr\end{split}$ (9)

$t_{0}\cong 13.7\times 10^{9}$ yrs is the age of the universe, $t_{tr}\sim
25t_{Pl}$ is the transition time between the Planck regime and the classical
Friedmann-Robertson-Walker(FRW) regime in the Planck scale cosmology
description of inflation in Ref. [70], $c_{2,eff}\cong 2.56\times 10^{4}$ is
defined in Refs. [62, 63, 64], $t_{eq}$ is the time of matter-radiation
equality, and $M_{Pl}$ is the Planck mass.

One of us (BFLW) has discussed [66, 85] the reliability, consistency and
implications of the prediction’s closeness to the observed value [86, 87, 88],
$\rho_{\Lambda}(t_{0})|_{\text{expt}}\cong((2.37\pm 0.05)\times
10^{-3}eV)^{4}$. He argues that its uncertainty is at the level of a factor of
${\cal O}(10)$. There follow constraints on susy GUT’s as well [66]. This
means that RQG is now rife for further confrontations with observation, such
as that suggested in Ref. [89] in which the RQG prediction for the behavior of
Newton’s law at the Planck scale could be probed by appropriate observables.

##  6 Summary

The exact amplitude-level resummation of the IR regime of quantum field
theory, coupled with exact results to a given order in an exact re-arrangement
of the original Feynman series, has wide applicability for precision
phenomenology. We see that the range of this applicability is a broad one,
spanning, as it does, from the current precision LHC physics to the futuristic
FCC precision physics program and reliable estimates for the quantum theory of
gravity by one of us (BFLW). The future of particle physics is intimately
interwoven with continued progress on exact amplitude-level IR-improved
(resummed) quantum field theoretic predictions, when taken together with
control of the concurrent collinear and UV limits. More precise data require
more precise theory.

## Acknowledgments

This work was supported in part by the Programme of the French–Polish
Cooperation between IN2P3 and COPIN within the Collaborations Nos. 10-138 and
11-142 and by a grant from the Citadel Foundation. The authors also thank
Prof. G. Giudice for the support and kind hospitality of the CERN TH
Department.

## References

  * [1] B.F.L. Ward _et al._ ,  _Role of IR-Improvement in Precision LHC/FCC Physics and in Quantum Gravity_ ,  PoS RADCOR2019, 086 (2020).
  * [2] B. W. Lynn _et al._ (2019),  Proc. SM@50, Cambridge University Press, 2019, in press.
  * [3] T. Rizzo _et al._ (2018),  Proc. 46th SLAC Summer Institute, C18-07-30, 2018.
  * [4] S. L. Glashow,  _Partial Symmetries of Weak Interactions_ ,  Nucl. Phys. 22, 579 (1961).
  * [5] A. Salam,  _Elementary Particle Theory_ ,  N. Svartholm (Almqvist and Wiksell), Stockholm (1968).
  * [6] S. Weinberg,  _A Model of Leptons_ ,  Phys. Rev. Lett. 19, 1264 (1967).
  * [7] G. ’t Hooft and M. Veltman,  Nucl. Phys. B 44, 189 (1972).
  * [8] D. J. Gross and F. Wilczek,  Phys. Rev. Lett. 30, 1343 (1973).
  * [9] H. D. Politzer,  Phys. Rev. Lett. 30, 1346 (1973).
  * [10] D. J. Gross (2019),  in Proc. SM@50, Cambridge University Press, 2019, in press.
  * [11] K. Jacobs,  Talk in ICHEP2020 (2020).
  * [12] R. Carlin,  Talk in ICHEP2020 (2020).
  * [13] F. Gianotti (2020),  in FCCIS Kickoff Meeting, CERN, Geneva, Switzerland, Nov. 9, 2020.
  * [14] S.K. Majhi _et al._ ,  Phys. Lett. B 719, 367 (2013),  hep-ph/1208.4750.
  * [15] A. Mukhopadhyay and B.F.L. Ward,  Mod. Phys. Lett. A 31, 1650063 (2016),  hep-ph/1412.8717.
  * [16] S. Frixione and B.Webber,  J. High Energy Phys. 0206, 029 (2002).
  * [17] S. Frixione _et al._ ,  J. High Energy Phys. 1101, 053 (2011),  hep-ph/1010.0568.
  * [18] M. Aaboud _et al._ ,  _Measurement of the W-boson mass in pp collisions at $\sqrt{s}=7$ TeV with the ATLAS detector_,  Eur. Phys. J. C 78, 110 (2018),  hep-ex/1701.07240.
  * [19] D. Froidevaux,  private communication, 2019 (2019).
  * [20] S. Jadach, B. F. L. Ward, Z. Wa̧s and S. Yost,  _${\cal KK}$ MC-hh: Resummed Exact ${\cal O}(\alpha^{2}L)$ EW Corrections in a Hadronic MC Event Generator_,  Phys. Rev. D 94, 074006 (2016),  hep-ph/1608.01260.
  * [21] S. Jadach, B. F. L. Ward, Z. Wa̧s and S. Yost,  _Systematic Studies of Exact ${\cal O}(\alpha^{2}L)$ CEEX EW Corrections in a Hadronic MC for Precision Z/$\gamma^{*}$ Physics at LHC Energies_,  Phys. Rev. D 99, 076016 (2019),  hep-ph/1707.06502.
  * [22] S. Jadach, B. F. L. Ward, Z. Wa̧s and S. Yost,  _IFI and ISR Effects for $Z/\gamma*$ Drell-Yan Observables using KKMC-hh_ (2020),  hep-ph/2002.11692.
  * [23] S. Jadach, B. F. L. Ward and Z. Wa̧s,  _Coherent Exclusive Exponentiation (CEEX):The Case of Resonant $e^{+}e^{-}$ Annihilation_,  Phys. Lett. B449, 97 (1999),  hep-ph/9905453.
  * [24] S. Jadach, B. F. L. Ward and Z. Wa̧s,  _Coherent Exclusive Exponentiation For Precision Monte Carlo Calculations_ ,  Phys. Rev. D 63, 113009 (2001),  Preprint UTHEP-99-09-01.
  * [25] E. Boos _et al._ (2001),  hep-ph/0109068.
  * [26] J. C. Collins and D. E. Soper,  _Angular distribution of dileptons in high-energy hadron collisions_ ,  Phys. Rev. D 16, 2219 (1977).
  * [27] E. Mirkes,  _Angular decay distribution of leptons from W-bosons at NLO in hadronic collisions_ ,  Nucl. Phys. B 387, 3 (1992),  .
  * [28] R. D. Ball _et al._ ,  _Parton distributions from high precision-collider data_ (2017), 1706.00428.
  * [29] L. A. Harland-Lang, A. D. Martin, P. Motylinski and R. S. Thorne,  _Parton distributions in the LHC era: MMHT 2014 PDFs_ (2015), 1412.3989.
  * [30] A. Manohar, P. Nason, G. P. Salam and G. Zanderighi,  _How Bright is the Proton? A Precise Determination of the Photon Parton Distribution Function_ ,  Phys. Rev. Lett. 117, 242002 (2016).
  * [31] S. Jadach _et al._ ,  to appear (2021).
  * [32] S. Jadach, W. Placzek, M. Skrzypek, B. F. L. Ward and S. A. Yost,  _The Path to 0.01 Percent Theoretical Luminosity Precision for the FCC-ee_ ,  Phys. Lett. B 790, 314 (2018),  10.1016/j.physletb.2019.01.012.
  * [33] S. Jadach, W. Placzek, E. Richter-Wa̧s, B. F. L. Ward and Z. Wa̧s,  _Upgrade of the Monte Carlo program BHLUMI for Bhabha scattering at low angles to version 4.04_ ,  Comput. Phys. Commun. 102, 229 (1997).
  * [34] P. Janot and S. Jadach,  Phys. Lett. B 803, 135319 (2020),  1912.02067.
  * [35] F. Jegerlehner,  _alphaQED(MZ) and future prospects with low energy $e^{+}e^{-}$ collider data_ (2016),  talk in FCC-ee Mini-Workshop, Physics Behind Precision, https://indico.cern.ch/event/469561/contributions /1977974/attachments/1221704/1786449/ SMalphaFCCee16.pdf.
  * [36] F. Jegerlehner,  CERN Yellow Reports: Monographs, eds. A. Blondel et al., CERN-2020-003 p. 9 (2020),  10.23731/CYRM-2020-003,  1905.05078.
  * [37] G. Montagna _et al._ ,  _Light pair correction to Bhabha scattering at small angle_ ,  Nucl. Phys. B 547, 39 (1999),  10.1016/S0550-3213(99)00064-4,  hep-ph/9811436.
  * [38] G. Montagna _et al._ ,  _Light pair corrections to small angle Bhabha scattering in a realistic setup at LEP_ ,  Phys. Lett. B459, 649 (1999),  10.1016/S0370-2693(99)00729-7,  hep-ph/9905235.
  * [39] S. Jadach, W. Placzek and B. Ward,  _Precision calculation of the gamma - Z interference effect in the SLC / LEP luminosity process_ ,  Phys. Lett. B 353, 349 (1995),  10.1016/0370-2693(95)00576-7.
  * [40] S. Jadach, E. E. Richter-Wa̧s, B. F. L. Ward and Z. Wa̧s,  _Analytical O(alpha) distributions for Bhabha scattering at low angles_ ,  Phys. Lett. B253, 469 (1991),  10.1016/0370-2693(91)91754-J.
  * [41] C. M. Carloni-Calame _et al._ ,  _Status of the BabaYaga event generator_ ,  EPJ Web Conf. PHIPSI17 218, 07004 (2019).
  * [42] F. Frixione _et al._ ,  J. High Energy Phys. 1911, 158 (2019),  1909.03886.
  * [43] V. Bertone _et al._ ,  J. High Energy Phys. 2003, 135 (2020),  1911.12040.
  * [44] F. Frixione,  J. High Energy Phys. 2021, 180 (2021),  2105.06688.
  * [45] P. Banerjee _et al._ ,  Phys. Lett. B 820, 136547 (2021),  2106.07469.
  * [46] S. Jadach, B. F. L. Ward and Z. Wa̧s,  _The Precision Monte Carlo Event Generator $KKMC$ For Two-Fermion Final States In $e^{+}e^{-}$ Collisions_,  Comput. Phys. Commun. 130, 260 (2000),  Up to date source available from http://home.cern.ch/jadach/.
  * [47] S. Jadach, B. F. L. Ward and Z. Wa̧s,  _KK MC 4.22: Coherent exclusive exponentiation of electroweak corrections for f f-bar – >f’ f’-bar at the LHC and muon colliders_,  Phys. Rev. D88(11), 114022 (2013),  10.1103/PhysRevD.88.114022,  1307.4037.
  * [48] G. Corcella, I. G. Knowles, G. Marchesini, S. Moretti, K. Odagiri, P. Richardson, M. H. Seymour and B. R. Webber,  _HERWIG 6.5: an event generator for Hadron Emission Reactions With Interfering Gluons (including supersymmetric processes)_ ,  JHEP 0101, 010 (2001),  hep-ph/0011363.
  * [49] S. Jadach, W. Płaczek and B. F. L. Ward,  _BHWIDE 1.00: ${\cal O}(\alpha)$ YFS exponentiated Monte Carlo for Bhabha scattering at wide angles for LEP1/SLC and LEP2_,  Phys. Lett. B390, 298 (1997),  10.1016/S0370-2693(96)01382-2,  The Monte Carlo program BHWIDE is available from http://http://placzek.home.cern.ch/placzek/bhwide/,  hep-ph/9608412.
  * [50] S. Jadach, W. Płaczek, M. Skrzypek, B. F. L. Ward and Z. Wa̧s,  Phys. Lett. B417, 326 (1998).
  * [51] S. Jadach, W. Płaczek, M. Skrzypek, B. F. L. Ward and Z. Wa̧s,  _The Monte Carlo Event Generator YFSWW3 Version 1.16 for W Pair Production and Decay at LEP-2/LC Energies_ ,  Comput. Phys. Commun. 140, 432 (2001).
  * [52] S. Jadach, W. Płaczek, M. Skrzypek, B. F. L. Ward and Z. Wa̧s,  _The Monte Carlo program KoralW version 1.51 and the concurrent Monte Carlo KoralW and YFSWW3 with all background graphs and first order corrections to W pair production_ ,  Comput. Phys. Commun. 140, 475 (2001).
  * [53] S. Jadach, W. Placzek and B. F. L. Ward,  _Gauge invariant YFS exponentiation of (un)stable Z pair production at and beyond LEP-2 energies_ ,  Phys. Rev. D 56, 6939 (1997),  hep-ph/9705430.
  * [54] M. Skrzypek,  Talk in 2020 FFC Workshop (2020).
  * [55] S. Jadach, W. Placzek and M. Skrzypek,  _QED exponentiation for quasi-stable charged particles: the $e^{-}e^{+}\rightarrow W^{-}W^{+}$ process_,  Eur.Phys. J. C 80, 499 (2020),  hep-ph/1906.09071.
  * [56] R. Kleiss and W. J. Stirling,  Nucl. Phys. B262, 235 (1985).
  * [57] B.F.L. Ward,  _Prediction for the Cosmological Constant and Constraints on SUSY GUT’s: Status Report for Resummed Quantum Gravity_ ,  PoS ICHEP2018, 620 (2018).
  * [58] B.F.L. Ward,  _Prediction for the Cosmological Constant and Constraints on Susy GUTS in Resummed Quantum Gravity_ ,  Int. J. Mod. Phys. A 33, 1830028 (2018).
  * [59] D. R. Yennie, S. Frautschi and H. Suura,  _The infrared divergence phenomena and high-energy processes_ ,  Ann. Phys. (NY) 13, 379 (1961),  10.1016/0003-4916(61)90151-8.
  * [60] S. Jadach and B. F. L. Ward,  Phys. Rev. D38, 2897 (1988).
  * [61] S. Weinberg,  Phys. Rev. 140, B516 (1965).
  * [62] B.F.L. Ward,  Open Nucl.Part.Phys.Jour. 2, 1 (2009).
  * [63] B.F.L. Ward,  Mod. Phys. Lett. A 17, 237 (2002).
  * [64] B.F.L. Ward,  Mod. Phys. Lett. 19, 143 (2004).
  * [65] B.F.L. Ward,  Mod. Phys. Lett. A 23, 3299 (2008).
  * [66] B.F.L. Ward,  Mod. Phys. Lett. A 30, 1550206 (2015).
  * [67] A. H. Guth,  Phys. Rev.D 23, 347 (1981).
  * [68] A. Linde,  Lecture Notes in Phys. 738, 1 (2008).
  * [69] A. Bonanno and M. Reuter,  Phys. Rev. D 65, 043508 (2002).
  * [70] A. Bonanno and M. Reuter,  Jour. Phys. Conf. Ser. 140, 012008 (2008).
  * [71] I. L. Shapiro and J. Sola,  Phys. Lett. B 475, 236 (2000).
  * [72] M. Reuter,  Phys. Rev. D 57, 971 (1998).
  * [73] O. Lauscher and M. Reuter,  Phys. Rev. D 66, 025026 (2002).
  * [74] E. Manrique, M. Reuter and Saueressig,  Ann. Phys. 326, 44 (2011).
  * [75] A. Bonanno and M. Reuter,  Phys. Rev. D 62, 043008 (2000).
  * [76] D. F. Litim,  Phys. Rev. Lett. 92, 201301 (2004).
  * [77] D. F. Litim,  Phys. Rev. D 64, 105007 (2001).
  * [78] P. Fischer and D. F. Litim,  Phys. Lett. B 638, 497 (2006).
  * [79] D. Don and R. Percacci, Class. Quant. Grav. 15 (1998) 3449,  Class. Quant. Grav. 15, 3449 (1998).
  * [80] R. Percacci and D. Perini,  Phys. Rev. D 67, 081503 (2003).
  * [81] R. Percacci,  Phys. Rev. D 73, 041501 (2006).
  * [82] R. Percacci and D. Perini,  Phys. Rev. D 68, 044018 (2003).
  * [83] A. Codello and R. Percacci and C. Rahmede,  Int.J. Mod. Phys. A 23, 143 (2008).
  * [84] V. Branchina and D. Zappala,  G. R. Gravit. 42, 141 (2010).
  * [85] B.F.L. Ward,  Phys. Rev. D 57, 971 (1998).
  * [86] A. G. Riess _et al._ ,  Astron. Jour. 116, 1009 (1998).
  * [87] S. Perlmutter _et al._ ,  Astrophys. Jour. 517, 565 (1999).
  * [88] C. Amsler _et al._ ,  Phys. Lett. B 667, 1 (2008).
  * [89] G. Calcagni _et al._ ,  J. Cos. Astropart. Phys 10, 012 (2019).


final final

# Enhancing Class Fairness in Classification
with A Two-Player Game Approach

Yunpeng Jiang
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
Paul Weng
Duke Kunshan University
<EMAIL_ADDRESS>&Yutong Ban∗
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
indicates the corresponding authors.

###### Abstract

Data augmentation is widely applied and has shown its benefits in different
machine learning tasks. However, as recently observed in some downstream
tasks, data augmentation may introduce an unfair impact on classifications.
While it can improve the performance of some classes, it can actually be
detrimental for other classes, which can be problematic in some application
domains. In this paper, to counteract this phenomenon, we propose a FAir
Classification approach with a Two-player game (FACT). We first formulate the
training of a classifier with data augmentation as a fair optimization
problem, which can be further written as an adversarial two-player game.
Following this formulation, we propose a novel multiplicative weight
optimization algorithm, for which we theoretically prove that it can converge
to a solution that is fair over classes. Interestingly, our formulation also
reveals that this fairness issue over classes is not due to data augmentation
only, but is in fact a general phenomenon. Our empirical experiments
demonstrate that the performance of our learned classifiers is indeed more
fairly distributed over classes in five datasets, with only limited impact on
the average accuracy.

##  1 Introduction

Data augmentation is a popular technique used in the field of computer vision
to improve both training efficiency and performance of models [DA_CV][DA_DRL].
It involves creating new training samples by applying semantically-preserving
transformations, such as random shift[DrQ] [drqv2], random convolution [SVEA]
and random resized cropping, to the original data. The aim is to increase the
diversity of the training dataset and prevent overfitting, leading to better
generalizability to the testing dataset.

However, recent research [BalestrieroEODA] has highlighted a potential issue
with data augmentation: although it is beneficial in improving the overall
performance, data augmentation can lead to disparities in performance across
different classes. This means that while there are significant improvements on
the performance of some classes, others may experience a decline or minimal
gains. This disparity raises concerns about fairness in classification tasks.
If certain classes are consistently under-performing, it can lead to biased
outcomes and a lack of equitable treatment for all
classes[focal_loss][DBLP:journals/corr/abs-2111-14581].

As a result, there is growing interest in developing methods to achieve
equitable performance among all classes within classification tasks. The goal
of fair classification problems is to ensure that the benefits of data
augmentation are evenly distributed across all classes, rather than being
concentrated in a few. This involves finding ways to mitigate the potential
negative impact of data augmentation on certain classes while maintaining the
positive effects on others.

In this paper, to address the challenge of achieving class fairness in
classification tasks without compromising the overall model performance, we
conceptualize the fair classification problem as an adversarial two-player
game. The max player aims at maximizing the overall accuracy of the model
while the min player focuses on ensuring fairness across different classes. We
show that existing methods based on the standard max-min fairness [Rawls71,
ZhangShah14] or using the generalized Gini social welfare function (GGF)
[WEYMARK1981409giniindex, pmlr-v119-siddique20a] fall under this formulation.
In the context of GGF, the min player is responsible for determining the
weight for each class.

Figure 1: The proposed FACT formulates the fair classification problem as an
adversarial two-player game, where $v_{i}$ represents the accuracy of the
$i^{th}$ class.

Meanwhile, the max player’s role is to take into account the weights assigned
by the min player and train the model such that the weighted accuracy of the
model is maximized.

In this strategic interaction, we propose employing


# On Learning Intrinsic Rewards for Faster Multi-Agent Reinforcement Learning
based MAC Protocol Design in 6G Wireless Networks

Luciano Miuccio1, Salvatore Riolo1, Mehdi Bennis2, and Daniela Panno1 emails:
<EMAIL_ADDRESS>{salvatore.riolo<EMAIL_ADDRESS><EMAIL_ADDRESS>1 Department of Electrical, Electronics and Computer
Engineering, University of Catania, Italy  2 Centre for Wireless
Communications, University of Oulu, Finland

###### Abstract

In this paper, we propose a novel framework for designing a fast convergent
multi-agent reinforcement learning (MARL)-based medium access control (MAC)
protocol operating in a single cell scenario. The user equipments (UEs) are
cast as learning agents that need to learn a proper signaling policy to
coordinate the transmission of protocol data units (PDUs) to the base station
(BS) over shared radio resources. In many MARL tasks, the conventional
centralized training with decentralized execution (CTDE) is adopted, where
each agent receives the same global extrinsic reward from the environment.
However, this approach involves a long training time. To overcome this
drawback, we adopt the concept of learning a per-agent intrinsic reward, in
which each agent learns a different intrinsic reward signal based solely on
its individual behavior. Moreover, in order to provide an intrinsic reward
function that takes into account the long-term training history, we represent
it as a long short-term memory (LSTM) network. As a result, each agent updates
its policy network considering both the extrinsic reward, which characterizes
the cooperative task, and the intrinsic reward that reflects local dynamics.
The proposed learning framework yields a faster convergence and higher
transmission performance compared to the baselines. Simulation results show
that the proposed learning solution yields 75% improvement in convergence
speed compared to the most performing baseline.

###### Index Terms:

6G, Intrinsic reward learning, MARL, Protocol learning.

##  I Introduction

The emergence of data-driven medium access control (MAC) protocols can provide
a cost-effective, flexible approach to boost the performance of beyond 5G
(B5G) and 6G networks. To address this problem, multi-agent reinforcement
learning (MARL) methods enable agents to learn an optimal policy by
interacting in the same environment [1]. Current works, such as [2, 3], have
studied the MAC protocol learning in a single cell scenario, where user
equipments (UEs) need to deliver MAC protocol data units (PDUs) to the base
station (BS) sharing the same radio channel. UEs are cast as reinforcement
learning (RL) agents that are trained to learn a new MAC protocol from their
partial observations of the global state.

However, despite the good performances shown at the end of the training
procedure, learning efficient and robust MAC protocols consisting of multiple
agents acting and learning in the same shared environment requires very long
training time. This aspect prevents the applicability of this approach to a
dynamic wireless environment that requires retraining to adapt the MAC
protocol to changing environments. The main causes of slowness in training
stem from the partial observability and non-stationarity of the MARL problem
(i.e., transitions from a state to another depend on the actions of all
agents) [4]. In addition to this, [2, 3] are based on the conventional
centralized training and decentralized execution (CTDE) paradigm and parameter
sharing technique [5] that further slows down the convergence time.

Specifically, on the one hand, CTDE allows agents to learn their local
policies in a centralized way while retaining the ability of decentralized
execution. During the training phase, the environment assigns the same global
reward to all agents without distinguishing their own contributions. As a
consequence, only a subset of agents contributes to the reward, and so, during
training an agent may be punished even if it has acted optimally, or rewarded
even if it has acted wrongly. Clearly, this approach induces slow and unstable
policy learning.

On the other hand, the parameter sharing technique consists in simultaneously
learning a single shared policy for multiple agents, which boosts scalability.
However, UEs may compete at the same time to transmit their own packets in the
same shared channel. In other words, even if two UEs have the same
observation, the action taken by one UE should be different from the action
taken by the other one to avoid interference. Updating the same policy for
every agent can create adverse actions that slow down convergence during
learning.

In light of the above considerations, the main contribution of this paper is
to propose a novel framework that provides faster convergence to the MAC
protocol learning problem. Specifically, we consider the same communication
scenario studied in [2, 3] and introduce our innovative learning framework
with the following features.

First, we adopt an enhanced version of the CTDE paradigm, which in addition to
the global reward signal, leverages for each agent, a different local
intrinsic reward signal based on its individual behavior. This idea is
inspired by intrinsic reward learning introduced in [6, 7] for a single-agent
environment. Different from the global reward given by the environment (termed
as “extrinsic reward”) that is hand-designed, the intrinsic reward is
automatically learned by each agent.

Second, the proposed solution avoids the use of the parameter sharing
technique and, instead, considers that each agent has two independent modules,
namely, a policy network and an intrinsic reward network. The policy network
learns the optimal policy per agent, while the intrinsic reward network
provides additional reward signal to the policy network.

Simulation results show that in complex scenarios and adopting the multi-agent
proximal policy optimization (MAPPO) algorithm, the proposed learning
framework yields a $75\%$ improvement in convergence speed, and about $4\%$
improvement in transmission performance compared to conventional CTDE without
parameter sharing technique, and even better results with respect to a
baseline consisting of both CTDE and parameter sharing.

The rest of the article is organized as follows. The system model and the
formalization of the cooperative MARL problem are described in Section II. The
proposed approach is detailed in Section III. Finally, the numerical
simulation results and conclusions are drawn in Section IV and Section V,
respectively.

##  II System Model and MARL Formulation

Consider a single BS serving a set $\mathcal{N}$ of $N$ homogeneous UEs
needing to deliver $P$ MAC PDUs to the BS. The network nodes exchange control
messages encapsulated inside signaling PDUs (sPDUs) through the downlink (DL)
and uplink (UL) control channels, which are assumed to be dedicated and error
free. As regards data transmission, UEs send data PDUs (dPDUs) using the same
physical uplink shared channel (PUSCH) operating according to a time division
multiple access (TDMA) scheme, which leads to possible collisions.
Specifically, for each time step $t$ a UE can send one dPDU, and this dPDU is
successfully received by the BS only if a single UE out of $N$ has transmitted
it.

Control plane: let $\mathcal{M}_{\text{UE,s}}=\\{0,1\\}$ be the set of
possible messages sent by the UEs, and $\mathcal{M}_{\text{BS}}=\\{0,1,2\\}$
be the set of DL messages. At each time step $t$, the BS can send to each UE
$i\in\mathcal{N}$ only one message $m_{i}^{t}\in\mathcal{M}_{\text{BS}}$, and
each UE $i$ can send one signaling message
$a_{i,\text{s}}^{t}\in\mathcal{M}_{\text{UE,s}}$ to the BS. Specifically,
$m_{i}^{t}=2$ represents an acknowledgement (ACK) message that confirms that a
dPDU sent from UE $i$ has been correctly received at the BS in the previous
time step $t-1$, $m_{i}^{t}=1$ refers to a scheduling grant message to UE $i$,
and $m_{i}^{t}=0$ to indicate that no access is granted for UE $i$. As for
UEs, $a_{i,\text{s}}^{t}=1$ means sending an access request to reserve time
step $t+1$ for transmission, while $a_{i,\text{s}}^{t}=0$ means do not
transmit any signaling message.

Data plane: each UE $i$ has a dPDUs storage capability, modeled as a buffer
with first-in first-out (FIFO) policy, which contains at most $P$ dPDUs. We
denote with $b_{i}^{t}\in\mathcal{B}=\\{0,1,\dots,P\\}$ the buffer status at
time $t$, and we assume that the buffer starts full. For each time step $t$,
UE $i$ is able to transmit a dPDU or to delete it. Specifically, the data
plane action is denoted as
$a_{i,\text{u}}\in\mathcal{M}_{\text{UE,u}}=\\{0,1,2\\}$, where
$a_{i,\text{u}}=1$ means that the UE transmits the first dPDU in its buffer
(if any), $a_{i,\text{u}}=2$ means it deletes the first dPDU in the buffer,
and $a_{i,\text{u}}=0$ to do nothing.

We assume that the BS is a MAC expert agent, i.e., it adopts a MAC protocol
that is not learned. In detail, at each time step $t$, if the BS receives more
scheduling requests from the UEs, then it chooses one of the requesting UEs at
random and a scheduling grant is sent in response. If the UE has made a
successful data transmission concurrently with the scheduling request, then
the BS will ignore this scheduling request and send only an ACK message to it.

###  II-A Multi-Agent Reinforcement Learning Formulation

The goal is to find the optimal MAC protocol adopted by UEs that maximizes the
number of unique dPDUs successfully received by the BS, while minimizing the
time spent to do so. To effectively reach this goal, we propose to cast the
UEs as MAC learning agents and the protocol learning problem as a cooperative
and multi-agent partially observable Markov decision process (MPOMDP). The
system can be described as a tuple as $\langle$$\mathcal{N}$, $\mathcal{A}$,
$\mathcal{S}$, $\mathcal{O}$, $\pi_{i}$, $R_{\text{ext}}$,$\gamma$$\rangle$.
Let $\mathcal{N}$ denote the set of $N$ homogeneous learning agents (i.e.,
UEs). Each agent $i\in\mathcal{N}$ at time step $t$ has a partial observation
of the global state defined as
$o_{i}^{t}=(b_{i}^{t},b_{i}^{t-1},a_{i}^{t-1},m_{i}^{t-1},\dots,b_{i}^{t-M},a_{i}^{t-M},m_{i}^{t-M})$,
where $M$ is the memory length. Accordingly, let
${a}_{i}^{t}=(a_{i,\text{u}}^{t},a_{i,\text{s}}^{t})$ indicate the action
taken by agent $i$, which involves both data and control plane. Each agent $i$
shares the same observation and action space, denoted as $\mathcal{O}$ and
$\mathcal{A}$, respectively. Clearly,
$\mathcal{A}=\mathcal{M}_{\text{UE,u}}\times\mathcal{M}_{\text{UE,s}}=\\{A_{1},\dots,A_{|\mathcal{A}|}\\}$.
Let $\pi_{i}\left(a_{i}^{t}\mid
o_{i}^{t}\right)\colon\mathcal{O}\times\mathcal{A}\rightarrow[0,1]$ be a
stochastic policy for agent $i$, that is, the probability of choosing a given
action $a_{i}$ given that agent $i$ is observing $o_{i}$. For sake of clarity,
we also introduce $\bm{o}^{t}=[o_{1}^{t},\dots,o_{N}^{t}]$,
$\bm{a}^{t}=[a_{1}^{t},\dots,a_{N}^{t}]$, and
$\bm{\pi}=[\pi_{1},\dots,\pi_{N}]$.

At time step $t$, each agent $i$ observes $o_{i}^{t}$ and selects an action
$a_{i}^{t}$ according to its own policy $\pi_{i}$. At time step $t+1$, in
conventional MPOMDP, each agent receives from the environment an extrinsic
reward $R_{\text{ext}}^{t+1}$, which is the same for all agents and quantifies
the benefit of the joint actions performed by all the $N$ agents. This design
decision reflects the objective of optimizing the performance of the whole
network, rather than that of individual agents. We define an episode as a
finite sequence of agent-environment interactions lasting $T_{\text{ep}}$ time
steps. For each episode, we define the episodic cumulative extrinsic return as

$G_{\text{ep},\text{ext}}=\sum_{t=0}^{T_{\text{ep}}-1}\gamma^{t}R_{\text{ext}}^{t+1},$
(1)

where $\gamma$ is a discount factor. Since maximizing
$G_{\text{ep},\text{ext}}$ represents the goal of the reinforcement learning
problem, the values of $R_{\text{ext}}^{t+1}$ in each time step $t$ should be
properly designed. Here, we leverage the simple approach [2], where
$R_{\text{ext}}^{t+1}\in\\{-1,0\\}$, with $R_{\text{ext}}^{t+1}$ equal to 0
only if a dPDU has been received correctly at time step $t$ or if all dPDUs
sent by each UE have been already received correctly in the previous time
steps. This means that $G_{\text{ep},\text{ext}}$ reaches its maximum value
(i.e., 0) when all packets have been received at minimum time, and the minimum
value of $G_{\text{ep},\text{ext}}$ is assumed when no packets have been
received correctly. We emphasize that the selection of extrinsic reward
functions are typically hand-designed. However, finding a good reward function
is not straightforward and requires a high expertise and domain knowledge of
the designer. Moreover, the extrinsic reward is strongly goal or task-
specific, which limits its applicability to other use cases and goals. Let
$J_{\text{ep},\text{ext}}(\bm{\pi})$ denote the expected episodic cumulative
extrinsic return obtained when each agent $i$ follows its own policy
$\pi_{i}\in\bm{\pi}$, i.e.,

$J_{\text{ep},\text{ext}}(\bm{\pi})=\mathbb{E}_{\textbf{o}^{0},\textbf{a}^{0},\dots\textbf{o}^{T_{\text{ep}}-1},\textbf{a}^{T_{\text{ep}}-1}}\left[G_{\text{ep},\text{ext}}\right],$
(2)

where $a_{i}^{t}\sim\pi_{i}\left(a_{i}^{t}\mid o_{i}^{t}\right),\forall
i\in\mathcal{N}$. The objective of the MARL problem is to find optimal
policies $\bm{\pi}^{*}$ that maximize $J_{\text{ep},\text{ext}}$. For doing
this, we adopt the CTDE paradigm. Several MARL techniques can be used, ranging
from simple value-based approaches (e.g., tabular Q-learning [8]) to on-policy
algorithms (e.g., MAPPO [9]). In general, all approaches consider an
independent policy parameterized by ${\theta_{i}}$ for each agent $i$ and
denoted as $\pi_{\theta_{i}}$. Each agent updates independently its own
parameter $\theta_{i}$ by maximizing the expected extrinsic reward. In
addition to this, thanks to the homogeneous nature of UEs and the use of the
same cumulative extrinsic reward $J_{\text{ep},\text{ext}}$, another possible
approach is to learn a shared optimal policy $\bm{\pi}^{*}$ by leveraging the
concept of parameter sharing [5].

##  III Proposed approach

In this section, we formally present our approach that aims to automatically
speed up the convergence time by adopting both the concept of extrinsic reward
empowered by an intrinsic reward [6] and the concept of lifetime [7].
Specifically, in [6] the authors investigated in the case of a single agent
environments the advantages of using an intrinsic reward function
parameterized by $\eta$ in addition to the conventional extrinsic reward.
Differently from the hand-designed extrinsic reward, the intrinsic reward
function is automatically learned by each agent to improve its learning
dynamics. In this case, both the policy and intrinsic reward parameters are
learned within a single episode. Conversely, in [7], the authors propose to
learn an intrinsic reward over a lifetime consisting of $N_{\text{ep}}$
episodes, instead of a single episode, to take into account the system
dynamics. In detail, the policy parameter $\theta$ is still updated episode-
by-episode by considering only the cumulative episodic intrinsic reward, while
the intrinsic reward parameter $\eta$ is updated within every lifetime to
maximize the cumulative extrinsic reward over an entire lifetime.

Therefore, inspired by these works, we propose a new approach that
incorporates the multi-agent intrinsic reward function in our system model.
For the sake of clarity, we first define some terminologies.

  * •

Intrinsic reward function for agent $i$. Defined as a function related to
agent $i$ and parameterized by $\eta_{i}$. At the end of time step $t$,
$R_{\text{in},\eta_{i}}^{t+1}$ is a scalar reward that takes into account the
history of the entire lifetime of agent $i$ until time step $t$, including all
its partial observations $([o_{i}^{0},\dots,o_{i}^{t}])$, its selected actions
$([a_{i}^{0},\dots,a_{i}^{t}])$, and extrinsic reward values
$[R_{\text{ext}}^{1},\dots,R_{\text{ext}}^{t}])$.

  * •

Overall reward function for agent $i$. Defined as a function related to agent
$i$ made of two contributions. First, the extrinsic reward value
$R_{\text{ext}}^{t+1}$ received from the environment, which is the same for
all agents and quantifies the benefit of joint actions performed by $N$
agents. Second, the intrinsic reward value $R_{\text{in},\eta_{i}}^{t+1}$ that
is learned independently by each agent $i$. For each agent $i$ and time step
$t+1$, the overall reward is given as

$R_{\text{ov},i}^{t+1}=R_{\text{ext}}^{t+1}+\lambda
R_{\text{in},\eta_{i}}^{t+1},$ (3)

where $\lambda\in[0,1]$ is a hyper-parameter that balances the weighted
summation between the extrinsic reward and the intrinsic reward.

  * •

Episodic overall return. For each episode $k$, we define the episodic overall
return of agent $i$ as

$G_{\text{ep},\text{ov},i}^{(k)}=\sum_{t=0}^{T_{\text{ep}}-1}\gamma^{t}R_{\text{ov},i}^{t+1}.$
(4)

  * •

Lifetime extrinsic return. At the end of a lifetime, we define the lifetime
extrinsic return as

$G_{\text{life},\text{ext}}=\sum_{t=0}^{T-1}\gamma^{t}R_{\text{ext}}^{t+1},$
(5)

where $T$ is the number of steps per lifetime, i.e.,
$T=N_{\text{ep}}T_{\text{ep}}$. Using the lifetime return
$G_{\text{life},\text{ext}}$ as the objective instead of the conventional
episodic return $G_{\text{ep},\text{ext}}$ allows exploration across multiple
episodes.

Figure 1: High-level depiction of the system model.

###  III-A Architecture of the multi-agent framework

Each agent $i\in\mathcal{N}$ is equipped with two neural networks, as depicted
in Fig. 2. The first one represents the policy function $\pi_{i}$, and the
second one represents the related intrinsic reward function. The policy
network (see Fig. 2a) is a multi-layer perceptron (MLP) with weights
$\theta_{i}$. The intrinsic reward function is represented by a neural network
providing as output a scalar reward that takes into account the long past
history of agent $i$. For this reason, instead of adopting a conventional MLP,
we exploit the characteristics of recurrent neural networks (RNNs). Unlike
MLP, in RNNs the output from previous step is fed as input to the current step
creating a feedback loop. As a consequence, the output provided at step $t$
takes into consideration not only the current input, but also what the network
has learned from the previous inputs, involving internal memory capabilities.
However, conventional RNNs are not able memorize data for long time and tend
to forget its previous inputs. To overcome this problem, we use an LSTM, which
is a type of recurrent neural network that expands the memory capacity for
long period of time [10]. The proposed LSTM is parameterized by $\eta_{i}$ for
representing the intrinsic reward function, as shown in Fig. 2b.

###  III-B Algorithm overview

A high level description of the proposed training algorithm related to each
agent $i$ is presented in Pseudo-code 1 and depicted in Fig. 3. As shown, the
updates of the policy network and the intrinsic rewards network are carried
out with a different periodicity, corresponding to one episode and one
lifetime, respectively. The periodicity of a lifetime permits to update the
intrinsic reward network taking into account the long-term system dynamics.

Pseudo-code 1 Network updates for agent $i$

1:learning parameters $\alpha$ and $\beta$, balancing parameter $\lambda$,
discount factor $\gamma$

2:Initialize a policy network with random weight $\theta_{i}^{(0)}$ and an
intrinsic reward network with random weight $\eta_{i}$.

3:repeat

4: for $k=1,2,\dots,N_{\text{ep}}$ do

5: Interact with the environment for one episode using
$\pi_{\theta_{i}^{(k-1)}}$.

6: Store the experience within the episode rollout $T_{\text{E},i}^{(k)}$ (6)
and the lifetime rollout $T_{\text{L},i}$ (7)

7: Update the policy parameter $\theta_{i}^{(k-1)}$ by exploiting
$T_{\text{E},i}^{(k)}$ as described in Section III-C

8: end for

9: Update the intrinsic reward network with parameter $\eta_{i}$ by exploiting
$T_{\text{L},i}$ as described in Section III-D.

10: Set $\theta_{i}^{(0)}\leftarrow\theta_{i}^{(N_{\text{ep}})}$.

11:until the intrinsic reward network converges.

At each episode $k$, each agent $i$ generates an experience interacting with
the environment for $T_{\text{ep}}$ time steps using its policy and its
intrinsic reward network. In detail, at each time step $t$, the experience of
agent $i$ is stored inside the episode rollout

$T_{\text{E},i}^{(k)}=\left\\{\left(o^{t}_{i},a^{t}_{i},\pi_{\theta^{(k-1)}_{i}}\left(a^{t}_{i}\mid
o^{t}_{i}\right),R_{\text{ext}}^{t+1},R_{\text{in},\eta_{i}}^{t+1}\right)\right\\}_{t=0}^{T_{\text{ep}}-1},$
(6)

and the lifetime rollout

$T_{\text{L},i}=\left\\{T_{\text{E},i}^{(k)}\right\\}_{k=1}^{N_{\text{ep}}}.$
(7)

Episode-by-episode, each agent $i$ updates its policy parameter
$\theta_{i}^{(k-1)}$ following the procedure is described in Section III-C. At
the end of a lifetime, each agent updates the intrinsic reward network
parameter $\eta_{i}$ following the procedure described in Section III-D. The
overall procedure is carried out until the intrinsic reward network reaches
convergence.

Figure 2: Learning networks inside each agent $i$. a) is the policy network
parameterized by $\theta_{i}$ that, given the current observation $o_{i}^{t}$,
outputs the probability of choosing the action $A_{j}$, for
$j\in\\{1,\dots,|\mathcal{A}|\\}$. b) is the intrinsic reward network
parameterized by $\eta_{i}$ that receives as input the current observation and
the selected action ($o_{i}^{t},a_{i}^{t}$) and is conditioned on the previous
hidden state and cell state ($h_{i}^{t},c_{i}^{t}$). It gives as output the
intrinsic reward value $R_{\text{in},\eta_{i}}^{t+1}$ and generates the next
tuple ($h_{i}^{t+1},c_{i}^{t+1}$). Figure 3: Training algorithm overview: for
each episode $k$, each agent $i$ updates its policy parameter
$\theta_{i}^{(k-1)}$ by exploiting the episode rollout $T_{\text{E},i}^{(k)}$
in the direction of the gradient of expected lifetime overall return
$J_{\text{ep},\text{ov},i}^{(k-1)}$. At the end of each lifetime, agent $i$
updates the intrinsic reward network parameter $\eta_{i}$ in the direction of
the gradient of the expected lifetime extrinsic return
$J_{\text{life},\text{ext}}$, by exploiting its lifetime rollout
$T_{\text{L},i}$.

###  III-C Updating the Policy Parameter $\theta_{i}$

In this subsection, we describe how to update the policy parameter of each
agent $i$. Specifically, at the end of episode $k$, the update of
$\theta^{(k-1)}_{i}$ is performed so as to maximize the expected episodic
cumulative overall return of episode $k-1$

$J_{\text{ep},\text{ov},i}^{(k-1)}=\mathbb{E}_{o_{i}^{0},a_{i}^{0},\dots
o_{i}^{T_{\text{ep}}-1},a_{i}^{T_{\text{ep}}-1}}\left[G_{\text{ep},\text{ov},i}^{(k-1)}\right],$
(8)

where $a_{i}^{t}\sim\pi_{\theta^{(k-1)}_{i}}\left(a_{i}^{t}\mid
o_{i}^{t}\right)$. This update can be done by using a simple policy gradient
method, as follows

$\begin{split}\theta^{(k)}_{i}&=\theta^{(k-1)}_{i}+\alpha\nabla_{\theta^{(k-1)}_{i}}J_{\text{ep},\text{ov},i}^{(k-1)}.\end{split}$
(9)

The policy gradient theorem [11] shows that, given the episode rollout
$T_{\text{E},i}^{(k)}$, the update can be computed as111Other approaches
exploiting other policy gradient methods (REINFORCE [12], PPO [13], TRPO [14])
can be used.

$\theta^{(k)}_{i}\approx\theta^{(k-1)}_{i}+\alpha
G_{\text{ep},\text{ov},i}^{(k-1)}\nabla_{\theta^{(k-1)}_{i}}\log{\pi_{\theta^{(k-1)}_{i}}}\left(a_{i}^{t}|o_{i}^{t}\right).$
(10)

###  III-D Updating the Intrinsic Reward Parameter ($\eta_{i}$)

Given a lifetime and the updated policy parameters at the end of the lifetime
$\left({\theta}_{i}^{\left(N_{\text{ep}}\right)}\right)$, we update the
intrinsic reward network parameter for each agent $i$ with the aim of
maximizing the expected lifetime extrinsic return

$J_{\text{life},\text{ext}}=\mathbb{E}_{o_{i}^{0},a_{i}^{0},\dots
o_{i}^{T-1},a_{i}^{T-1}}\left[G_{\text{life},\text{ext}}\right],$ (11)

where $a_{i}^{t}\sim\pi_{\theta^{(N_{\text{ep}})}_{i}}\left(a_{i}^{t}\mid
o_{i}^{t}\right)$. Similarly as the policy parameters update, this update can
be done by using a simple policy gradient method, as follows

$\eta_{i}^{\prime}=\eta_{i}+\beta\nabla_{\eta_{i}}J_{\text{life},\text{ext}}.$
(12)

Intuitively, updating $\eta_{i}$ requires estimating the effect such a change
would have on the extrinsic value through the change in the policy parameters.
To obtain this, we compute the meta-gradient
$\nabla_{\eta_{i}}J_{\text{ext},\text{life}}$ exploiting the chain rule as
follows:

$\nabla_{\eta_{i}}J_{\text{life},\text{ext}}=\nabla_{{\theta}_{i}^{\left(N_{\text{ep}}\right)}}J_{\text{life},\text{ext}}\nabla_{\eta_{i}}{\theta}_{i}^{\left(N_{\text{ep}}\right)}.$
(13)

Moreover, the first gradient can be approximated by means of the policy
gradient theorem [11] as

$\nabla_{{\theta}_{i}^{\left(N_{\text{ep}}\right)}}J_{\text{life},\text{ext}}\approx
G_{\text{life},\text{ext}}\nabla_{\theta^{(N_{\text{ep}})}_{i}}\log{\pi_{\theta^{(N_{\text{ep}})}_{i}}}\left(a_{i}^{t}|o_{i}^{t}\right).$
(14)

We note that a new lifetime should be computed with the updated policy
parameters $\theta^{(N_{\text{ep}})}_{i}$ to calculate this gradient. For
avoiding this, we reuse the lifetime generated by the original policy
parameters $\theta^{(k)}_{i}$, with $k=1,\dots,N_{\text{ep}}$, by means of the
_importance sampling ratio_ [15]. Hence, we exploit the lifetime rollout
$T_{\text{L},i}$, and rewrite the gradient computation as follows:

$\nabla_{{\theta}_{i}^{\left(N_{\text{ep}}\right)}}J_{\text{life},\text{ext}}\approx
G_{\text{life},\text{ext}}\frac{\nabla_{\theta^{(N_{\text{ep}})}_{i}}{\pi_{\theta^{(N_{\text{ep}})}_{i}}}\left(a_{i}^{t}|o_{i}^{t}\right)}{\pi_{\theta^{(k)}_{i}}\left(a_{i}^{t}|o_{i}^{t}\right)}.$
(15)

(a) $P=1$.

(b) $P=2$.

Figure 4: Convergence curves of all the considered algorithms varying the
number of packets to deliver and testing results. In the boxplots, the circle
is the median, the colored box represents the interquartile range (IQR) from
the first quartile (Q1) to the third quartile (Q3), the vertical lines
represent the minimum (Q1-1.5 IQR) and the maximum (Q3+1.5 IQR) values, the
diamonds are outliers.

##  IV Simulation Results and Analysis

In this section, we examine the convergence performance of the proposed
learning framework in terms of percentage of successfully delivered dPDUs vs.
the number of training episodes. The list of the simulation parameters is
reported in Table I.

TABLE I: Training algorithm Parameters

Parameter | Symbol | Value
---|---|---
Number of UEs | $N$ | 2
Discount factor | $\gamma$ | 0.99
Memory length | $M$ | $3$
Duration of episode | $T_{\text{ep}}$ | 32
Number of episodes per lifetime | $N_{\text{ep}}$ | 250
Number of dPDUs to deliver | $P$ | $\\{1,2\\}$
Balancing parameter | $\lambda$ | 1
Act. function, intrinsic reward network |  | {i, t}a
Learning rate, intrinsic reward network | $\beta$ | $7\cdot 10^{-4}$
Neurons per layer, intrinsic reward network |  | 128
Act. function per layer, policy network |  | {t, t, s}a
Learning rate, policy network | $\alpha$ | $3\cdot 10^{-4}$
Neurons per layer, policy network |  | 64

  * a

i = identity function, t = tanh function, s = softmax function.

The results of the proposed method are compared against the following
baselines.

  * •

Extrinsic-NPS: An independent policy is trained for each agent $i$ to maximize
the expected episodic cumulative extrinsic return
($J_{\text{ep},\text{ext}}$).

  * •

Extrinsic-PS: A shared policy is trained among all the agents to maximize
$J_{\text{ep},\text{ext}}$, as in [2].

  * •

Random Uniform (RU): Regardless of the observation $o_{i}^{t}$, each agent $i$
select its action uniformly from $\mathcal{A}$.

Fig. 4 plots the percentage of successfully delivered packets vs. training
episodes with respect to baselines averaged over 10 independent training
sessions. After assessing the training phase, we select the best trained
instance for each solution in terms of average percentage of successfully
delivered packets. Then, we test them in 1000 episodes and show the related
statistics by using boxplots.

Specifically, Fig. 4a shows the simulation results in the case of $P=1$ packet
to deliver. We observe that the proposed algorithm and Etrinsic-NPS require
$5.1\cdot 10^{3}$ iterations to reach convergence. Conversely, Etrinsic-PS has
not converged within $7\cdot 10^{3}$ episodes. This shows that the
introduction of additional features (intrinsic reward and lifetime update),
does not introduce any significant improvement in the case of a simple
transmission scenario. In Fig. 4b we show the performances when $P=2$ packets
need to be delivered. Therein, the proposed method reaches convergence in
almost $2\cdot 10^{4}$ training episodes, which is $75\%$ less than the number
of episodes required for the Extrinsic-NPS method. This is because, in this
more complex scenario, the proposed method provides additional information
with the correct periodicity to the policy update process. Fig. 4b shows also
that the proposed method achieves a maximum service success rate of $81\%$,
that is $4\%$ better than the maximum performances of the Extrinsic-NPS one.
As regards the other baseline, Extrinsic-PS does not reach convergence within
the considered training interval. As concerns the testing phase, our solution
exhibits an interquartile range between $75\%$ and $100\%$ of packet
successfully delivered, which is the best result, as show in Fig. 4b.

Summarizing, the proposed method yields better convergence speed with better
transmission performances in the case of a complex scenario in which the
additional information is key for policy parameter tuning.

##  V Conclusions

We have proposed a novel multi-agent reinforcement learning framework for MAC
protocol learning, which in addition to using the classical extrinsic team
reward, learns an individual intrinsic reward for each agent based on its
history. Each agent uses two modules, namely a policy network and an intrinsic
reward network. These two modules are updated with a different periodicity to
obtain better learning results in terms of convergence speed. Specifically,
the policy network is trained within an episode, while the intrinsic reward
network is trained over a fixed number of subsequent episodes, called
lifetime. We formulated an optimization problem that seeks to maximize the
number of successfully transmitted packets. Our results demonstrate that
exploiting these two modules with two different learning periodicities induces
a faster convergence speed compared to several baseline solutions.

## Acknowledgments

This work was partially supported by the European Union under the Italian
National Recovery and Resilience Plan (NRRP) of NextGenerationEU, partnership
on “Telecommunications of the Future” (PE00000001 - program “RESTART”), by the
Italian MUR PON 2014-2020 under Project “reCITY - Resilient City - Everyday
Revolution” (cod. ARS01_00592, CUP B69C21000390005), and by the European
Union’s Horizon Europe program through the project CENTRIC.

## References

  * [1] A. Lazaridou and M. Baroni, “Emergent multi-agent communication in the deep learning era,” 2020. [Online]. Available: https://arxiv.org/abs/2006.02419
  * [2] A. Valcarce and J. Hoydis, “Toward joint learning of optimal MAC signaling and wireless channel access,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 7, no. 4, pp. 1233–1243, 2021.
  * [3] L. Miuccio, S. Riolo, S. Samarakoon, D. Panno, and M. Bennis, “Learning generalized wireless MAC communication protocols via abstraction,” in _GLOBECOM 2022 - 2022 IEEE Global Communications Conference_ , 2022, pp. 2322–2327.
  * [4] G. Papoudakis, F. Christianos, A. Rahman, and S. V. Albrecht, “Dealing with non-stationarity in multi-agent deep reinforcement learning,” 2019. [Online]. Available: https://arxiv.org/abs/1906.04737
  * [5] X. Chu and H. Ye, “Parameter sharing deep deterministic policy gradient for cooperative multi-agent reinforcement learning,” 2017. [Online]. Available: https://arxiv.org/abs/1710.00336
  * [6] Z. Zheng, J. Oh, and S. Singh, “On learning intrinsic rewards for policy gradient methods,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18, 2018, pp. 4644–4654.
  * [7] Z. Zheng, J. Oh, M. Hessel, Z. Xu, M. Kroiss, H. van Hasselt, D. Silver, and S. Singh, “What can learned intrinsic rewards capture?” 2019. [Online]. Available: https://arxiv.org/abs/1912.05500
  * [8] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [9] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. M. Bayen, and Y. Wu, “The surprising effectiveness of MAPPO in cooperative, multi-agent games,” _CoRR_ , vol. abs/2103.01955, 2021.
  * [10] J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” _CoRR_ , vol. abs/1412.3555, 2014. [Online]. Available: http://arxiv.org/abs/1412.3555
  * [11] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in _Advances in Neural Information Processing Systems_ , S. Solla, T. Leen, and K. Müller, Eds., vol. 12. MIT Press, 1999.
  * [12] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine Learning_ , vol. 8, pp. 229–256, 2004.
  * [13] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
  * [14] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _Proceedings of the 32nd International Conference on Machine Learning_. PMLR, July 2015, pp. 1889–1897.
  * [15] R. Marini, S. Park, O. Simeone, and C. Buratti, “Continual meta-reinforcement learning for UAV-aided vehicular wireless networks,” 2022. [Online]. Available: https://arxiv.org/abs/2207.06131