# Active Reconfigurable Intelligent Surface for Mobile Edge Computing

Zhangjie Peng, Ruisong Weng, Zhenkun Zhang,
Cunhua Pan, , and Jiangzhou Wang Z. Peng, R. Weng and Z. Zhang are with the
College of Information, Mechanical and Electrical Engineering, Shanghai Normal
University, Shanghai 200234, China (e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>1000479070@smail.shnu.edu.cn).C. Pan is with the
National Mobile Communications Research Laboratory, Southeast University,
China. (e-mail: cpan@seu.edu.cn).J. Wang is with the School of Engineering,
University of Kent, CT2 7NT Canterbury, U.K. (e-mail: j.z.wang@kent.ac.uk).

###### Abstract

This paper investigates an active reconfigurable intelligent surface
(RIS)-aided mobile edge computing (MEC) system. Compared with passive RIS, the
active RIS is equipped with active reflective amplifier, which can effectively
circumvent the “double path loss” attenuation. We propose a joint computing
and communication design to minimize the maximum computational latency (MCL),
subject to both the phase shift constraints and the edge computing capability
constraints. Specifically, the original problem is decoupled into four
subproblems, and then the block coordinate descent (BCD) method and the
successive convex approximation (SCA) method are applied to alternately
optimize the subproblems. The simulation results show that with the same power
budget, the performance gain achieved by the active RIS is much larger than
that by the passive RIS.

###### Index Terms:

Mobile edge computing (MEC), latency minimization, Internet of things,
reconfigurable intelligent surface (RIS), active RIS.

##  I Introduction

With the rapid increase in the amount of data generated by the Internet of
Things (IoT), the traditional cloud computing is difficult to meet the high
quality of service (QoS) requirements of the emerging applications [1]. Mobile
edge computing (MEC), a new paradigm that is regarded as one of the most
promising techniques in the future communication system, was proposed in [2].
In MEC systems, powerful and reliable computing resources are deployed at the
edge of the wireless networks to serve the mobile devices, enabling them to
execute computation-intensive and latency-critical applications. However, the
computation offloading links may suffer from severe signal attenuation or
blockage, which will increase the offloading latency, and inevitably
deteriorate the overall performance of MEC system [3].

Recently, reconfigurable intelligent surfaces (RISs) has been envisioned as an
effective solution for enhancing the channel gains of offloading links in MEC
systems [4]. RIS is a thin planar board that is composed of an array of low-
cost and passive reflecting elements integrated with controllable electronics.
By reconfiguring the electromagnetic propagation environment and constructing
the reflection links, RISs can effectively improve the offloading data rate of
the edge devices in MEC systems [5, 6]. However, when an RIS is deployed
between the receiver and the transmitter with favourable communication link,
the performance gain is limited [7]. Besides, the potential of RIS is
significantly restricted by the inevitable “double path loss” attenuation in
the reflection link, i.e., the signal received through this link will suffer
from the large-scale fading twice [8]. To this end, the concept of active RIS,
which is expected to circumvent these disadvantages, has been proposed in [8,
9, 10, 11]. Different from passive RIS, the active RIS is equipped with an
integrated active reflection-type amplifier, which is able to amplify the
received signal by consuming additional power. The application of this device
has been explored in various scenarios, such as multiple-input single-output
(MISO) systems[8], multiple-input multiple-output (MIMO) systems[7] and IoT
networks [10], and the results in [8]-[10] showed that active RIS can achieve
significant performance gain in terms of data rate [7, 12, 9] and physical
layer security [13, 14].

However, there is a lack of studies on the use of active RIS in MEC systems.
In this paper, we propose to minimize the maximum computational latency (MCL)
of the MEC system by jointly optimizing the receive beamforming of the access
point (AP), the reflection coefficients of the active RIS, the computation
offloading volume and the computing resource allocation. The MCL minimization
problem is solved by applying the block coordinate descent (BCD) method and
the successive convex approximation (SCA) method. The simulation results not
only verify the effectiveness of our algorithm, but also show that the
performance gain achieved by the active RIS is much larger than that by using
the passive RIS with the same power budget.

##  II System Model

Figure 1: An active RIS-aided MEC system with an $N$-antenna AP and $K$
single-antenna users.

###  II-A Signal Transmission Model

Consider an MEC system as shown in Fig. 1, which consists of an AP equipped
with $N$ receive antennas and $K$ single-antenna users. An MEC server has a
wired connection to the AP with negligible data transmission latency, to which
each user can offload a certain fraction of or all of its computational tasks
through the communication link. To assist the users’ computation offloading,
an active RIS with $M$ reflecting elements is deployed to reconfigure the
propagation environment and enhance the reflected signals.

The baseband equivalent channels from the RIS to the AP, from the $k$-th user
to the RIS, and from the $k$-th user to the AP are denoted by
$\mathbf{H}\in\mathbb{C}^{N\times M}$, $\mathbf{h}_{k}\in\mathbb{C}^{M\times
1}$ and $\mathbf{g}_{k}\in\mathbb{C}^{N\times 1}$, respectively.
$\theta_{m}=\lambda_{m}e^{j\phi_{m}}$ is the reflection coefficient of the
$m$-th reflecting element of the RIS, and the phase shift
$\phi_{m}\in\left[0,2\pi\right]$. Then, the reflecting beamforming of the RIS
can be represented as the reflection coefficient vector
$\bm{\theta}=\left[\theta_{1},\theta_{2},\dotsc,\theta_{M}\right]^{\mathrm{T}}$.
Different from passive RISs, the reflecting elements of an active RIS are
equipped with amplifiers that consume additional power, and thus the thermal
noise at the active RIS is not negligible [7]. Hence, the signal received at
the AP is modelled as

$\mathbf{y}=\sum_{k=1}^{K}{\sqrt{p_{k}}\left(\mathbf{H\Theta
h}_{k}+\mathbf{g}_{k}\right)s_{k}+\mathbf{H\Theta v}+\mathbf{n}},$ (1)

where $\mathbf{\Theta}\triangleq\mathrm{diag}\left(\bm{\theta}\right)$,
$s_{k}$ is the data symbol sent by the $k$-th user, $p_{k}$ denotes the
corresponding computation offloading power, and
$\mathbf{v}\in\mathbb{C}^{M\times
1}\sim\mathcal{C}\mathcal{N}\left(\mathbf{0}_{M},\sigma^{2}\mathbf{I}_{M}\right)$
and $\mathbf{n}\in\mathbb{C}^{N\times
1}\sim\mathcal{C}\mathcal{N}\left(\mathbf{0}_{N},\delta^{2}\mathbf{I}_{N}\right)$
denote the thermal noise at the RIS and the AP, respectively. Define
$\mathcal{M}\\!=\\!\left\\{1,2,\dotsc,M\right\\}$ and
$\mathcal{K}=\left\\{1,2,\dotsc,K\right\\}$. It is assumed that each
$s_{k}\sim\mathcal{C}\mathcal{N}\left(0,1\right)$ for $\forall
k\in\mathcal{K}$.

We assume that the AP decodes the received signal by using linear receive
beamforming. By denoting
$\mathbf{F}=\left[\mathbf{f}_{1},\dotsc,\mathbf{f}_{K}\right]\in\mathbb{C}^{N\times
K}$ as the set of the receiving beamforming vectors, the signal recovered at
the AP is given by

$\mathbf{\hat{s}}=\mathbf{F}^{\mathrm{H}}\mathbf{y},$ (2)

where
$\mathbf{\hat{s}}=\left[\hat{s}_{1},\dotsc,\hat{s}_{K}\right]^{\mathrm{T}}$
and $\hat{s}_{k}$ is the recovered signal for the $k$-th user. Then, the
achievable computation offloading rate (bit/s) of the $k$-th user is given by

$R_{k}=B\log_{2}\left(1+\gamma_{k}\right),$ (3)

where $B$ is the channel bandwidth and

$\gamma_{k}=\frac{p_{k}\left|\mathbf{f}_{k}^{\mathrm{H}}\left(\mathbf{H\Theta
h}_{k}+\mathbf{g}_{k}\right)\right|^{2}}{\sum\limits_{\begin{subarray}{c}i=1\\\
i\neq
k\end{subarray}}^{K}{p_{i}\left|\mathbf{f}_{k}^{\mathrm{H}}\left(\mathbf{H\Theta
h}_{i}+\mathbf{g}_{i}\right)\right|^{2}}+\sigma^{2}\left\|\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H\Theta}\right\|^{2}+\delta^{2}\left\|\mathbf{f}_{k}^{\mathrm{H}}\right\|^{2}}.$
(4)

###  II-B Computation Task Model

We consider the partial offloading scheme by applying the data-partition model
[2], which is widely used in the existing literature [5, 15]. Specifically,
the task-input bits are assumed to be bit-wise independent. Each user divides
its computational task into two parts, which are respectively handled locally
and offloaded to the edge computing node. The latency model of the considered
MEC system is detailed as follows:

####  II-B1 Local Computing

The total computational task and the offloading volume of the $k$-th user in
terms of the number of bits are denoted by $L_{k}$ and $l_{k}$, respectively.
In addition, the computing power of the $k$-th user is quantified by its
number of central processing unit (CPU) cycles per second $f_{\mathrm{L},k}$.
Denote by $c_{k}$ the number of CPU cycles required to process each bit. Then,
the latency at the $k$-th user induced by local computing can be expressed as

$T_{\mathrm{L},k}=\frac{\left(L_{k}-l_{k}\right)c_{k}}{f_{\mathrm{L},k}}.$ (5)

####  II-B2 Edge Computing

The MEC server is assumed to start the process of the offloaded task of the
$k$-th user at the time when all $l_{k}$ bits are completely received.
Compared with offloaded computational task, the amount of bits of the
computational results is ususally considered to be much smaller, so the
feedback delay can be reduced to a negligible level [16]. Hence, the time
required for executing the $k$-th user’s offloaded computational task is
mainly composed of that required for computation offloading and edge
computing, i.e.,

$T_{\mathrm{E},k}=\frac{l_{k}}{R_{k}}+\frac{l_{k}c_{k}}{f_{\mathrm{E},k}},$
(6)

where $f_{\mathrm{E},k}$ (cycle/s) is the edge computing resource allocated to
the $k$-th user.

It is assumed that the users are able to carry out the local computing and
computation offloading in parallel. In this case, the total latency of the
$k$-th user can be formulated as

$T_{k}=\max\left\\{T_{\mathrm{L},k},T_{\mathrm{E},k}\right\\}.$ (7)

###  II-C Problem Formulation

To solve the problem of minimizing the MCL of the MEC system, we propose to
jointly optimize the receive beamforming matrix $\mathbf{F}$, the reflection
coefficient vector $\bm{\theta}$, the computation offloading power
$\mathbf{p}=\left[\sqrt{p_{1}},\dotsc,\sqrt{p_{K}}\right]^{\mathrm{T}}$, the
computation offloading volume
$\bm{l}=\left[l_{1},\dotsc,l_{K}\right]^{\mathrm{T}}$ and the computing
resource allocation
$\bm{f}_{\mathrm{E}}=\left[f_{\mathrm{E},1},\dotsc,f_{\mathrm{E},K}\right]^{\mathrm{T}}$.
Specifically, the MCL minimization problem is formulated as

$\displaystyle\min_{\mathbf{F},\bm{\theta},\mathbf{p},\bm{l},\bm{f}_{\mathrm{E}}}\quad$
$\displaystyle\max_{k\in\mathcal{K}}\left\\{T_{k}\right\\}$ (8a) s.t.
$\displaystyle p_{k}\in\left[0,p_{max}\right],k\in\mathcal{K},$ (8b)
$\displaystyle l_{k}\in\left\\{0,1,\dotsc,L_{k}\right\\},k\in\mathcal{K},$
(8c) $\displaystyle f_{\mathrm{E},k}\geqslant 0,\;k\in\mathcal{K},$ (8d)
$\displaystyle\sum_{k=1}^{K}{f_{\mathrm{E},k}}\leqslant
f_{\mathrm{E}}^{\mathrm{tot}},$ (8e)
$\displaystyle\sum_{k=1}^{K}{p_{k}\left\|\mathbf{\Theta
h}_{k}\right\|^{2}}+\left\|\mathbf{\Theta}\right\|^{2}\sigma^{2}\leqslant
P_{\mathrm{aRIS}},$ (8f)

where $f_{\mathrm{E}}^{\mathrm{tot}}$ denote the computing power of the MEC
server. In the power constraint (8f), the amplification power budget is
modelled as
$P_{\mathrm{aRIS}}=\xi\left(P_{\mathrm{tot}}-M\left(P_{\mathrm{DC}}+P_{\mathrm{c}}\right)\right)$,
where $\xi$, $P_{\mathrm{tot}}$, $P_{\mathrm{DC}}$ and $P_{\mathrm{c}}$ are
the amplifier efficiency, the total power budget, the DC biasing power
consumption and the circuit power consumption of the active RIS, respectively
[12].

##  III Joint Computing and Communication Design

In this section, an effective algorithm is proposed to solve the MCL
minimization problem (8) based on the BCD method, where the variables are
optimized alternately.

###  III-A Local and Edge Computing Design

By fixing the receive beamforming matrix $\mathbf{F}$, the reflection
coefficient vector $\bm{\theta}$ and the computation offloading power
$\mathbf{p}$, we can formulate the subproblem for local and edge computing
design as follows

$\displaystyle\min_{\bm{l},\bm{f}_{\mathrm{E}}}\quad$
$\displaystyle\max_{k\in\mathcal{K}}\left\\{T_{k}\right\\}$ (9) s.t.
$\displaystyle\eqref{constraint_l_k},\eqref{constraint_f_E>0},\eqref{constraint_f_E}.$

In the following, Problem (9) is solved by alternately optimizing each subset
of the variables.

####  III-A1 Optimizing the computation offloading volume $\bm{l}$

Note that the feasible region of $\bm{l}$ is discrete. By relaxing $\bm{l}$ to
continuous variables, i.e., $\tilde{l}_{k}\in\left[0,L_{k}\right]$ and
$\bm{\tilde{l}}=\left[\tilde{l}_{1},\dotsc,\tilde{l}_{K}\right]^{\mathrm{T}}$,
we have

$\displaystyle
T_{k}\left(\tilde{l}_{k}\right)=\max\left\\{T_{\mathrm{L},k}(\tilde{l}_{k}),T_{\mathrm{E},k}(\tilde{l}_{k})\right\\}$
$\displaystyle=\begin{cases}\frac{\left(L_{k}-\tilde{l}_{k}\right)c_{k}}{f_{\mathrm{L},k}},\\!\\!&0\leqslant\tilde{l}_{k}\leqslant\frac{L_{k}c_{k}R_{k}f_{\mathrm{E},k}}{f_{\mathrm{L},k}f_{\mathrm{E},k}+c_{k}R_{k}\left(f_{\mathrm{L},k}+f_{\mathrm{E},k}\right)}\\\
\frac{\tilde{l}_{k}}{R_{k}}+\frac{\tilde{l}_{k}c_{k}}{f_{\mathrm{E},k}},\\!\\!&\frac{L_{k}c_{k}R_{k}f_{\mathrm{E},k}}{f_{\mathrm{L},k}f_{\mathrm{E},k}+c_{k}R_{k}\left(f_{\mathrm{L},k}+f_{\mathrm{E},k}\right)}\\!\leqslant\\!\tilde{l}_{k}\leqslant
L_{k}\\\ \end{cases}\\!.$ (10)

It can be readily verified that $T_{k}\left(\tilde{l}_{k}\right)$ is minimized
at

$\tilde{l}_{k}^{\mathrm{opt}}=\frac{L_{k}c_{k}R_{k}f_{\mathrm{E},k}}{f_{\mathrm{L},k}f_{\mathrm{E},k}+c_{k}R_{k}\left(f_{\mathrm{L},k}+f_{\mathrm{E},k}\right)}.$
(11)

Then, the optimal $l_{k}$ can be obtained as

$l_{k}^{\mathrm{opt}}={\mathrm{arg}\min}_{\tilde{l}_{k}\in\left\\{\lfloor\tilde{l}_{k}^{\mathrm{opt}}\rfloor,\lceil\tilde{l}_{k}^{\mathrm{opt}}\rceil\right\\}}\,\,T_{k}\left(\tilde{l}_{k}\right),$
(12)

where $\lceil\cdot\rceil$ and $\lfloor\cdot\rfloor$ represent the ceiling and
floor operations, respectively.

####  III-A2 Optimizing the computing resource allocation
$\bm{f}_{\mathrm{E}}$

By letting $l_{k}=\tilde{l}_{k}^{\mathrm{opt}}$, $k\in\mathcal{K}$ in Problem
(9), the subproblem corresponding to $\bm{f}_{\mathrm{E}}$ can be reformulated
as

$\displaystyle\min_{\bm{f}_{\mathrm{E}}}\quad$
$\displaystyle\max_{k\in\mathcal{K}}\left\\{\frac{L_{k}c_{k}f_{\mathrm{E},k}+L_{k}c_{k}^{2}R_{k}}{f_{\mathrm{L},k}f_{\mathrm{E},k}+c_{k}R_{k}\left(f_{\mathrm{L},k}+f_{\mathrm{E},k}\right)}\right\\}$
(13) s.t. $\displaystyle\eqref{constraint_f_E>0},\eqref{constraint_f_E}.$

By introducing an auxiliary variable $\eta$, Problem (13) can be transformed
as

$\displaystyle\min_{\eta,\bm{f}_{\mathrm{E}}}\quad$ $\displaystyle\eta$ (14a)
s.t.
$\displaystyle\frac{L_{k}c_{k}f_{\mathrm{E},k}+L_{k}c_{k}^{2}R_{k}}{f_{\mathrm{L},k}f_{\mathrm{E},k}+c_{k}R_{k}\left(f_{\mathrm{L},k}+f_{\mathrm{E},k}\right)}\leqslant\eta,\;\forall
k\in\mathcal{K},$ (14b)
$\displaystyle\eqref{constraint_f_E>0},\eqref{constraint_f_E}.$

To solve this problem, the method based on the SCA method is applied. Firstly,
by introducing auxiliary variables
$\bm{a}=\left[a_{1},\dotsc,a_{K}\right]^{\mathrm{T}}$ and
$\bm{b}=\left[b_{1},\dotsc,b_{K}\right]^{\mathrm{T}}$, constraint (14b) can be
replaced with the following constraints

$\displaystyle\frac{L_{k}c_{k}}{a_{k}}\leqslant\frac{c_{k}R_{k}f_{\mathrm{L},k}}{f_{\mathrm{E},k}}+f_{\mathrm{L},k}+c_{k}R_{k}\\!\triangleq\\!g_{k}(f_{\mathrm{E},k}),\forall
k\in\mathcal{K},$ (15)
$\displaystyle\frac{L_{k}c_{k}^{2}R_{k}}{b_{k}}\leqslant\left(f_{\mathrm{L},k}+c_{k}R_{k}\right)f_{\mathrm{E},k}+c_{k}R_{k}f_{\mathrm{L},k},\forall
k\in\mathcal{K},$ (16) $\displaystyle a_{k}+b_{k}\leqslant\eta,\;\forall
k\in\mathcal{K}.$ (17)

Note that constraint (15) is non-convex. Then, by utilizing the inequality
(3.2) in [17] that a convex function is lower-bounded by its first-order
Taylor expansion, we have

$\displaystyle\frac{L_{k}c_{k}}{a_{k}}\leqslant\frac{c_{k}R_{k}f_{\mathrm{L},k}}{f_{\mathrm{E},k}^{r}}-\frac{c_{k}R_{k}f_{\mathrm{L},k}}{f_{\mathrm{E},k}^{r,2}}\left(f_{\mathrm{E},k}-f_{\mathrm{E},k}^{r}\right)+f_{\mathrm{L},k}+c_{k}R_{k}$
$\displaystyle\triangleq h_{k}\left(f_{\mathrm{E},k}\mid
f_{\mathrm{E},k}^{r}\right),\forall k\in\mathcal{K},$ (18)

where $f_{\mathrm{E},k}^{r}$ is the value of $f_{\mathrm{E},k}$ at the $r$-th
iteration. It can be readily verified that when
$f_{\mathrm{E},k}=f_{\mathrm{E},k}^{r}$, $h_{k}(f_{\mathrm{E},k}\mid
f_{\mathrm{E},k}^{r})=g_{k}(f_{\mathrm{E},k})$. Finally, the problem of
$\bm{f}_{\mathrm{E}}$ to be solved at the $r$-th SCA iteration is formulated
as

$\displaystyle\min_{\eta,\bm{f}_{\mathrm{E}},\bm{a},\bm{b}}\quad$
$\displaystyle\eta$ (19) s.t.
$\displaystyle\eqref{constraint_f_E>0},\eqref{constraint_f_E},\eqref{constraint_SCA_b}-\eqref{constraint_SCA_a}.$

The globally optimal solution of this problem can be obtained by the common
optimization tools, e.g., CVX. By denoting the objective function of Problem
(8) as
$\mathcal{T}\left(\mathbf{F},\bm{\theta},\bm{l},\bm{f}_{\mathrm{E}}\right)$,
the detail of the proposed SCA algorithm for solving Problem (19) is
summarized in Algorithm 1, which is guaranteed to converge to the Karush-Kuhn-
Tucker solution of Problem (14), as proved in [18].

Algorithm 1 SCA Algorithm for solving Problem (19)

Initialize: Initialize the number of iterations as $r=1$, set feasible
$\bm{f}_{\mathrm{E}}^{1}$, the maximum number of iterations $r_{\max}$ and the
error tolerance $\varrho$.

1:while
$\left|\mathcal{T}\left(\bm{f}_{\mathrm{E}}^{r+1}\right)-\mathcal{T}\left(\bm{f}_{\mathrm{E}}^{r}\right)\right|/\mathcal{T}\left(\bm{f}_{\mathrm{E}}^{r}\right)\geqslant\varrho$
and $n\leqslant r_{\max}$ do

2: Calculate $\bm{f}_{\mathrm{E}}^{r+1}$ by solving Problem (19);

3: Set $r\leftarrow r+1$;

4:end while

###  III-B Signal Transmission Design

Given the computation offloading volume $\bm{l}$ and the computing resource
allocation $\bm{f}_{\mathrm{E}}$, the optimal signal transmission design can
be obtained by solving the following subproblem

$\displaystyle\min_{\mathbf{F},\bm{\theta},\mathbf{p}}\quad$
$\displaystyle\max_{k\in\mathcal{K}}\left\\{T_{k}\right\\}$ (20) s.t.
$\displaystyle\eqref{constraint_p},\eqref{constraint_P}$

As discussed in Section III-A, the optimal value of the objective function of
Problem (8) satisfies
$T_{k}^{\mathrm{opt}}=T_{\mathrm{L},k}^{\mathrm{opt}}=T_{\mathrm{E},k}^{\mathrm{opt}}$.
In addition, the expression of $T_{\mathrm{L},k}$ in (5) is independent of
$\mathbf{F}$ and $\bm{\theta}$. Thus, we replace $T_{k}$ with
$T_{\mathrm{E},k}$ in (III-B). Furthermore, by introducing an auxiliary
variable $\varepsilon$, Problem (III-B) can be transformed as

$\displaystyle\min_{\mathbf{F},\bm{\theta},\mathbf{p},\varepsilon}\quad$
$\displaystyle\varepsilon$ (21a) s.t.
$\displaystyle\frac{l_{k}}{R_{k}}+\frac{l_{k}c_{k}}{f_{\mathrm{E},k}}\leqslant\varepsilon,\;\forall
k\in\mathcal{K},$ (21b)
$\displaystyle\eqref{constraint_p},\eqref{constraint_P}.$

Note that constraint (8f) is convex. Next, we transform constraint (21b) by
utilizing the equivalence between data rate and the mean-square error [19].
From (2), we can derive the mean-square error of the signal recovered at the
AP corresponding to the $k$-th user as follows

$\displaystyle d_{k}$
$\displaystyle=\sum_{i=1}^{K}{\left|\sqrt{p_{i}}\mathbf{f}_{k}^{\mathrm{H}}\left(\mathbf{H\Theta
h}_{i}+\mathbf{g}_{i}\right)\right|^{2}}+\sigma^{2}\left\|\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H\Theta}\right\|^{2}+\delta^{2}\left\|\mathbf{f}_{k}^{\mathrm{H}}\right\|^{2}$
$\displaystyle\quad-2\mathrm{Re}\left\\{\sqrt{p_{k}}\mathbf{f}_{k}^{\mathrm{H}}\left(\mathbf{H\Theta
h}_{k}+\mathbf{g}_{k}\right)\right\\}+1.$ (22)

By introducing a set of auxiliary variables
$\mathcal{V}=\left\\{v_{k}\geqslant 0,k\in\mathcal{K}\right\\}$, $R_{k}$ can
be reformulated as

$\tilde{R}_{k}(\mathbf{F},\mathbf{\Theta},\mathbf{p},\mathcal{V})=B\left(\log_{2}\left|v_{k}\right|-\frac{v_{k}d_{k}}{\log_{\mathrm{e}}2}+\frac{1}{\log_{\mathrm{e}}2}\right).$
(23)

Note that $\tilde{R}_{k}(\mathbf{F},\mathbf{\Theta},\mathbf{p},\mathcal{V})$
is a concave function with respect to each variable when the others are fixed,
which is more tractable than $R_{k}$. Therefore, Problem (21) can be
reformulated as

$\displaystyle\min_{\mathbf{F},\bm{\theta},\mathbf{p},\mathcal{V},\varepsilon}\quad$
$\displaystyle\varepsilon$ (24a) s.t.
$\displaystyle\tilde{R}_{k}\geqslant\frac{l_{k}{f_{\mathrm{E},k}}}{\varepsilon{f_{\mathrm{E},k}}-l_{k}c_{k}},\;\forall
k\in\mathcal{K},$ (24b)
$\displaystyle\eqref{constraint_p},\eqref{constraint_P}.$

Then, we alternately optimize the objective function over $\mathcal{V}$,
$\mathbf{F}$, $\bm{\theta}$ and $\mathbf{p}$ to solve Problem (24).

####  III-B1 Optimizing the Auxiliary Variables $\mathcal{V}$ and the Receive
Beamforming Matrix $\mathbf{F}$

For $k\in\mathcal{K}$, the optimal $v_{k}$ can be obtained by setting the
first-order derivative of
$\tilde{R}_{k}(\mathbf{F},\bm{\lambda},\bm{\phi},\mathcal{V})$ with respect to
$v_{k}$ to zero, which is given by

$\displaystyle v_{k}^{\mathrm{opt}}=d_{k}^{-1}.$ (25)

Similarly, when the other variables are fixed, the optimal $\mathbf{f}_{k}$
can be derived as

$\displaystyle\mathbf{f}_{k}^{\mathrm{opt}}$
$\displaystyle=\sqrt{p_{k}}\left(\sum_{i=1}^{K}{p_{i}\left(\mathbf{H\Lambda\Phi
h}_{i}+\mathbf{g}_{i}\right)\left(\mathbf{H\Lambda\Phi
h}_{i}+\mathbf{g}_{i}\right)^{\mathrm{H}}}+\delta^{2}\mathbf{I}_{N}\right.$
$\displaystyle\quad+\left.\sigma^{2}\mathbf{H\Lambda\Phi\Phi}^{\mathrm{H}}\mathbf{\Lambda}^{\mathrm{H}}\mathbf{H}^{\mathrm{H}}\right)^{-1}\left(\mathbf{H\Lambda\Phi
h}_{k}+\mathbf{g}_{k}\right).$ (26)

####  III-B2 Optimizing the Reflection Coefficient $\bm{\theta}$

Based on the equality in [20, Eq. (1.10.6)], we can rewrite (III-B) as

$\displaystyle d_{k}$
$\displaystyle=\mathrm{Tr}\left(\mathbf{\Theta}^{\mathrm{H}}\mathbf{H}^{\mathrm{H}}\mathbf{f}_{k}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H\Theta
A}\right)+\sigma^{2}\mathrm{Tr}\left(\mathbf{\Theta}^{\mathrm{H}}\mathbf{H}^{\mathrm{H}}\mathbf{f}_{k}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H\Theta}\right)$
$\displaystyle\quad+2\mathrm{Re}\left\\{\mathrm{Tr}\left(\mathbf{W}_{k}\mathbf{\Theta}\right)\right\\}+a_{k}$
$\displaystyle=\bm{\theta}^{\mathrm{H}}\mathbf{B}_{k}\bm{\theta}+2\mathrm{Re}\left\\{\mathbf{w}_{k}\bm{\theta}\right\\}+a_{k},$
(27)

where

$\displaystyle\mathbf{A}$
$\displaystyle\triangleq\sum_{i=1}^{K}{p_{i}\mathbf{h}_{i}\mathbf{h}_{i}^{\mathrm{H}}},\mathbf{w}_{k}\triangleq\left[\left[\mathbf{W}_{k}\right]_{1,1},\dotsc,\left[\mathbf{W}_{k}\right]_{M,M}\right]^{\mathrm{T}},$
$\displaystyle\mathbf{W}_{k}$
$\displaystyle\triangleq\sum_{i=1}^{K}{p_{i}\mathbf{h}_{i}\mathbf{g}_{i}^{\mathrm{H}}}\mathbf{f}_{k}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H}-\sqrt{p_{k}}\mathbf{h}_{k}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H},$
$\displaystyle a_{k}$
$\displaystyle\triangleq\mathbf{f}_{k}^{\mathrm{H}}\left(\sum_{i=1}^{K}{p_{i}\mathbf{g}_{i}\mathbf{g}_{i}^{\mathrm{H}}}\right)\mathbf{f}_{k}+\delta^{2}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{f}_{k}\\!-\\!2\mathrm{Re}\left\\{\sqrt{p_{k}}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{g}_{k}\right\\}\\!+\\!1,$
$\displaystyle\mathbf{B}_{k}$
$\displaystyle\triangleq\left(\mathbf{H}^{\mathrm{H}}\mathbf{f}_{k}\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H}\right)\odot\left(\mathbf{A}+\sigma^{2}\mathbf{I}_{M}\right)^{\mathrm{T}}.$

By substituting (III-B2) into (23), we have

$\bar{R}_{k}(\bm{\theta})=-\bm{\theta}^{\mathrm{H}}\mathbf{\bar{B}}_{k}\bm{\theta}-2\mathrm{Re}\left\\{\mathbf{\bar{w}}_{k}^{\mathrm{H}}\bm{\theta}\right\\}+\bar{a}_{k},$
(28)

where $\bar{a}_{k}\triangleq
B\left(\log_{2}\left|v_{k}\right|-\frac{v_{k}a_{k}}{\log_{\mathrm{e}}2}+\frac{1}{\log_{\mathrm{e}}2}\right)$,
$\mathbf{\bar{B}}_{k}\triangleq\frac{v_{k}B\mathbf{B}_{k}}{\log_{\mathrm{e}}2}$
and
$\mathbf{\bar{w}}_{k}\triangleq\frac{v_{k}B\mathbf{w}_{k}}{\log_{\mathrm{e}}2}$.
Similarly, constraint (8f) can be rewritten as

$\bm{\theta}^{\mathrm{H}}\left(\mathbf{I}_{M}\odot\mathbf{A}^{\mathrm{T}}+\sigma^{2}\mathbf{I}_{M}\right)\bm{\theta}\leqslant
P_{\mathrm{aRIS}}.$ (29)

Then, the subproblem of $\bm{\theta}$ is formulated as

$\displaystyle\min_{\bm{\theta},\varepsilon}\quad$ $\displaystyle\varepsilon$
(30a) s.t.
$\displaystyle\bar{R}_{k}\left(\bm{\theta}\right)\geqslant\frac{l_{k}{f_{\mathrm{E},k}}}{\varepsilon{f_{\mathrm{E},k}}-l_{k}c_{k}},\;\forall
k\in\mathcal{K},$ (30b) $\displaystyle\eqref{constraint_theta}.$

From (6), we have $\varepsilon\geqslant
T_{\mathrm{E},k}>\frac{l_{k}c_{k}}{f_{\mathrm{E},k}}$. Problem (30) is a
second-order cone programming problem, and its globally optimal solution can
also be obtained by existing optimization tools.

####  III-B3 Optimizing the Computation Offloading Power $\mathbf{p}$

We introduce a selection vector $\mathbf{t}_{k}\in\mathbb{R}^{K\times 1}$, in
which the $k$-th element is equal to 1, while other elements are 0. Then,
(III-B) can be rewritten as

$\displaystyle
d_{k}=\mathbf{p}^{\mathrm{T}}\mathbf{C}_{k}\mathbf{p}-2\mathrm{Re}\left\\{\mathbf{p}^{\mathrm{T}}\mathbf{j}_{k}\right\\}+m_{k},$
(31)

where

$\displaystyle\mathbf{C}_{k}$
$\displaystyle\triangleq\sum_{i=1}^{K}{\mathbf{t}_{k}\mathbf{f}_{k}^{\mathrm{H}}\left(\mathbf{H\Theta
h}_{i}+\mathbf{g}_{i}\right)\left(\mathbf{H\Theta
h}_{i}+\mathbf{g}_{i}\right)^{\mathrm{H}}\mathbf{f}_{k}\mathbf{t}_{k}^{\mathrm{H}}},$
$\displaystyle\mathbf{j}_{k}$
$\displaystyle\triangleq\mathbf{t}_{k}\mathbf{f}_{k}^{\mathrm{H}}\left(\mathbf{H\Theta
h}_{k}+\mathbf{g}_{k}\right),$ $\displaystyle m_{k}$
$\displaystyle\triangleq\sigma^{2}\left\|\mathbf{f}_{k}^{\mathrm{H}}\mathbf{H\Theta}\right\|^{2}+\delta^{2}\left\|\mathbf{f}_{k}^{\mathrm{H}}\right\|^{2}+1.$

By substituting (31) into (23), we have

$\widehat{R}_{k}(\mathbf{p})=-\mathbf{p}^{\mathrm{T}}\mathbf{\widehat{C}}_{k}\mathbf{p}+2\mathrm{Re}\left\\{\mathbf{p}^{\mathrm{T}}\mathbf{\widehat{j}}_{k}\right\\}+\widehat{m}_{k},$
(32)

where $\widehat{m}_{k}\triangleq
B\left(\log_{2}\left|v_{k}\right|-\frac{v_{k}m_{k}}{\log_{\mathrm{e}}2}+\frac{1}{\log_{\mathrm{e}}2}\right)$,
$\mathbf{\widehat{C}}_{k}\triangleq\frac{v_{k}B\mathbf{C}_{k}}{\log_{\mathrm{e}}2}$
and
$\mathbf{\widehat{w}}_{k}\triangleq\frac{v_{k}B\mathbf{j}_{k}}{\log_{\mathrm{e}}2}$.

Then, constraint (8f) can be rewritten as

$\displaystyle\mathbf{p}^{\mathrm{T}}(\sum_{k=1}^{K}{\mathbf{t}_{k}\left\|\mathbf{\Theta
h}_{k}\right\|^{2}}\mathbf{t}_{k}^{\mathrm{H}})\mathbf{p}+\left\|\mathbf{\Theta}\right\|^{2}\sigma^{2}\leqslant
P_{\mathrm{aRIS}}.$ (33)

Similarly, the subproblem of $\mathbf{p}$ is formulated as

$\displaystyle\min_{\mathbf{p},\varepsilon}\quad$ $\displaystyle\varepsilon$
(34a) s.t.
$\displaystyle\widehat{R}_{k}\left(\mathbf{p}\right)\geqslant\frac{l_{k}{f_{\mathrm{E},k}}}{\varepsilon{f_{\mathrm{E},k}}-l_{k}c_{k}},\;\forall
k\in\mathcal{K},$ (34b)
$\displaystyle\eqref{constraint_p},\eqref{constraint_P_p}.$

Similar to Problem (30), Problem (34) is also a second-order cone programming
problem, whose globally optimal solution can also be obtained by existing
optimization tools.

Algorithm 2 BCD algorithm for solving Problem (8)

Initialize: Initialize
$\mathbf{F}^{1},\bm{\theta}^{1},\mathbf{p}^{1},\bm{l}^{1},\bm{f}_{\mathrm{E}}^{1}$
to feasible values. Set $n=1$, the maximum number of iterations $n_{\max}$ and
the error tolerance $\zeta$.

1:while $n\leqslant n_{\max}$ and
$\mathcal{T}\left(\mathbf{F}^{n},\bm{\theta}^{n},\mathbf{p}^{n},\bm{l}^{n},\bm{f}_{\mathrm{E}}^{n}\right)\zeta\leqslant$
$\left|\mathcal{T}\\!\left(\mathbf{F}^{n+1},\bm{\theta}^{n+1},\mathbf{p}^{n+1},\bm{l}^{n+1},\bm{f}_{\mathrm{E}}^{n+1}\right)\\!\\!-\\!\\!\mathcal{T}\\!\left(\mathbf{F}^{n},\bm{\theta}^{n},\mathbf{p}^{n},\bm{l}^{n},\bm{f}_{\mathrm{E}}^{n}\right)\right|$
do

2: Calculate $\bm{l}^{n+1}$ by using (12);

3: Calculate $\bm{f}_{\mathrm{E}}^{n+1}$ by using Algorithm 1;

4: Calculate $\mathbf{F}^{n+1}$ by solving Problem (19);

5: Calculate $\bm{\theta}^{n+1}$ by solving Problem (30);

6: Calculate $\mathbf{p}^{n+1}$ by solving Problem (34);

7: Set $n\leftarrow n+1$;

8:end while

TABLE I: Default Simulation Parameter Setting Description | Parameter and Value
---|---
Computing model |  $[L_{1},L_{2},L_{3}]=[250,300,350]$ Kb $[f_{\mathrm{L},1},f_{\mathrm{L},2},f_{\mathrm{L},3}]=[4,5,6]\times 10^{8}$ cycle/s $[c_{1},c_{2},c_{3}]=[700,750,800]$ cycle/s $f_{\mathrm{E}}^{\mathrm{tot}}=50\times 10^{9}$ cycle/s
Signal transmission model |  $B=1$ MHz, $M=16$, $N=4$ $\alpha_{\mathrm{UR}}=\alpha_{\mathrm{RA}}=2.2$, $\alpha_{\mathrm{UA}}=2.8$ $\xi=0.8$, $P_{\mathrm{DC}}=-5$ dBm, $P_{\mathrm{c}}=-10$ dBm $\delta^{2}=-80$ dBm, $\sigma^{2}=-70$ dBm
Others |  $n_{\max}=100$, $\zeta=10^{-4}$

###  III-C Algorithm Development

Based on the above discussions, a BCD-based algorithm for solving Problem (8)
is proposed, the detail of which are summarized in Algorithm 2. Since the
value of the objective function
$\mathcal{T}\left(\mathbf{F},\bm{\theta},\mathbf{p},\bm{l},\bm{f}_{\mathrm{E}}\right)$
is nondecreasing at each iteration and has an upper bound, the convergence of
Algorithm 2 is guaranteed. The computational complexity mainly depends on
solving the subproblems (19) and (30). According to [21], the computational
complexity of Problem (19) is $\mathcal{O}\left(N^{3}\right)$. In addition,
the complexity of solving Problem (30) is given by
$\mathcal{O}\left(MK^{3.5}+M^{3}K^{2.5}\right)$. Denote the number of
iterations required for Algorithm 1 to converge by $r_{\mathrm{act}}$. Hence,
the overall computational complexity of Algorithm 2 for each iteration is
$\mathcal{O}\left(r_{\mathrm{act}}N^{3}+MK^{3.5}+M^{3}K^{2.5}\right)$.

##  IV Simulation Results

In the simulation scenario, the locations of the RIS and the AP are set at
$($260 m, 0$)$ and $($0, 0$)$, respectively. Three users with computation
offloading power $p_{k}=1$ mW are randomly located in a 10 m $\times$ 10 m
area centered at $($280 m, 10 m$)$. The heights of the users, the RIS and the
AP are set to 1.5 m, 10 m and 30 m, respectively. The path loss exponents of
the reflected user-RIS, RIS-AP and user-AP channel are set to
$\alpha_{\mathrm{UR}}$, $\alpha_{\mathrm{RA}}$ and $\alpha_{\mathrm{UA}}$,
respectively. For $q\in\left\\{\mathrm{UR},\mathrm{RA},\mathrm{UA}\right\\}$,
the large-scale path loss is modelled as
$\mathrm{PL}_{q}=-10\alpha_{q}\log_{10}d_{q}-30$ dB, where $d_{q}$ is the
distance of link $q$. The default settings of these parameters are specified
in the “Signal transmission model” block of Table I. Besides, the variables
$L_{k}$, $f_{\mathrm{L},k}$, $c_{k}$ and $f_{\mathrm{E}}^{\mathrm{tot}}$ are
specified in the “Computing model” block of Table I. The default setting of
maximum number of iterations $n_{\max}$ and error tolerance $\zeta$ are
specified in the “Others” block of Table I. We consider practical scenarios,
where 2-bit control of each reflecting element is used in our simulation,
where each optimal phase shift $\phi_{m}^{\mathrm{opt}}$ generated by
Algorithm 2 is quantized as 0, $\pi/2$, $\pi$ or $3\pi/2$. In addition, we
investigate the passive RIS as a performance benchmark, where the
amplification power budget is modelled as
$P_{\mathrm{pRIS}}=P_{\mathrm{tot}}-MP_{\mathrm{c}}$ [12], and the subproblem
for optimizing the passive reflection coefficients is solved by using the
popular semidefinite relaxation method.

Fig. 2 depicts the convergence behaviour of Algorithm 2 by fixing the
amplification power budget to $P_{\mathrm{aRIS}}=10$ mW. It can be observed
that although the convergence will be slightly slower as the number of
reflection coefficients increases, the algorithm can converge within 15
iterations, which demonstrates the effectiveness of the proposed algorithm.

Figure 2: Convergence of the proposed algorithm for $M=[8,16,32]$. Figure 3:
Impact of RIS reflecting element number.

Fig. 3 and Fig. 4 plot the latency versus the number of reflecting elements
and the x coordinate of the RIS $x_{\mathrm{RIS}}$, respectively. The total
power budget is set to $P_{\mathrm{tot}}=$ 10 and 20 mW. In these cases, the
maximum numbers of reflecting elements the passive RIS can employ are 100 and
200, respectively. From Fig. 3, it is observed that when the direct link is
not severely obscured, compared to the schemes with passive RIS, the schemes
with active RIS and the corresponding 2 bit schemes can reduce the required
number of reflecting elements while achieving lower latency. Hence, active RIS
is expected to effectively overcome the double path loss attenuation, and
offers advantages in terms of size and deployment flexibility. The results in
Fig. 4 again demonstrate the advantages of active RIS in improving the
performance of MEC systems. Moreover, it is shown that the achieved latency
increases as the distance between the active RIS and the users increases. This
is because the noise power $\sigma^{2}$ at the active RIS is assumed to be
constant, while the incident signal power is stronger when the RIS is located
near the users. Therefore, the best place to deploy active RIS is close to the
users in the MEC system. In addition, Fig. 4 illustrates the individual
latency, which shows the fairness achieved by our proposed algorithm.

##  V Conclusion

In this work, we proposed to reduce the MCL of the MEC system by deploying an
active RIS. A joint computing and communication design was proposed to solve
the MCL minimization problem, subject to both the active RIS phase shift
constraints and the edge computing capability constraints. The simulation
results showed that under the condition of the same power budget, the active
RIS can achieve much better performance than the passive RIS.

Figure 4: Impact of the RIS location for $P_{\mathrm{tot}}=[10,20]$ mW.

## References

  * [1] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and challenges,” _IEEE Internet Things J._ , vol. 3, no. 5, pp. 637–646, Oct. 2016.
  * [2] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge computing: The communication perspective,” _IEEE Commun. Survey Tuts._ , vol. 19, no. 4, pp. 2322–2358, 2017.
  * [3] T. Bai, C. Pan, C. Han, and L. Hanzo, “Reconfigurable intelligent surface aided mobile edge computing,” _IEEE Wireless Commun._ , vol. 28, no. 6, pp. 80–86, Dec. 2021.
  * [4] Q. Wu, S. Zhang, B. Zheng, C. You, and R. Zhang, “Intelligent reflecting surface-aided wireless communications: A tutorial,” _IEEE Trans. Commun._ , vol. 69, no. 5, pp. 3313–3351, May 2021.
  * [5] T. Bai _et al._ , “Latency minimization for intelligent reflecting surface aided mobile edge computing,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 11, pp. 2666–2682, July 2020.
  * [6] Z. Chu, P. Xiao, M. Shojafar, D. Mi, J. Mao, and W. Hao, “Intelligent reflecting surface assisted mobile edge computing for internet of things,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 3, pp. 619–623, Mar. 2021.
  * [7] Z. Zhang _et al._ , “Active RIS vs. passive RIS: Which will prevail in 6G?” Mar 2021. [Online]. Available: https://arxiv.org/abs/2103.15154
  * [8] K. Zhi, C. Pan, H. Ren, K. K. Chai, and M. Elkashlan, “Active RIS versus passive RIS: Which is superior with the same power budget?” _IEEE Commun. Lett._ , vol. 26, no. 5, pp. 1150–1154, May 2022.
  * [9] C. You and R. Zhang, “Wireless communication aided by intelligent reflecting surface: Active or passive?” 2021. [Online]. Available: https://arxiv.org/abs/2106.10963
  * [10] G. Chen _et al._ , “Active IRS aided multiple access for energy-constrained IoT systems,” 2022. [Online]. Available: https://arxiv.org/abs/2201.12565
  * [11] Z. Peng, X. Liu, C. Pan, L. Li, and J. Wang, “Multi-pair D2D communications aided by an active RIS over spatially correlated channels with phase noise,” _IEEE Wireless Commun. Lett._ , early access, July. 2022.
  * [12] R. Long, Y.-C. Liang, Y. Pei, and E. G. Larsson, “Active reconfigurable intelligent surface-aided wireless communications,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 8, pp. 4962–4975, Aug. 2021.
  * [13] L. Dong, H.-M. Wang, and J. Bai, “Active reconfigurable intelligent surface aided secure transmission,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 2, pp. 2181–2186, Feb. 2022.
  * [14] M. H. Khoshafa, T. M. N. Ngatched, M. H. Ahmed, and A. R. Ndjiongue, “Active reconfigurable intelligent surfaces-aided wireless communication system,” _IEEE Commun. Lett._ , vol. 25, no. 11, pp. 3699–3703, Nov. 2021.
  * [15] Y. Wang, M. Sheng, X. Wang, L. Wang, and J. Li, “Mobile-edge computing: Partial computation offloading using dynamic voltage scaling,” _IEEE Trans. Commun._ , vol. 64, no. 10, pp. 4268–4282, Oct 2016.
  * [16] M. Bennis, M. Debbah, and H. V. Poor, “Ultrareliable and low-latency wireless communication: Tail, risk, and scale,” _Proc. IEEE_ , vol. 106, no. 10, pp. 1834–1853, Oct 2018.
  * [17] S. Boyd and L. Vandenberghe, _Convex Optimization_ , Cambridge, U.K.: Cambridge Univ. Press, 2004.
  * [18] Y. Mao, B. Clerckx, J. Zhang, V. O. K. Li, and M. A. Arafah, “Max-min fairness of K-user cooperative rate-splitting in MISO broadcast channel with user relaying,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 10, pp. 6362–6376, Oct 2020.
  * [19] Q. Shi, M. Razaviyayn, Z.-Q. Luo, and C. He, “An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel,” _IEEE Trans. Signal Process._ , vol. 59, no. 9, pp. 4331–4340, Sep. 2011.
  * [20] X. Zhang, _Matrix analysis and applications_. Beijing, CHN: Tsinghua University Press, 2004.
  * [21] A. Ben-Tal and A. Nemirovski, _Lectures on modern convex optimization: Analysis, algorithms, and engineering applications_. Philadelphia, PA, USA: Society for Industrial and Applied Mathematics, 2001.


# TRILLsson: Distilled Universal Paralinguistic Speech Representations

###### Abstract

Recent advances in self-supervision have dramatically improved the quality of
speech representations. However, deployment of state-of-the-art embedding
models on devices has been restricted due to their limited public availability
and large resource footprint. Our work addresses these issues by publicly
releasing a collection of paralinguistic speech
models111https://tfhub.dev/s?q=trillsson that are small and near state-of-the-
art performance. Our approach is based on knowledge distillation, and our
models are distilled on public data only. We explore different architectures
and thoroughly evaluate our models on the Non-Semantic Speech (NOSS)
benchmark. Our largest distilled model is less than 15% the size of the
original model (314MB vs 2.2GB), achieves over 96% the accuracy on 6 of 7
tasks, and is trained on 6.5% the data. The smallest model is 1% in size
(22MB) and achieves over 90% the accuracy on 6 of 7 tasks. Our models
outperform the open source Wav2Vec 2.0 model on 6 of 7 tasks, and our smallest
model outperforms the open source Wav2Vec 2.0 on both emotion recognition
tasks despite being 7% the size.

Index Terms—  speech, representations, on-device, paralinguistic speech

##  1 Introduction

Self-supervised learning for audio has improved the quality of speech
representations, resulting in huge performance gains on downstream tasks [1,
2, 3]. However, a few obstacles prevent these representations from being
widely adopted on devices, particularly for paralinguistic speech tasks which
focus on aspects of speech other than the textual meaning. First, recent self-
supervised models are often extremely large, making them challenging to use in
resource-constrained environments like mobile phones (e.g. HuggingFace’s
Wav2Vec 2.0 [3] at 360 MB). Second, due to the private nature of most speech
data, high-performance models are often not publicly released (e.g. CAP12
[4]). In this work, we overcome both constraints using public data and
knowledge distillation [5]. Specifically, we knowledge distill a recent state-
of-the-art paralinguistic speech representation “Conformer Applied to
Paralinguistics” (CAP12 [4]) into a series of models, which we call TRILLsson.

Our approach primarily relies on “knowledge distillation” [5]. We train small
student models on several fixed-length input architectures, including ResNets,
EfficientNets, and Transformers, to match the arbitrary-length input CAP12
(teacher) Transformer embeddings. Our architectures explore the model size
versus performance tradeoff. To our knowledge, this is the first successful
cross-architecture distillation from a Transformer to non-Transformer model
using a different dataset.

We use nearly 58K hours of publicly available speech data from Libri-light [6]
and Audioset [7] for distillation. For evaluation, we use the “Non-Semantic
Speech Benchmark” (NOSS) [8]. NOSS includes 7 tasks such as emotion
recognition and speaker identification that require slow-time features.
Additionally, we demonstrate the superior performance of TRILLsson models by
comparing them with existing publicly available representations such as TRILL
[8] and Wav2Vec$2.0$ [3]. Our contributions are:

  1. 1.

Create generally-useful paralinguistic models that are small enough to run on-
device.

  2. 2.

Demonstrate successful cross-architecture knowledge distillation from
Transformers to fixed-context convolutional networks.

  3. 3.

Publicly release models at different points of the model size and performance
trade-off curve.

  4. 4.

Identify the best paralinguistic representation in the public Wav2Vec2.0 model
and demonstrate that our models outperform it.

##  2 Background and related works

Self-supervised representation learning has shown remarkable success in vision
[9] and speech recognition [3]. The Wav2Vec$2.0$ [3] and Conformer models [10]
are most relevant to our work. [3] was one of the first frameworks to
successfully combine Transformers [11] and a self-supervised contrastive
learning objective for speech. The same training objective was subsequently
combined with Conformer architectures [10], which added convolution filters to
Transformer layers, producing further improvements in semi-supervised speech
recognition applications [1]. Recently, [4] developed Conformer-based models
(CAP12) that created representations for non-ASR speech analysis and
paralinguistics tasks. Linear models on time-averaged CAP12 representations
performed at or above state-of-the-at across several tasks simultaneously.
However, as with most self-supervised models, their resource footprint (memory
and compute) makes them less suitable for on-device applications. In this
work, we distill the CAP12 model from [4] to several “lite” architectures for
use on mobile devices, and we release them publicly.

Distillation [5] has been popular for transferring knowledge from large models
to smaller ones. We distill the Conformer model to a variety of smaller,
fixed-length input architectures that have been used in audio classification,
such as ResNets [7], EfficientNets [12], and ASTs [13]. Other works have
applied distillation to speech representations [14], but to our knowledge this
work is the first successful speech embedding distillation from arbitrary-
length input Transformers to fixed-length input models.

##  3 Approach

Table 1: Training datasets. We only use AudioSet and Libri-light for distillation. YT-U was used to train CAP12 [4].  Dataset | Total (Hours) | Samples | Avg len (s)
---|---|---|---
Speech AudioSet | 4.9K | 1.8M | 9.9
Libri-light | 53K | 6.0M | 31.8
YT-U | 900K | 295M | 11.0
Table 2: Downstream evaluation datasets. ∗Results in our study used a subset of Voxceleb filtered according to YouTube’s privacy guidelines. Dataset | Target | Classes | Samples |  | Avg
---
length (s)
VoxCeleb∗ [15] | Speaker ID | 1,251 | 12,052 | 8.4
VoxForge [16] | Language ID | 6 | 176,438 | 5.8
| Speech
---
Commands[17]
Command | 12 | 100,503 | 1.0
ASVSpoof [18] |  | Synthetic
---
or not
2 | 121,461 | 3.2
Euphonia [19] | Dysarthria | 5 | 15,224 | 6.4
CREMA-D [20] | Emotion | 6 | 7,438 | 2.5
IEMOCAP [21] | Emotion | 4 | 5,531 | 4.5
Fig. 1: Depiction of the evaluation process.

Shor et. al. [4] demonstrated that the 1024 dimension representation of the
12th layer of the CAP Conformer model (referred to as “CAP12”) achieved at or
near state-of-the-art performance across all tasks in the paralinguistic Non-
Semantic Speech Benchmark (NOSS) [8]. CAP12 is a 606M parameter (2.2GB)
Conformer model trained via a modified Wav2Vec$2.0$ self-supervised training
loss on a 900M+ hour speech dataset derived from YouTube (YT-U [1], Tab. 1).
In this work, we use CAP12 as the teacher and distill this model to several
“lite” architectures based on the teacher-student distillation approach [5].

###  3.1 Student architectures.

We explore 3 different student architectures of varying sizes.

  1. 1.

Audio Spectrogram Transformer (AST) [13] is a Transformer-based model for
audio classification. We train student models with different depths and
widths.

  2. 2.

EfficientNetv2 [12] was designed by neural architecture search on image
classification. The architecture is mobile friendly. Different versions of
this architecture vary in terms of depths and filters.

  3. 3.

Resnetish [7] are modified ResNet-50 architectures designed to take audio
spectral features as input. Different versions of this architecture include
different depths and different number of filters per layer.

Table 3: Test performance on the NOSS Benchmark and extended tasks. “Prev. SoTA” are usually domain specific, but all other rows are linear models on time-averaged input. TRILLsson model sizes are shown without frontends. $\uparrow$ indicates higher values are better, and $\downarrow$ indicates lower is better. †We use a filtered subset of Voxceleb1 according to YouTube’s privacy guidelines. We omit previous SoTA results on this dataset, since they used the entire dataset. ∗∗ASVSpoof uses equal error rate [18]. We report the best single-model performance (as compared to model ensembles). #Euphonia is the only non-public dataset. We use a larger dataset than was reported on in [4]. ∗We use the public Wav2Vec 2.0 model from Hugging Face [22] | Model
---
(input size)
| Params
---
(M)
| Size
---
(MB)
Public |  Voxceleb1† $\uparrow$ |  Voxforge $\uparrow$ |  |  Speech $\uparrow$
---
Commands
| ASVSpoof
---
2019∗∗ $\downarrow$
Euphonia# $\uparrow$ |  CREMA-D $\uparrow$ |  IEMOCAP $\uparrow$
Prev. SoTA | - | - | - | - |  99.8 [4] |  97.9 [23] |  2.5 [4] | - |  88.2 [4] |  79.2 [4]
CAP12 |  |  |  |  |  |  |  |  |  |
(full) | 606 | 2,200 | ✗ | 51.0 | 99.7 | 97.1 | 2.5 | 46.9 | 88.2 | 75.5
(3s) | 606 | 2,200 | ✗ | 47.9 | 99.4 | 97.1 | 3.8 | 46.9 | 88.1 | 74.3
(2s) | 606 | 2,200 | ✗ | 48.1 | 99.4 | 97.0 | 6.9 | 46.9 | 85.3 | 72.7
Baselines |  |  |  |  |  |  |  |  |  |
Wav2Vec2 Sm. L6∗ | 93.4 | 360 | ✓ | 17.9 | 98.5 | 95.0 | 6.7 | 48.2 | 77.4 | 65.8
Wav2Vec2 Sm.∗ | 93.4 | 360 | ✓ | 1.7 | 95.9 | 89.3 | 11.2 | 50.0 | 58.0 | 52.4
TRILL | 24.5 | 87 | ✓ | 13.8 | 84.5 | 77.6 | 6.3 | 47.0 | 65.7 | 55.4
YAMNet | 3.7 | 17 | ✓ | 9.6 | 79.8 | 78.5 | 6.7 | 43.8 | 66.4 | 57.5
TRILLsson |  |  |  |  |  |  |  |  |  |
5 (AST) | 88.6 | 314 | ✓ | 46.2 | 99.7 | 93.9 | 5.4 | 48.1 | 86.1 | 72.7
4 (AST) | 63.4 | 224 | ✓ | 43.1 | 99.6 | 94.5 | 7.1 | 50.7 | 86.2 | 73.2
3 (EffNetv2) | 21.5 | 99 | ✓ | 40.1 | 99.2 | 93.2 | 6.8 | 47.4 | 83.2 | 70.3
2 (EffNetv2) | 8.1 | 42 | ✓ | 37.5 | 99.2 | 92.1 | 6.6 | 44.6 | 82.6 | 69.8
1 (ResNet) | 5.0 | 22 | ✓ | 36.6 | 98.6 | 91.2 | 7.5 | 43.3 | 81.3 | 68.5

Based on the Figure 1 lower in [4] and the fact that some benchmark datasets
have audio that’s mostly less than 3 seconds (Tab. 2) we focus on student
architectures that process 2 seconds of audio at a time. When operating on
audio less than 2 seconds, we symmetrically pad the audio around the end.

For the frontend, we use window length of 25ms and a hop length of 10ms. We
use log-magnitude mel-frequency spectrograms with 80 bins ranging from 125 Hz
to 7500 Hz. The frame width for all models is 2 seconds of audio, and we treat
the frame advance between successive model patches as a hyperparameter. Since
the model sees audio of exactly 2 seconds during training (see Sec 3.2), the
frame advance hyperparameter can be explored after the model is fully trained.

###  3.2 Distillation targets: global vs local matching

There are two paradigms for generating targets from a teacher that borrow
ideas from distilling large vision models [24]. In both cases the audio is
first chunked into context windows which the student processes. Then the
student is trained to match a target embedding generated using the teacher
model. In “global matching”, the target is the average teacher embedding from
the entire un-chunked audio clip. In the “local matching” paradigm, the target
is generated by averaging the teacher’s output on the same context window that
the student sees. In this work, we focus on “local matching.”

###  3.3 Training datasets

We perform distillation on two open source datasets which together contain
about 58K hours of speech data (Tab. 1). Audioset [25] clips are collection
from YouTube, so it represents a variety of settings and acoustic
environments. We use the speech subset of this data, which yields
approximately 5K hours. Libri-light [6] contains 60k hours of audio derived
from open-source audio books in the LibriVox project. It is the largest
publicly available, unlabeled semi-supervised dataset to date. We note that
the CAP12 teacher model [4] is trained on YT-U [1], which is a 900K hour
dataset derived from YouTube.

Fig. 2: Comparison of “average $d^{\prime}$” vs. “model size” for various
student model architectures and sizes. Performance in this figure is across
test sets, although only dev-set performance is used to select ”best” models.

###  3.4 Representation Evaluation

We compare and evaluate the distilled representations on several tasks
including detection of speaker, language, command, synthetic speech,
dysarthria, and emotion. Table 2 provides an overview of the 7 datasets these
tasks come from [8, 4, 26]. As depicted in Fig 1, for each (model, eval
dataset) pair, we first generate the candidate embeddings for the train, dev,
and test splits. We then train three types of linear models on the train set
embeddings. We take the model that performs best on the dev set, and report
the model’s performance on the test set as the score for that (model, eval
dataset) pair.

####  3.4.1 Aggregation metric: Equivalent D-Prime ($d^{\prime}$)

To fairly compare with previous results, we report on the typical performance
metric for each dataset (accuracy or equal error rate). However, to aggregate
performance into a single scalar, for the purpose of comparing embedding
quality, we average the “equivalent d-prime” ($d^{\prime}$) metric, defined as
follows:

$d^{\prime}=\sqrt{2}Z(AUC)$ (1)

where “AUC” is the “Area Under the Receiver Operating Characteristic Curve”
(ROC AUC), and “$Z()$” is the inverse Cumulative Distribution Function for the
standard normal distribution.

$d^{\prime}$ is better suited to aggregate performance on multiple datasets
for two reasons. First, ROC AUC and $d^{\prime}$, unlike accuracy, take into
account performance at various levels along the accuracy/recall curve. This
makes it more reflective of the overall performance of a particular
representation. Second, unlike ROC AUC, $d^{\prime}$ doesn’t saturate in the
highly performant regime. Thus, 1 unit of $d^{\prime}$ is in some sense more
equivalent, so averaging $d^{\prime}$ values across datasets is more natural
than averaging AUC values. We use the average $d^{\prime}$ scores to sort and
select models.

###  3.5 Models for comparison

To contextualize performance on the NOSS benchmark, we compare our models to
1) previous state-of-the-art (SoTA) results, 2) publicly available speech
representation models, and 3) CAP12 with different input sizes. Previous SoTA
models are been mostly domain-specific. The public models we compare to are:

  1. 1.

Hugging Face’s Wav2Vec 2.0 [3, 22]: This model was trained on the
approximately 1K hour Librispeech [27]. We use the TensorFlow model from the
Hugging Face library. We compute the performances for each layer, and find
that layer 6 of 11 performs the best overall.

  2. 2.

TRILL [8]: A Resnet triplet-loss model trained on AudioSet. We access this
model from TensorFlow Hub.

  3. 3.

YAMNet [7]: This is a supervised audio classification network, also trained on
AudioSet. We use layer 19 as in [8]. We access this model through TensorFlow
Hub.

##  4 Results

Table 3 shows the sizes and performances of 5 TRILLsson models as compared to
CAP12 and other publicly available models. We see that the two largest
TRILLsson models outperform all publicly available models on all datasets
except Wav2Vec2 layer 6 on speech commands, and that even TRILLsson1, the
smallest model, outperforms previously available embeddings on emotion
recognition tasks. In particular, TRILLsson2 outperforms the best Wav2Vec 2.0
representation on 5 of 6 public tasks despite being less than 12% the size.

Compared to CAP12, the distilled models maintain performance on language
identification and dysarthria detection. TRILLsson models suffer minor
degradation on speech emotion recognition tasks, and major degradation on
speaker identification, speech commands, and fake speech detection.

Fig 2 shows the relationship between size and performance across different
architectures. We see that Resnetish models perform best at the low end,
EfficientNets perform best at sizes between 40-230MB, and AST performs best
when larger then 230MB.

Training on AudioSet improves performance on emotion recognition tasks: Using
a dependent t-test on paired samples between models trained on Libri-light and
AudioSet versus just Libri-light, we observed statistically significant
improvements in dev and test set accuracies on the speech emotion recognition
tasks (CREMA-D and IEMOCAP, $n=86$, $p<0.03$). Interestingly, training on the
combined dataset hurt performance on ASVSpoof2019 test set compared to either
dataset on its own ($n=86$, $p<10^{-3}$), but the paired t-test did not reject
the null hypothesis on the ASVSpoof2019 dev set.

Best model curve robust to removing a single benchmark task: Removing any
single task keeps the size versus performance curve relatively stable. Either
TRILLsson5 or 4 are the best models with any single task removed. TRILLsson3
and 2 appear in every optimal curve. TRILLsson1 appears in every optimal curve
unless ’voxforge’ is removed.

Table 4: Kendall rank coefficient between model orderings according to different criteria. For all computations, n = 435. |  $d^{\prime}$ Dev. |  $d^{\prime}$ Test | Acc. Dev | Acc. Test
---|---|---|---|---
$d^{\prime}$ Dev | - | - | - | -
$d^{\prime}$ Test | 0.898 | - | - | -
Acc Dev | 0.893 | 0.854 | - | -
Acc Test | 0.860 | 0.821 | 0.820 | -

$d^{\prime}$ orderings: Table 4 justifies our choice of $d^{\prime}$ on the
dev set as our ordering criteria. The Kendall rank coefficient, a correlation
statistic for different orderings, shows that average $d^{\prime}$ on the dev
sets more closely correlates to average accuracy and average $d^{\prime}$ than
average accuracy on the dev set.

##  5 Conclusion

In this work, we demonstrate that it’s possible to distill huge models trained
on large datasets to obtain much smaller models that perform well on
paralinguistic speech tasks. The distillation uses only 7% of the training
data and is entirely from public sources. The models we obtain are between
22MB and 314MB, and achieve between 90% and 96% of the larger CAP12 accuracy
on 6 of 7 tasks. These models are between 1% and 15% the size of the original
model. We release the model to allow the research community to benefit from
the practical applications of self-supervised representations for
paralinguistic speech.

##  6 Acknowledgements

We want to thank Aren Jansen, Channing Moore, Wei Han, Daniel Park, Yu Zhang,
and the entire author list of [1], without which this work wouldn’t have been
possible.

## References

  * [1] Y. Zhang _et al._ , “BigSSL: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,” 2021.
  * [2] W. Hsu _et al._ , “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _CoRR_ , vol. abs/2106.07447, 2021\. [Online]. Available: https://arxiv.org/abs/2106.07447
  * [3] A. Baevski _et al._ , “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
  * [4] J. Shor _et al._ , “Universal paralinguistic speech representations using self-supervised conformers,” 2021.
  * [5] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _NIPS Deep Learning and Representation Learning Workshop_ , 2015\. [Online]. Available: http://arxiv.org/abs/1503.02531
  * [6] J. Kahn _et al._ , “Libri-Light: A benchmark for ASR with limited or no supervision,” in _ICASSP_ , 2020, pp. 7669–7673.
  * [7] S. Hershey _et al._ , “Cnn architectures for large-scale audio classification,” 2017.
  * [8] J. Shor _et al._ , “Towards Learning a Universal Non-Semantic Representation of Speech,” in _Proc. Interspeech 2020_ , 2020, pp. 140–144. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-1242
  * [9] T. Chen _et al._ , “A simple framework for contrastive learning of visual representations,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 119, 13–18 Jul 2020.
  * [10] A. Gulati _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” in _Interspeech_. ISCA, 2020, pp. 5036–5040.
  * [11] A. Vaswani _et al._ , “Attention is all you need,” 2017.
  * [12] M. Tan and Q. V. Le, “Efficientnetv2: Smaller models and faster training,” 2021\.
  * [13] Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” 2021\.
  * [14] J. Peplinski _et al._ , “FRILL: A Non-Semantic Speech Embedding for Mobile Devices,” in _Proc. Interspeech 2021_ , 2021.
  * [15] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” _arXiv preprint arXiv:1706.08612_ , 2017.
  * [16] K. MacLean, “Voxforge,” _Ken MacLean.[Online]. Available: http://www.voxforge.org/home.[Acedido em 2012]_ , 2018.
  * [17] P. Warden, “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition,” _ArXiv e-prints_ , Apr. 2018. [Online]. Available: https://arxiv.org/abs/1804.03209
  * [18] M. Todisco _et al._ , “Asvspoof 2019: Future horizons in spoofed and fake audio detection,” 2019.
  * [19] R. L. MacDonald _et al._ , “Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia,” in _Interspeech_ , 2021, pp. 4833–4837.
  * [20] H. Cao _et al._ , “CREMA-D: Crowd-sourced emotional multimodal actors dataset,” _IEEE transactions on affective computing_ , vol. 5, no. 4, pp. 377–390, 2014.
  * [21] C. Busso _et al._ , “IEMOCAP: Interactive emotional dyadic motion capture database,” _Language Resources and Evaluation_ , vol. 42, pp. 335–359, 12 2008.
  * [22] T. Wolf _et al._ , “Transformers: State-of-the-art natural language processing,” in _EMNLP_ , 2020.
  * [23] D. Seo, H.-S. Oh, and Y. Jung, “Wav2kws: Transfer learning from speech representations for keyword spotting,” _IEEE Access_ , vol. 9, pp. 80 682–80 691, 2021.
  * [24] M. Caron _et al._ , “Emerging properties in self-supervised vision transformers,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * [25] J. F. Gemmeke _et al._ , “Audio set: An ontology and human-labeled dataset for audio events,” in _ICASSP_. IEEE, 2017, pp. 776–780.
  * [26] S. Venugopalan _et al._ , “Comparing Supervised Models and Learned Speech Representations for Classifying Intelligibility of Disordered Speech on Selected Phrases,” in _Proc. Interspeech 2021_ , 2021, pp. 4843–4847.
  * [27] V. Panayotov _et al._ , “Librispeech: An asr corpus based on public domain audio books,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.


# Exponentially sized pointsets with angles less than 61 degrees

Miroslav Marinov<EMAIL_ADDRESS>
###### Abstract

We prove that any maximal set of points in $\mathbb{R}^{d}$, any three of
which form an angle less than $\frac{\pi}{3}+c$, has size $(1+\Theta(c))^{d}$
for sufficiently small $c>0$. The proof is based on a refinement of an
approach by Erdős and Füredi. The lower bound is relying on a problem about
large uniform hypergraphs with small edge intersections, while the upper bound
is tightly connected to the problem of packing disjoint caps on a sphere.

Keywords: exponential size, set of points, acute angles

2020 Mathematics Subject Classification: 51M04, 52C10

##  1 Introduction

Around $1950$ Erdős conjectured that $2^{d}$ is the greatest possible number
of points in $\mathbb{R}^{d}$ such that no three of them determine an obtuse
angle. This was proven in $1962$ by Danzer and Grünbaum ([2], [1]). The only
examples attaining the bound turn out to be affine images of the hypercube. As
it contains a lot of angles equal to $\pi/2$, the following question becomes
interesting: what is the maximum number of points in $\mathbb{R}^{d}$ with all
angles strictly less than $\pi/2$? Very recently, in 2017, Gerenscér and
Harangi [4] made a breakthrough by providing a surprisingly simple
construction of $2^{d-1}+1$ points. Regarding an upper bound, currently it is
not known even whether sets of $2^{d}-1$ points are possible.

The above problem can be naturally generalized by considering, for a positive
integer $d\geq 2$ and a real number $\alpha\in(0,\pi)$, the maximal
cardinality $g_{\alpha}(d)$ of a set of points in $\mathbb{R}^{d}$ such that
any angle formed by three of the points is less than or equal to $\alpha$;
define $f_{\alpha}(d)$ analogously, requiring the angles to be strictly less
than $\alpha$. Clearly, $f_{\alpha}(d)\leq g_{\alpha}(d)$.

For $\alpha\neq\frac{\pi}{2}$ the problem was considered by Erdős and Füredi
[3], Li and Otsetarova [6] and Mironenko [7]. The value $g_{\pi/3}(d)=d+1$
appears in [6], [7] and we outline a simplified proof in Section 2. A
probabilistic argument for large non-obtuse $\alpha$ and all $d\geq 2$ is
examined in [6]. Explicit approaches for $\alpha=\pi-c$ and
$\alpha=\frac{\pi}{3}+c$, for small $c$ and large $d$, can be found in [3].

Regarding $\alpha=\frac{\pi}{3}+c$ for sufficiently small $c>0$, the work of
Erdős and Füredi [3] addresses the bounds $(1+c^{2})^{d}\leq
f_{\pi/3+c}(d)\leq(1+4c)^{d}$. We refine their approach on the lower bound to
obtain the asymptotic $f_{\pi/3+c}(d)=(1+\Theta(c))^{d}$. (All arguments apply
equally well for $g$.) More precisely:

######  Theorem 1.1.

For any $c\in(0,1)$, $\delta>0$ and sufficiently large $d$,

$f_{\pi/3+c}(d)>\left(1+\left(0.0094-\delta\right)c\right)^{d}.$

######  Theorem 1.2.

For any $c\in(0,0.02)$ and sufficiently large $d$,

$f_{\pi/3+c}(d)<(1+3.75c)^{d}.$

The only substantial difference between our arguments and these of Erdős and
Füredi is in the analysis of the hypergraph construction discussed in Section
3. All other statements and proofs in the paper are not claimed to be
original, but we include them for completeness, often with additional details
for the sake of readability.

##  2 The case $\alpha=\frac{\pi}{3}$

As the sum of angles in a triangle is $\pi$, we immediately have
$f_{\alpha}(d)=2$ for $\alpha\leq\frac{\pi}{3}$ and $g_{\alpha}(d)=2$ for
$\alpha<\frac{\pi}{3}$. The following result concerns the value of
$g_{\pi/3}(d)$.

######  Proposition 2.1.

For all $d\geq 1$, we have $g_{\pi/3}(d)=d+1$.

###### Proof.

We simplify the approach in [7]. Let
$S=\\{a_{0},a_{1},\ldots,a_{n}\\}\in\mathbb{R}^{d}$ be a set with
$\angle(a_{i},a_{j},a_{k})\leq\frac{\pi}{3}$ for all $i,j,k$. This holds if
and only if any triangle $a_{i}a_{j}a_{k}$ is equilateral which occurs if and
only if the distances between points in $S$ are equal. Without loss of
generality we assume these distances to be $1$. Note also that $\langle
a_{i}-a_{j},a_{k}-a_{j}\rangle=\cos\frac{\pi}{3}=\frac{1}{2}$.

For the lower bound, a working construction is the well known regular
$d$-simplex – one way to explicitly describe it in $\mathbb{R}^{d+1}$ is to
take the points $a_{i}=(0,0,\ldots,1/\sqrt{2},\ldots,0)$, $1\leq i\leq d+1$,
which lie on the $d$-dimensional hyperplane
$\sum_{i=1}^{d+1}x_{i}=1/\sqrt{2}$.

To prove the upper bound for $|S|$, it suffices to show the following weaker
statement: if $n\geq d$ and $a_{0},a_{1},\ldots,a_{n}$ are points in
$\mathbb{R}^{d}$ such that

$|a_{i}-a_{0}|=1,\ \ \ \ \ \ \langle
a_{i}-a_{0},a_{j}-a_{0}\rangle=\frac{1}{2}$

for all non-zero $i\neq j$, then the vectors $(a_{i}-a_{0})$, $i=1,\ldots,n$
are linearly independent. Indeed, $\dim\mathbb{R}^{d}=d$ would then imply
$n=d$.

So suppose $a_{0},a_{1},\ldots,a_{n}$ are as above and assume, for the sake of
contradiction, that $(a_{i}-a_{0})$, $i=1,\ldots,n$ are dependent. Then there
is a linear combination of them with at least one non-zero scalar – thus by
relabeling the points and dividing by this scalar, if necessary, we may treat
$a_{n}-a_{0}=\sum_{i=1}^{n-1}k_{i}(a_{i}-a_{0})$ for some
$k_{i}\in\mathbb{R}$.

Taking the inner product with $(a_{j}-a_{0})$ for $j=1,\ldots,n-1$ gives

$\frac{1}{2}=k_{j}+\sum_{\begin{subarray}{c}i=1\\\ i\neq
j\end{subarray}}^{n-1}\frac{k_{i}}{2}=\frac{k_{j}}{2}+\sum_{i=1}^{n-1}\frac{k_{i}}{2}=\frac{k_{j}}{2}+M,$

and as $M$ does not depend on $j$, we get $k_{j}=\frac{1}{n}$ for all $j$.
Substituting and using the Triangle inequality gives

$|a_{n}-a_{0}|\leq\frac{1}{n}\sum_{i=1}^{n-1}|(a_{i}-a_{0})|=\frac{n-1}{n}<1,$

which is the required contradiction. ∎

##  3 The construction – embedding a uniform hypergraph into the cube

Recall that a hypergraph is $k$-uniform if each edge contains exactly $k$
vertices.

We construct greedily a large uniform hypergraph in which every two edges have
a tiny overlap. Then we map each edge to a vertex of the hypercube in a way
that this overlap results in a small range of possible distances between image
points – this would force the angles to be close to $\pi/3$.

The difference between the upcoming proposition with its analogue in [3] is
that the number of “bad choices” in the greedy algorithm is shown to be
smaller, with a slightly stronger argument. However, as we shall see later on,
this improvement comes at the price of a more technical treatment.

######  Proposition 3.1.

Fix $c\in(0,1)$ and positive integers $k$ and $d$. There exists a $k$-uniform
hypergraph $G$ on $[d]:=\\{1,2,\ldots,d\\}$ with $|F_{i}\cap F_{j}|<ck$ for
any $F_{i},F_{j}\in E(G)$, $i\neq j$, and

$|E(G)|\geq\frac{\binom{d}{k}}{\sum_{j=\lceil
ck\rceil}^{k}\binom{k}{j}\binom{d-k}{k-j}}.$

###### Proof.

Choose the subset $F_{1}$ arbitrarily. Now suppose $F_{1},\ldots,F_{i}$ are
already chosen. For $k>0$ to be chosen later, consider the set of “bad”
subsets

$B_{i}=\\{F\subseteq[d]:|F|=k,|F\cap F_{j}|\geq\lceil ck\rceil\mbox{ for at
least one }j\in[i]\\}.$

Any $k$-element bad subset can be formed by choosing exactly $j$ of its $k$
elements to be in one of $F_{1},\ldots,F_{i}$ (here $j$ ranges from $\lceil
ck\rceil$ to $k$), say $F_{m}$, and the other $k-j$ to be in
$[d]\setminus\\{F_{m}\\}$. Hence $|B_{i}|\leq i\sum_{j=\lceil
ck\rceil}^{k}\binom{k}{j}\binom{d-k}{k-j}$. The number of $k$-element subsets
of $[d]$ is $\binom{d}{k}$, hence we can choose $F_{i+1}\not\in B_{i}$ as long
as $|B_{i}|<\binom{d}{k}$. Thus the process can be performed for at least
$\binom{d}{k}/\sum_{j=\lceil ck\rceil}^{k}\binom{k}{j}\binom{d-k}{k-j}$ steps.
Each step yields an edge of the hypergraph, so we are done. ∎

Now we focus on estimating the summation in the conclusion of Proposition 3.1.
For two functions $f,g:\mathbb{N}\to\mathbb{R}$ we write $f\gtrapprox g$ if
for all $\delta>0$ and sufficiently large $d$ we have $f(d)-g(d)>-d\delta$ and
also let $f\approx g$ if $f\gtrapprox g$ and $g\gtrapprox f$. Observe that if
$f\sim g$, that is, $f(d)/g(d)\to 1$ as $d\to\infty$, and also if $d/f(d)\to
0$, then $d/g(d)\to 0$ and $f\approx g$.

######  Lemma 3.2.

Let $0<y<x<1$ be real numbers and $a,b:\mathbb{N}\to\mathbb{N}$ be functions
such that $a(n)\approx xn$ and $b(n)\approx yn$. Then
$\log\binom{a(n)}{b(n)}\approx n[x\log x-y\log y-(x-y)\log(x-y)]$.

###### Proof.

By Stirling’s approximation we have, as $k,\ell\to\infty$, that

$\binom{k}{\ell}\sim\sqrt{\frac{k}{2\pi\ell(k-\ell)}}\frac{k^{k}}{\ell^{\ell}(k-\ell)^{k-\ell}}$

and hence

$\displaystyle\log\binom{a(n)}{b(n)}$ $\displaystyle\approx\frac{1}{2}(\log
a(n)-\log b(n)-\log((a(n)-b(n))-\log 2\pi)$ $\displaystyle+a(n)\log
a(n)-b(n)\log b(n)-(a(n)-b(n))\log((a(n)-b(n)).$

Since $\log n\approx 0$, the terms on the first line contribute overall with
$0$. Substituting $a(n)\approx xn$ and $b(n)\approx yn$ in the second line, as
well as cancelling terms, now gives the result. ∎

######  Proposition 3.3.

With the notation in Proposition 3.1, for any $\delta>0$ and sufficiently
large $d$, and for any $c\in(0,1)$, there exists a positive integer $k$ such
that

$A(d,k,c):=\frac{\binom{d}{k}}{\sum_{j=\lceil
ck\rceil}^{k}\binom{k}{j}\binom{d-k}{k-j}}\geq\left(1+\left(\log
5-\frac{8}{5}-\delta\right)c\right)^{d}.$

###### Proof.

We begin with the crude bound

$\sum_{j=\lceil ck\rceil}^{k}\binom{k}{j}\binom{d-k}{k-j}\leq(k-\lceil
ck\rceil)\max_{j=\lceil ck\rceil,\ldots,k}\binom{k}{j}\binom{d-k}{k-j}\leq
d\max_{j=\lceil ck\rceil,\ldots,k}\binom{k}{j}\binom{d-k}{k-j}.$

Since $-\log d\approx 0$, we deduce $\log A(d,k,c)\geq\log\binom{d}{k}-\log
d-\log\max_{j=\lceil
ck\rceil,\ldots,k}\binom{k}{j}\binom{d-k}{k-j}\gtrapprox\log\binom{d}{k}-\max_{j=\lceil
ck\rceil,\ldots,k}\left(\log\binom{k}{j}+\log\binom{d-k}{k-j}\right)$. Let
$k\approx ad$ for some $a\in(0,1)$ to be chosen later and write $j\approx
bk\approx abd$ where $b\in[c,1]$. From Lemma 3.2 we obtain

$\log\binom{d}{k}\approx d(-a\log a-(1-a)\log(1-a))$

and

$\log\binom{k}{j}+\log\binom{d-k}{k-j}\approx d(a\log
a-ab\log(ab)-(a-ab)\log(a-ab)$
$+(1-a)\log(1-a)-(a-ab)\log(a-ab)-(1-2a+ab)\log(1-2a+ab)).$

This implies $\log A(d,k,c)\gtrapprox d\inf_{b\in[c,1]}H(a,b)$ where

$H(a,b)=-ab\log a-2(1-a)\log(1-a)+ab\log
b+2a(1-b)\log(1-b)+(1-2a+ab)\log(1-2a+ab).$

So we should look for $a\in(0,1)$ and a constant $\rho$ such that
$H(a,b)\geq\log(1+\rho c)$ for all $b\in[c,1]$. Furthermore, let us impose the
condition $a\leq c$. We compute $\frac{dH}{db}=-a\log a+a\log
b-2a\log(1-b)+a\log(1-2a+ab)$ and observe that this derivative increases and
is zero when $b=a\leq c$ – hence for $b\in(c,1]$ it is positive and so it
remains to establish $H(a,c)\geq\log(1+\rho c)$ for suitable $a\leq c$ and
$\rho$. Pick111Having in mind that the derivative $\frac{dH}{da}$ vanishes at
$a=c$ and at $a\sim\frac{c}{5}$, as well as checking the second derivative for
these. $a=\frac{c}{5}$ and note the inequality
$H(\frac{c}{5},c)\geq\log(1+(\log 5-\frac{8}{5})c)$. (Indeed, the ratio
$(e^{H(\frac{c}{5},c)}-1)/c$ is strictly increasing and tends to $\log
5-\frac{8}{5}$ as $c\to 0$.) Therefore $\log A(d,k,c)\gtrapprox d\log(1+(\log
5-\frac{8}{5})c)$, so for any $\delta_{0}$ and large $d$ we have $\log
A(d,k,c)-d\log(1+(\log 5-\frac{8}{5})c)+\delta_{0}d\geq 0$. By choosing
$\delta_{0}=\log(1+(\log 5-\frac{8}{5})c)/(1+(\log 5-\frac{8}{5}-\delta)c)$,
we conclude that $A(d,k,c)\geq(1+(\log 5-\frac{8}{5}-\delta)c)^{d}$ for all
sufficiently large $d$, as desired. ∎

With the next proposition we illustrate how to obtain the large set of points
from the large hypergraph.

######  Proposition 3.4.

Let $k$ be a positive integer, let $c\in(0,1)$ and let $G$ be a $k$-uniform
hypergraph on $[d]$ such that $|F_{i}\cap F_{j}|<ck$ for any two distinct
$F_{i},F_{j}\in E(G)$. Then there is a set of points in $\mathbb{R}^{d}$ of
size $|E(G)|$ such that any three points form an angle less than
$\frac{\pi}{3}+c$.

###### Proof.

Consider the injective map $\theta$ from $E(G)$ to the vertex set of the
$d$-dimensional cube $\\{0,1\\}^{d}$, which sends $F_{i}$ to the vertex
$v_{i}$, where the $m$-th coordinate $(v_{i})_{m}$, $1\leq m\leq d$, of
$v_{i}$ is $1$ if $m\in F_{i}$ and is $0$ otherwise. (In other words, $v_{i}$
is the characteristic vector of $F_{i}$.) Note that, by the construction of
$G$, each $v_{i}$ has exactly $k$ coordinates which are $1$. Hence on one hand

$|v_{i}-v_{j}|=\sqrt{|\\{1\leq m\leq
d:(v_{i})_{m}\neq(v_{j})_{m}\\}|}\leq\sqrt{2k}$

as the total number of $1$s in $v_{i}$, $v_{j}$ is $2k$ and they may have no
coordinate which is $1$ for both. On the other hand, for $v_{i},v_{j}\in$ Im
$\theta$

$|v_{i}-v_{j}|>\sqrt{2k-2ck}$

since, by construction of $G$, $v_{i}$, $v_{j}$ have less than $ck$
coordinates in common, both of which are $1$s. It remains to show that if
$v_{i},v_{j},v_{k}\in$ Im $\theta$, then $\angle
v_{i}v_{j}v_{k}<\frac{\pi}{3}+c$. By the Law of Cosines

$\cos\angle
v_{i}v_{j}v_{k}=\frac{|v_{j}-v_{i}|^{2}+|v_{j}-v_{k}|^{2}-|v_{i}-v_{k}|^{2}}{2|v_{j}-v_{i}||v_{j}-v_{k}|}>\frac{2(2k-2ck)-2k}{2\cdot
2k}=\frac{1}{2}-c.$

The result now follows from $\cos(\frac{\pi}{3}+c)-\frac{1}{2}+c>0$ for $c>0$.
∎

Combinining Propositions 3.1, 3.3 and 3.4 proves Theorem 1.1.

##  4 The upper bound

In order to obtain a bound from above, we are going to cover our set by a
reasonably small ball, project it onto its surface and then bound the
cardinality of the image in terms of the minimal distance between points from
it. We denote by $\mathbb{S}^{d-1}$ the unit sphere
$x_{1}^{2}+x_{2}^{2}+\cdots+x_{d}^{2}=1$ with center $O$. The hyperspherical
cap $C(P,\alpha)$ with centre $P\in\mathbb{S}^{d-1}$ and angle $\alpha\leq\pi$
is the set of points $Q\in\mathbb{S}^{d-1}$ such that $\angle POQ\leq\alpha$.

######  Theorem 4.1.

$($Rankin $1955$, [8, Theorem 2]$)$ Fix $\alpha\in(0,\frac{\pi}{4})$ to be
independent of $d$. Then the maximum possible number of pairwise disjoint
hyperspherical caps in $\mathbb{S}^{d-1}$ does not exceed $f(\alpha)$, where

$f(\alpha)\sim\frac{\sqrt{\frac{1}{2}\pi d^{3}\cos
2\alpha}}{(\sqrt{2}\sin\alpha)^{d-1}}\mbox{ as }d\to\infty.$

######  Corollary 4.2.

Fix $\alpha\in(0,\frac{\pi}{4})$ and let $d$ be sufficiently large. Let $S$ be
a pointset on the sphere $\frac{1}{\sqrt{2}}\mathbb{S}^{d-1}$ of radius
$\frac{1}{\sqrt{2}}$, such that for some $y\in(0,1)$ and all $P_{i},P_{j}\in
S$ we have $|P_{i}P_{j}|>1-y$. Then $|S|\leq d^{2}(1-y)^{-d}$.

###### Proof.

Begin by projecting $S$, through the origin, onto the unit sphere
$\mathbb{S}^{d-1}$ – let $Q_{i}$ be the image of $P_{i}$. Then
$|Q_{i}Q_{j}|>(1-y)\sqrt{2}$ and so $\sin\frac{\angle
Q_{i}OQ_{j}}{2}>\frac{1-y}{\sqrt{2}}$. In particular, with
$\alpha=\arcsin(\frac{1-y}{\sqrt{2}})\in(0,\frac{\pi}{4})$, the hyperspherical
caps $C(Q_{i},\alpha)$ are pairwise disjoint and so Theorem 4.1 gives, for
sufficiently large $d$, that $|S|\leq 2f(\alpha)$ and hence that $|S|\leq
d^{2}(1-y)^{1-d}$. ∎

Our tool of covering the pointset by a small enough ball is as follows.

######  Theorem 4.3.

(Jung 1901, [5]) Any set $A\subseteq\mathbb{R}^{d}$ of diameter $1$ is
contained in a closed ball of radius $\sqrt{\frac{d}{2(d+1)}}$.

Before moving on to the main proof, we include separately a small triangle
lemma.

######  Lemma 4.4.

Let $ABC$ be an isosceles triangle $(|AC|=|BC|)$ and assume there exist points
$M\in AC$ and $N\in BC$ such that $|MN|>|AC|$. Then $|AB|>|MN|$.

###### Proof.

Without loss of generality treat $|CM|\geq|CN|$. Let the line through $A$,
parallel to $MN$, intersect the segment $BN$ at the point $K$ (which may
coincide with $B$) and the line through $M$, parallel to $BC$, intersect $AK$
at the point $P$. Then $|AK|>|PK|=|MN|$ (since $PKNM$ is a parallelogram) and
now the condition of the lemma gives $|AK|>|AC|=|BC|>|CK|$. In particular
$\angle ACK>60^{\circ}$ as it is the largest angle in triangle $ACK$, hence
$\angle AKB$ is the largest angle in triangle $AKB$. Therefore
$|AB|>|AK|>|MN|$. ∎

We are now ready to prove the upper bound.

###### Proof of Theorem 1.2.

Consider a pointset $S=\\{P_{1},P_{2},\ldots,P_{|S|}\\}$ with angles less than
$\frac{\pi}{3}+c$. In any triangle $P_{i}P_{j}P_{k}$ the largest angle is less
than $\frac{\pi}{3}+c$ and the smallest angle exceeds $\frac{\pi}{3}-2c$,
hence the ratio of the smallest and largest sides, by the Sine Law, exceeds

$\frac{\sin(\frac{\pi}{3}-2c)}{\sin(\frac{\pi}{3}+c)}\geq 1-1.744c$

for $c\in(0,0.024)$.

Next, we may assume that $|P_{i}P_{j}|=1$ is the largest distance in $S$. We
now bound the smallest distance in $S$, say $|P_{k}P_{l}|$. If without loss of
generality $P_{k}P_{j}$ is the largest side in $\triangle P_{j}P_{k}P_{l}$,
then $|P_{k}P_{l}|\geq(1-1.75c)|P_{k}P_{j}|$ and in $\triangle
P_{i}P_{k}P_{j}$ we have $|P_{k}P_{j}|\geq(1-1.744c)|P_{i}P_{j}|$. Hence

$|P_{k}P_{l}|\geq(1-1.744c)^{2}|P_{i}P_{j}|>1-3.488c>\frac{1}{\sqrt{2}}.$

On the other hand, by Theorem 4.3 there is a closed ball containing $S$ and of
radius $\sqrt{\frac{d}{2(d+1)}}<\frac{1}{\sqrt{2}}$ – so there is such a ball
$B$ (centered at $O$, say) with radius $\frac{1}{\sqrt{2}}$. For
$i=1,\ldots,m$ denote by $Q_{i}$ the intersection of the line connecting $O$
and $P_{i}$ with the sphere $\partial B$ and note that $P_{i}$ is between $O$
and $Q_{i}$. As $|P_{m}P_{n}|>\frac{1}{\sqrt{2}}=|OQ_{i}|$, by Lemma 4.4 it
follows that $|Q_{m}Q_{n}|>|P_{m}P_{n}|$ for all $m$, $n$ and hence
$|Q_{m}Q_{n}|>1-3.5c$. Finally, by Corollary 4.2 and $c<0.02$ (so that
$(1+3.75c)(1-3.488c)>1$), we conclude

$|S|<d^{2}(1-3.488c)^{-d}<(1+3.75c)^{d}$

for sufficiently large $d$. ∎

Acknowledgments. This research was supported by an undergraduate course for
projects at the University of Oxford. The author would like to thank the
supervisor David Conlon for introduction to the problem, as well as for
helpful discussions. Thanks also to team Germany South from the 2020 edition
of the International Tournament of Young Mathematicians, for discussions on
the technical Proposition 3.3, and to Lyuben Lichev, for remarks on the
details in the paper.

## References

  * [1] M. Aigner, G. M. Ziegler, _Proofs from THE BOOK_ (4th ed.), Springer-Verlag, (2010), 89-94.
  * [2] L. Danzer, B. Grünbaum, ‘Uber zwei Probleme bezüglich konvexer Körper von P. Erdős und von V.L. Klee’, Math. Zeitschrift 79 (1962), 95-99.
  * [3] P. Erdős, Z. Füredi, ‘The greatest angle among $n$ points in the $d$-dimensional Euclidean space’, Ann. Discrete Math. 17 (1983), 275-283.
  * [4] B. Gerencsér, V. Harangi, ‘Acute Sets of Exponentially Optimal Size’, Discrete $\&$ Computational Geometry 62:4 (2019), 775-780.
  * [5] H. Jung, ‘Über die kleinste Kugel, die eine räumliche Figur einschließt’, J. Reine Angew. Math. (in German), 123 (1901), 241-257.
  * [6] Z. Li, T. Otsetarova, ‘Boundaries on the Number of Points in Acute Sets’, Research Science Institute (2017).
  * [7] F. Mironenko, ‘Maximal Cardinalities of Bounded-angled Sets’, International Tournament of Young Mathematicians (2018).
  * [8] R. Rankin, ‘The closest packing of spherical caps in $n$ dimensions’, Glasg. Math. J. 2 (1955), 139-144.