# Matrix Low-Rank Approximation For Policy Gradient Methods

###### Abstract

Estimating a policy that maps states to actions is a central problem in
reinforcement learning. Traditionally, policies are inferred from the so
called value functions (VFs), but exact VF computation suffers from the curse
of dimensionality. Policy gradient (PG) methods bypass this by learning
directly a parametric _stochastic_ policy. Typically, the parameters of the
policy are estimated using neural networks (NNs) tuned via stochastic gradient
descent. However, finding adequate NN architectures can be challenging, and
convergence issues are common as well. In this paper, we put forth _low-rank_
matrix-based models to estimate efficiently the parameters of PG algorithms.
We collect the parameters of the stochastic policy into a matrix, and then, we
leverage matrix-completion techniques to promote (enforce) _low rank_. We
demonstrate via numerical studies how _low-rank_ matrix-based policy models
reduce the computational and sample complexities relative to NN models, while
achieving a similar aggregated reward.

Index Terms—  Reinforcement Learning, Low-rank optimization, Policy gradients,
Actor-critic methods.

##  1 Introduction

In the era of big data, complex dynamical systems call for intelligent
algorithms that use observations to adapt the way they behave with the world.
Reinforcement Learning (RL) addresses this by implementing _stochastic_
algorithms that learn by trial and error how to interact with the environment
[1, 2]. Technically speaking, RL models decision-making problems: given an
environment defined by a set of states, the RL goal is to learn the best
action to be made in each state. Then, RL is about estimating the reward
associated with state-action pairs, which is referred to as value function
(VF), and inferring a function that maps the states into actions, which is
referred to as policy. Typically, RL algorithms estimate first the VF
(strictly speaking, the expected accumulated reward associated with each
state-action pair) and, then, implement a policy that maximizes the VF.
However, value-based methods are algorithmically challenging, and suffer from
the curse of dimensionality. To overcome these issues, policy-based methods
were introduced. They bypass these problems by estimating the policy directly.
To render the design of these methods tractable, policies are typically
assumed stochastic and parametric, so that the problem boils down to
estimating the parameters of some probability distribution.

The workhorse approach in policy approaches is to consider normal
distributions and learn the parameters of those Gaussians using a neural
network (NN) architecture that uses the states as inputs [3]. NNs, and
therefore, most policy-based methods tend to suffer from convergence issues,
as they strongly depend on the architecture employed. As a result, postulating
(finding) the proper NN architecture for the RL setup at hand is usually a
problem in itself.

In the context of stochastic policy methods for RL, this paper takes an
alternative path and, leveraging matrix completion results [4, 5], puts forth
a low-rank algorithm to estimate stochastic policy models. The ultimate goal
is to design an estimation scheme that i) is sufficiently generic to learn the
policy and ii) mitigates some of the problems present in NN-based schemes. The
rest of this section is devoted to explaining the state of the art and the
contribution in more detail. Fundamentals of RL are discussed in Section 2,
and our new low-rank scheme is introduced in Section 3. Numerical experiments
showcasing some of the benefits of our approach are provided in Section 4.

Detailed contribution and related work. This paper investigates the design of
_low-rank_ schemes for (actor/critic) policy RL methods leveraging matrix
completion techniques. The central aspects of such a design are i) the
consideration of a policy method based on an actor and a critic (where the
latter estimates a VF); ii) modeling the actor parameters and the critic VF as
matrices (typically associated with a clusterization of the different
dimensions of the input state); and iii) to regularize the estimation problem
by enforcing a low-rank structure via matrix factorization. Although those
techniques are well-known in the context of low-rank optimization and matrix
completion [4, 5, 6, 7], they have not been exploited in the context of
policy-based RL methods. While closely-related ways of modeling parsimony in
RL have been investigated, e.g. sparsity [8, 9], the use of low-rank
optimization in the general context of RL has been limited. Notable exceptions
include efforts to approximate some structures of the Markov Decision Process
(MDP), such as the transition matrix or the reward function, as low rank [10,
11, 12]. Matrix-completion techniques have also been used to compress the
estimated $Q$-functions of an MDP [13]. In energy storage applications, low-
rank (rank-one) methods have been proposed for the estimation of the VF [14,
15]. Along the same lines, _linear models_ have been used to approximate VFs
on-the-fly [16]. These methods operate by defining a set of features [17] and
then modeling the VF as a weighted sum of the features associated with the
state-action pair. More recently, in the context of value-based methods,
schemes that estimate the VF promoting a low-rank matrix structure [18, 19,
20] and a low-rank tensor structure [21] have been proposed.

##  2 Fundamentals of RL and notation.

RL models the world as a closed-loop setup where agent(s) sequentially
interact with the environment. The environment consists of i) the space of
states ${\mathcal{S}}$; ii) the space of action agent(s) can take
${\mathcal{A}}$; and iii) the reward associated with every state-action pair.
In each (typically multidimensional) state, the agent takes an action and
obtains a reward. To be more specific, let $t=1,...,T$ be a time index. Given
a particular state $s_{t}$, the agent takes an action $a_{t}$, and obtains a
reward $r_{t}$. The reward $r_{t}$ quantifies the _instantaneous_ value of the
state-action pair. However, the action $a_{t}$ impacts $s_{t^{\prime}}$ for
$t^{\prime}>t$ and, as a result, affects future rewards $r_{t^{\prime}}$ for
$t^{\prime}>t$. This illustrates that actions are coupled across time and that
the aggregated (long-term) reward must be considered for decision-making.
Furthermore, the dependence of $r_{t}$ on $s_{t}$ and $a_{t}$ is usually
stochastic, rendering the optimization more challenging. Markovianity is
commonly used to mitigate these issues, so that the optimization can be recast
as a MDP.

In this context, i) a policy $\pi:{\mathcal{S}}\mapsto{\mathcal{A}}$ is a
function that maps states into actions; and ii) the VF associated with
$(s_{t},a_{t})$ is the expected cumulative reward
$\mathbb{E}[\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}}|s_{t},a_{t}]$,
with $\gamma\in(0,1)$ being a discount factor that places more focus on short-
term rewards [1, 2]. For a specific policy $\pi$, the VF quantifies the
expected reward associated with a particular state by taking the actions
dictated by $\pi$. Value-based methods focus on estimating the VFs (using
parametric or non-parametric approaches) of the MDP, to infer a policy later
on from the VFs by greedily following the path of highest value [22, 23].
Policy-based methods, on the other hand, focus on directly learning the policy
$\pi$, typically using a stochastic parametric models [24, 25]. To estimate
the parameters, a set of trajectories sampled from the environment is assumed
to be available, and those are used to run a stochastic gradient descent
scheme. Each trajectory $\tau_{T}:(s_{0},a_{0},r_{0}),...,(s_{T},a_{T},r_{T})$
is a set of state-action-rewards triplets obtained from the sequential
interaction of an agent with the environment.

From an optimization perspective, policy-based methods aim at maximizing the
reward of a trajectory ${\mathcal{R}}(\tau_{T})$ over all possible
trajectories $\tau$. To be precise, consider the following objective function:

$J(\theta)=\mathbb{E}_{{\mathcal{T}}_{\theta}(\tau_{T})}[{\mathcal{R}}(\tau_{T})]$
(1)

where $\theta$ denotes the parameters of a policy $\pi_{\theta}$. Then,
${\mathcal{T}}_{\theta}(\cdot)$ is the probability density function of the
trajectories under the given policy $\pi_{\theta}$. Leveraging Markovianity,
it follows that
${\mathcal{T}}_{\theta}(\tau_{T})={\mathcal{D}}(s_{0})\prod_{t=0}^{T}\pi_{\theta}(a_{t}|s_{t})P(s_{t+1}|s_{t},a_{t})$,
where ${\mathcal{D}}(\cdot)$ is the initial distribution of the MDP. Since
direct optimization of (1) is intractable, the standard approach is to
estimate the parameters $\theta$ of the policy $\pi_{\theta}$ via (stochastic)
gradient ascent methods [26]. The update rule takes the form
$\theta_{h+1}=\theta_{h}+\alpha_{\theta}\nabla_{\theta}J(\theta_{h})$, where
$h$ is the iteration index, and $\alpha_{\theta}$ is the gradient step, or
learning rate. Obtaining the gradient of $J$ is challenging since it involves
computing derivatives of an expectation across ${\mathcal{T}}_{\theta}$. The
policy gradient theorem [27] provides an elegant reformulation of
$\nabla_{\theta}J(\theta)$

$\nabla_{\theta}J(\theta)=\mathbb{E}_{{\mathcal{T}}_{\theta}}[\sum_{t=0}^{T}{\mathcal{R}}(\tau_{T})\nabla_{\theta}log\;\pi_{\theta}(a_{t}|s_{t})],$
(2)

which is more tractable. One of the most celebrated policy-based algorithms is
REINFORCE [24], or Monte-Carlo policy gradient, which samples a full
trajectory from the environment to avoid computing the expectation, and uses
(2) to update the parameters using the actual total return
$G_{t}=\sum_{t^{\prime}=t}^{T}r_{t^{\prime}}$ in lieu of
${\mathcal{R}}(\tau_{T})$.

Actor critic methods. The main drawback of Monte-Carlo methods is that they
exhibit a high variance, due to so called credit assignment problem. This
issue is usually addressed by subtracting a baseline value $b_{t}$ from the
return $G_{t}$ to reduce the variance of the gradient estimate while keeping
the bias unchanged [28]. A choice that provides several benefits is setting
$b_{t}$ to the VF. The quantity resultant from this subtraction is denoted as
$A(s_{t})=G_{t}-V(s_{t})$ and is called advantage function. The price to pay
in this case is that a method to estimate the VFs is needed as well. This
leads to a two-step method (actor-critic) that involves the estimation of the
parameters of the policy $\pi_{\theta}$ (actor) as well as those of the VF
(critic). The parameters of the actor $\theta$ are updated via gradient ascent
using

$\nabla_{\theta}J(\theta)=\sum_{t=0}^{T}A(s_{t})\nabla_{\theta}log\;\pi_{\theta}(a_{t}|s_{t}),$
(3)

where $A(s_{t})=G_{t}-V_{\omega}(s_{t})$, and $V_{\omega}(s_{t})$ is
considered given. In contrast, the parameters $\omega$ of the VF are obtained
as

$\omega^{*}=\arg\min_{\omega}{\mathcal{L}}(\omega):=\arg\min_{\omega}\frac{1}{2}\sum_{t=0}^{T}(G_{t}-V_{\omega}(s_{t}))^{2}.$
(4)

While some parametric models (4) can be solved in closed form, $\omega$ are
typically estimated using gradient descent methods of the form

$\omega_{h+1}=\omega_{h}-\alpha_{\omega}\nabla_{\omega}{\mathcal{L}}(\omega_{h}),$
(5)

where $h$ is the iteration index and $\alpha_{\omega}$ is the gradient step.

Gaussian policies. In setups where the action state ${\mathcal{A}}$ is
continuous and one-dimensional, a commonly adopted approach is to model
actions as samples of a univariate Gaussian distribution
$a_{t}\sim{\mathcal{N}}(a|\mu(s_{t}),\sigma(s_{t}))$ [29, 30]. The problem of
finding the optimal action associated with a state is reformulated as finding
the mean $\mu(s_{t})$ and the standard deviation $\sigma(s_{t})$ associated
with the state. The functions $\mu:{\mathcal{S}}\mapsto{\mathbb{R}}$ and
$\sigma:{\mathcal{S}}\mapsto{\mathbb{R}}^{+}$ can be either parametric or non-
parametric, with a stronger preference in the literature for the former.
Clearly, to find the gradient on the right-hand side of (3) w.r.t. those
parameters, one needs to resort to the chain rule and find first the
derivative of $J$ w.r.t. $\mu$ and $\sigma$. Upon setting the distribution of
the policy to a Gaussian, this approach leads to the following partial
derivatives

$\displaystyle\nabla_{\mu}J(\mu,\sigma)$
$\displaystyle=\sum_{t=0}^{T}A(s_{t})\frac{a_{t}-\mu(s_{t})}{\sigma(s_{t})^{2}},$
(6) $\displaystyle\nabla_{\sigma}J(\mu,\sigma)$
$\displaystyle=\sum_{t=0}^{T}A(s_{t})\left(\frac{(a_{t}-\mu(s_{t}))^{2}}{2\sigma(s_{t})^{3}}-\frac{1}{\sigma(s_{t})}\right).$
(7)

Practical implementations often consider simpler models for $\sigma(s_{t})$.
For example, it is not uncommon to assume that $\sigma$ is constant and
independent of the state $s_{t}$, reducing the problem to learn this single
value (or even fixing it using prior knowledge).

After describing the basics of policy methods in RL, we are ready to introduce
our scheme. The reader should notice that al- though in this paper we have
focused on Gaussian policies, the proposed approach can be extended to other
policy models.

##  3 Policy Gradients and Matrix Factorization

The state space ${\mathcal{S}}$ in most MDPs is discrete, either because the
number of states is finite or because it represents the discretized (sampled)
version of a continuous space. As a result, we have a finite number
$N_{\mathcal{S}}$ of states. Since each state is mapped to a mean and standard
deviation, the goal in this section is to provide a (low-rank based) scheme
that yields an efficient estimator for the $N_{\mathcal{S}}$ mean and standard
deviation pairs.

The first step in our approach is to introduce a matrix representation of the
states. In most applications states are multi-dimensional and as a result,
they can be indexed by a tuple of indexes. To keep the exposition simple,
suppose that two indices are used (either because there are only two
dimensions, or because the different dimensions are clustered into two
groups)111Generalizations of this approach to (low-rank) tensor models are
straighforward and well-motivated, but for simplicity, we limit this
conference paper to the matrix case.. With this in mind, define
${\mathbf{X}}_{\mu}\in{\mathbb{R}}^{N\times M}$ as the matrix that collects
the means associated with each of the $N_{\mathcal{S}}=N\cdot M$ states. The
idea under this approach is that every state $s\in{\mathcal{S}}$ is coded into
two indices $i_{s}\in\\{1,...,N\\}$ and $j_{s}\in\\{1,...,M\\}$ and, then, the
mean associated with $s_{t}$ is simply obtained as
$\mu(s_{t})=[{\mathbf{X}}_{\mu}]_{i_{s_{t}},j_{s_{t}}}$. Analogously, we
define ${\mathbf{X}}_{\sigma}\in{\mathbb{R}}^{N\times M}$ and set
$\sigma(s_{t})=[{\mathbf{X}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}$. If no structure
is imposed on ${\mathbf{X}}_{\mu}$ and ${\mathbf{X}}_{\sigma}$, the estimation
of $\mu$ and $\sigma$ is non-parametric and, as a result, the updates in (6)
and (7) suffice to estimate the policy.

However, such an approach would suffer from the curse of dimensionality. To
avoid this, we impose additional structure on ${\mathbf{X}}_{\mu}$ and
${\mathbf{X}}_{\sigma}$, forcing them to be low rank222Alternative schemes
that _promote_ low-rank via, e.g., nuclear norm regularizers are also
possible.. In particular, this entails introducing the matrices
${\mathbf{L}}_{\mu}\in{\mathbb{R}}^{M\times K}$,
${\mathbf{R}}_{\mu}\in{\mathbb{R}}^{K\times N}$,
${\mathbf{L}}_{\sigma}\in{\mathbb{R}}^{M\times K}$ and
${\mathbf{R}}_{\sigma}\in{\mathbb{R}}^{K\times N}$, and write the mean and
standard deviation matrices as
${\mathbf{X}}_{\mu}={\mathbf{L}}_{\mu}{\mathbf{R}}_{\mu}$ and
${\mathbf{X}}_{\sigma}={\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}$. This model
is still non-parametric, but it helps alleviating the curse of dimensionality.
The implications of this design decision are twofold: i) the rank of
${\mathbf{X}}_{\mu}$ and ${\mathbf{X}}_{\sigma}$ is at most $K$, limiting the
degrees of freedom of those matrices (hence, facilitating its estimation from
a limited number of observations); and ii) the parameters to estimate are no
longer the entries of ${\mathbf{X}}_{\mu}$ and ${\mathbf{X}}_{\sigma}$, but
the entries of ${\mathbf{L}}_{\mu}$, ${\mathbf{R}}_{\mu}$,
${\mathbf{L}}_{\sigma}$ and ${\mathbf{R}}_{\sigma}$. To see the latter point
more clearly, note that the mapping from the state to the parameters of the
Gaussian under the low-rank models is given by

$\mu(s_{t}\\!)\\!=\\!\\!\sum_{k=1}^{K}[{\mathbf{L}}_{\mu}]_{i_{s_{t}}\\!,k}[{\mathbf{R}}_{\mu}]_{k,j_{s_{t}}}\;|\;\sigma(s_{t}\\!)\\!=\\!\\!\sum_{k=1}^{K}[{\mathbf{L}}_{\sigma}]_{i_{s_{t}}\\!,k}[{\mathbf{R}}_{\sigma}]_{k,j_{s_{t}}}\\!.$
(8)

The next step is to optimize/estimate the values of the entries of
${\mathbf{L}}_{\mu}$, ${\mathbf{R}}_{\mu}$, ${\mathbf{L}}_{\sigma}$ and,
${\mathbf{R}}_{\sigma}$. To that end, we resort to non-convex gradient-based
matrix factorization approaches [5] along with the expressions for the
gradients provided in (6)-(7). To be more specific, in our approach the
parameters $\theta$ are the entries of
$\\{{\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma}\\}$
and our goal is to find an expression for the (entries of the) gradient in
(3). For simplicity, let us focus first on ${\mathbf{L}}_{\mu}$ and
${\mathbf{R}}_{\mu}$. We aim at finding the expression for the entries of
$\nabla_{{\mathbf{L}}_{\mu}}J({\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma})$,
and
$\nabla_{{\mathbf{R}}_{\mu}}J({\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma})$.
To that end, we need to apply the chain rule and combine the partial
derivatives of the cost $J$ w.r.t. $\mu$ in (6) with the partial derivatives
of the $\mu$ function in (8) w.r.t. each entry of the matrices
${\mathbf{L}}_{\mu}$, and ${\mathbf{R}}_{\mu}$. The result of this is

$\displaystyle\frac{\partial
J({\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma})}{\partial[{\mathbf{L}}_{\mu}]_{i,k}}=\sum_{t=0}^{T}\mathbb{I}_{i=i_{s_{t}}}A(s_{t})$
$\displaystyle\frac{a_{t}-[{\mathbf{L}}_{\mu}{\mathbf{R}}_{\mu}]_{i_{s_{t}},j_{s_{t}}}}{[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}^{2}}[{\mathbf{R}}_{\mu}]_{k,j_{s_{t}}}$
(9) $\displaystyle\frac{\partial
J({\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma})}{\partial[{\mathbf{R}}_{\mu}]_{k,j}}=\sum_{t=0}^{T}\mathbb{I}_{j=j_{s_{t}}}A(s_{t})$
$\displaystyle\frac{a_{t}-[{\mathbf{L}}_{\mu}{\mathbf{R}}_{\mu}]_{i_{s_{t}},j_{s_{t}}}}{[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}^{2}}[{\mathbf{L}}_{\mu}]_{i_{s_{t}},k}$
(10)

where $\mathbb{I}_{i=i_{s_{t}}}$, and $\mathbb{I}_{j=j_{s_{t}}}$ are indicator
functions. An analogous approach considering the derivatives of the cost $J$
w.r.t. $\sigma$ in (6), along with the derivatives of $\sigma$ w.r.t. the
entries of ${\mathbf{L}}_{\sigma}$ and ${\mathbf{R}}_{\sigma}$ yields:

$\displaystyle\frac{\partial
J({\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma})}{\partial[{\mathbf{L}}_{\sigma}]_{i,k}}=\sum_{t=0}^{T}\mathbb{I}_{i=i_{s_{t}}}A(s_{t})$
$\displaystyle\left(\frac{(a_{t}-[{\mathbf{L}}_{\mu}{\mathbf{R}}_{\mu}]_{i_{s_{t}},j_{s_{t}}})^{2}}{2[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}^{3}}-\frac{1}{[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}}\right)[{\mathbf{R}}_{\sigma}]_{k,j_{s_{t}}}$
(11) $\displaystyle\frac{\partial
J({\mathbf{L}}_{\mu},{\mathbf{R}}_{\mu},{\mathbf{L}}_{\sigma},{\mathbf{R}}_{\sigma})}{\partial[{\mathbf{R}}_{\sigma}]_{k,j}}=\sum_{t=0}^{T}\mathbb{I}_{j=j_{s_{t}}}A(s_{t})$
$\displaystyle\left(\frac{(a_{t}-[{\mathbf{L}}_{\mu}{\mathbf{R}}_{\mu}]_{i_{s_{t}},j_{s_{t}}})^{2}}{2[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}^{3}}-\frac{1}{[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}}\right)[{\mathbf{L}}_{\sigma}]_{i_{s_{t}},k}$
(12)

Once the actor is properly set up, the last step is to define the scheme to
estimate $\omega$, the parameters of the critic. As for the actor, we use a
matrix representation ${\mathbf{X}}_{\omega}\\!\in\\!{\mathbb{R}}^{N\times M}$
for the VF and then postulate that the VF matrix is low rank, so that it can
be factorized as the product of
${\mathbf{L}}_{\omega}\\!\in\\!{\mathbb{R}}^{N\times K}$ and
${\mathbf{R}}_{\omega}\\!\in\\!{\mathbb{R}}^{K\times M}$ with $K\ll$
$\min\\{N,M\\}$. As a result, the functional expression of the critic takes
the form
$V(s_{t})\\!=\\!\\!\sum_{k=1}^{K}[{\mathbf{L}}_{\omega}]_{i_{s_{t}}\\!,k}[{\mathbf{R}}_{\omega}]_{k,j_{s_{t}}\\!}$.
We then optimize the critic using alternating gradient descent, with the
partial derivatives of the critic cost in (4) w.r.t. the entries of
${{\mathbf{L}}_{\omega}}$ and ${\mathbf{R}}_{\omega}$ being

$\displaystyle\frac{\partial{\mathcal{L}}({\mathbf{L}}_{\omega},{\mathbf{R}}_{\omega})}{\partial[{\mathbf{L}}_{\omega}]_{i,k}}=\sum_{t=0}^{T}\mathbb{I}_{i=i_{s_{t}}}(G_{t}-[{\mathbf{L}}_{\omega}{\mathbf{R}}_{\omega}]_{i_{s_{t}},j_{s_{t}}})[{\mathbf{R}}_{\omega}]_{k,j_{s_{t}}}$
(13)
$\displaystyle\frac{\partial{\mathcal{L}}({\mathbf{L}}_{\omega},{\mathbf{R}}_{\omega})}{\partial[{\mathbf{R}}_{\omega}]_{k,j}}=\sum_{t=0}^{T}\mathbb{I}_{j=j_{s_{t}}}(G_{t}-[{\mathbf{L}}_{\omega}{\mathbf{R}}_{\omega}]_{i_{s_{t}},j_{s_{t}}})[{\mathbf{L}}_{\omega}]_{i_{s_{t}},k}$
(14)

The algorithm. A low-rank policy gradient (LRPG) algorithm flows from the
definitions above. The agent samples a full trajectory using the Gaussian
policy $\pi_{\mu,\sigma}={\mathcal{N}}(a_{t}|\mu(s_{t}),\sigma(s_{t}))$, where
the mean is
$\mu(s_{t})=[{\mathbf{L}}_{\mu}{\mathbf{R}}_{\mu}]_{i_{s_{t}},j_{s_{t}}}$, and
the standard deviation is
$\sigma(s_{t})=[{\mathbf{L}}_{\sigma}{\mathbf{R}}_{\sigma}]_{i_{s_{t}},j_{s_{t}}}$.
Then, the actor matrices ${\mathbf{L}}_{\mu}$, ${\mathbf{R}}_{\mu}$,
${\mathbf{L}}_{\sigma}$, and ${\mathbf{R}}_{\sigma}$ are updated via
stochastic gradient ascent using (9)–(12). Finally, the critic matrices
${\mathbf{L}}_{\omega}$, and ${\mathbf{R}}_{\omega}$ are updated via
stochastic gradient descent using (13)–(14). The algorithm is depicted in
Algorithm LABEL:alg:pg.

##  4 Numerical experiments

We test LRPG in three standard continuous-action RL problems of the toolkit
OpenAI Gym [31]. The first scenario is the inverted pendulum, where an agent
tries to keep a pendulum upright. The second one is the acrobot, which mimics
a gymnast trying to swing up. The third one is the Goddard problem, a rocket
optimizing its peak altitude ascending vertically. We test and compare NN-
based vs low-rank-based policy methods. The figures of merit reported are i)
parametrization-efficiency, ii) convergence rate, and iii) return obtained by
the estimated policy. Details can be found in [32].

Experimental setup. We compared the LRPG algorithm against REINFORCE with VF-
baselines (RVFB). In both scenarios, given a state $s_{t}$, the action $a_{t}$
is sampled from the normal distribution ${\mathcal{N}}(a|\mu(s_{t}),\sigma)$.
For the sake of simplicity, $\sigma$ does not depend on $s_{t}$ and the
difference between the schemes resides in the model to estimate the mean
$\mu(s_{t})$. While the parameters for LRPG are the entries of
${\mathbf{L}}_{\mu}$ and ${\mathbf{R}}_{\mu}$, the RVFB algorithm uses an NN
model $\mu_{\theta}(s_{t})=\text{NN}_{\theta}(s_{t})$. Similarly, LRPG models
the VF as
$V(s_{t})=[{\mathbf{L}}_{\omega}{\mathbf{R}}_{\omega}]_{i_{s_{t}},j_{s_{t}}}$,
and RVFB as $V_{\omega}(s_{t})=\text{NN}_{\omega}(s_{t})$. Since the three
setups tested are continuous, LRPG discretizes the state space
${\mathcal{S}}$. Note that the finer the discretization, the larger the number
of entries of ${\mathbf{X}}_{\mu}$ and ${\mathbf{X}}_{\omega}$. Imposing low
rank can drastically reduce the number of parameters while keeping a fine
sampling resolution. The number of total states is defined by the Cartesian
product of a regularly-sampled grid. On the other hand, NNs can deal with
continuous setups, but choosing the proper architecture is usually a hard
task. For the presented scenarios, we run $100$ simulations of each algorithm.
In each episode, we measure the return per episode
$\check{{\mathcal{R}}}=\sum_{t=0}^{T}r_{t}$. In Fig. 1 the median
$\check{{\mathcal{R}}}$ (across the $100$ simulations) is shown. We have
exhaustively examined the space of all potential fully-connected NN
architectures to find the smallest one that solves the problem. A summary of
the parametrizations and associated median returns is provided in Table 1.

Fig. 1: Median return per episode in three standard RL problems: (a) the pendulum, (b) the acrobot, and (c) the Goddard problem. Env. | LRPG | RVFB
---|---|---
Parameters | Return | Parameters | Return
Pendulum | 160 | 99.34 | 4,098 | 98.81
Acrobot | 16 | 94.45 | 386 | 94.25
Rocket | 80 | 1.17 | 1,282 | 1.16
Table 1: Parameters vs. Median return.

Convergence properties. The rate of convergence of LRPG and RVFB can be
observed in Fig. 1. Here, we use the term convergence loosely to refer to the
fact that the return reaches a steady state. In all scenarios, LRPG needs
fewer episodes to converge to the final policy than RVFB. The strength of this
finding varies across scenarios, being conspicuous in some of them. In the
Pendulum scenario, LRPG algorithm needs almost $500$ episodes less to converge
than RVFB (5 times faster). Similarly, in the Goddard problem, LRPG converges
around episode $7,000$, while RVFB converges in episode $22,000$. Note,
however, that this is not the case in the acrobot setup, where LRPG converges
only slightly faster than RVFB. The faster convergence of LRPG is likely due
to the fact that low-rank factorized models are simpler and have fewer
parameters to estimate (see Table 1). Furthermore, trading off between
convergence rate and obtained return is significantly harder in NN-based
setups since exploring and testing NN architectures is challenging.

Parametrization efficiency. As summarized in Table 1, LRPG needs far fewer
parameters than RVFB to estimate its associated policy. In the pendulum and
acrobot environments, the size of the LRPG low-rank matrix-based models is
approximately $~{}4\%$ of that of the NNs in RVFB. In the case of the Goddard
problem, the size of the matrix model is around $~{}6\%$ of the size of the NN
model. As mentioned previously, no smaller fully-connected NN was found to
converge in these problems. Yet the policies estimated by LRPG lead to higher
returns in all scenarios. NNs tend to have problems with local optima,
especially in RL setups, and the space of all possible architectures is vast.
Hence, the problem of finding the adequate architecture not only affects the
reward, but also the convergence properties of the algorithm. In contrast,
tuning the LRPG algorithm boils down to designing the discretization and
indexing of the state space ${\mathcal{S}}$, along with the rank of the model
$K$.

##  5 Conclusions

This paper introduced a _low-rank_ policy gradient (LRPG) algorithm, which
leverages matrix completion in (discrete) matrix-based Gaussian policy models.
This approach portrays a natural way of introducing parsimony in policy-based
RL techniques. The low-rank policy is very efficient in terms of parameters,
and it is easy to interpret. LRPG was tested in three OpenAI Gym classical
control environments, comparing it with standard NN-based schemes. The limited
but meaningful tests reveal that LRPG achieves high rewards, usually better
than NN-based models, with a relatively small number of parameters.
Furthermore, it requires significantly less number of episodes to converge to
the final policy.

## References

  * [1] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. The MIT Press, 2nd Ed., 2018\.
  * [2] D. P. Bertsekas, _Reinforcement learning and optimal control_. Athena Scientific, 2019.
  * [3] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 26–38, 2017.
  * [4] C. Eckart and G. Young, “The approximation of one matrix by another of lower rank,” _Psychometrika_ , vol. 1, no. 3, pp. 211–218, 1936.
  * [5] I. Markovsky, _Low rank approximation_. Springer, 2019.
  * [6] M. Udell, C. Horn, R. Zadeh, S. Boyd _et al._ , “Generalized low rank models,” _Foundations and Trends® in Machine Learning_ , vol. 9, no. 1, pp. 1–118, 2016.
  * [7] M. Mardani, G. Mateos, and G. B. Giannakis, “Decentralized sparsity-regularized rank minimization: Algorithms and applications,” _IEEE Trans. Signal Processing_ , vol. 61, no. 21, pp. 5374–5388, 2013.
  * [8] E. Tolstaya, A. Koppel, E. Stump, and A. Ribeiro, “Nonparametric stochastic compositional gradient descent for q-learning in continuous markov decision problems,” in _2018 Annual American Control Conference (ACC)_. IEEE, 2018, pp. 6608–6615.
  * [9] G. Lever, J. Shawe-Taylor, R. Stafford, and C. Szepesvári, “Compressed conditional mean embeddings for model-based reinforcement learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * [10] A. M. Barreto, R. L. Beirigo, J. Pineau, and D. Precup, “Incremental stochastic factorization for online reinforcement learning,” in _Proc. AAAI Conf. Artificial Intelligence_ , 2016.
  * [11] N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire, “Contextual decision processes with low bellman rank are pac-learnable,” in _Proc. Intl. Conf. Machine Learning-Volume 70_. JMLR. org, 2017, pp. 1704–1713.
  * [12] A. Mahajan, M. Samvelyan, L. Mao, V. Makoviychuk, A. Garg, J. Kossaifi, S. Whiteson, Y. Zhu, and A. Anandkumar, “Tesseract: Tensorised actors for multi-agent reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 7301–7312.
  * [13] H. Y. Ong, “Value function approximation via low-rank models,” _arXiv preprint arXiv:1509.00061_ , 2015.
  * [14] B. Cheng and W. B. Powell, “Co-optimizing battery storage for the frequency regulation and energy arbitrage using multi-scale dynamic programming,” _IEEE Trans. Smart Grid_ , vol. 9.3, pp. 1997–2005, 2016.
  * [15] B. Cheng, T. Asamov, and W. B. Powell, “Low-rank value function approximation for co-optimization of battery storage,” _IEEE Trans. Smart Grid_ , vol. 9.6, pp. 6590–6598, 2017.
  * [16] F. S. Melo and M. I. Ribeiro, “Q-learning with linear function approximation,” in _Intl. Conf. Comp. Learning Theory_. Springer, 2007, pp. 308–322.
  * [17] B. Behzadian, S. Gharatappeh, and M. Petrik, “Fast feature selection for linear value function approximation,” in _Proc. Intl. Conf. Automated Planning and Scheduling_ , vol. 29, no. 1, 2019, pp. 601–609.
  * [18] Y. Yang, G. Zhang, Z. Xu, and D. Katabi, “Harnessing structures for value-based planning and reinforcement learning,” in _International Conference on Learning Representations_ , 2020.
  * [19] D. Shah, D. Song, Z. Xu, and Y. Yang, “Sample efficient reinforcement learning via low-rank matrix estimation,” in _Proc. of the 34th International Conference on Neural Information Processing Systems_ , ser. NIPS’20. Red Hook, NY, USA: Curran Associates Inc., 2020.
  * [20] S. Rozada, V. Tenorio, and A. G. Marques, “Low-rank state-action value-function approximation,” in _2021 29th European Signal Processing Conference (EUSIPCO)_ , 2021, pp. 1471–1475.
  * [21] S. Rozada and A. G. Marques, “Tensor and matrix low-rank value-function approximation in reinforcement learning,” _arXiv preprint arXiv:2201.09736_ , 2022.
  * [22] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * [23] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [24] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine learning_ , vol. 8, no. 3, pp. 229–256, 1992.
  * [25] J. Peters and S. Schaal, “Natural actor-critic,” _Neurocomputing_ , vol. 71, no. 7-9, pp. 1180–1190, 2008.
  * [26] D. Lee, N. He, P. Kamalaruban, and V. Cevher, “Optimization for reinforcement learning: From a single agent to cooperative agents,” _IEEE Signal Processing Magazine_ , vol. 37, no. 3, pp. 123–135, 2020.
  * [27] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” _Advances in neural information processing systems_ , vol. 12, 1999.
  * [28] E. Greensmith, P. L. Bartlett, and J. Baxter, “Variance reduction techniques for gradient estimates in reinforcement learning.” _Journal of Machine Learning Research_ , vol. 5, no. 9, 2004.
  * [29] S. Levine and P. Abbeel, “Learning neural network policies with guided policy search under unknown dynamics,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [30] K. Ciosek and S. Whiteson, “Expected policy gradients,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [31] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [32] S. Rozada, “Online code repository: Matrix low-rank approximation for policy gradient methods,” https://github.com/sergiorozada12/matrix-low-rank-pg, 2022.


# Physics-informed generative neural networks
for RF propagation prediction with application to
indoor body perception ††thanks: This work is funded by the European Union.
Views and opinions expressed are however those of the author(s) only and do
not necessarily reflect those of the European Union or European Innovation
Council and SMEs Executive Agency (EISMEA). Neither the European Union nor the
granting authority can be held responsible for them. Grant Agreement No:
101099491.

Federica Fieramosca1, Vittorio Rampa2, Michele D’Amico1, Stefano Savazzi2 1 __
DEIB, Politecnico di Milano, Piazza Leonardo da Vinci 32, I-20133, Milano,
Italy
2 _Consiglio Nazionale delle Ricerche_ _,__IEIIT_ institute, Piazza Leonardo
da Vinci 32, I-20133, Milano, Italy.

###### Abstract

Electromagnetic (EM) body models designed to predict Radio-Frequency (RF)
propagation are time-consuming methods which prevent their adoption in strict
real-time computational imaging problems, such as human body localization and
sensing. Physics-informed Generative Neural Network (GNN) models have been
recently proposed to reproduce EM effects, namely to simulate or reconstruct
missing data or samples by incorporating relevant EM principles and
constraints. The paper discusses a Variational Auto-Encoder (VAE) model which
is trained to reproduce the effects of human motions on the EM field and
incorporate EM body diffraction principles. Proposed physics-informed
generative neural network models are verified against both classical
diffraction-based EM tools and full-wave EM body simulations.

###### Index Terms:

EM body models, generative models, variational autoencoders, generative
adversarial networks, radio tomography, integrated sensing and communication,
localization.

##  I Introduction

Communication while sensing tools employ ambient radio signals used for
wireless communications also to detect, locate, and track people that do not
carry any electronic device, namely device-free [1, 2, 3]. Data analytic
tools, such as Bayesian [4] and machine learning approaches [5], can be used
to extract information about body movements from the observed RF radiation
field.

Approaches proposed for solving the radio sensing problem require an
approximated knowledge of a physics-informed (prior) model used to interpret
the effects of human subjects on radio propagation [6]. The impact of body
movements on radio signals can be interpreted using electromagnetic (EM)
propagation theory considerations [7]. To this aim, several physical and
statistical models have been then proposed to exploit full wave approaches,
ray tracing, moving point scattering [8] and diffraction theory [9, 10, 11,
12]. On the other hand, a general EM model for the prediction of body-induced
effects on propagation is still under scrutiny [13] while current full-wave
models are also too complex to be of practical use for real-time sensing
scenarios [14].

Physics-informed generative modeling [15] is an emerging field in several
application contexts ranging from EM field computation, imaging and inverse
problems. For example, Generative Neural Networks (GNN) generate observations
drawn from a distribution which reflects the complex underlying physics of the
environment under study. Generative models for prediction of RF propagation
find applications in communications and sensing [16]. For example, [17]
discussed a convolutional encoder-decoder structure that can be trained to
reproduce the results of a ray-tracer, while the generation of body induced RF
excess attenuation values to simulate diffraction effects was considered in
[18]. A ML-assisted channel modeling approach was proposed in [19] to generate
site-specific mmWave channel characteristics. Proposed EM-informed GNN models
can be trained with samples obtained either by EM diffraction-based body
models [10, 11, 12, 14], as also done in [18], or using training samples
obtained from EM body simulation tools, such as FEKO® software.

Figure 1:  Link geometry with body, sketched as a 2D EM perfectly absorbing
sheet, and Multiple-Input-Multiple-Output (MIMO) antennas placed in the
monitored area. The top view of this area is shown on the right as well.
Figure 2:  Conditional VAE system, encoder and decoder GNN structures.

Contributions. The paper discusses the adoption of EM-informed generative
models inspired by VAE tools [20, 18]. VAE methods can be used to build a
surrogate model for EM field analysis comprising of a generator of EM field
samples which is trained to reproduce EM diffraction effects [11, 12, 14]. The
paper extends [18] as it proposes a tool to reproduce the effects of body
movements on the complex EM field, rather than the scalar body-induced RF
attenuation (in fact, complex values of EM field are needed for array
processing [21]). As depicted in Fig. 1, the proposed generators are also
optimized for multiple antenna settings, and can be used to reproduce the
responses of conventional array processing. In line with initial studies [15,
22, 23], the proposed generators are purely based on sequences of
convolutional (and de-convolutional) layers.

The paper is organized as follows. Sect. II introduces the problem and the
setup adopted for simulation, while Sect. III discusses relevant EM body
models for passive radio sensing with multiple antennas. Sect. IV targets the
problem of generative modelling, and discusses adaptations to VAE tools. Sect.
V verifies the models effectiveness in reproducing the array response in
typical dense MIMO configurations and compared with two EM body models.
Conclusions are finally discussed in Sect. VI.

##  II GNNs for reproducing RF propagation effects

The goal of radio sensing is to extract the EM effects ($\mathbf{E}_{\theta}$)
of human body movements on propagation, based on the noisy measurements
$\mathbf{S}_{t}$ of the RF radiation at time $t$. The subjects, or targets,
are characterized by an unknown state $\mathbf{\theta}$ which we want to
recover in real-time. Subject state might provide information about motions,
location, size/shape, and orientation [4, 11, 12] of the bodies. As depicted
in Fig. 1, we consider a scenario where the EM body effects
$\mathbf{E}_{\theta}$ are observed by a dense array of receivers. Following
the classical Bayesian formulation, the objective is to maximize the a
posterior distribution $p(\mathbf{E}_{\theta}|\mathbf{S}_{t})$:

$p(\mathbf{E}_{\theta}|\mathbf{S}_{t})=\frac{p(\mathbf{S}_{t}|\mathcal{\mathbf{E}_{\theta}})\cdot
p(\mathbf{E}_{\theta})}{p(\mathbf{S}_{t})}$ (1)

of the (unknown) effects $\mathbf{E}_{\theta}$, given the measurements
$\mathbf{S}_{t}$. Measurements considered here are in the form of complex
base-band channel impulse response (CIR) [3] obtained from radio devices
equipped with multiple antenna frontends. The observations $\mathbf{S}_{t}$
are perturbed by body movements according to the prior distribution
$p(\mathbf{E}_{\theta})$, which models the expected effects of the body in the
state $\mathbf{\theta}$ as the result of the RF propagation.

Solving the radio sensing problem requires the knowledge of the RF
measurements model $p(\mathbf{S}_{t}|\mathbf{E}_{\theta})$, and the prior
distribution $p(\mathbf{E}_{\theta})$. While the first term depends on the
data collection process, the prior distribution
$p(\mathcal{\mathbf{E}_{\theta}})$ is usually hard to model as it often
requires time-consuming full-wave EM approaches. Therefore, these methods are
commonly considered of limited practical use for real-time sensing scenarios
[14]. In line with [18], the proposed GNN model approach is designed to
generate samples from the prior distribution
$p(\mathcal{\mathbf{E}_{\theta}})$. Generation times are analyzed to assess
the feasibility of using the proposed model during real-time target tracking
as well.

##  III EM body modelling with multiple antennas

In this section, we discuss relevant EM body models to reproduce the EM
effects $\mathbf{E}_{\theta}$ of body movements. As far as the diffraction-
based model is concerned, we adopted the body model originally proposed in
[11] for a single link scenario and extended in [21] for the case of multiple
array receivers.

Full-wave electromagnetic simulations were performed with Altair® FEKO®, a
commercial software package that implements, among others, the Method of
Moments (MoM), a full-wave solution of Maxwell’s integral equations in the
frequency domain. These calculations are computationally-intensive: in fact,
as described in Sect. V, the surface of the humanoid obstacle is discretised
using triangles, then the Surface Equivalent Principle (SEP) is applied by
introducing equivalent electric and magnetic currents on the surface of the
dielectric body.

Besides full-wave calculations, that are needed to represent the EM field,
most of physical models designed to quantify realistic body motions must also
take into account imperfect knowledge of the environment, namely small and
involuntary body movements [11] which cause the subject to change its
orientation or pose w.r.t. to the receivers. All these impairments make body
features hard to obtain with an acceptable level of accuracy. A statistical
approach is often adopted, being in line with classical Bayesian informative-
prior modeling. We assume that EM effects $\mathbf{E}_{\theta}$ are obtained
for random instances of body configuration/features $\theta$, which follow the
probability function $p(\theta|\theta_{k})$. Formally, the received EM field
$\mathbf{E}_{\theta}$ is thus defined in probability as it is sampled from a
distribution $p(\mathbf{E}_{\theta})$ defined as:

$p(\mathbf{E}_{\theta}|\theta_{k})=\mathbf{\mathbf{E}}_{\theta\sim
p(\theta|\theta_{k})},$ (2)

where the function $p(\theta|\theta_{k})$ models the uncertainty with respect
to the _nominal body features_ $\theta_{k}$.

##  IV Physics-informed generative neural network

The generative models considered here are able to reproduce different body-
induced EM effects $\mathbf{E}_{\theta}$ as sampled from the ground-truth
prior distribution $p(\mathbf{E}_{\theta}|\theta_{k})$ in (2), and conditioned
on the input nominal body features, or configurations, $\theta_{k}$. Unlike
[18], body effects are represented here by the EM field responses
$\mathbf{E}_{\theta}$, while the generative tool is designed to reproduce the
body effects observed for each receiving antenna/link $\ell$. We also limit
our focus on simple body features
$\theta_{k}=\left[\mathbf{p}_{k},\varphi_{k},h_{S},w_{S,1},w_{S,2}\right]$
that include body locations $\mathbf{p}_{k}$, relative orientations
$-\pi/2\leq\varphi_{k}\leq\pi/2$, height $h_{S}$, max $w_{S,1}$ and min
$w_{S,2}$ traversal sizes of the target, which could, for example, represent a
change in the subject pose (see Sect. V-B).

###  IV-A Conditional Variational Auto-Encoder

As depicted in Fig. 2, the proposed VAE model, namely conditional VAE, or
C-VAE for short, uses an encoder
$\mathcal{Q}(\mathbf{z}|\mathbf{E}_{\theta},\theta_{k};\mathbf{W}_{E})$,
parameterized by the NN parameters $\mathbf{W}_{E}$, which learns the latent
space
$p_{\mathcal{Z}}(\mathbf{z}|\theta_{k})\sim\mathcal{N}(\mathbf{\boldsymbol{\mu}_{\mathit{k}}},\boldsymbol{\sigma}_{k}^{2})$
for inputs $\theta_{k}$, assumed here as multivariate Gaussian distributed,
with mean $\mathbf{\boldsymbol{\mu}_{\mathit{k}}}$ and standard deviation
$\boldsymbol{\sigma}_{k}$ parameters. Note that other choices are not
investigated here. The encoder uses training samples of (true) body-induced
values $\mathbf{E}_{\theta}$ and the corresponding features $\theta_{k}$ of
interest. The decoder produces a distribution
$\widehat{\mathbf{E}}_{\theta}^{\mathrm{VAE}}\sim
p_{\mathrm{gen}}^{\mathrm{VAE}}(\mathbf{E}_{\theta}|\theta_{k})$

$p_{\mathrm{gen}}^{\mathrm{VAE}}(\mathbf{E}_{\theta}|\theta_{k})=\int_{\mathcal{Z}}p_{\mathrm{gen}}^{\mathrm{VAE}}(\mathbf{E}_{\theta}|\mathbf{z},\theta_{k};\mathbf{W}_{D})\,p_{\mathcal{Z}}(\mathbf{z}|\theta_{k})\,d\mathbf{z},$
(3)

which is the marginalization of the conditional probability
$p_{\mathrm{gen}}^{\mathrm{VAE}}(\mathbf{E}_{\theta}|\mathbf{z},\theta_{k};\mathbf{W}_{D})$
function of the NN parameters $\mathbf{W}_{D}$. The goal is to maximize the
likelihood bound namely Evidence Lower BOund (ELBO)
$\mathcal{L}_{\mathrm{ELBO}}$ as described in [24].

TABLE I: C-VAE vs diffraction model and FEKO® simulations: model configuration and EM field generation time analysis. Model | Configuration | Generation time
---|---|---
| ($L$ links) | [s/sample/link]
|  $Z=16$, $L=1$ | $4.6\times 10^{-5}$
C-VAE |  $Z=32$, $L=1$ | $6.3\times 10^{-5}$
$(\widehat{\mathbf{E}}_{\theta}$ generation) |  $Z=16$, $L=81$ | $5.3\times 10^{-2}$
|  $Z=32$, $L=81$ | $6.1\times 10^{-2}$
|  $\epsilon=10^{-3}$ , $L=1$ | $5.4\times 10^{-3}$
EM Diffraction |  $\epsilon=10^{-6}$, $L=1$ | $3.81\times 10^{-2}$
(omnidirectional antennas) |  $\epsilon=10^{-3}$, $L=81$ | $0.78$
|  $\epsilon=10^{-6}$, $L=81$ | $3.1$
EM FEKO® simulations | $L=81$ | $>240$

###  IV-B Implementation considerations

C-VAE pre-trained models reproducing diffraction effects are available on-line
[25] together with example codes for generating EM field samples
($\widehat{\mathbf{E}}_{\theta}$) according to specific body configurations.
Generation times of C-VAE are compared in Tab. I with those obtained by
classical diffraction models [12] as well as popular EM body simulator tools
(Altair® FEKO® software, see Sect. V). In the example, samples are generated
assuming $L=81$ MIMO links with variable number of antennas at TX and RX. VAE-
based generation of the samples is about one order of magnitude faster than EM
diffraction model computation, which also depends on the chosen numerical
integration configurations (i.e., tiled integration method, and absolute error
tolerance $\epsilon$), target size, and antenna configuration. The generative
model can produce up to $50\div 100$ samples per second, which is reasonable
considering typical body movement speeds (max. $1$m/s).

##  V Generation of EM body model samples

C-VAE sample generation is assessed here at carrier frequency $f_{c}=2.4$ GHz.
First, we analyze the reproduction accuracy against EM body diffraction [12].
Next, we compare the results with FEKO® simulations. The RX node is equipped
with a Uniform Linear Array (ULA) of $9$ omni-directional antennas and spaced
at $\triangle=\lambda/2$. The TX node is equipped with a single antenna while
$L=9$ links are considered. All the links of the array are horizontally placed
at height $h=0.99$ m from the ground. Distance between the TX and the central
RX of the array is $d=4$ m. C-VAE is trained here using examples of EM
diffraction from: 1) $75$ marked locations inside the first Fresnel ellipsoid
of the central link of the array; 2) varying target dimension $h_{S}$ in the
interval $[1.65-2.00]$ m.

Figure 3:  Array layout setups (top) and corresponding responses (bottom)
$R_{\theta}(\gamma)$ for $\gamma=\pi/2$. Target has dimensions $h_{S}=1.65$ m,
$w_{S,1}=0.55$ m, $w_{S,2}=0.25$ m. Two subject positions are considered,
namely $x=2$ m, $y=0.25$ m (left) and $x=2$ m, $y=-0.25$ m (right). Response
is obtained using VAE-generated field samples
$\widehat{\mathbf{E}}_{\theta}^{\mathrm{VAE}}$ (green/red solid lines) and
compared with the array response obtained with the EM body diffraction model
(dashed lines).

In the following, we verify the ability of the proposed C-VAE model to
generate samples of the EM field
$\widehat{\mathcal{\mathbf{E}}}_{\theta}^{\mathrm{VAE}}=\left[\widehat{E}_{\ell,\theta}^{\mathrm{VAE}}\right]{}_{\ell=1}^{L}$
which are able to reproduce the array excess attenuation response of the
conventional linear beamforming processing [21, 26]. For $L=2M+1$ antennas
($M=4$), we consider the vector
$\mathbf{w}(\gamma)=\left[w_{k}\right]_{k=-M,...,0,...,+M}=\left[w_{-M}\,...\,w_{-1}\,w_{0}\,w_{1}\,...\,w_{M}\right]^{T}$
of linear beamforming coefficients designed to _steer_ the antenna array in a
direction $\gamma$. The received base-band signal $r_{\theta}(\gamma)$ at the
output of the beamforming processing is:

$r_{\theta}(\gamma)=\sum_{m=-M}^{+M}w_{m}^{*}(\gamma)\,\left[E_{\ell,\theta}+n_{\ell}\right]=\mathbf{w(\gamma)}^{H}\cdot\left[\mathbf{E}_{\theta}+\mathbf{n}\right],$
(4)

where $n_{\ell}$ is the Additive White Gaussian Noise (AWGN) complex vector
$\mathbf{n}=\left[n_{-M}\,...n_{-1}\,n_{0}\,n_{1}\,...\,n_{M}\right]^{T}$ of
size $2M+1$, with covariance $\sigma^{2}\mathbf{I}$. We define the body-
induced excess attenuation response $F_{\theta}(\gamma)$ with target in state
$\theta$ as

$F_{\theta}(\gamma|\mathbf{E}_{\theta})=\sum_{m=-M}^{+M}w_{m}^{*}(\gamma)\,\frac{E_{\ell,0}}{E_{\ell,\theta}}=\mathbf{w(\gamma)}^{H}\cdot\frac{\mathbf{E}_{0}}{\mathbf{E}_{\theta}}.$
(5)

with $\mathbf{E}_{0}$ being the EM field measured in free space and $\gamma$
the DoA (Direction Of Arrival), namely the direction of propagation of the
impinging wavefront w.r.t. the array axis.

###  V-A Generation of antenna array responses

Figure 4:  Body-shape model built in FEKO® with size $h_{S}=1.80$ m,
$w_{S,1}=0.52$ m, and $w_{S,2}=0.32$ m. Simulation settings are shown as well.
Figure 5:  Excess attenuation values obtained by FEKO® software (solid) and
compared with C-VAE generation (dashed) for varying antennas of the array (9
antennas) and target dimensions. Three nominal subject positions are
considered, namely $x=2$ m, $y=0.25$ m (red), $x=2$ m, $y=0$ m (black) and
$x=2$ m, $y=-0.25$ m (green). For all cases, attenuation responses account for
small body movements around the nominal positions. Figure 6:  DoA analysis for
a subject moving across the LOS ($-0.25$ m $\leq y$ $\leq 0.25$ m, $x=2$ m).
Comparative analysis of array responses obtained from C-VAE generation and
FEKO® simulations.

The C-VAE model is set here to reproduce the response obtained by EM body
diffraction
$F_{\theta}(\gamma|\widehat{\mathbf{E}}_{\theta}^{\mathrm{VAE}})\sim
F_{\theta}(\gamma|\mathbf{E}_{\theta})$ through the generation of the
corresponding EM field samples $\widehat{\mathbf{E}}_{\theta}^{\mathrm{VAE}}$.
For conventional ULA scenarios, that assume planar wavefront propagation, the
steering vector $\mathbf{w}(\gamma)=\left[a_{m}\right]$ for the array is given
by eq. (7) of [21] (see also [26] for details).

Fig. 3 shows the body-induced array response, in terms of the excess
attenuation $20\,\log_{10}|F_{\theta}(\gamma)|$ as a function of the DoA
$\gamma$ and for different values of the $y$ displacement of the target
(w.r.t. the central LOS) and $x=2$ m. The human target has height $h_{S}=1.65$
m, traversal max. and min. sizes equal to $w_{S,1}=0.55$ m and $w_{S,2}=0.25$
m, respectively. The array signal processing extracts the response for varying
DoA $\gamma$ and is based on Fast Fourier Transform (FFT) with $N_{FFT}=257$
points. The figure shows two responses (red and green lines) for corresponding
nominal displacements $y=0.25$ m and $y=-0.25$ m, respectively. Dashed lines
show the responses $F_{\theta}(\gamma|\mathbf{E}_{\theta})$ obtained using EM
body diffraction, while solid lines represent the reproduced response
$F_{\theta}(\gamma|\widehat{\mathbf{E}}_{\theta}^{\mathrm{VAE}})$ using C-VAE.
Notice that, in the example, each array response represents a random movement
of the body in the surrounding of the nominal displacement. The maximum
response of the array, namely the dominant DoA
$\gamma_{\mathrm{max}}=\arg\max_{\gamma}F_{\theta}(\gamma|\mathbf{E}_{\theta})$,
is perturbed by the presence of the subject and such alteration is generally
well reproduced by the C-VAE model.

###  V-B Analysis with EM body simulations

In this section, we verify the ability of the C-VAE model to reproduce the
results of EM field simulation obtained by diffraction and MoM methods (i.e.,
using Altair® FEKO® tools). The setup is again composed by a transmitter (TX)
and a receiver array (RX) at $4$ m distance. The former is an electric dipole
transmitting at $f=2.4868$ GHz, the latter is composed by 9 isotropic elements
spaced at $\lambda/2$. For FEKO® simulations, rather than a 2D perfectly
absorbing sheet, as assumed in the scalar diffraction models [12], an
anthropomorphic obstacle is introduced between the two devices. The obstacle
is built as shown in Fig. 4: a body-shape model, $h_{S}=1.80$ m,
$w_{S,1}=0.52$ m and $w_{S,2}=0.32$ m, is used, which is an improvement over
the 2D PEC object considered in [11, 12, 18, 27, 28, 21]. The material of the
body is retrieved from a simulator available online [29]. Assuming the body
mainly constituted by muscles, we set: the relative permittivity
$\epsilon_{r}=60$, the dielectric loss tangent tan$\delta=0.242$, and the mass
density $\rho=1040.0$ kg/m3. Simulations are taken with the body located in
three positions: $1$) $x=2$ m and $y=0$ m (LOS, black); $2$) $x=2$ m, and
$y=0.25$ m (red); $3$) $x=2$ m, and $y=-0.25$ m (green).

In Fig. 5, we report the body-induced excess attenuation values measured by
each antenna element for the corresponding subject positions (black, red and
green colors). Responses consider the effect of small body movements ($5$ cm)
around the nominal subject position. FEKO® simulated results are shown in
solid lines, while in dashed lines we reported the corresponding C-VAE
generated attenuation samples reproducing EM body diffraction effects. In
general, the GNN mostly underestimates the FEKO® results, due to different
hypotheses about body composition and shape, namely about $4$ dB for positions
on the LOS and $2$ dB for positions outside ($y=0.25$ m, $y=-0.25$ m). A
possible remedy, to be investigated in the future, is to train the GNN network
using examples from both diffraction model and FEKO® simulation samples, with
the goal of improving the GNN generalization capabilities.

Fig. 6 compares the array responses and the DoA $\gamma_{\mathrm{max}}$
obtained from the C-VAE generator
$\gamma_{\mathrm{max}}=\arg\max_{\gamma}F_{\theta}(\gamma|\widehat{\mathbf{E}}_{\theta}^{\mathrm{VAE}})$
(on the left) and FEKO® simulated array response under the same configuration
(right figure). In all cases the target is moving across the LOS ($-0.25$ m
$\leq y\leq 0.25$ m, $x=2$ m), with speed $0.5$ m/s. The movements of the
target are distinguishable, as associated with separable DoAs, considering
both generated samples (reproducing diffraction effects) and FEKO®
simulations.

##  VI Concluding remarks

The paper considered the adoption of an EM-informed Generative Neural Network
(GNN) model to predict the effects of body movements on radio propagation,
with applications to passive radio localization and sensing. A variational
auto-encoder (VAE) tool is designed to generate samples of EM field to
reproduce body-induced EM diffraction effects, considering small and
involuntary movements of the body, as well as multiple antenna settings.
Moreover, the C-VAE is several orders of magnitude faster than diffraction and
full-wave methods. The paper verified the model effectiveness in reproducing
the array response in typical MIMO configurations and the perturbations
induced by body movements on such responses. Results are compared with scalar
diffraction models as well as EM simulations based on FEKO® software.

## References

  * [1] M. Youssef, et al., “Challenges: Device-free passive localization for wireless environments,“ 13th Annu. ACM Intl. Conf. Mobile Comput. Netw. (MobiCom’07), pp. 222–229, 2007.
  * [2] J. Wilson, et al., “Radio tomographic imaging with wireless networks,“ IEEE Trans. on Mob. Comp., vol. 9, no.5, pp. 621–632, May 2010.
  * [3] S. Savazzi, et al., “On the use of stray wireless signals for sensing: A look beyond 5G for the next generation of industry,“ Computer, vol. 52, no. 7, pp. 25–36, Jul. 2019.
  * [4] O. Kaltiokallio, et al., “A Novel Bayesian Filter for RSS-Based Device-Free Localization and Tracking,“ IEEE Trans. on Mob. Comp., vol. 20, no. 3, pp. 780–795, Mar. 2021.
  * [5] S. Palipana, et al., “Recent advances in RF-based passive device-free localisation for indoor applications,“ Ad Hoc Netw., vol. 64, pp. 80–98, 2017.
  * [6] Z. Wang, et al., “A Diffraction Measurement Model and Particle Filter Tracking Method for RSS-Based DFL,“ IEEE J. on Select. Areas in Comm., vol. 33, no. 11, pp. 2391–2403, Nov. 2015.
  * [7] Z. Krupka, “The effect of the human body on radiation properties of small-sized communication systems,“ IEEE Trans. on Ant. Prop., vol. 16, no. 2, pp. 154–163, Mar. 1968.
  * [8] A. Abdelgawwad, et al., “A Trajectory-Driven 3D Channel Model for Human Activity Recognition,“ IEEE Access, vol. 9, pp. 103393–103406, 2021.
  * [9] G. Koutitas, “Multiple human effects in body area networks,“ IEEE Ant. Wir. Prop. Letters, vol. 9, pp. 938–941, 2010.
  * [10] E. Plouhinec, et al., “UTD Human Body Models Comparison based on dual Motion Capture and Radio Measurements,“ IEEE-APS Topical Conf. on Ant. and Prop. in Wir. Comm. (APWC’19), Granada, pp. 192–197, 2019.
  * [11] V. Rampa, et al., “EM models for passive body occupancy inference,“ IEEE Ant. Wir. Prop. Letters, vol. 17, no. 16, pp. 2517–2520, 2017.
  * [12] V. Rampa, et al., “Electromagnetic Models for Passive Detection and Localization of Multiple Bodies,“ IEEE Trans. on Ant. Prop., vol. 70, no. 2, pp. 1462–1745, 2022.
  * [13] B. R. Hamilton, et al., “Propagation modeling for radio frequency tomography in wireless networks,“ IEEE J. Sel. Topics Sig. Process., vol. 8, no. 1, pp. 55–65, Feb. 2014.
  * [14] A. Eleryan, et al., “Synthetic generation of radio maps for device-free passive localization,“ IEEE Global Telecomm. Conf., pp. 1–5, 2011.
  * [15] Z. Zhao, et al., “Generative Models for Inverse Imaging Problems: From mathematical foundations to physics-driven applications,“ IEEE Signal Processing Magazine, vol. 40, no. 1, pp. 148–163, Jan. 2023.
  * [16] R. He, et al., “Guest Editorial Artificial Intelligence in Radio Propagation for Communications,“ IEEE Trans. on Ant. Prop., vol. 70, no. 6, Jun. 2022.
  * [17] S. Bakirtzis, et al. “EM DeepRay: An expedient, generalizable and realistic data-driven indoor propagation model,“ IEEE Trans. on Ant. Prop., vol. 70, no. 6, pp. 4140–4154, Jun. 2022.
  * [18] S. Savazzi et al., “A physics-informed generative model for passive radio-frequency sensing,“ IEEE-APS Topical Conf. on Ant. and Prop. in Wir. Comm. (APWC’23), Venice, pp. 1–5, 2023.
  * [19] P. Zhang, et al., “Predictive modeling of millimeter-wave vegetation scattering effect using hybrid physics-based and data-driven approach,“ IEEE Trans. on Ant. Prop., vol. 70, no. 6, pp. 4056–4068, Jun. 2022.
  * [20] S. Bond-Taylor, et al., “Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and autoregressive models,“ IEEE Trans. Patt. Anal. Mach. Intell., vol. 44, no. 11, pp. 7327–7347, Nov. 2021.
  * [21] V. Rampa, et al., “An EM body model for Device-Free Localization with multiple antenna receivers: A first study,“ IEEE-APS Topical Conf. on Ant. and Prop. in Wir. Comm. (APWC’23), Venice, pp. 1–5, 2023.
  * [22] Paolo Di Barba, “Future Trends in Optimal Design in Electromagnetics,“ IEEE Trans. on Magnetics, vol.58, no.9, pp. 1–4, 2022.
  * [23] M Baldan, et al., “Physics-Informed Neural Networks for Inverse Electromagnetic Problems,“ IEEE Trans. on Magnetics, vol. 59, no. 5, pp. 1–5, 2023.
  * [24] D. P. Kingma, et al., “Auto-encoding variational Bayes,“ Proc. Intl. Conf. Learn. Represent (ICLR), 2013. [Online] Available: https://arxiv.org/abs/1312.6114
  * [25] Code and Data: diffraction model generation pre-trained network and sample codes, [Online], accessed: Apr. 28, 2023. Available: https://github.com/labRadioVision/generative_diffraction_model
  * [26] J. Benesty, et al., “A Brief Overview of Conventional Beamforming,“ in Array Beamforming with Linear Difference Equations, Springer Topics in Signal Processing, vol 20, pp. 13–21, 2021.
  * [27] F. Fieramosca, et al., “Modelling of the Floor Effects in Device-Free Radio Localization Applications,“ 17th EU Conf. on Ant. and Prop. (EuCap’23), Mar. 26-31, Florence, pp. 1–5, 2023.
  * [28] F. Fieramosca, et al., “On the impact of the antenna radiation patterns in passive radio sensing,“ IEEE Ant. Wir. Prop. Letters, to appear, 2024.
  * [29] Body Tissue Dielectric Parameters: https://www.fcc.gov/general/body-tissue-dielectric-parameters, FCC, US. Accessed: 11 Oct. 2023.


# Finite groups with large Chermak-Delgado lattices

Georgiana Fasolă and Marius Tărnăuceanu

(September 1, 2022)

###### Abstract

Given a finite group $G$, we denote by $L(G)$ the subgroup lattice of $G$ and
by ${\cal CD}(G)$ the Chermak-Delgado lattice of $G$. In this note, we
determine the finite groups $G$ such that $|{\cal CD}(G)|=|L(G)|-k$, $k=1,2$.

MSC2020 : Primary 20D30; Secondary 20D60, 20D99.

Key words : Chermak-Delgado measure, Chermak-Delgado lattice, subgroup
lattice, generalized quaternion $2$-group.

##  1 Introduction

Let $G$ be a finite group and $L(G)$ be the subgroup lattice of $G$. The
Chermak-Delgado measure of a subgroup $H$ of $G$ is defined by

$m_{G}(H)=|H||C_{G}(H)|.$

Let

$m^{*}(G)={\rm max}\\{m_{G}(H)\mid H\leq G\\}\mbox{ and }{\cal CD}(G)=\\{H\leq
G\mid m_{G}(H)=m^{*}(G)\\}.$

Then the set ${\cal CD}(G)$ forms a modular, self-dual sublattice of $L(G)$,
which is called the Chermak-Delgado lattice of $G$. It was first introduced by
Chermak and Delgado [4], and revisited by Isaacs [5]. In the last years there
has been a growing interest in understanding this lattice (see e.g. [1, 2, 3,
6, 7, 8, 11, 12, 13, 14]). We recall several important properties of the
Chermak-Delgado measure:

  * $\bullet$

if $H\leq G$ then $m_{G}(H)\leq m_{G}(C_{G}(H))$, and if the measures are
equal then $C_{G}(C_{G}(H))=H$;

  * $\bullet$

if $H\in{\cal CD}(G)$ then $C_{G}(H)\in{\cal CD}(G)$ and $C_{G}(C_{G}(H))=H$;

  * $\bullet$

the maximal member $M$ of ${\cal CD}(G)$ is characteristic and satisfies
${\cal CD}(M)={\cal CD}(G)$;

  * $\bullet$

the minimal member $M(G)$ of ${\cal CD}(G)$ (called the Chermak-Delgado
subgroup of $G$) is characteristic, abelian and contains $Z(G)$.

In [12], the Chermak-Delgado measure of $G$ has been seen as a function

$m_{G}:L(G)\longrightarrow\mathbb{N}^{*},\,H\mapsto m_{G}(H),\,\forall\,H\in
L(G).$

If $G$ is non-trivial, then $m_{G}$ has at least two distinct values, or
equivalently ${\cal CD}(G)\neq L(G)$ (see Corollary 3 of [11]). This leads to
the following natural question:

How large can the lattice ${\cal CD}(G)$ be?

Note that the dual problem of finding finite groups with small Chermak-Delgado
lattices has been studied in [6, 7].

Our main result is stated as follows.

######  Theorem 1.1.

Let $G$ be a finite group. Then

  * a)

$|{\cal CD}(G)|=|L(G)|-1$ if and only if $G\cong\mathbb{Z}_{p}$ or $G\cong
Q_{8}$;

  * b)

$|{\cal CD}(G)|=|L(G)|-2$ if and only if $G\cong\mathbb{Z}_{p^{2}}$.

For the proof of the above theorem, we need the following well-known result
(see e.g. (4.4) of [10], II).

######  Theorem A.

A finite $p$-group has a unique subgroup of order $p$ if and only if it is
either cyclic or a generalized quaternion $2$-group.

We recall that a generalized quaternion $2$-group is a group of order $2^{n}$
for some positive integer $n\geq 3$, defined by

$Q_{2^{n}}=\langle a,b\mid
a^{2^{n-2}}=b^{2},a^{2^{n-1}}=1,b^{-1}ab=a^{-1}\rangle.$

We also need the following theorem taken from [12].

######  Theorem B.

Let $G$ be a finite group. For each prime $p$ dividing the order of $G$ and
$P\in{\rm Syl}_{p}(G)$, let $|Z(P)|=p^{n_{p}}$. Then

$|{\rm Im}(m_{G})|\geq 1+\sum_{p}n_{p}\,.$ (1)

Finally, we indicate a natural open problem concerning the above study.

Open problem. Determine the finite groups $G$ such that $|{\cal
CD}(G)|=|L(G)|-k$, where $k\geq 3$.

Most of our notation is standard and will usually not be repeated here.
Elementary notions and results on groups can be found in [5]. For subgroup
lattice concepts we refer the reader to [9].

##  2 Proof of the main result

First of all, we solve the problem for generalized quaternion $2$-groups.

######  Lemma 2.1.

Under the above notation, we have:

  * a)

$|{\cal CD}(Q_{2^{n}})|=|L(Q_{2^{n}})|-1$ if and only if $n=3$, i.e. $G\cong
Q_{8}$;

  * b)

$|{\cal CD}(Q_{2^{n}})|\neq|L(Q_{2^{n}})|-2$ for all $n\geq 3$.

###### Proof.

We easily obtain

$m^{*}(Q_{2^{n}})=2^{2n-2},\,\forall\,n\geq 3$

and

${\cal CD}(Q_{2^{n}})=\left\\{\begin{array}[]{lll}\\{Q_{8},\langle
a\rangle,\langle b\rangle,\langle ab\rangle,\langle a^{2}\rangle\\},&n=3\\\
\\{\langle a\rangle\\},&n\geq 4.\end{array}\right.$

These lead immediately to the desired conclusions. ∎

We are now able to prove our main result.

Proof of Theorem 1.1.

a) Since $|{\cal CD}(G)|=|L(G)|-1$, we have ${\cal
CD}(G)=L(G)\setminus\\{H_{0}\\}$, where $H_{0}\leq G$. We infer that $|{\rm
Im}(m_{G})|=2$ and so $G$ is a $p$-group with $|Z(G)|=p$ by Theorem B. Then
$m_{G}(1)<m_{G}(Z(G))$, implying that

$H_{0}=1\mbox{ and }m^{*}(G)=m_{G}(Z(G))=p^{n+1},$

where $|G|=p^{n}$.

Assume that there exists $H\leq G$ with $|H|=p$ and $H\neq Z(G)$. Then
$H\notin{\cal CD}(G)$, which shows that $H=1$, a contradiction. Thus $G$ has a
unique subgroup of order $p$ and Theorem A leads to

$G\cong\mathbb{Z}_{p^{n}}\mbox{ or }G\cong Q_{2^{n}}\mbox{ for some }n\geq 3.$
(2)

In the first case we easily get $n=1$, i.e. $G\cong\mathbb{Z}_{p}$, while in
the second one we get $G\cong Q_{8}$ by item a) in Lemma 2.1.

b) The condition $|{\cal CD}(G)|=|L(G)|-2$ means ${\cal
CD}(G)=L(G)\setminus\\{H_{1},H_{2}\\}$, where $H_{1},H_{2}\leq G$. Then $|{\rm
Im}(m_{G})|\leq 3$. Recall that we cannot have $|{\rm Im}(m_{G})|=1$.

If $|{\rm Im}(m_{G})|=2$, then

$m_{G}(H_{1})=m_{G}(H_{2})\neq m^{*}(G)$

and again $G$ is a $p$-group with $|Z(G)|=p$. It is clear that one of the two
subgroups $H_{1}$ and $H_{2}$ must be trivial, say $H_{1}=1$. Then $G$ has at
most two subgroups of order $p$, namely $Z(G)$ and possibly $H_{2}$. This
implies that it has exactly one subgroup of order $p$ because the number of
subgroups of order $p$ in a finite $p$-group is congruent to $1$ $({\rm
mod}\,p)$. Consequently, one obtains again (2). For $G\cong\mathbb{Z}_{p^{n}}$
we easily get $n=2$, i.e. $G\cong\mathbb{Z}_{p^{2}}$, while for $G\cong
Q_{2^{n}}$ we get no solution by item b) in Lemma 2.1.

If $|{\rm Im}(m_{G})|=3$, then $m_{G}(H_{1})$, $m_{G}(H_{2})$ and $m^{*}(G)$
are distinct. Also, the inequality (1) becomes

$3\geq 1+\sum_{p}n_{p}\,.$

Since $n_{p}\geq 1$ for all $p$, we have the following two possibilities:

Case 1. $|G|=p^{n}$ and $|Z(G)|\in\\{p,p^{2}\\}$

Obviously, if $G$ is abelian, we get $G\cong\mathbb{Z}_{p^{2}}$. Assume that
$G$ is not abelian. Since $m_{G}(1)<m_{G}(Z(G))=m_{G}(G)$, we infer that one
of the two subgroups $H_{1}$ and $H_{2}$ is trivial, and that

$m^{*}(G)=m_{G}(Z(G))=m_{G}(G).$

If $|Z(G)|=p$, then $G$ has a unique subgroup of order $p$ and so it is a
generalized quaternion $2$-group, contradicting item b) in Lemma 2.1. The same
thing can be also said when $|Z(G)|=p^{2}$ because all subgroups of order $p$
of $G$ are outside of ${\cal CD}(G)$.

Case 2. $|G|=p^{n}q^{m}$ and the Sylow $p$-subgroups and $q$-subgroups of $G$
have centers of orders $p$ and $q$, respectively

Let $P$ be a Sylow $p$-subgroup and $Q$ be a Sylow $q$-subgroup of $G$. Since
$P\subseteq C_{G}(Z(P))$, we have

$m_{G}(Z(P))=p\,|C_{G}(Z(P))|=p^{n+1}q^{x}\mbox{ for some }0\leq x\leq m,$

and similarly

$m_{G}(Z(Q))=p^{y}q^{m+1}\mbox{ for some }0\leq y\leq n.$

Also, we have

$m_{G}(1)=p^{n}q^{m}\mbox{ and }m_{G}(G)=p^{n}q^{m}|Z(G)|.$

We observe that the measures $m_{G}(Z(P))$, $m_{G}(Z(Q))$ and $m_{G}(1)$ are
distinct, and consequently they are all possible measures of the subgroups of
$G$. We distinguish the following two subcases:

Subcase 2.1. $Z(G)=1$

Then $m^{*}(G)=m_{G}(1)=m_{G}(G)$. Indeed, if $m^{*}(G)=m_{G}(Z(P))$, then
$1$, $G$ and $Z(Q)$ will be outside of ${\cal CD}(G)$, a contradiction. In the
same way, we cannot have $m^{*}(G)=m_{G}(Z(Q))$. Since $m_{G}(P)$ is divisibly
by $p^{n+1}$ and $m_{G}(Q)$ is divisibly by $q^{m+1}$, we infer that
$m_{G}(P)=m_{G}(Z(P))$ and $m_{G}(Q)=m_{G}(Z(Q))$. Thus

$P,Z(P),Q,Z(Q)\notin{\cal CD}(G)$

and our hypothesis implies that $P=Z(P)$ and $Q=Z(Q)$, i.e. $G$ is a non-
abelian group of order $pq$. Assume that $p<q$. Then ${\cal CD}(G)$ consists
of the unique subgroup of order $q$ of $G$ and therefore we obtain $|L(G)|=3$,
a contradiction.

Subcase 2.2. $Z(G)\neq 1$

Then $m_{G}(1)<m_{G}(G)$, which shows that $m_{G}(G)$ equals either
$m_{G}(Z(P))$ or $m_{G}(Z(Q))$. Assume that $m_{G}(G)=m_{G}(Z(P))$. Then $x=m$
and $|Z(G)|=p$, implying that

$Z(G)=Z(P).$ (3)

Note that we cannot have $m^{*}(G)=m_{G}(Z(Q))$ because in this case $1$,
$Z(G)$ and $G$ will be outside of ${\cal CD}(G)$, a contradiction.
Consequently, we have

$m^{*}(G)=m_{G}(Z(P))=m_{G}(P)=m_{G}(Z(G))=m_{G}(G).$

It follows that $1$, $Z(Q)$ and $Q$ are not contained in ${\cal CD}(G)$, which
leads to $Q=Z(Q)$. In other words, we have ${\cal
CD}(G)=L(G)\setminus\\{1,Q\\}$. Thus ${\cal CD}(G)$ is the lattice interval

$[G/Z(G)]=\\{H\in L(G)\mid Z(G)\leq H\leq G\\}$

and Corollary 2 of [11] shows that $G$ is nilpotent. Then $G=P\times Q$ and so
$Z(G)=Z(P)\times Q$, contradicting (3).

This completes the proof.∎

## References

  * [1] L. An, J.P. Brennan, H. Qu and E. Wilcox, Chermak-Delgado lattice extension theorems, Comm. Algebra 43 (2015), 2201-2213.
  * [2] B. Brewster and E. Wilcox, Some groups with computable Chermak-Delgado lattices, Bull. Aus. Math. Soc. 86 (2012), 29-40.
  * [3] B. Brewster, P. Hauck and E. Wilcox, Groups whose Chermak-Delgado lattice is a chain, J. Group Theory 17 (2014), 253-279.
  * [4] A. Chermak and A. Delgado, A measuring argument for finite groups, Proc. AMS 107 (1989), 907-914.
  * [5] I.M. Isaacs, Finite group theory, Amer. Math. Soc., Providence, R.I., 2008.
  * [6] R. McCulloch, Chermak-Delgado simple groups, Comm. Algebra 45 (2017), 983-991.
  * [7] R. McCulloch and M. Tărnăuceanu, Two classes of finite groups whose Chermak-Delgado lattice is a chain of length zero, Comm. Algebra 46 (2018), 3092-3096.
  * [8] R. McCulloch and M. Tărnăuceanu, On the Chermak-Delgado lattice of a finite group, Comm. Algebra 48 (2020), 37-44.
  * [9] R. Schmidt, Subgroup lattices of groups, de Gruyter Expositions in Mathematics 14, de Gruyter, Berlin, 1994.
  * [10] M. Suzuki, Group theory, I, II, Springer Verlag, Berlin, 1982, 1986.
  * [11] M. Tărnăuceanu, A note on the Chermak-Delgado lattice of a finite group, Comm. Algebra 46 (2018), 201-204.
  * [12] M. Tărnăuceanu, Finite groups with a certain number of values of the Chermak-Delgado measure, J. Algebra Appl. 19 (2020), article ID 2050088.
  * [13] E. Wilcox, Exploring the Chermak-Delgado lattice, Math. Magazine 89 (2016), 38-44.
  * [14] A. Morresi Zuccari, V. Russo and C.M. Scoppola, The Chermak-Delgado measure in finite $p$-groups, J. Algebra 502 (2018), 262-276.

Georgiana Fasolă

Faculty of Mathematics

”Al.I. Cuza” University

Iaşi, Romania

e-mail<EMAIL_ADDRESS>
Marius Tărnăuceanu

Faculty of Mathematics

”Al.I. Cuza” University

Iaşi, Romania

e-mail<EMAIL_ADDRESS>