Metrics Overview sec:metrics

We have developed several metrics for detecting depegs for Curve’s StableSwap
pool. The metrics are summarized in Table table:metrics. With each metric we
describe the intuition for why this metric might be a useful indicator of
potential depegs. We look to identify changes in market behavior that precede
changes in token prices.

Table : Summary of Studied Metrics. Metric |  Description
---|---
Gini Coefficient and Shannon’s Entropy |  A measurement of the relative balances of a pool’s tokens.
Net Swap Flows |  The net amount swapped into or out of a pool for a particular token.
Net LP Flows |  The net amount deposited into or withdrawn from a pool for a particular token.
Price Volatility |  The rolling log returns for a token’s price.
PIN |  The probability of informed trading developed by Easley et al. [easley1996].
Markouts |  A short-term measurement of a trade’s profits.
Shark Trades |  A classification of traders/LPs as “sharks” based on their past performance.
|

##  Gini Coefficient and Shannon’s Entropy

We begin with two metrics meant to capture changes in the token balances of
each pool. Intuitively, abrupt changes in the relative token balances in the
pool could signal new information regarding one or more of the underlying
tokens. For example, informed traders might sell the depegging token to the
pool in large amounts. The fire sale of the depegging token might be abrupt,
as with UST, or gradual, as with USDN. Measuring changes in the relative
weights of each token in the pool might therefore be a useful indicator of
potential depegs. We consider two related metrics for measuring relative
balances of pools with $2$ or more tokens: the Gini Coefficient and Shannon’s
Entropy. Both metrics allow us to produce a scalar value based on pool
composition for pools with an arbitrary number of tokens. Ultimately, we find
that Shannon’s entropy produces a useful signal for detecting depegs.

###  The Gini Coefficient

is a measure of inequality, originally developed to quantify income or wealth
distribution within a population. The Gini coefficient ranges from 0
(representing perfect equality, where the observed Lorenz Curve matches the
Line of Perfect Equality) to 1 (representing perfect inequality). For a
discrete probability distribution $p(x)$ of $N$ values sorted in non-
decreasing order, the Gini coefficient $G$ can be expressed as:

$G=\frac{1}{N-1}\sum_{i=1}^{N}(2i-N-1)p(x_{i})$ (1)

where $x_{i}$ is the $i^{th}$ value in the sorted list, and the sum runs over
all values. The Gini coefficient has been widely adopted in various fields due
to its interpretative simplicity and its sensitivity to the changes at the
extremes of distributions. In the context of Curve’s StableSwap pools, the
Gini Coefficient is calculated on the token balances of the pool. A perfectly
balanced pool, where each token has an equal share of the pool’s total
balance, would have a Gini Coefficient of zero.

###  Shannon’s Entropy

is a fundamental concept in information theory, introduced by Claude E.
Shannon in his seminal 1948 paper A Mathematical Theory of Communication
[shannon2001mathematical]. It quantifies the uncertainty or randomness in a
dataset, essentially measuring the informational diversity. In the AMM
context: imagine a pool as a basket where each unit of each token is a ball
with that token’s color. Shannon entropy loosely measures how easy it is to
predict the color of a ball picked at random from that basket. It can
therefore be conceptualized as a variation of the Urn problem. For a discrete
random variable $X$ with possible values $\\{x_{1},x_{2},\ldots,x_{n}\\}$ and
probability mass function $p(x)$, the Shannon entropy $H(X)$ is defined as:

$H(X)=-\sum_{i=1}^{n}p(x_{i})\log_{2}p(x_{i})$ (2)

In this equation, $p(x_{i})$ is the probability of event $x_{i}$ occurring.
The logarithm base is typically 2 if the measure of entropy is in bits, but
any base can be used. The entropy is maximized when all outcomes are equally
likely, indicating maximum uncertainty or randomness. We may produce a signal
with either the Gini Coefficient or Shannon’s Entropy by continuously
measuring the diffs (or log diffs) on a periodic (per block, or per minute)
basis. Ideally, our changepoint detection algorithm would signal significant
changes in the variance of these returns, as shown in Fig. . We take the
logarithmic difference of our metrics in an attempt to create a stationary
timeseries, and mitigate the noise in their variances.

[width=]figs/shannon_ust_3crv.png

Figure : Shannon’s entropy calculated on a minutely basis for the UST-3Crv
pool. The log diffs are calculated as $\log{(1+\frac{x_{t+1}}{x_{t}})}$.
Notice that as entropy decreases, smaller absolute changes in entropy lead to
much larger percentage changes, which is why we observe a large spike on May
15th.

We will refer to Shannon’s entropy as entropy for the remainder of this paper.

##  Net Swap Flows

Whenever a swap is conducted in an AMM it moves the relative prices of the
swapped tokens. This creates an arbitrage opportunity for traders to buy low
and sell high relative to other markets. This means that over sufficiently
long windows of time, denoted as $\Delta t$, trades will be executed to
rebalance AMM prices immediately following a swap (i.e. backrunning). It
follows that the net swap flow for the pool over $\Delta t$ is zero in
expectation, if token prices stay approximately constant. For example, if
USDC/USDT prices are approximately constant around $1$, we would expect the
net amount of USDC or USDT being swapped into or out of Curve’s 3pool to be
approximately zero for both tokens. This maintains the relative prices of USDC
and USDT within the 3pool approximately constant at $1$111This assumes
balanced withdrawals by LPs. If LPs withdraw significant amounts of just one
token, arbitrageurs will be incentivized to sell that token to the pool, and
we will observe a net positive swap flow.. However, if informed traders begin
to price a token below its peg, it is very likely that we will observe non-
zero net swap flows. Since takers are generally more informed than makers,
large non-zero net swap flows might be highly predictive of a change in a
token’s price. Ideally, this metric would allow us to detect a potential depeg
before this information dissipates, and the market prices in the new price of
the token. That is, the increase or decrease in net swap flows would ideally
precede the change in the market price of a token. This is might be especially
true if we filter the net swap flows to only track informed traders. We
describe an attempt in doing so with our “sharks” metric. Formally, we define
the net swap flow over rolling window $\Delta t$ for token $i$ on pool $P$ as:
netSwapFlow(P, i Δt) = Token $i$ buy volume on pool $P$ over $\Delta t$
\- Token $i$ sell volume on pool $P$ over $\Delta t$

[width=]figs/3pool_swap_flows.png

Figure : Net swap flows for different rolling window sizes in Curve’s 3pool
during the collapse of Silicon Valley Bank. The mean in each plot is zero.

Figure  shows a snapshot of our net swap flow metric for Curve’s 3pool during
the collapse of Silicon Valley Bank. Notice that swap flows usually fluctuate
around 0 as expected, but with non-constant variance.

##  Net Deposit and Withdrawal Flows

Similarly to net swap flows, we measure the rolling window of net deposits and
withdrawals into the pool. We denote this as the net LP flow, defined very
similarly for each token as in Eq. 1222Note that the deposited token for a
MetaPool is the LP token of its base pool. That is, the deposited tokens for
the UST-3Crv MetaPool are UST and 3Crv (the LP token for the 3pool), instead
of UST, USDC, USDT, and DAI.. We posit that many of the large liquidity
providers on Curve are actively and systematically providing liquidity to
particular pools. A sudden change in the net deposit and withdrawal flows for
a pool might be indicative that these LPs are anticipating an increase in risk
- such as a depeg. Tracking this metric would allow all LPs to “copy trade”
the first-movers in a particular pool.

##  Price Volatility

Intuitively the price for each token would be a reasonable indicator of a
potential depeg. However, looking directly at the price of a token could not
produce a leading indicator in itself. Instead, we consider the volatility of
returns for a particular token, and test whether abrupt changes in the token’s
volatility precede its eventual decline in price. For some window $\Delta t$,
and prices $p_{i,t}$, the rolling price volatility for token $i$ is: R(i, t_0,
t_1) = {R_i, t ∈[t_0, t_1]} = {ln(pi, t+1pi, t)}
volatility = σ_R
Δt = t_1 - t_0 Where $\sigma_{R}$ is the standard deviation of
$R(i,t_{0},t_{1})$.

##  PIN

The probability of informed trading (PIN) is a metric proposed by Easley et
al. in 1996 [easley1996] that measures information asymmetry in a market. The
idea is that in a given window of time, traditionally a trading day, an
imbalance in buy and sell orders indicates an increased probability of an
information event. For example, if a trading day exhibits significantly more
buy orders than sell orders compared to previous trading days, it is likely
that good news for the underlying asset arrived before or during the trading
day, prompting informed selling [foucault2013market]. In theory, a sudden
increase in PIN suggests an increase in informed buying or selling, which
could then be a useful predictor of a potential depeg. Let $\alpha$ denote the
probability that an information event has occured (such as a depeg), and let
$\theta$ define the probability that this news was positive. For our purposes,
it is likely that $\theta$ is very close to zero (i.e. informed selling or
shorting, as opposed to informed buying). The authors posit that informed
trades arrive as a Poisson process with intensity $\epsilon_{i}$, whereas
uninformed buys arrive with intensity $\epsilon_{b}$ and uninformed sells
arrive with intensity $\epsilon_{s}$. As these three Poisson point processes
are independent, the likelihood of any trade being informed can be shown to
be:

$\text{PIN}=\frac{\alpha\epsilon_{i}}{\epsilon_{b}+\epsilon_{s}+\alpha\epsilon_{i}}.$
(3)

These parameters are often estimated with maximum likelihood estimators. The
following is the likelihood function for $B$ buy and $S$ sell orders in one
trading day [easley2010factoring], assuming orders are sampled i.i.d. from
their respective Poisson point processes: L((B, S)—ϵ_i, ϵ_b, ϵ_s, α, θ) = α(1
- θ)e^-(ϵ_i + ϵ_b + ϵ_s)(ϵi\+ ϵb)BϵsSB!S!
\+ αθe^-(ϵ_i + ϵ_b + ϵ_s (ϵi\+ ϵs)SϵbBB!S! + (1 - α)e^(ϵ_b + ϵ_s)ϵbBϵsSB!S!.
This likelihood function can be generalized for $(B_{t},S_{t})|_{t=1}^{T}$
orders in a period $T$ using a log-likelihood function, which is what we
implement for our modeling [easley2010factoring]. Once we have estimated the
relevant parameters $(\epsilon_{i},\epsilon_{b},\epsilon_{s},\alpha,\theta)$
we may calculate the PIN for the relevant window $T$, giving us our rolling
PIN metric. Notice that rolling PIN is a computationally expensive metric to
calculate relative to other metrics displayed here, as it requires a several
steps of numerical approximation of the parameters until convergence, which
must be performed at every time step.

[width=]figs/pin_ust.png

Figure : The probability of informed trading fit on weekly rolling window of
daily aggregate data for the UST-3Crv pool. Unexpectedly, PIN peaks in March
of 2022, and sees only a marginal increase during the depeg in May.

As shown in Fig. , we find that PIN does not behave as expected in the depegs
we have examined, and is not a useful metric for our purposes.

##  Short-Term Markouts

Markouts are a common way to evaluate trade performance in the context of
high-frequency trading, and have lately been used to evaluate LP performance
in AMMs [holloway2022, crocswap2022]. Generally, markouts compare the
execution price of a trade to its price at some “mark” in the future. Markouts
are a powerful way of measuring the “informedness” of a trade, meaning that
traders with consistently positive markouts for some window $W$ tend to be
good at predicting price moves within that time window, such as 5 minutes.
Formally, we define the markout $m$ for trade $\phi=(x_{i},x_{j})$ as

$m_{\phi}=x_{i}\cdot\left(P_{t+h}-\bar{p}(\phi)\right),$ (4)

where $x_{i}$ quantity of token $i$ is being bought, $P_{t+h}$ is the relative
price of tokens $i,j$ at time $t+h$, $h$ being the markout window, and
$\bar{p}(\phi)=\frac{x_{i}}{x_{j}}$ is the trade’s execution price (accounting
for fees)333We have defined markouts from the perspective of takers, we may
similarly define them for LPs by multiplying $m_{\phi}$ by negative one (i.e.
the LP is selling token $i$).. Given that Curve pools might have multiple
tokens, we normalize the units of our markout by taking the numeraire prices
of each token.

$m_{\phi}=x_{i}\cdot p_{i,t+h}-x_{j}\cdot p_{j,t},$ (5)

where $p_{i,t}$ is the numeraire price (e.g. USD price) of token $i$ at time
$t$. By considering the numeraire-price of each token we it becomes possible
for both the taker and the LP to profit off the same trade if the numeraire-
price of both tokens increase. However, as we will discuss in the following
subsection, we would like our markout metric to clearly determine whether the
taker or the LP were more informed at the time of the trade, by assigning the
“less informed” side a negative markout. We modify the markout metric to allow
us to compare the numeraire-price of each token at the mark time.

$m_{\phi}=x_{i}\cdot p_{i,t+h}-x_{j}\cdot p_{j,t+h}.$ (6)

This allows us to parse through all swaps and deposits/withdrawals made on
Curve’s major StableSwap pools and rank the users who have historically taken
the most advantage of token depegs by selling the depegging token for a pegged
token. We expect markouts over time to have a mean around 0 for pegged assets
with respect to their numeraires. Large fluctuations in markouts, resulting in
changes to the variance of the markouts timeseries, may then be used to alert
LPs of potential depegs. As a depeg is about to occur, we expect informed
traders to place larger-than-usual bets against the depegging token, which
would lead to sudden spikes in markouts that are highly unlikely unless prices
begin to depeg, as shown in Fig. . While markouts can be leading indicators by
providing us with very large signals despite relatively small deviations in
price, they must necessarily follow price action, instead of precede it. Note
that markouts are very sensitive to the markout window being considered. Since
we are not interested in measuring the LP’s or taker’s performance per se with
markouts, we simply look to construct a useful signal for our changepoint
detection. It is then better to observe shorter markout windows, since we are
able to observe abrupt changes in markouts more quickly.

[width=]figs/3crv_lp_markouts.png

Figure : LP markouts for various windows.

##  Sharks

As described in the previous subsection, we use our markouts metric to
identify informed takers. We define these takers, called “sharks”, as the
takers with the top $1\%$ cumulative markouts. We measure cumulative markouts
using all trades executed on the pools considered in our research (refer to
Appendix LABEL:app:pools), from January 1st 2022 to May 1st 2023. By
specifically looking at previous markout performance, we identify which
accounts successfully traded previous depegs in size. Using this
classification of sharks, we may then measure the net swap flows for each
token based only on shark volume. In theory, this could significantly reduce
the amount of noise in our net swap flow metric, creating a more predictive
signal. For example, a high net shark volume may indicate sharks have
information that the pool does not. However, there are a number of
complications in measuring shark performance. As previously mentioned,
markouts are very sensitive to the markout window; a shark under a 5 minute
markout might not be a shark under a 1 day markout and vice-versa.
Furthermore, indexing sharks by Ethereum addresses is not “Sybil resistant”:
we are not capturing the cumulative markout of a “trader” per se, just one of
their addresses. This means, for example, that if a shark trader begins to use
a new Ethereum address, we have no way of classifying their new address as a
shark address. Finally, markouts are not representative of any other trades or
hedges performed on other trading venues, meaning it is not representative of
a trader’s profit and loss (PnL). Still, we find that 5 minute markouts
provide a useful heuristic for the top performing addresses on Curve.

[width=]figs/sharflow_vs_swapflow.png

Figure : Net shark flow vs net swap flow for the 3pool during the SVB
collapse. Notice how these metrics are very closely related, despite us having
filtered out $75\%$ of takers in constructing the net shark flow metric. We
show these metrics perform similarly in the results section.

For the purposes of classifying sharks that have successfully traded depegs in
the past, we employ a 1-day markout (instead of the 5 minute markout) to
create our sharkflow metric.


11institutetext: Utrecht University, Princetonplein 5, 3584CC, Utrecht, the
Netherlands 22institutetext: Boğaziçi University, Bebek, 34342, Istanbul,
Turkey
22email<EMAIL_ADDRESS>
# Automatic Analysis of Human Body Representations in Western Art

Shu Zhao 11 0000-0001-8129-2583 Almıla Akdağ Salah 11 0000-0002-7204-5633
Albert Ali Salah 1122 0000-0001-6342-428X

###### Abstract

The way the human body is depicted in classical and modern paintings is
relevant for art historical analyses. Each artist has certain themes and
concerns, resulting in different poses being used more heavily than others. In
this paper, we propose a computer vision pipeline to analyse human pose and
representations in paintings, which can be used for specific artists or
periods. Specifically, we combine two pose estimation approaches (OpenPose and
DensePose, respectively) and introduce methods to deal with occlusion and
perspective issues. For normalisation, we map the detected poses and contours
to Leonardo da Vinci’s Vitruvian Man, the classical depiction of body
proportions. We propose a visualisation approach for illustrating the
articulation of joints in a set of paintings. Combined with a hierarchical
clustering of poses, our approach reveals common and uncommon poses used by
artists. Our approach improves over purely skeleton based analyses of human
body in paintings.

###### Keywords:

Human Pose Estimation, Hierarchical Clustering, Painting Analysis

##  1 Introduction

The human body is expressive of mood and emotions, as well as intentions, and
artists have used the expressive possibilities of the body pose to its fullest
extent. Body language can reflect embedded societal or gender differences
[18], or convey intense emotions, which cannot be discriminated by facial
expressions only [1]. Consequently, art historians analyse the poses of
subjects in paintings in depth.

The portrayal of emotions via poses of a human body was first documented by
the cultural and art historian Aby Warburg with his concept of Pathosformel
[9]. The term comes from the combination of “Pathos” (emotion) and “formel” (a
formula). Warburg traces the Pathosformel back to ancient Greek vase paintings
[15], where a narrative is illustrated in the interactions and compositional
relationships between characters. The same visual elements of postures and
gestures are used in recurrent narratives. Warburg’s Bilderatlas contains a
rich collection of artifacts through which he studied the adoption of poses by
various painters and influences between them. This is one of the reasons why
automatic pose estimation in paintings is a useful tool for art historians, as
composition transfer can be identified through the similarities between the
postures of individual characters [10]. Painters often incorporate stylistic
elements by copying human poses depicted by other artists.

The automatic detection and statistical analysis of body shapes and poses in
paintings can provide art historians with an overview of artists from a new
angle. The statistical analysis in artworks usually focus on genre, style, or
artist identification [25, 23], whereas pose analysis is relatively rare [9,
15, 14]111For example IconClass is a classification system for image content
and includes human body poses (https://iconclass.org/31A23).. It will
furthermore be useful in image retrieval for art datasets and archives, as
well as for tracking the re-use of visual elements [4] and help with recent
efforts in automatic captioning of paintings [6, 2, 22].

The aim of this paper is to use automatic pose estimation methods to create a
tool which will allow pose based analyses of paintings. For this purpose, we
start from off-the-shelf pose estimators, and then seek to resolve specific
issues pertaining to poses in paintings. More specifically, we combine two
pose estimators (to complement each other’s shortcomings), followed by an
artist-specific normalisation step that corrects occlusions and perspective
related distortions based on average body poses depicted by a specific artist.
We contribute an artistic pose dataset of Western art that we have semi-
automatically annotated for pose ground truth. By applying hierarchical
clustering on the detected and corrected poses, we perform a detailed analysis
of the joint angles used in paintings in our dataset, and show how the
analysis reveals the common and niche pose depictions used and re-used
throughout Western art.

This paper is structured as follows. In Section 2, we briefly summarise
related work on human pose estimation. Section 3 describes our algorithmic
pipeline and its various components. Section 4 describes the dataset we have
used and annotated in the study. We illustrate our methods via experimental
results in Section 5, and conclude in Section 6.

##  2 Related Work

Human pose estimation from images can be achieved with 2D or 3D models. There
are three different types of approaches, using kinematic models (used for 2D /
3D), planar models (used for 2D), and volumetric models (used for 3D). For the
kinematic and planar models, body joints are represented with keypoints, and
limbs are represented with lines joining those keypoints. While this is
suitable for pose analysis in paintings, we will need more than skeleton
representations for body depictions, as the same pose can be depicted with
different body representations.

We distinguish between top-down and bottom-up approaches for pose estimation.
In the top-down pipeline, a human body detector is used to obtain each
person’s bounding box and a single-person pose estimator is used to predict
the locations of keypoints within the bounding box. In the bottom-up pipeline,
body joint detectors are used to extract human body joint candidates which are
clustered into individual bodies. In general, the bottom-up methods outperform
the top-down methods [28]. OpenPose is one of the most commonly used bottom-up
models with a real-time performance to estimate the poses for multiple people
in one image [3], and we use it to detect keypoints in this work. This is a
rapidly growing area, and the keypoint extractor can be updated with more
promising approaches. For instance HRNet, while not as widely tested as
OpenPose, shows good performance in certain benchmarks [24].

Keypoint estimation is not sufficient for representing body shapes in images.
DensePose [7] is a widely used top-down method, which is built on the Faster
R-CNN architecture [19]. In addition to keypoints, it provides a segmentation
of body parts (i.e. head, torso, arms, hands, legs and feet) for multiple
people in the image. We use this approach in this work for detecting body
segments, but further enhance the results with keypoints detected by the more
accurate OpenPose.

Pose estimation approaches (as well as face detection), are used for automatic
analysis of human representations in paintings [21, 15, 5], but also, the
analysis of these key-points and landmarks are further used as a way to
statistically analyse artistic datasets [9] and to design higher level
analysis and synthesis methods [26, 14, 4]. Here we detail three related
studies further.

Madhu et al. focused on the analysis of figures on Greek vase paintings, which
are full of visual narratives, in which the protagonists are depicted by their
actions and interactions conveyed through their poses composed against a
certain scene [15]. To automatically detect these poses, a styled dataset is
generated from the COCO-Persons dataset and a style transfer approach [8] is
applied to convert these images to a style similar to Greek vase paintings.
Combining a person detector based on Faster R-CNN, and a pose estimator based
on HRNet, a model is created and trained on this styled dataset. This model is
fine-tuned on a classical archaeology dataset with $2.629$ person annotations
and $1.728$ pose annotations from over $1.000$ Greek vase paintings.

In another example by Yaniv et al., facial landmarks are estimated on the
portraits [26]. First, a custom artistic portrait dataset is generated with
$160$ paintings from 10 artists. A multi-task cascaded CNN [27] is used to
automatically detect the faces, which are then cropped and resized to images
of $256\times 256$ pixels. A landmark detection algorithm is applied to
extract $68$ facial landmarks using the Dlib-ml toolkit [11]. With the help of
a natural-faces dataset with $68$ landmark annotations per face [20], the
geometric differences between natural faces and artistic faces is documented.
Not surprisingly, artistic faces have a larger geometric variation due to
artistic exaggeration and deformation.

In the third example, a portion of Aby Warburg’s Bilderatlas collection is
manually annotated via crowdsourcing for pose key-points [9], and analysed
using statistical methods such as hierarchical and two stage clustering to
generate an overview of body pose-clusters. To determine the number of
clusters automatically, a two-sample Kolmogorov-Smirnov test is run on the
distributions of each joints’ angle. The results show that certain poses from
antiquity indeed resurface in Renaissance, however the context of these poses
(and hence their emotional content) is changed [9].

While these studies show the potential of computer vision based analyses in
the domain of paintings, annotated datasets are very rare for pose keypoints,
body segments or facial landmarks. However, the investment to generate these
datasets is worthwhile. One possibility is to apply style transfer to already-
annotated datasets of photographic images, and use such an augmented dataset
as a training set for pose and landmark estimation in paintings. However,
paintings may contain more stylised poses than naturally occurring poses in
photographs, and manual annotation of paintings can potentially result in
better model training. In the end, both approaches are costly, either in terms
of computing or manpower. In this paper, we explore to what extent automatic
pose estimation with widely used off-the-shelf tools can be used for body pose
and shape analysis in paintings.

##  3 Methodology

Our general analysis approach is illustrated in Figure 1. We use OpenPose and
DensePose in parallel to obtain a set of keypoints and segments for the
depicted bodies in the paintings. For pose analysis of a group of paintings
(such as from a single painter or a style), we use the keypoints from
OpenPose, prepare visualisations that depict distributions of joint angles,
and following [9], dendrograms to find pose groups. For body shape analysis,
we use a shape normalisation step, and generate the average contours from all
normalised segments to superpose them on original poses.

Figure 1:  The pipeline of pose analysis.

###  3.1 Average contours of body segments

In Western art historical literature, the archetype, or the canon is an
example that is considered as the highest standard that transcends the
aesthetics of a given era [12]. Such an example can be an artwork, but also
the depiction of a pose, or more importantly, the human body. The canonical
body, i.e. the perfect proportions of a the human body has been a topic of
discussion and elaboration since the ancient Greeks [17]. There have been many
descriptions of these proportions, as well as drawings. Among these the
Vitruvian Man by Leonardo da Vinci is one of the most well known [16].

In our approach, we use the Vitruvian Man as a standard for normalising
extracted body contours of artistic poses, as it has been influential in
defining the ideal body proportions in Western art. This allows us to compare
different poses with each other at the same scale. Using contours, we propose
an approach to mitigate the occlusion and perspective issues that are
generally present in automatic pose estimation, as well as generate an
overview for each artist’s preferred style in drawing human bodies: Do they
use proportions closer to norm, or do they prefer exaggeration of these by
elongating or thickening of body segments? Lastly, through the comparison of
average contours we illustrate the differences between the contours of natural
poses and artistic poses.

To carry out the normalisation process we follow several assumptions: 1) The
head size is different for men and women. The (vertical) size of the head is
equal to the vertical distance between the nose and the top of the chest. 2)
The body proportions are the same for men and women; 3) The body segments are
convex, and can be abstracted as rectangles; 4) The body segments are
symmetrical between left and right sides. A specific segment is also
symmetrical around its centroid.

We use the head size as a normalisation reference, based on which we scale the
length of limbs and area of segments. The baseline Vitruvian Man is referenced
by a fixed dimension, i.e., $\left(624\times 624\right)$ in pixels, with male
head size being fixed at 62 pixels, and the female head size by 58 pixels. The
median size, i.e. $23.2$ centimetres for men and $21.8$ centimetres for women,
is taken as the norm222https://en.wikipedia.org/wiki/Human_head. We transform
this proportion to 62 pixels for men, and 58 pixels for women. For an input
pose from any image, the scale factor is calculated by dividing the standard
head size to the actual head size in pixels. Other limbs are scaled
accordingly by the same scale factor.

The output of normalisation is a T-pose figure superimposed on the Vitruvian
Man with $10$ segments, i.e., head, torso, upper and lower arms, upper and
lower legs, respectively. We use the keypoints detected by OpenPose and match
these to the DensePose segments, as OpenPose has a higher accuracy for
keypoint detection (see tab:accuracy-keypoints). If a painting has multiple
interacting poses, bounding boxes of segments may overlap. In such cases we
will discard the data during matching. We also discard instances if the torso
is only partially detected since we use this information for rotating the
whole pose to a vertical position. We furthermore rotate each segment
separately to a vertical or a horizontal position, as required by a T-pose.
Lastly we dilate all the segments to their tightest-fitting rectangles.

###  3.2 Visualising joint distributions

To summarise artists style when it comes to pose geometry, we prepare and
visualise distributions for male and female poses by using angle of joints as
a reference point. This type of analysis gives on overview of the range of
poses for a group of paintings (examples shown in Section 5.3).

The procedure starts with the normalisation of keypoints following these three
steps: 1) Validate whether a set of indicative keypoints (we use nose, neck
and midhip), are detected. 2) Rotate the whole pose to a vertical position, so
that the spine is vertical. 3) Use $60$ pixels (size of the head) as a scaling
factor to normalise all the poses.

After this procedure is applied to all poses in the selected set, the
distributions of the keypoints are visualised by fitting Gaussian ellipsoids
on the keypoint locations. Each ellipsoid can be thought of a visualisation of
the covariance matrix of the spread of the points. Hence, each ellipsoid
stands for the standard distribution of one keypoint for a group of poses.
However, while traditionally one standard deviation is used for the contours,
we use here half a standard deviation for better visibility. The covariance
matrix $\sum$ of the vector for one keypoint $X=[X1,X2]$ can be calculated and
represented as $\sum=\begin{bmatrix}\sigma_{11}&\sigma_{12}\\\
\sigma_{21}&\sigma_{22}\end{bmatrix}$.

The coordinate $(x,y)$ of the ellipsoid for the keypoint with its
corresponding radius and scale around it is as follows:

$\displaystyle x$ $\displaystyle=\frac{\sum_{i=1}^{n}X1_{i}}{n}$
$\displaystyle radius_{x}$
$\displaystyle=\sqrt{1+\frac{\sigma_{12}}{\sqrt{\sigma_{11}\times\sigma_{22}}}}$
$\displaystyle scale_{x}=\sqrt{\sigma_{11}}\times 0.5$ $\displaystyle y$
$\displaystyle=\frac{\sum_{i=1}^{n}X2_{i}}{n}$ $\displaystyle radius_{y}$
$\displaystyle=\sqrt{1-\frac{\sigma_{12}}{\sqrt{\sigma_{11}\times\sigma_{22}}}}$
$\displaystyle scale_{y}=\sqrt{\sigma_{22}}\times 0.5$

###  3.3 Hierarchical clustering

The visualisation of joint distributions explains the range of joints depicted
in the poses, whereas by looking at joints in relation to each other it is
possible to understand the nature of poses as well. To do that, we carry out
hierarchical clustering for artistic poses [9]. This process can also
illustrate the common and niche poses in detail, which can to some extent
explain why a pose detector performs better or worse for some poses.

For hierarchical clustering, the keypoints are normalised in three steps: 1)
Validate whether all the $6$ torso keypoints are detected, i.e., neck, right
and left shoulders, midhip, right and left hips. 2) Rotate the whole pose to a
standing-up position, with the spine being vertical. 3) Calculate the inner
angles for all the triplets of joints. In total, there are $13$ such joint
triplets, i.e., $1$ triplet of (nose, neck, midhip), $6$ triplets of the right
body: (shoulder, neck, midhip), (elbow, shoulder, neck), (wrist, elbow,
shoulder), (hip, midhip, neck), (knee, hip, midhip), (ankle, knee, hip), and
$6$ symmetric triplets of the left body. The inner angles of each pose on the
2-dimensional plane are treated as a $13$-dimensional vector, representative
of each pose. These pose vectors are used for agglomerative hierarchical
clustering, in which each pose starts in its own cluster, which is merged with
other clusters if their pair-wise Euclidean distance is the smallest. The
results are depicted in a dendrogram (see Section 5.4), where similar poses
are connected with each other by shorter distances, whereas different pose
clusters are connected with each other by longer distance.

##  4 The Artistic Pose (AP) Dataset

To test the inference accuracy of OpenPose and DensePose we decided to
generate a dataset by selecting 10 artists from the Painter by Numbers dataset
which has $103,250$ paintings from Western art that range from the early 11th
century to the 2010s. This is similar in size to earlier studies in the
literature [26]. We chose artists with more nude paintings, as these were
expected to generate the least problems for pose estimation, but the earlier
painters have less nude paintings. We also tried to find a gender balance, as
well as a genre balance in choosing our artists. The resulting Artistic Pose
dataset (from now on called the AP Dataset333The dataset with manual
annotations, as well as all the code are made publicly available at
https://github.com/tintinrevient/joints-data.) covers a wide range both in
time and style: Michelangelo (Renaissance), El Greco (Mannerism), Artemisia
Gentileschi (Baroque), Pierre-Paul Prud’hon (Romantism), Pierre-Auguste Renoir
(Early Impressionism), Paul Gauguin (Post Impressionism), Felix Vallotton
(Magical Realism) and Amedeo Modigliani (Expressionism), Tamara de Lempicka
(Art Deco), and Paul Delvaux (Surrealism).

We furthermore make use of a natural-pose dataset to bring forth an
understanding of the geometric style of natural poses and to compare them to
the artistic poses. COCO Persons dataset is one of these datasets with a wide
variety of common activities that are manually annotated with respect to the
joints and body segments [13]. More importantly, OpenPose and DensePose are
trained with this dataset.

##  5 Experimental Results

###  5.1 Comparison of OpenPose and DensePose

We first apply OpenPose and DensePose on the AP dataset and evaluate their
performances. To measure accuracy of keypoint detection, we use PCK
(Percentage of Correct Keypoints). The keypoints are considered correct if the
distance between the inferred and the true keypoint is within a certain
threshold. PCK for each artist is shown in Table 1. For pose analysis, we only
use $15$ keypoints, namely the nose, neck, midhip, left and right shoulders,
elbows, and wrists, left and right hips, knees, and ankles, respectively.

On average for all artists, the inference accuracy of OpenPose is around
$80\%$, and $66\%$ for DensePose. The inference of joints by the paintings of
Tamara de Lempicka and Amedeo Modigliani have the least accuracy due to their
exaggeration of shapes and proportions. In contrast, Paul Delvaux’s paintings
have the highest accuracy by both OpenPose and DensePose, because most of the
poses are resting or standing with naturally hanging arms, which constitute
easy cases for prediction.

Table 1: The measurement of PCK. The columns list the artists, the total number of people depicted (Male and Female), the number of valid keypoints and segments, accuracy of DensePose (Acc.), PCK of OpenPose (PCK-O) and DensePose (PCK-D) Artist | Subjects | # Keypoints | # Segments | Acc. | PCK-O | PCK-D
---|---|---|---|---|---|---
Michelangelo | 15 (14M, 1F) | 214 | 163 | 79% | 87% | 63%
El Greco | 34 (32M, 2F) | 375 | 259 | 42% | 73% | 55%
Artemisia Gentileschi | 21 (6M, 15F) | 214 | 182 | 59% | 86% | 60%
Pierre-Paul Prud’hon | 15 (7M, 8F) | 216 | 196 | 65% | 93% | 82%
Pierre-Auguste Renoir | 19 (19F) | 237 | 200 | 49% | 80% | 63%
Paul Gauguin | 31 (4M, 27F) | 414 | 335 | 64% | 82% | 74%
Felix Vallotton | 20 (1M, 19F) | 243 | 206 | 66% | 86% | 67%
Amedeo Modigliani | 15 (15F) | 180 | 147 | 21% | 40% | 30%
Tamara de Lempicka | 18 (1M, 17F) | 227 | 188 | 54% | 69% | 52%
Paul Delvaux | 35 (4M, 31F) | 455 | 371 | 82% | 89% | 89%

Figure 2:  The joints inferred by OpenPose (first row) and DensePose (second
row) under scenarios such as multiple people, low contrast, and niche poses.
The last row shows the more difficult scenarios for both OpenPose and
DensePose.

In general, OpenPose outperforms DensePose, especially under scenarios such as
multiple people, low contrast, and niche poses, as shown in Figure 2. The
universally difficult scenarios are when (1) two people hug or interact
closely with each other, (2) the poses have twisted limbs, (3) there are niche
perspectives, (4) there are exaggerated shapes, (5) there are exaggerated body
proportions. These challenging scenarios are illustrated in the last row of
Figure 2, in the mentioned order.

Next, we test the accuracy of DensePose body segment detection, by the
percentage of correctly inferred segments over the total number of visible
segments (max. 14), as shown in the last column of Table 1. The factors that
impact detection are the number of people in the painting, the occlusions of
clothes, interacting people, niche poses and perspectives, artistic effects
including unusual colouring and brush usage for body segments, and unusual
shapes of body segments.

The paintings of Michelangelo and Paul Delvaux have the highest accuracy. In
their paintings, most of the poses are nude, hence without interference of
clothes and in nude skin colours. For Michelangelo, we have one pose per
painting to detect, and for Delvaux, the people are distanced from each other.
All the poses from Michelangelo are sitting, and most poses from Delvaux are
standing, which form the two most common pose groups, and all of their poses
are depicted from a frontal view. Lastly, all of their poses follow natural
body proportions. Figure 3 shows successful and failed pose detections
including these two artists.

Figure 3:  First row: The detection of body segments for Michelangelo (first
three) and Paul Delvaux (last two). Middle row: The inference of segments for
Modigliani (first two) and Renoir (last three). Last row, from left to right:
The inference of segments for El Greco, Prud’hon, Gauguin, Vallotton and
Lempicka.

As shown in Table 1, the paintings of Amedeo Modigliani and Pierre-Auguste
Renoir have the least accuracy, as most of their poses are very challenging.
Lying poses of Modigliani, along with his use of overly slender and elongated
body proportions are hard to detect. Similarly, El Greco’s elongated bodies
are not always easy to segment. If the bodies are painted by blobs of various
colours (e.g. Renoir), or if the colour contrast is low (e.g. Gaugin) the
inference suffers. The niche perspective poses are also a challenge for
inference (e.g. Lempicka). Examples are given in Figure 3.

The root causes of these failures may include the fact that the COCO dataset
used in the training of these models has a lower density of natural poses,
such as lying, sitting with back turned to observers, crowds of overlapping
people, and such. Instead, many training samples depict single persons, doing
sports or with only the upper torso captured. Furthermore, the training data
collects mostly common people with common body proportions and height, which
are neither too slender, nor too plump. Yet artistic depictions can go to
extremes.

###  5.2 Average Contours of Body Segments

We use Michelangelo and Paul Delvaux as examples to demonstrate the
normalisation process and the results (see Figure 4). When we compare these
two normalised poses, we see that the body drawn by Michelangelo is more
inflated and fills to the brim the under the contours of the Vitruvian Man,
whereas Paul Delvaux’s woman is slender. Compared with Paul Delvaux’s standing
pose, Michelangelo’s sitting pose exposes three major issues: (1) The right
thigh is occluded by the right arm, which leads to the partial segment with a
gap in the normalised pose. (2) The thighs are not fully stretched out due to
the sitting pose and the viewing perspective. Thus, the corresponding
normalised thighs are both shorter. (3) The lower left arm is retracted a
little behind the torso, which is further away from the observer. Thus, the
corresponding normalised lower left arm is shorter and thinner, compared to
the lower right arm.

Figure 4:  Michelangelo’s man (first row) and Paul Delvaux’s woman (middle
row). For both rows, left to right: Keypoints by OpenPose, segments by
DensePose, normalised DensePose on Vitruvian Man and on the original image.
Last row, left to right: 1) Average contour of the COCO men. 2) Average
contour of the COCO women. 3) Michelangelo pose imposed on the contour of the
COCO men. 4) The Delvaux pose imposed on the contour of the COCO women.

Figure 4 shows the average contours of the COCO men and women respectively,
calculated with the same approach, using 144 men and 150 women from the COCO
Person dataset. Superimposing the same pose from Michelangelo and Paul Delvaux
on the natural men and women contours shows that Michelangelo tends to
exaggerate the muscles of men, as every segment is inflated outside the brim.
On the contrary, Paul Delvaux tends to draw the women more slender than their
counterparts in the natural poses, as the torso and the limbs shrink a bit
width-wise within the borders (see Figure 4).

In summary, the mean contours can give us an intuitive view of artistic and
natural poses, focusing on height and width of body segments. For art
historical analysis, the segment contours of each artist can be visualised as
to how each artist tends to draw human bodies. The contours can be compared
with each other to further explore whether there exist significant differences
between the drawing styles for each artist, and with that of the COCO men and
women contours to analyse whether the artistic contour conforms to or deviates
from the natural contours.

###  5.3 Visualising joint distributions

For the AP dataset, after the first step of the validity check, we have $211$
poses. Out of these, 67 poses are male, and 144 female (Figure 5). Different
from the distributions of facial landmarks [26], the keypoints are prone to
high variance of articulation. For the upper limbs, the wrists occupy a larger
ellipsoid than the elbows, as do the elbows compared to the shoulders.
Similarly, for the lower limbs, the areas of the ellipsoids are in descending
order: ankle $>$ knee $>$ hip. In general, the outer joints swing over a
larger circle than the inner joints. We also observe that the joint
distributions are similar for both male and female poses, as they are both
with arms and legs hanging alongside the torso.

Figure 5:  The mean and $0.5$ standard deviation distribution for artistic
(AP) and natural poses (COCO). Joints used in the normalisation have smaller
variances. From left to right: Males in AP; Females in AP; Males in COCO;
Females in COCO.

As a reference, we also analyse the gestures of daily activities captured in
photographs. After filtering through the COCO people dataset and a validity
check, we use $653$ male and $222$ female poses for the mean and standard
deviation analysis. Compared with artistic poses, natural poses tend to be
more varied with wider movement of joints. The natural poses are formed by a
wide variety of daily activities and sports, whereas the artistic poses are
staged and hence restrained to certain poses (see Figure 5).

###  5.4 Hierarchical clustering

We use the poses from the paintings of Felix Vallotton as an example to
demonstrate the hierarchical clustering results. Among the $15$ full-body
depictions of Felix Vallotton figures, one can find variations such as lying,
sitting, and standing poses. The hierarchical clustering, as shown in Figure
6, cannot distinguish whether a person lies down, sits or stands. But the
clusters can differentiate between the stretched and compressed limbs. When
the number of clusters is set to $5$, cluster 2 and 3 contain the poses with
arms thrown up; cluster 4 contains the standing poses with arms relaxed
alongside the body; cluster 5 contains the twisted legs and arms.

Figure 6:  Dendrogram for all $15$ poses drawn by Felix Vallotton from $15$
paintings.

When the number of clusters is set to $10$ in order to generate the
hierarchical clustering for all the artists, the outcome shows that there are
no clear boundaries between artists. Out of these $10$ clusters, there are $2$
outstanding clusters with only a small fraction of poses. These are all
standing poses with the arms thrown upward. The pose of opening arms appears
for example in the religious paintings of El Greco, but similar poses are
depicted by Artemisia Gentileschi, Pierre-Paul Prud’hon and is later used by
Paul Delvaux as well (see Figure 7).

Figure 7:  First row: The standing poses with arms thrown upward into the air,
as an example niche pose from the AP dataset. Middle row: The niche natural
poses in the COCO dataset for only men. Last row: The niche natural poses in
the COCO dataset for only women.

For comparison, we carried out hierarchical clustering in the COCO dataset to
find niche poses for men and women. The rare poses for men are usually with
more stretched arms, i.e., swinging back and forth, or the ones with more
twisted legs. Moreover, the lying poses barely appear for men. The rare poses
for women are the ones with twisted or overly stretched legs, or those with
arms thrown upward (Figure 7).

In summary, the hierarchical clustering of the pose vectors can tell the
difference of poses in: (1) the orientation, (2) whether the limbs are
stretched or compressed, and (3) the completeness of the joints. Via this
visualisation method it is also possible to track the transformation and re-
use of similar poses by different artists over time. When we compare the
artistic and natural poses we see that the poses with arms stretched upward
are quite rare for natural poses as they appear only in women’s hugging or
lying poses, or in sports. It might be assumed that if trained with more niche
poses and niche perspectives from natural poses, OpenPose and DensePose might
also perform better at inference of artistic poses.

##  6 Conclusions

In this paper, we evaluated automated human body analysis for paintings in
Western artworks. Specifically, we tested two state of the art pose estimation
models, namely DensePose and OpenPose, on a curated and manually annotated
artistic dataset of 10 Western Artists from various genres and periods, and
reported their performances. OpenPose in general performed better than
DensePose, especially for paintings with more than one figure where figures
are interacting with each other. There is room for improvement in pose
segmentation, and fine-tuning on annotated painting datasets may help.

We used a simple, rule-based approach to improve the performance of DensePose
by combining it with the keypoints of OpenPose, and proposed a way of
visualising a painter’s preferred body shapes using a normalisation method
using Leonardo’s Vitruvian Man. This way, we generate average contours for all
poses drawn by a given artist, which can also be used to summarise differences
in body shape preferences of artists. For example, a body that looks fat with
respect to an average body can be classified as skinny with respect to the
artist’s specific style. The same holds true for other shape parameters. Our
proposed approach improves over purely skeleton-based analysis of human poses.

Finally, we have analysed the distribution of joints in natural photographs
and paintings, and shown how our approach can automatically find niche poses
preferred by artist. The niche poses we found in the AP dataset via the
agglomerative hierarchical clustering are also documented in Impett and
Susstrunk’s study of Aby Warburg’s Bilderatlas [9], which shows that our
automatic approach is useful. Our findings also point out to what kind of
poses are needed for the training of pose detectors for paintings.

## References

  * [1] Aviezer, H., Trope, Y., Todorov, A.: Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 338(6111), 1225–1229 (2012)
  * [2] Bai, Z., Nakashima, Y., Garcia, N.: Explain me the painting: Multi-topic knowledgeable art description generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5422–5432 (2021)
  * [3] Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intelligence 43(1), 172–186 (2019)
  * [4] Castellano, G., Lella, E., Vessio, G.: Visual link retrieval and knowledge discovery in painting datasets. Multimedia Tools and Applications 80(5), 6599–6616 (2021)
  * [5] Castellano, G., Vessio, G.: Deep learning approaches to pattern extraction and recognition in paintings and drawings: An overview. Neural Computing and Applications 33(19), 12263–12282 (2021)
  * [6] Cetinic, E.: Iconographic image captioning for artworks. In: International Conference on Pattern Recognition. pp. 502–516. Springer (2021)
  * [7] Güler, R.A., Neverova, N., Kokkinos, I.: DensePose: Dense human pose estimation in the wild. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [8] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1501–1510 (2017)
  * [9] Impett, L., Süsstrunk, S.: Pose and pathosformel in Aby Warburg’s Bilderatlas. In: European Conference on Computer Vision. pp. 888–902. Springer (2016)
  * [10] Jenicek, T., Chum, O.: Linking art through human poses. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1338–1345. IEEE (2019)
  * [11] King, D.E.: Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research 10, 1755–1758 (2009)
  * [12] Langfeld, G.: The canon in art history: concepts and approaches. Journal of Art Historiography (19), 152–180 (2018)
  * [13] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [14] Madhu, P., Marquart, T., Kosti, R., Bell, P., Maier, A., Christlein, V.: Understanding compositional structures in art historical images using pose and gaze priors. In: European Conference on Computer Vision. pp. 109–125. Springer (2020)
  * [15] Madhu, P., Villar-Corrales, A., Kosti, R., Bendschus, T., Reinhardt, C., Bell, P., Maier, A., Christlein, V.: Enhancing human pose estimation in ancient vase paintings via perceptually-grounded style transfer learning. arXiv preprint arXiv:2012.05616 (2020)
  * [16] Magazù, S., Coletta, N., Migliardo, F.: The Vitruvian Man of Leonardo da Vinci as a representation of an operational approach to knowledge. Foundations of Science 24(4), 751–773 (2019)
  * [17] Murtinho, V.: Leonardo’s Vitruvian Man drawing: A new interpretation looking at Leonardo’s geometric constructions. Nexus Network Journal 17(2), 507–524 (2015)
  * [18] Noroozi, F., Kaminska, D., Corneanu, C., Sapinski, T., Escalera, S., Anbarjafari, G.: Survey on emotional body gesture recognition. IEEE transactions on affective computing (2018)
  * [19] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497 (2015)
  * [20] Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: The first facial landmark localization challenge. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 397–403 (2013)
  * [21] Sarı, C., Salah, A.A., Akdag Salah, A.A.: Automatic detection and visualization of garment color in Western portrait paintings. Digital Scholarship in the Humanities 34(Supplement_1), i156–i171 (2019)
  * [22] Sheng, S., Moens, M.F.: Generating captions for images of ancient artworks. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 2478–2486 (2019)
  * [23] Silva, J.M., Pratas, D., Antunes, R., Matos, S., Pinho, A.J.: Automatic analysis of artistic paintings using information-based measures. Pattern Recognition 114, 107864 (2021)
  * [24] Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5693–5703 (2019)
  * [25] Wang, J.Z., Kandemir, B., Li, J.: Computerized analysis of paintings. In: The Routledge Companion to Digital Humanities and Art History, pp. 299–312. Routledge (2020)
  * [26] Yaniv, J., Newman, Y., Shamir, A.: The face of art: landmark detection and geometric style in portraits. ACM Transactions on graphics (TOG) 38(4), 1–15 (2019)
  * [27] Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10), 1499–1503 (2016)
  * [28] Zheng, C., Wu, W., Yang, T., Zhu, S., Chen, C., Liu, R., Shen, J., Kehtarnavaz, N., Shah, M.: Deep learning-based human pose estimation: A survey. arXiv preprint arXiv:2012.13392 (2020)


# Towards Generalized Speech Enhancement
with Generative Adversarial Networks

###### Abstract

The speech enhancement task usually consists of removing additive noise or
reverberation that partially mask spoken utterances, affecting their
intelligibility. However, little attention is drawn to other, perhaps more
aggressive signal distortions like clipping, chunk elimination, or frequency-
band removal. Such distortions can have a large impact not only on
intelligibility, but also on naturalness or even speaker identity, and require
of careful signal reconstruction. In this work, we give full consideration to
this generalized speech enhancement task, and show it can be tackled with a
time-domain generative adversarial network (GAN). In particular, we extend a
previous GAN-based speech enhancement system to deal with mixtures of four
types of aggressive distortions. Firstly, we propose the addition of an
adversarial acoustic regression loss that promotes a richer feature extraction
at the discriminator. Secondly, we also make use of a two-step adversarial
training schedule, acting as a warm up-and-fine-tune sequence. Both objective
and subjective evaluations show that these two additions bring improved speech
reconstructions that better match the original speaker identity and
naturalness.

Index Terms: speech enhancement, generative adversarial networks, acoustic
regression loss.

##  1 Introduction

Speech enhancement comprises the improvement of intelligibility and quality of
speech contaminated by noise, typically in an additive form [1]. It is thus
broadly applicable to scenarios that need to palliate masking artifacts over
the speech, like communication systems, hearing aids, or cochlear implants,
where enhancing the signal prior to amplification significantly reduces
discomfort and increases intelligibility [2]. Deep learning has been broadly
applied to this field, either in the form of regression of waveform samples or
clean spectral components, as well as prediction of masking elements over
spectral bins [3, 4, 5, 6, 7, 8, 9]. A recent trend in speech enhancement is
also the use of generative adversarial networks (GANs) [10], with its first
application being the speech enhancement GAN (SEGAN) [11]. Other GAN
variations have been used to avoid using aligned clean/noisy corpora [12] or
using other adversarial losses and/or domains [13, 14]. SEGAN has also been
applied to speech regeneration as a whispered-to-voiced alaryngeal speech
conversion [15], thus extending the previous denoising approach.

Athough GANs were designed as an unsupervised learning strategy, they are
proven to be more stable and effective when coupled with labels that are
either injected as input conditionings [16, 17] or that help classify some
traits about the data being generated [18, 19]. Also the successful speech
synthesis model parallel WaveNet [20] proved that a multi-task aggregation on
top of the speech generative model is beneficial to improve speaker identity,
prosodic, and content traits.

In this work, we first extend SEGAN towards a more generalized speech
enhancement such that we recover cleaner speech out of severely distorted
utterances. This generalization towards different speech distortions is
interesting and directly applicable to modern communication technologies,
where connections are interrupted (i.e. losing voice packets), and voice
processing pipelines can distort our signals through amplifiers, compression
of transmission encoders, etc. In this work we emulate these effects by
introducing four applied distortions in this work, with different severity
levels per distortion. Then, we propose the use of SEGAN to palliate them with
three different schemes. First, a plain adversarially trained SEGAN version
serves us as a baseline for our recovery experiments. Then, we propose a new
regression component on top of the discriminator that serves as an additional
self-supervised task, boosting the generated recovered speech quality. This
new task is specially effective when we use the second modification, a two-
stage adversarial training schedule as a warm up and fine-tunning sequence.
Objective and subjective results show the effectiveness of both proposed
mechanisms when they are jointly applied.

##  2 Speech enhancement GAN Review

GANs define a deep generative framework, involving two neural networks and an
adversarial training mechanism [10]. The two networks are the generator ($G$)
and the discriminator ($D$). Both $G$ and $D$ are trained with opposite
objectives, hence the adversarial nature of the framework. On the one hand,
$D$ has to increasingly get better at classifying real or fake features. On
the other hand, $G$ has to make up samples realistic enough to make $D$ fail
in its prediction. This process is formulated as a minimax game between $G$
and $D$, and we can use different loss functions at the output of $D$ to
express it. For instance a least squares loss makes it an LSGAN [21]. It is
often interesting to add a conditioning factor to be used for a specific task,
where we have some additional information $\tilde{\textbf{x}}$ injected into
$G$ and $D$ to perform the mapping and classification (see [22] and references
therein). The formulation of $G$ and $D$ losses for a conditional LSGAN with
these components is expressed as

$\begin{split}\underset{G}{\min}~{}V(G)&=\operatorname{\mathbb{E}}_{\textbf{z}\sim
p_{\textbf{z}}(\textbf{z}),\tilde{\textbf{x}}\sim
p_{\text{data}}(\tilde{\textbf{x}})}[(D(G(\textbf{z},\tilde{\textbf{x}}),\tilde{\textbf{x}})-c)^{2}],\\\
\underset{D}{\max}~{}V(D)&=\frac{1}{3}\operatorname{\mathbb{E}}_{\textbf{x},\tilde{\textbf{x}}\sim
p_{\text{data}}(\textbf{x},\tilde{\textbf{x}})}[(D(\textbf{x},\tilde{\textbf{x}})-b)^{2}]\\\
&+\frac{1}{3}\operatorname{\mathbb{E}}_{\textbf{z}\sim
p_{\textbf{z}}(\textbf{z}),\tilde{\textbf{x}}\sim
p_{\text{data}}(\tilde{\textbf{x}})}[(D(G(\textbf{z},\tilde{\textbf{x}}),\tilde{\textbf{x}})-a)^{2}]\\\
&+\frac{1}{3}\operatorname{\mathbb{E}}_{\textbf{x},\textbf{x}^{r}\sim
p_{\text{data}}(\textbf{x})}[(D(\textbf{x},\tilde{\textbf{x}}^{r})-a)^{2}],\end{split}$
(1)

where $a=-1$, $b=1$, and $c=0$ fulfill the condition to minimize Pearson
$\chi^{2}$ divergence [21]. Also, $\tilde{\textbf{x}}^{r}$ are conditioning
samples unaligned with x, thus creating a fake signal that captures that any
misalignment between x and $\tilde{\textbf{x}}$ is a mistake [15].

In the speech enhancement setting, we have an input noisy signal
$\tilde{\textbf{x}}$ that has to be cleaned to obtain the enhanced signal
$\hat{\textbf{x}}$. SEGAN decimates and expands the input signal feature-wise,
through a number of strided convolutional layers with multiparametric
rectified linear activations (PReLUs) [23]. Decimation is applied until we
obtain a compact representation of a few time samples c. This result is
concatenated with the generative noise component
$\textbf{z}\sim\mathcal{N}(0,I)$, which adds the generative stochastic
behavior to the generator predictions $\hat{\textbf{x}}$. Decimation
discourages learning a trivial identity function in the reconstruction
$\tilde{\textbf{x}}$, and also accelerates the convolution operations when we
go deeper in the decimation structure of the encoder (shorter sequence lengths
involve faster processing). It also reduces the memory footprint by using
shorter feature maps. The encoding process is then reversed in the decoding
stage by means of transposed convolutions, followed again by PReLUs until last
layer which is $\tanh$ to bound the output between $-$1 and 1, stabilizing the
adversarial process [16]. Additionally, skip connections linking the encoder
and the decoder of $G$ boosted stability and generated quality.The enhanced
output signal is thus defined at the output of the decoder as
$\hat{\textbf{x}}=G(\textbf{z},\tilde{\textbf{x}})$.

Successive SEGAN works have shown its applicability to new languages with
little adaptation data [24] and its capacity to reconstruct damaged speech in
the context of whispered-to-voiced alaryngeal speech conversion, also named
dewhispering [15]. For reconstruction, however, some details of the original
architecture were modified, which led to the whisper SEGAN (WSEGAN). The most
critical ones were the addition of learnable skip connections and the removal
of the $L_{1}$ regularization term at waveform level. This eliminated a strict
temporal alignment between x and $G(\textbf{z},\tilde{\textbf{x}})$.

##  3 Generalized Speech Enhancement

In this work, we depart from the classical denoising task and consider a more
general class of enhancement problems. More specifically, we consider the
problem of reconstructing speech that has been degraded by a number of
(different) signal manipulations, each of them being potentially highly
perceptually harmful. We propose to mix up such manipulations together with
several speaker identities at training time, with the objective of being able
to train generative algorithms to recover multiple (and diverse) speech
components simultaneously. As a first approximation towards this generalized
speech enhancement task, we consider the following signal manipulations:

  * •

Whispered speech: This is the distortion is similar to the one introduced in
the WSEGAN work [15]. However, it differs in the fact that instead of using
magnetic sensors, we synthesize whispering speech by encoding the clean speech
with a vocoder, removing the log-F0 (i.e. making all frames unvoiced), and
recovering the signal into a version that whispers, hence artificially
removing voicing. The vocoder used is Ahocoder with the default parameters
[25].

  * •

Bandwith reduction: We downsample the audio signals with different severity
factors, ranging from $\times 2$ to $\times 8$, thus reducing the bandwidth.
Then the generalized enhancement model must reconstruct entire frequency
bands, according to the clean signals seen in training.

  * •

Chunk removal: For the parts of the waveform that contain speech, a random
number of chunks are subtracted by inserting silences. Thus, we cancel the
signal at random temporal sections that contain speech. The length of a
silence is sampled from one of two distributions $\mathcal{N}(0.05,0.025)$,
and $\mathcal{N}(0.1,0.05)$ (numerical values in seconds).

  * •

Clipping: The waveform is clipped globally by different severity factors
relative to the maximum absolute peak of the whole utterance (e.g., 30%). The
regenerated signal thus has to re-condition the signal inside a proper, non-
distorted range of amplitudes.

##  4 Acoustic Mapping Discriminator

To better deal with the added difficulty that the previous signal
manipulations can introduce to the speech signal, we propose two modifications
to the existing SEGAN pipeline: the addition of two acoustic losses and the
introduction of a two-stage training schedule for applying such losses.

###  4.1 Acoustic Losses

In general, $D$ can be understood as a learnable loss function, where the
realistic features we want to generate are implicit in the back-propagation,
and $G$ gets better because of $D$’s gradient flows [10]. This builds a need
for $D$ to be a competitive feature extractor, so that the better the features
extracted by $D$, the better $G$ can capture reality. The use of auxiliary
classifying labels in $D$ is generally helpful (Sec. 1). Additionally, multi-
task setups can boost generative modeling performance, where multiple factors
of the signal are predicted at the output of the generative model to enforce
modeling better perceptual qualities of the signal [20]. This is applicable to
the SEGAN setup where the discriminator only learns features that lead to a
real or fake decision, but is not concerned about specific factors of the
signal that are important to remark the reconstructed speaker identity,
prosody, or contents. Note that we can also have some additional
regularization loss in the output of $G$ too, aggregated to the adversarial
loss coming from $D$. For instance the first version of SEGAN used an $L_{1}$
regularization term that made $G$ output a zero-centered signal [11], and the
posterior WSEGAN implementation substituted it by a less restrictive power
loss [20, 26], enforcing proper energy allocation in the frequency bands.

Following the aforementioned multi-task improvement criteria, we propose the
use of an additional acoustic loss that opens a new branch in the end of $D$.
This makes $D$ necessarily aware of additional acoustic components that can
inform $G$ about important mistakes when mismatching identities, intonations,
or contents in the reconstruction without necessarily injecting any additional
conditioning in the input of $G$ besides the distorted signal
$\tilde{\textbf{x}}$. This is advantageous in terms of avoiding restricting
the generalization of $G$, as we do not inject any code limited by a set of
possible classes to specify identities, distortion types, or any other kind of
information. This new $D$ output branch grows from a certain level in the
convolutional structure, where the time decimation factor is 256, thus each
time-step corresponds to 16 ms stride over the waveform (emulating a short-
time Fourier transform sliding window). At each output frame, it predicts a
concatenation of log-power spectral (LPS) bins, Mel-frequency cepstral
coefficients (MFCC), and prosodic features. We design it using this collection
of features as we hypothesize they properly convey different levels of
acoustic cues, ranging from power allocation expressing identity, to content
and intonation clues. This branch is just in addition to the binary output
neuron that tells real and fake, and is only active when this binary
activation is real with input x or $\tilde{ve{x}}$. The proposed acoustic loss
in $D$ and power regularization in $G$ redefines the loss function as

$\begin{split}\underset{G}{\min}~{}V(G)&=\frac{1}{2}\operatorname{\mathbb{E}}_{\textbf{z}\sim
p_{\textbf{z}}(\textbf{z}),\tilde{\textbf{x}}\sim
p_{\text{data}}(\tilde{\textbf{x}})}[(D(G(\textbf{z},\tilde{\textbf{x}}),\tilde{\textbf{x}})-c)^{2}]\\\
&+\frac{1}{2}\operatorname{\mathbb{E}}_{\textbf{z}\sim
p_{\textbf{z}}(\textbf{z}),\textbf{x},\tilde{\textbf{x}}\sim
p_{\text{data}}(\textbf{x},\tilde{\textbf{x}})}[(\delta(G(\textbf{z},\tilde{\textbf{x}}),\tilde{\textbf{x}})-\Theta({\textbf{x}}))^{2}]\\\
&+\alpha\operatorname{\mathbb{E}}_{\textbf{z}\sim
p_{\textbf{z}}(\textbf{z}),\textbf{x},\tilde{\textbf{x}}\sim
p_{\text{data}}(\textbf{x},\tilde{\textbf{x}})}[\Phi(G(\textbf{z},\tilde{\textbf{w}}))-\Phi(\textbf{x})],\\\
\underset{D}{\max}~{}V(D)&=\frac{1}{4}\operatorname{\mathbb{E}}_{\textbf{x},\tilde{\textbf{x}}\sim
p_{\text{data}}(\textbf{x},\tilde{\textbf{x}})}[(D(\textbf{x},\tilde{\textbf{x}})-b)^{2}]\\\
&+\frac{1}{4}\operatorname{\mathbb{E}}_{\textbf{x},\tilde{\textbf{x}}\sim
p_{\text{data}}(\textbf{x},\tilde{\textbf{x}})}[(\delta(\textbf{x},\tilde{\textbf{x}})-\Theta({\textbf{x}}))^{2}]\\\
&+\frac{1}{4}\operatorname{\mathbb{E}}_{\textbf{z}\sim
p_{\textbf{z}}(\textbf{z}),\tilde{\textbf{x}}\sim
p_{\text{data}}(\tilde{\textbf{x}})}[(D(G(\textbf{z},\tilde{\textbf{x}}),\tilde{\textbf{x}})-a)^{2}]\\\
&+\frac{1}{4}\operatorname{\mathbb{E}}_{\textbf{x},\textbf{x}^{r}\sim
p_{\text{data}}(\textbf{x})}[(D(\textbf{x},\textbf{x}^{r})-a)^{2}],\end{split}$

where x, $\tilde{\textbf{x}}$, $\textbf{x}^{r}$, z, $a$, $b$, and $c$ are the
same as in Eq. 1, $\Phi(\textbf{x})$ corresponds to the short-time Fourier
transform magnitude in dBs (20 ms windows, 10 ms stride, and 2048 bias), and
$\alpha=10^{-3}$ is a weighting term [15]. The $\delta$ function is the
discriminator output at some intermediate layer of our choice, and
$\Theta(\textbf{x})$ is the feature extractor of LPS, MFCC and prosodic
features (see sec. 5.4). Hereafter, we will refer to the joint power loss
regularization and acoustic loss as acoustic losses.

Figure 1: Setup of new multi-task framework with acoustic loss, power loss,
and adversarial outcomes (F and R).

###  4.2 Adversarial Pre-Training

Whereas SEGAN with only an adversarial loss is stable and learns in a steady
equilibrium (eq. 1), the addition of the acoustic losses induces a particular
unbalance effect during training. The addition of these terms makes $D$ learn
quicker and converge faster in both losses, conversely making $G$ converge
slower. Importantly, both $G$ and $D$ should maintain an equilibrium learning
from each other, so making one of them quickly better discourages the other to
perform properly [10]. We hypothesize that scheduling the learning of the
discriminator as a two-stage process makes the addition of these acoustic
losses more effective for it first gets high-level representations
classifying, and then focuses on specific speech properties by doing
regressions. Hence we first do an adversarial warm up in $D$ with Eq. 1
formulation, and then add the acoustic losses.

##  5 Experimental Setup

###  5.1 Dataset

We employ the VCTK Corpus for our experiments [27]. We select 80 speakers for
training and 14 for test. We trim large silence regions to 100 ms with the
help of a voice activity detector. After this, we get roughly 20 h of training
speech and 3 h of test speech. We incorporate the aforementioned distortions
(Sec. 3) with an online process jointly working with training. When we
construct a training minibatch, we (1) load the clean utterance, (2) get a
random $\approx$1 s chunk (16384 samples) inside the utterance, and (3) apply
a series of transforms that get activated independently under a probability
$p=0.4$. This way one or two distortions coincide often, and zero (thus purely
autoencoder) or more than two coincide less. Table 1 shows this activation
distribution with up to the 4 mentioned combinations. In addition to being
active or not, each transform has a certain severity level or factor, as
mentioned before (sec. 3). The only transform with just one level of severity
is the whispering transform. Hence every time a transformation is activated in
the pipeline, one of the possible factors is randomly selected. These
possibilities are: clip factors of 30%, 40% and 50%, signal resampling factors
of 2, 4, and 8 and, in the case of chunk removal, we allow the system to zero
out up to $5$ chunks from speech regions. Then, inside each region, the length
of the chunk is sampled from one of the two mentioned Gaussian distributions
(sec. 3).

Table 1: Frequency of number of training active transformations using $p=0.4$. Num. of transforms | 0 | 1 | 2 | 3 | 4
---|---|---|---|---|---
Relative frequency | 0.14 | 0.34 | 0.33 | 0.15 | 0.04

For the test split, we have two different setups, one for an objective
evaluation and another one for a subjective one. For the objective test set,
we proceed as with the training set. For the subjective test, we generate a
special split with two subsets in order to account for two different effects
present in the system outcomes. The first subset is designed to focus on
generated speaker identity (SpkID pool). To do so, we just use the most severe
version of bandwidth reduction ($\times$8) and the whispering manipulations,
so the ones that affect the most this feature. Then, for each test speaker we
degrade four utterances, two with bandwidth reduction and two with whispering.
The second subset focuses on the generated speech naturalness (Nat pool). To
evaluate this aspect, we pick 4 test speakers (two male and two female) and
apply clipping, resampling, and whispering to 6 utterances in total (two
utterances per distortion). Both clipping and resampling have severity factors
30% and $\times$8, respectively.

###  5.2 Experiments

We experiment with three models applied over the aforementioned data. Firstly,
we use SEGAN in a plain adversarial setup with the LSGAN loss as in Eq. 1,
where the only signal comes from real or fake decisions. This is the baseline,
which is trained for 400 epochs with two-timescale update rule (TTUR) learning
rates [28] $\eta_{D}=4\cdot 10^{-4}$ and $\eta_{G}=1\cdot 10^{-4}$. Secondly,
we apply the acoustic losses presented in Sec. 4.1. This system trains also
for 400 epochs, but the learning rates are balanced equal
$\eta_{D}=\eta_{G}=5\cdot 10^{-5}$, because the discriminator already has an
advantage with the extra signals and TTUR involved a noisier result on $G$. We
name this approach SEGAN-Aco. Finally, we include our improved proposal, for
which we pre-train the system over 100 epochs the same way we do with the
baseline, and then we activate the acoustic losses for the remaining 300
epochs, also lowering the learning rates to $\eta_{D}=\eta_{G}=5\cdot
10^{-5}$. We name this approach SEGAN-PTAco.

###  5.3 Evaluation

To assess each model result we perform two evaluations. First, we run a number
of objective distortion metrics, often applied in speech synthesis problems:
Mel cepstral distortion (MCD; in dB) [29], F0 root mean squared error (RMSE;
in Hz), and the voiced/unvoiced frame prediction error (UV) [30]. These errors
give us a first clue on how close is each system to the clean original signal
in terms of content, identity, and intonation.

Given the importance of perceptual scores, we also conduct a subjective
evaluation with 26 listeners. This has two stages, with two tasks to be
evaluated by the listeners: speaker identification and naturalness rating. For
speaker identification, listeners are asked to determine how close a
reconstructed signal is towards the original speaker identity. They are
presented with 4 randomly-selected utterances out of the SpkID pool (Sec.
5.1). For each utterance, the clean reference is shown, as well as the four
systems to be rated: (1) the degraded input signal to $G$, (2) the SEGAN
baseline, (3) SEGAN-Aco, and (4) SEGAN-PTAco. The rating is as simple as
ordering them by preference, such that position 1 is for closest system
towards the reference and position 4 is the furthest one. For naturalness
rating, 6 utterances taken randomly out of the Nat pool (Sec. 5.1) are shown
to each listener. For each utterance, the four different aforementioned
systems are shown and asked to be ranked from most natural (1) to least
natural (4). There is no reference shown for this case as there is no
similarity trait like a speaker identity, it is only comparing the perceptual
quality of the synthesized utterances. In all subjective evaluations, systems
are shown in random order at every utterance.

###  5.4 SEGAN Setup

We use the same kernel widths, strides, and feature map configurations as in
the WSEGAN work [15]. These are kernels of width 31 in all convolutional
layers for both $G$ and $D$. The feature maps are incremental in the encoder
and decremental in the decoder, having
$\\{64,128,256,512,1024,512,256,128,64,1\\}$ in $G$ and
$\\{64,128,256,512,1024\\}$ in $D$. The discriminator is then the one
diverging from its predecessors in earlier works [11, 15]. Firstly, it has a
multi-layer perceptron (MLP) with 16384 inputs (1024$\times$16 feature map,
unrolled), 256 hidden PReLU units, and the single output for real and fake
predictions. Secondly, we find the acoustic branch for SEGAN-Aco and SEGAN-
PTAco models at the fourth convolutional layer, which outputs 512$\times$64
feature maps for an input of approximately 1 s at 16 kHz sampling rate. These
corresponds to 64 frames, each one injected into the acoustic prediction MLP
of 128 hidden PReLU units and 277 linear outputs. These outputs predict 257
log-power spectral bins, 16 MFCCs, 1 log-F0 value of the frame, 1
voiced/unvoiced frame flag, 1 frame energy coefficient, and 1 frame zero
crossing rate. Mini-batches of 150 samples are used for all models.
Additionally, $D$ implements spectral normalization to avoid sudden exploding
gradients leading to training collapse [31, 32] and phase shift of $5$ to
reduce high-frequency artifacts in the output of $G$ [33]. All the experiments
were developed with the public SEGAN framework at https://github.com/santi-
pdp/segan_pytorch.

##  6 Results

Objective results are shown in Table 3. We can see that the distorted signals
themselves have the noisiest behaviors, with large variances in all error
metrics. This is expectable, as we have a wide range of distortion conditions.
We can see how, in all metrics, SEGAN-PTAco is the best system and the least
noisy one. A lower MCD can be assumed to correlate with better content
preservation, identity reconstruction, and naturalness generation [34]. The
lower values obtained for F0-RMSE and UV error also typically denote better
intonation schemes matching the input identity with prosodic contents.

Table 2: Objective metrics for the different systems (standard deviation into parenthesis). For each metric, lower is better. Model | MCD [dB] | RMSE [Hz] | UV [%]
---|---|---|---
Distorted | 7.6 (5.6) | 52.9 (75.9) | 22.3 (25.5)
SEGAN | 8.2 (3.5) | 37.2 (43.5) |  9.1 (11.4)
SEGAN-Aco | 7.4 (2.7) | 37.5 (53.5) | 20.0 (23.4)
SEGAN-PTAco | 6.5 (2.7) | 22.7 (28.5) |  5.8 ( 6.3)

Subjective results are shown in Table 3. In general, they also highlight the
importance of both the acoustic losses and the two-stage adversarial training
schedule. We can see that the baseline and SEGAN-Aco are clustered together
with the distorted signals for the Speaker ID test. This implies that, for
both bandwidth extension and dewhispering problems, neither systems
reconstruct a proper speaker identity from the input signal. A qualitative
listening in this case tells us that the baseline imposes an arbitrary
identity to the reconstruction, and SEGAN-Aco sounds robotic and muffled. In
contrast, SEGAN-PTAco is consistently ranked above the distorted signal, and
we can clearly hear a competitive identity consistency with the conditioning.
In the naturalness task, the baseline and SEGAN-Aco systems do better than the
distorted signals. This was expected, as they involve an enhancement process
that removes very noticeable artifacts and recover speech nuances.
Nevertheless, SEGAN-PTAco is still the best system by a large margin. Some
audio samples are available at http://veu.talp.cat/gsegan/.

Table 3: Subjective mean ranking score (lower is better). Model | Speaker ID | Naturalness
---|---|---
Distorted | 2.87 (1.07) | 3.00 (1.04)
SEGAN | 2.85 (1.03) | 2.69 (1.04)
SEGAN-Aco | 2.88 (0.83) | 2.78 (0.92)
SEGAN-PTAco | 1.41 (0.74) | 1.52 (0.84)

##  7 Conclusion

In this work, we propose the application of speech enhancement GANs in a more
generalized signal recovery framework. We first introduce four aggressive
distortions applied in our experiments emulating aggressive distortions to the
speech signals. Next, we introduce a new acoustic regression component in the
SEGAN $D$ and, in addition, we propose a two-stage adversarial training method
to make the new acoustic regression component reach an appropriate
performance. Objective and subjective results are provided, showing how both
the regression losses and and the two-stage schedule outperform the regression
losses alone and the purely adversarial approach.

##  8 Acknowledgements

This research was partially supported by the project TEC2015-69266-P
(MINECO/FEDER, UE). We deeply thank the participants of the subjective
evaluation.

## References

  * [1] P. C. Loizou, _Speech Enhancement: Theory and Practice_ , 2nd ed. Boca Raton, FL, USA: CRC Press, Inc., 2013.
  * [2] L.-P. Yang and Q.-J. Fu, “Spectral subtraction-based speech enhancement for cochlear implant patients in background noise,” _Journal of the Acoustical Society of America_ , vol. 117, no. 3, pp. 1001–1004, 2005.
  * [3] X. Lu, Y. Tsao, S. Matsuda, and C. Hori, “Speech enhancement based on deep denoising autoencoder.” in _Proc. of the Conf. of the Int. Speech Communication Association (INTERSPEECH)_ , 2013, pp. 436–440.
  * [4] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “A regression approach to speech enhancement based on deep neural networks,” _IEEE/ACM Transactions on Audio, Speech and Language Processing_ , vol. 23, no. 1, pp. 7–19, 2015.
  * [5] D. S. Williamson and D. Wang, “Time-frequency masking in the complex domain for speech dereverberation and denoising,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 25, no. 7, pp. 1492–1501, 2017\.
  * [6] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in _Proc. of the Int. Conf. on Latent Variable Analysis and Signal Separation_ , 2015, pp. 91–99.
  * [7] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,” in _Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 708–712.
  * [8] D. Rethage, J. Pons, and X. Serra, “A wavenet for speech denoising,” in _Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5069–5073.
  * [9] S. R. Park and J. Lee, “A fully convolutional neural network for speech enhancement,” in _Proc. of the Conf. of the Int. Speech Communication Association (INTERSPEECH)_ , 2017.
  * [10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2014, pp. 2672–2680.
  * [11] S. Pascual, A. Bonafonte, and J. Serrà, “Segan: Speech enhancement generative adversarial network,” in _Proc. of the Conf. of the Int. Speech Communication Association (INTERSPEECH)_ , 2017, pp. 3642–3646.
  * [12] T. Higuchi, K. Kinoshita, M. Delcroix, and T. Nakatani, “Adversarial training for data-driven speech enhancement without parallel corpus,” in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2017, pp. 40–47.
  * [13] S. Qin and T. Jiang, “Improved wasserstein conditional generative adversarial network speech enhancement,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2018, no. 1, p. 181, 2018.
  * [14] C. Donahue, B. Li, and R. Prabhavalkar, “Exploring speech enhancement with generative adversarial networks for robust speech recognition,” in _Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5024–5028.
  * [15] S. Pascual, A. Bonafonte, J. Serrà, and J. A.-G. López, “Whispered-to-voiced alaryngeal speech conversion with generative adversarial networks,” _Proc. IberSPEECH 2018_ , pp. 117–121, 2018.
  * [16] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _ArXiv: 1511.06434_ , 2015.
  * [17] A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” _arXiv preprint arXiv:1809.11096_ , 2018\.
  * [18] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier gans,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 2642–2651.
  * [19] M. Lucic, M. Tschannen, M. Ritter, X. Zhai, O. Bachem, and S. Gelly, “High-fidelity image generation with fewer labels,” _arXiv preprint arXiv:1903.02271_ , 2019.
  * [20] A. v. d. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. v. d. Driessche, E. Lockhart, L. C. Cobo, F. Stimberg _et al._ , “Parallel wavenet: Fast high-fidelity speech synthesis,” _arXiv preprint arXiv:1711.10433_ , 2017.
  * [21] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, “Least Squares Generative Adversarial Networks,” in _Proc. of the IEEE Int. Conf. on Computer Vision (ICCV)_. IEEE, 2017, pp. 2813–2821.
  * [22] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proc. of the IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2017, pp. 5967–5976.
  * [23] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _Proc. of the IEEE Int. Conf. on Computer Vision (ICCV)_ , 2015, pp. 1026–1034.
  * [24] S. Pascual, M. Park, J. Serrà, A. Bonafonte, and K.-H. Ahn, “Language and noise transfer in speech enhancement generative adversarial network,” in _Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5019–5023.
  * [25] D. Erro, I. Sainz, E. Navas, and I. Hernáez, “Improved hnm-based vocoder for statistical synthesizers,” in _Twelfth Annual Conference of the International Speech Communication Association_ , 2011.
  * [26] W. Ping, K. Peng, and J. Chen, “Clarinet: Parallel wave generation in end-to-end text-to-speech,” _arXiv preprint arXiv:1807.07281_ , 2018.
  * [27] C. Veaux, J. Yamagishi, K. MacDonald _et al._ , “Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,” 2016.
  * [28] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6626–6637.
  * [29] R. Kubichek, “Mel-cepstral distance measure for objective speech quality assessment,” in _Proc. of the IEEE Pacific Rim Conf. on Communications Computers and Signal Processing (PACRIM)_ , vol. 1. IEEE, 1993, pp. 125–128.
  * [30] S. Pascual and A. Bonafonte, “Multi-output rnn-lstm for multiple speaker speech synthesis and adaptation,” in _Proc. of the European Signal Processing Conference (EUSIPCO)_. IEEE, 2016, pp. 2325–2329.
  * [31] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” _arXiv preprint arXiv:1805.08318_ , 2018.
  * [32] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” _arXiv preprint arXiv:1802.05957_ , 2018\.
  * [33] C. Donahue, J. McAuley, and M. Puckette, “Synthesizing audio with generative adversarial networks,” _ArXiv: 1802.04208_ , 2018.
  * [34] S. Pascual de la Puente, “Deep learning applied to speech synthesis,” Master’s thesis, Universitat Politècnica de Catalunya, 2016.