# Person Re-Identification with a Locally Aware Transformer

Charu Sharma*, Siddhant R. Kapil*, David Chapman*
* Contributed Equally
Department of Computer Science
University of Maryland Baltimore County
Baltimore MD 21250
{charus2, skapil1<EMAIL_ADDRESS>

###### Abstract

Person Re-Identification is an important problem in computer vision-based
surveillance applications, in which the same person is attempted to be
identified from surveillance photographs in a variety of nearby zones. At
present, the majority of Person re-ID techniques are based on Convolutional
Neural Networks (CNNs), but Vision Transformers are beginning to displace pure
CNNs for a variety of object recognition tasks. The primary output of a vision
transformer is a global classification token, but vision transformers also
yield local tokens which contain additional information about local regions of
the image. Techniques to make use of these local tokens to improve
classification accuracy are an active area of research. We propose a novel
Locally Aware Transformer (LA-Transformer) that employs a Parts-based
Convolution Baseline (PCB)-inspired strategy for aggregating globally enhanced
local classification tokens into an ensemble of $\sqrt{N}$ classifiers, where
$N$ is the number of patches. An additional novelty is that we incorporate
blockwise fine-tuning which further improves re-ID accuracy. LA-Transformer
with blockwise fine-tuning achieves rank-1 accuracy of $98.27\%$ with standard
deviation of $0.13$ on the Market-1501 and $98.7\%$ with standard deviation of
$0.2$ on the CUHK03 dataset respectively, outperforming all other state-of-
the-art published methods at the time of writing.

##  1 Introduction

In recent years, Person Re-Identification(re-ID) has gained a lot of attention
due to its foundational role in computer vision based video surveillance
applications. Person re-ID is predominantly considered as a feature embedding
problem. Given a query image and a large set of gallery images, person re-ID
generates the feature embedding of each image and then ranks the similarity
between query and gallery image vectors. This can be used to re-identify the
person in photographs obtained by nearby surveillance cameras.

Recently, Vision Transformer (ViT) as introduced by Dosovitskiy et al. (2020)
is gaining substantial traction for image recognition problems. While some
methods for image classification (Dosovitskiy et al., 2020; Touvron et al.,
2020), and for image retrieval (El-Nouby et al., 2021) are focused only on the
classification token, some approaches utilize the fact that local tokens,
which are also outputs of the transformer encoder, can be used to improve
performance of many computer vision applications including image segmentation
(Wu et al., 2020; Wang et al., 2021; Chen et al., 2021), object detection
(Beal et al., 2020; Wang et al., 2021) and even person re-ID (He et al.,
2021). Nevertheless, at present, approaches to make use of local and global
tokens are an active area of research.

In the words of Beal et al. (2020), "The remaining tokens in the sequence are
used only as features for the final class token to attend to. However, these
unused outputs correspond to the input patches, and in theory, could encode
local information useful for performing object detection".Beal et al. (2020)
observed that the local tokens, although theoretically influenced by global
information, also have substantial correspondence to the original input
patches. One might therefore consider the possibility of using these local
tokens as an enhanced feature representation of the original image patches to
more strongly couple vision transformer encoders to fully connected (FC)
classification techniques. This coupling of local patches with FC
classification techniques is the primary intuition behind the LA-Transformer
architectural design.

Part-based Convolutional Baseline (PCB) (Sun et al., 2018) is a strong
convolutional baseline technique for person re-ID and has inspired many state-
of-the-art models (Yao et al., 2018; Guo et al., 2019; Zheng et al., 2019).
PCB partitions the feature vector received from the backbone network into six
vertical regions and constructs an ensemble of regional classifiers with a
voting strategy to determine the predicted class label. A limitation of PCB is
that each regional classifier ignores the global information which is also
very important for recognition and identification. Nevertheless, PCB has
achieved much success despite this limitation, and as such the design of LA-
Transformer uses a PCB-like strategy to combine globally enhanced local
tokens.

Our work also improves on the recent results of He et al. (2021), who was the
first to employ Vision Transformers to person re-ID and achieved results
comparable to the current state-of-the-art CNN based models. Our approach
extends He et al. (2021) in several ways but primarily because we aggregate
the globally enhanced local tokens using a PCB-like strategy that takes
advantage of the spatial locality of these tokens. Although He et al. (2021)
makes use of fine-grained local tokens, it does so with a ShuffleNet (Zhang et
al., 2017) like Jigsaw shuffling step which does not take advantage of the 2D
spatial locality information inherent in the ordering of the local tokens. LA-
Transformer overcomes this limitation by using a PCB-like strategy to combine
the globally enhanced local tokens while first preserving their ordering in
correspondence with the image dimension.

An additional novelty of our approach is the use of blockwise fine-tuning
which we find is able to further improve the classification accuracy of LA-
Transformer for person re-ID. Blockwise fine-tuning is viable as a form of
regularization when training models with a large number of parameters over
relatively small in-domain datasets. Howard and Ruder (2018) advocate for
blockwise fine-tuning or gradual unfreezing particularly when training
language models due to a large number of fully connected layers. As vision
transformers also have high connectivity, we find that this approach is able
to further improve the classification accuracy for LA-Transformer.

This paper is organized as follows: Firstly, we discuss related work involving
Transformer architectures and other related methodologies in person re-ID.
Secondly, we describe the architecture of LA-Transformer, including the novel
locally aware network and blockwise fine-tuning techniques. Finally, we
present quantitative results of the person re-ID including mAP and rank-1
analysis on the market-1501 and CUHK03 datasets.

##  2 Related Work

For many years CNN based models have dominated image recognition tasks
including person re-ID. A vast body of research has been performed to
determine the best strategy to extract features using CNNs to address issues
like appearance ambiguity, background perturbance, partial occlusion, body
misalignment, viewpoint changes, and pose variations, etc. Sarfraz et al.
(2018) proposed a Pose-Sensitive Embedding to incorporate information
associated with poses of a person in the model, Yu et al. (2020) used a Graph
Convolution Network (Kipf and Welling, 2017) to generate a conditional feature
vector based on the local correlation between image pairs, Herzog et al.
(2021) used global channel-based and part-based features, Zhang et al. (2018)
used global pooling to extract global features and horizontal pooling followed
by $1\times 1$ CNN for local features. CNN based methods have led to many
advances in recent years and are continuing to be developed for person re-ID.

Another branch of techniques for person re-ID focuses on the development of
highly engineered network designs that incorporate additional domain knowledge
to improve re-ID performance. Ding et al. (2020) used a part-aware approach
for which the model performs the main task as well as auxiliary tasks for each
body part. Zhou and Shao (2018) and Zhu et al. (2019) use viewing angles as
additional features. Yao et al. (2017) introduced the idea of calculating part
loss and Sun et al. (2018) (Part-based Convolutional Backbone a.k.a. PCB)
improved on it. Even current top-performing models like Yao et al. (2018) used
PCB along with domain-specific Spatio-temporal distribution information to
achieve good results on the Market-1501 dataset. In our work we incorporate
PCB-like local classifiers with Vision Transformers, and furthermore we find
that our model performs better if we pass global information along with local
features. LA-Transformer achieves results with comparable and slightly higher
rank-1 accuracy than the reported results of Yao et al. (2018) over
Market-1501 and does so without the use of additional Spatio-temporal
information.

Interest in Vision Transformers grew initially from attention mechanisms which
were first employed for language translation problems in NLP (Bahdanau et al.,
2016), Attention mechanisms have been employed to great effect in image
recognition. Wang et al. (2018a) introduced parameter-free spatial attention
to integrating spatial relations to Global Average Pooling (GAP). Xie et al.
(2020) used Spatial Attention Module (SAM), and Channel Attention Module (CAM)
to deliver prominent spatial and channel information. Chen et al. (2019)
propose Position Attention Module (PAM) for semantically related pixels in the
spatial domain along with CAM. Attention mechanisms continue to be an active
area of research for many problems related to object detection and
recognition.

Transformers were first introduced in NLP problems by Vaswani et al. (2020),
and now Transformers are contributing to many new developments in machine
learning. Dosovitskiy et al. (2020) introduced transformers to images by
treating a 16x16 patch as a word and treating image classification as
analogous to text classification. This approach showed promising results on
ImageNet and it was soon adopted in many image classification problems (Parmar
et al., 2018; Wang et al., 2018b). Object detection is another highly related
problem for which vision transformers have been recently applied (Carion et
al., 2020; Beal et al., 2020). Beal et al. (2020) described a correspondence
between local tokens and input patches and combined local tokens to create
spatial feature maps. At present, this observation of the correspondence
between local tokens and input patches has yet to be applied to a wide variety
of computer vision problems, nor has it been previously explored in the
context of person re-ID. One exception is in the area of image segmentation,
for which recent works are beginning to take advantage of the 2D ordering of
the local tokens in order to produce more accurate predicted masks (Wu et al.,
2020; Wang et al., 2021; Chen et al., 2021). Our approach builds upon the
recent work of He et al. (2021) who was the first to apply vision transformers
to object and person re-ID. Although the approach of He et al. (2021) makes
use of global and local tokens, He et al. (2021) combines the local tokens
using a jigsaw classification branch which shuffles the ordering of the local
features. Shuffling the order of local features does not take advantage of the
observation of Beal et al. (2020) in that local features correspond strongly
with input patches and therefore have a natural ordering in the form of a 2D
image grid. Conversely, LA-Transformer takes advantage of the spatial locality
of these local features by combining globally enhanced local tokens with a
PCB-like strategy (Sun et al., 2018). Furthermore, LA-Transformer incorporates
the blockwise fine-tuning strategy as described by Howard and Ruder (2018) as
a form of regularization for high-connectivity pre-trained language models. As
such LA-Transformer builds upon recent advances in the application of vision
transformers in tandem with novel training techniques to achieve state of the
art accuracy in person re-ID.

##  3 Methodology

LA-Transformer combines vision transformers with an ensemble of FC classifiers
that take advantage of the 2D spatial locality of the globally enhanced local
tokens. Section 3.1 describes the overall architecture including the backbone
vision transformer (section 3.1.1), as well as the PCB inspired classifier
network ensemble (section 3.1.2). The blockwise fine-tuning strategy is
described in section 3.2. As such, these sections describe the major elements
of the LA-Transformer methodology.

Figure 1: Architecture of LA-Transformer. Part (a) \- shows the backbone
architecture. The input image is converted into patch embedding using 2D
convolution. The class embedding (cls embedding) is prepended to the patch
embedding. Then the position embeddings are added and this resulting sequence
is fed to the transformer encoder. $F={f_{0},f_{1},...,f_{N}}$ is the output
of the transformer encoder where $f_{0}$ is the global vector $G$ and
remaining tokens from $f_{1}$ to $f_{N}$ are local tokens $Q$. $G$ and $Q$ are
then combined using weighted averaging and are called Globally Enhanced Local
Tokens (GELT). GELT are then arranged into a 2D spatial grid. Part (b) \-
shows the Locally Aware Network. $L$ is the row-wise average of GELTs and is
performed using average pooling. $L$ is then fed to the locally aware
classification ensemble. Part (c) and (d) describes the architecture of
transformer encoder and FC classifier respectively.

###  3.1 Locally Aware Transformer

LA-Transformer (figure 1) consists of two main parts: a backbone network and a
locally aware network. Both components are interconnected and trained as a
single neural network model.

The backbone network is the ViT architecture as proposed by Dosovitskiy et al.
(2020). ViT generates tokens $F={f_{0},f_{1},..,f_{N}}$. The token $f_{0}$,
also known as the global classification token and we refer to this token as
the global token $G$. Supplementary outputs ${f_{1}..f_{196}}$ are referred to
as local tokens which we denote to collectively as $Q$. Globally Enhanced
Local Tokens (GELT) are obtained by combining global tokens and local tokens
($G$ and $Q$) using weighted averaging and are arranged into a $14\times 14$
2D spatial grid as seen in Figure 1(a). The row-wise averaged GELTs are then
fed to the locally aware classification ensemble as seen in Figure 1(b) to
classify during the training process and to generate feature embedding (by
concatenating L) during the testing process. These steps are described in
greater detail in the following sections 3.1.1 and 3.1.2

####  3.1.1 LA-Transformer Backbone

The backbone network of LA-Transformer is the ViT vision transformer
(Dosovitskiy et al., 2020). ViT requires extensive training data on the order
of $14M-300M$ images to train effectively, but the Market1501 and CUHK-03
datasets are relatively small (Table 1) in comparison on the order of $10$’s
of thousands of images. As such we employed a pre-trained ViT model, and
further made use of blockwise fine-tuning to improve accuracy as described in
section 3.2

##### Embeddings

The backbone ViT architecture takes images of size $224\times 224$ as input,
and as such the Market1501 and CUHK-03 images are re-sampled to this
resolution during training. First, the image is converted into $N$ number of
patches $x^{i}_{p}|i=1,..,N$. Each patch is then linearly projected into $D$
dimensions using the patch embedding function ($E(x^{i}_{p})|i=1,..,N$) (eq.
2), which is obtained using a convolution layer with a kernel size of
$16\times 16$. For non-overlapping patches, a stride equal to $16$ is used.
$D$ is the number of channels and is set to $768$ which represents the size of
the embedding. The total number of patches $N$ depends on kernel size, stride,
padding, and size of the image. $N$ can be easily calculated using the eq. 1.
Assuming padding is 0, and ${H}$, ${W}$ are height and width of an image,
${K_{H}}$, ${K_{W}}$ are height and width of the kernel and $S$ is kernel
stride.

$N=\left(\frac{H-K_{H}}{S}+1\right)\times\left(\frac{W-K_{W}}{S}+1\right)$ (1)

Afterward, the learnable class embedding $x_{class}$ is prepended with the
patch embedding ($E(x^{i}_{p}$)) whose output state keeps the information of
the entire image and serves as the global vector. The resulting vectors are
then added with position embeddings $P$ to preserve the positional
information. Subsequently, the final sequence of vectors $z_{0}$ (eq. 2) is
fed into the transformer encoder (figure1) to generate $N+1$ feature vectors
where $N$ is the number of patches plus class embedding.

$z_{0}=[x_{class};E(x^{1}_{p});E(x^{2}_{p});....;E(x^{N}_{p})]+P$ (2)

##### Transformer Encoder

The transformer encoder consist of total $B=12$ blocks. Each block contains
alternating MSA (Multiheaded Self-Attention) introduced by Vaswani et al.
(2020) and MLP blocks. The Layernorm (LN) is applied before MSA and MLP blocks
and a residual connections is applied after each encoder block. The output of
transformer encoder $F$ described in eq. 5 passes through all the $B$ blocks
(eq. 3 and 4 ).

$z^{\prime}_{b}=z_{b-1}+MSA(LN(z_{b-1}))$ (3)
$z_{b}=z^{\prime}_{b}+MLP(LN(z^{\prime}_{b}))$ (4) $F=LN(z_{B})$ (5)

While the seminal work of Dosovitskiy et al. (2020) only uses classification
token $z^{0}_{B}$ for classification, LA-Transformer makes use of all of the
features $z_{B}$ eq. 5. Though the class embedding can be removed from the
backbone network, our experiments show promising results with class embedding
serving as a global vector (Table 2). From our experiments, it is clear that
ViT as a backbone network is a good choice for person re-ID based problems.
Further, we believe that any transformer based model like Diet by Touvron et
al. (2020), or DeepViT by Zhou et al. (2021) can be used as a backbone
network.

####  3.1.2 Locally Aware Network

The Locally Aware Network is a classifier ensemble similar to the PCB
technique of Sun et al. (2018) but with some differences. Firstly, in Sun et
al. (2018) the input features are purely local, whereas in LA-Transformer, we
find that the inclusion of global vectors along with local vectors via
weighted averaging can increase the network accuracy. Secondly, although in
Sun et al. (2018) the image is divided into six input regions, we divide the
2D spatial grid of tokens into $\sqrt{N}=14$ regions as seen in Figure 1.
Finally, while PCB uses a convolutional backbone, LA-Transformer uses the ViT
backbone.

In Figure 1, the transformer encoder outputs $N+1$ feature vectors. The global
tokens $G=f_{0}$ and local tokens $Q=[f_{1},f_{2},f_{3},...,f_{N}]$ are
obtained for which $N$ is number of patches. $N_{R}$ is defined as the total
number of patches per row and $N_{C}$ as the total number of patches per
column. In our case, $N_{R}=N_{C}=\sqrt{N}$. Then we define $L$ as the
averaged GELT obtained after average pooling of $Q$ and $G$ as follows,

$L_{i}=\frac{1}{N_{R}}\sum_{j=i*N_{R}+1}^{(i+1)*N_{R}}\frac{(Q_{j}+\lambda
G)}{(1+\lambda)}\qquad\qquad i=0...N_{C}-1$ (6)

In eq. 6 all the patches in a row are averaged to create one local vector per
row . The total number of FC classifiers is equal to $N_{C}$. Each FC
classifier contains two fully connected layers with RELU and Batch
Normalization. We define $y$ as the output of LA-Transformer as follows,

$y_{i}=FC_{i}(L_{i})\qquad\qquad i=1...N_{C}$ (7)

The outputs $y$ are passed through softmax and the softmax scores are summed
together. The argument of the maximum score represents the ID of the person as
follows.

$score=\sum_{i=0}^{N_{C}}softmax(y_{i})$ (8) $prediction=argmax(score)$ (9)

###  3.2 Fine-tuning Strategies

According to the recent studies of Touvron et al. (2020) and Dosovitskiy et
al. (2020), training a vision transformer from scratch requires about 14M-300M
images. Person re-ID datasets are known for their small size and training a
transformer on these datasets can quickly lead to overfitting. As such, ViT
was pre-trained on ImageNet (Ridnik et al. (2021)), and then fine-tuned on
person re-ID datasets. Blockwise fine-tuning was applied which is highly
similar to the gradual unfreezing method described by Howard and Ruder (2018)
for the purposes of training large language models in the event of limited
training data from a target domain.

##### Blockwise Fine-tuning

In blockwise fine-tuning, all transformer blocks are frozen in the start
except for the bottleneck model. After every $t$ epochs (where $t$ is a hyper-
parameter), one additional transformer encoder block is unfrozen and the
learning rate is reduced as described by Alg1. Blockwise fine-tuning helps in
mitigating the risk of catastrophic forgetting of the pre-trained weights
(Howard and Ruder, 2018). The learning rate decay helps in reducing the
gradient flow in the subsequent layers hence prevent abrupt weight updates.

Algorithm 1 Blockwise Fine-tuning

1:Freeze all the transformer blocks $B$

2:Initialize parameters: $t=2$, $b=12$, $lr=3e-4$, $lr-decay=0.85$

3:while $0<=i<epochs$ do

4: if $i\%t==0\ and\ b>0$ then $\triangleright$ ViT has 12 blocks

5: $unfreeze\ B[b]$ $\triangleright$ Unfreeze the last block first

6: $b\leftarrow b-1$ $\triangleright$ Reduce block index counter

7: $lr\leftarrow lr*lr\\_decay$ $\triangleright$ Reduce lr rate

##  4 Experiments

###  4.1 Datasets and Metrics

Table 1: Datasets Overview Dataset | Classes | Train | Query | Gallery
---|---|---|---|---
Market-1501 | 751 | 12192 | 3368 | 19744
CUHK-03 | 1367 | 13131 | 965 | 965

##### Datasets

LA Transformer is trained over two benchmark datasets; Market-1501 and
CUHK-03. Table 1 gives the overview of datasets used to train the model. The
Market-1501 dataset (Zheng et al., 2015) contains total 1501
classes/identities captured by six different cameras. Out of 1501 classes, the
train set contains 750 classes, and the test set consists of 751 classes. A
total of 12,192 images are present in the train set. The test set is divided
into a query set of 3,368 images and a gallery set of 19744 images. CUHK-03
dataset (Li et al., 2014) contains a total of 1,367 classes captured by six
cameras. There are 13,131 images in the train set and 1,930 images in the test
set (965 in query and 965 in the gallery set).

##### Evaluation protocol

By convention, re-ID is evaluated over two standard evaluation metrics;
Cumulative Matching Characteristics (CMC) and Mean Average Precision (mAP). We
apply these metrics to assess the performance of the LA-Transformer and other
experiments.

###  4.2 Model Implementation Details

ViT was pre-trained on ImageNet-21K and used as a backbone network as well as
a baseline model (Dosovitskiy et al., 2020; Ridnik et al., 2021). All the
images are resized into $224\times 224$ as this resolution is compatible with
the backbone network. The model is trained over 30 epochs with a batch size of
32. We used the Adam optimizer with an initial learning rate of $3e-5$, step
decay of $0.8$, ${t}=2$ and ${\lambda}=0.8$. For testing, we concatenated all
of the averaged GELTs $L$ to generate the feature embedding. To efficiently
calculate the Euclidean norm between the query and gallery vectors, we use the
FAISS library Johnson et al. (2017). All the models are trained and tested on
a single GPU machine with an Nvidia RTX2080 Ti with 11 GB VRAM, and 64 GB RAM.

Table 2: Ablation result of the influence of global and local features on baseline ViT and LA-Transformer with and without blockwise fine-tuning over Market-1501 | Without BW-FT | With BW-FT
---|---|---
# | Model | Classifiers | Tokens | Rank-1 | mAP | Rank-1 | mAP
1 | ViT | 1 | Global | 95.8 | 89.55 | 96.2 | 90.5
2 | ViT | 1 | Local | 95.1 | 86.5 | 95.45 | 87.8
3 | ViT | 1 | Global+Local | 96.3 | 90.1 | 96.6 | 90.30
4 | LA | 14 | Global | 96.9 | 92.5 | 97.74 | 92.68
5 | LA | 14 | Local | 96.1 | 91.1 | 97.23 | 91.93
6 | LA | 14 | Global+ Local | 97.55 | 93.3 | 98.27 | 94.46
Figure 2: Part(a)- compares the validation loss of LA-Transformer with and
without Blockwise Fine-tuning (BWFT). Part(b) - compares validation accuracy
of LA-Transformer with and without BWFT. Both graphs show results using
Market-1501.

###  4.3 Ablation study of LA Transformer

The table 2 compares the performance of variations of LA-Transformer versus
the same variations of baseline ViT using the Market-1501 dataset. All six
experiments are performed with and without blockwise fine-tuning. Experiment 1
is the baseline model that uses only the global token to generate feature
embedding. Experiment 2 uses only the local tokens of the transformer encoder
and exhibits the lowest rank-1 accuracy ($95.1$) and mAP score ($86.5$) out of
all of the variations of ViT. Experiment 3 combines the the first and second
experiments by utilizing the globally enhanced local tokens. The impact of
global and local features is also compared using LA-Transformer via three
variations: global, local, and globally enhanced local tokens. All of the
experiments with LA-Transformer perform better than the baseline ViT and its
variations. LA-Transformer increases the rank-1 accuracy by ${+2.3\%}$ and mAP
score by ${+2.6}$ on an average versus ViT over the experiments in table 2.
Similar to ViT with local features, LA-Transformer with only local features
achieves the lowest accuracy of the LA-Transformer only experiments.
Therefore, we conjecture that using only local vectors to predict the output
and generate the final embedding is not sufficient. Nevertheless, using
globally enhanced local tokens outperforms the local only results by
${+1.31\%}$ rank-1 and ${+0.265}$ mAP and improves over the global only
results by ${+0.62\%}$ rank-1 and ${+0.5}$ mAP. Therefore LA-Transformer using
globally enhanced local tokens achieves the highest rank-1 and mAP scores of
all technique $+$ feature embedding designs in this comparison.

##### Ablation study of Blockwise fine-tuning

Blockwise fine-tuning achieves higher rank-1 and mAP scores in all experiments
as compared against similar experiments without blockwise fine-tuning over the
Market-1501 dataset. As seen in Table 2, blockwise fine-tuning increases the
rank-1 accuracy by ${+0.6\%}$ and mAP score by ${+0.77}$ on average across all
of the experiments in this Ablation study. During blockwise fine-tuning, the
hyperparameter $t$ is set to 2 which means, after every 2 epochs one
additional block is unfrozen. The baseline ViT model has 12 blocks. Therefore,
it takes 22 epochs to unfreeze and train on all the layers. However, for most
models, we found the best validation score is reached before the 22nd epoch,
but rather after the 18th epoch yielding 10 trainable blocks during fine
tuning. Figure 2 shows the comparison of validation results for LA-Transformer
trained with and without blockwise fine-tuning. It can be clearly seen that
blockwise fine-tuning leads to faster convergence and better results than the
training model without blockwise fine-tuning.

Table 3: Comparison with State of the Art methods Market-1501 | CUHK-03
---|---
Model | Rank-1 | mAP | Model | Rank-1 | mAP
PCB | 92.3 | 77.4 | UTAL | 56.3 | 42.3
AANet | 93.9 | 83.4 | k-reciprocal 46 | 61.6 | 67.6
IANet | 94.4 | 83.1 | DG-Net 61.1 65.6 |  |
DG-Net | 94.8 | 86 | OIM | 77.5 | 72.5
TransReID | 95.2 | 88.9 | VI+LSRO | 84.6 | 87.6
Flip reid | 95.8 | 94.7 | TriNet | 89.63 | -
VA-reid | 96.79 | 95.4 | FD-GAN | 92.6 | 91.3
st-Reid | 98 | 95.5 | DCDS | 95.8 | -
CTL Model | 98 | 98.3 | AlignedReID | 97.8 | -
LA-Transformer | 98.27 | 94.46 | LA-Transformer | 98.7 | 96.4

###  4.4 Comparison with state of the art

To evaluate the performance of LA-Transformer, it is trained and evaluated
five times on Market-1501 and CUHK03 and the mean results are reported in
Table 3. On Market-1501, the rank-1 accuracy of LA-Transformer is $98.27\%$
with standard deviation of $0.13$ with blockwise fine-tuning and $97.55$ with
standard deviation of $0.49$ without blockwise fine-tuning. On CUHK03, the
rank-1 accuracy of LA-Transformer is $98.7\%$ with a standard deviation of
$0.2$ with blockwise fine-tuning.

Table 3 compares LA Transformer with the state-of-the-art (SOTA) models on two
benchmarks of person re-ID; Market-1501 and CUHK-03. On the Market-1501
dataset, LA-transformer achieves the highest reported rank-1 accuracy of all
models in this comparison, and outperforms the rank-1 accuracy of the next
highest SOTA model by $+0.27\%$. On Market-1501, the mAP score lies among the
top five SOTA models. In case of CUHK-03, LA-Transformer achieves both the
highest rank-1 accuracy as well as the highest mAP score, and outperforms the
next highest SOTA models score by $+0.9\%$ (rank-1) and $+5.1$ (mAP)
respectively.

##  5 Conclusion

We present a novel technique for person re-ID called Locally Aware Transformer
(LA-Transformer) which achieves state of the art performance on the
Market-1501 and CHUK-03 datasets. This approach makes two contributions toward
solving the person re-ID problem. First, we show that the global token and
local token outputs of vision transformers can be combined with a PCB-like
strategy to improve re-ID accuracy. Secondly, we incorporate blockwise fine-
tuning to regularize the fine tuning of a pre-trained vision transformer
backbone network. We believe that vision transformers will continue to have a
major positive impact in the field of computer vision, and we are hopeful that
the architectural design of LA-transformer will lead to further innovation and
the development of new and novel techniques to advance our understanding of
person re-ID.

## References

  * Bahdanau et al. [2016] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate, 2016.
  * Beal et al. [2020] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk.  Toward transformer-based object detection, 2020.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers, 2020.
  * Chen et al. [2021] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation, 2021.
  * Chen et al. [2019] Tianlong Chen, Shaojin Ding, Jingyi Xie, Ye Yuan, Wuyang Chen, Yang Yang, Zhou Ren, and Zhangyang Wang.  Abd-net: Attentive but diverse person re-identification, 2019.
  * Ding et al. [2020] Changxing Ding, Kan Wang, Pengfei Wang, and Dacheng Tao.  Multi-task learning with coarse priors for robust part-aware person re-identification.  In _Computer Vision and Pattern Recognition (cs.CV)_ , 2020.  URL arXiv:2003.08069.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Computer Vision and Pattern Recognition (cs.CV)_ , 2020.  URL arXiv:2010.11929.
  * El-Nouby et al. [2021] Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou.  Training vision transformers for image retrieval, 2021.
  * Guo et al. [2019] Jianyuan Guo, Yuhui Yuan, Lang Huang, Chao Zhang, Jinge Yao, and Kai Han.  Beyond human parts: Dual part-aligned representations for person re-identification, 2019.
  * He et al. [2021] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification, 2021.
  * Herzog et al. [2021] Fabian Herzog, Xunbo Ji, Torben Teepe, Stefan Hörmann, Johannes Gilg, and Gerhard Rigoll.  Lightweight multi-branch network for person re-identification, 2021.
  * Howard and Ruder [2018] Jeremy Howard and Sebastian Ruder.  Fine-tuned language models for text classification.  _CoRR_ , abs/1801.06146, 2018.  URL http://arxiv.org/abs/1801.06146.
  * Johnson et al. [2017] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  _arXiv preprint arXiv:1702.08734_ , 2017.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks, 2017\.
  * Li et al. [2014] Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang.  Deepreid: Deep filter pairing neural network for person re-identification.  In _CVPR_ , 2014.
  * Parmar et al. [2018] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer, 2018.
  * Ridnik et al. [2021] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses, 2021.
  * Sarfraz et al. [2018] M. Saquib Sarfraz, Arne Schumann, Andreas Eberle, and Rainer Stiefelhagen.  A pose-sensitive embedding for person re-identification with expanded cross neighborhood re-ranking, 2018.
  * Sun et al. [2018] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline), 2018.
  * Touvron et al. [2020] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  _CoRR_ , abs/2012.12877, 2020.  URL https://arxiv.org/abs/2012.12877.
  * Vaswani et al. [2020] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Neural Information Processing Systems_ , 2020.  URL arXiv:2010.11929.
  * Wang et al. [2018a] Haoran Wang, Yue Fan, Zexin Wang, Licheng Jiao, and Bernt Schiele.  Parameter-free spatial attention network for person re-identification, 2018a.
  * Wang et al. [2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021.
  * Wang et al. [2018b] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks, 2018b.
  * Wu et al. [2020] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda.  Visual transformers: Token-based image representation and processing for computer vision, 2020.
  * Xie et al. [2020] Ben Xie, Xiaofu Wu, Suofei Zhang, Shiliang Zhao, and Ming Li.  Learning diverse features with part-level resolution for person re-identification, 2020.
  * Yao et al. [2017] Hantao Yao, Shiliang Zhang, Yongdong Zhang, Jintao Li, and Qi Tian.  Deep representation learning with part loss for person re-identification.  In _Computer Vision and Pattern Recognition (cs.CV)_ , 2017.  URL arXiv:1707.00798.
  * Yao et al. [2018] Hantao Yao, Shiliang Zhang, Yongdong Zhang, Jintao Li, and Qi Tian.  Spatial-temporal person re-identification.  In _Computer Vision and Pattern Recognition (cs.CV)_ , 2018.  URL arXiv:1812.03282.
  * Yu et al. [2020] Fufu Yu, Xinyang Jiang, Yifei Gong, Shizhen Zhao, Xiaowei Guo, Wei-Shi Zheng, Feng Zheng, and Xing Sun.  Devil’s in the details: Aligning visual clues for conditional embedding in person re-identification, 2020.
  * Zhang et al. [2017] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017.
  * Zhang et al. [2018] Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun.  Alignedreid: Surpassing human-level performance in person re-identification, 2018.
  * Zheng et al. [2019] Feng Zheng, Cheng Deng, Xing Sun, Xinyang Jiang, Xiaowei Guo, Zongqiao Yu, Feiyue Huang, and Rongrong Ji.  Pyramidal person re-identification via multi-loss dynamic training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Zheng et al. [2015] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  pages 1116–1124, 12 2015.  doi: 10.1109/ICCV.2015.133.
  * Zhou et al. [2021] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng.  Deepvit: Towards deeper vision transformer.  _CoRR_ , abs/2103.11886, 2021.  URL https://arxiv.org/abs/2103.11886.
  * Zhou and Shao [2018] Yi Zhou and Ling Shao.  Viewpoint-aware attentive multi-view inference for vehicle re-identification.  In _Computer Vision and Pattern Recognition (cs.CV)_ , 2018.
  * Zhu et al. [2019] Zhihui Zhu, Xinyang Jiang, Feng Zheng, Xiaowei Guo, Feiyue Huang, Weishi Zheng, and Xing Suno.  Viewpoint-aware loss with angular regularization for person re-identification.  In _Computer Vision and Pattern Recognition (cs.CV)_ , 2019.  URL arXiv:1912.01300.


# EMOPIA: A Multi-modal Pop Piano Dataset for Emotion Recognition and Emotion-
based Music Generation

###### Abstract

While there are many music datasets with emotion labels in the literature,
they cannot be used for research on symbolic-domain music analysis or
generation, as there are usually audio files only. In this paper, we present
the EMOPIA (pronounced ‘yee-mò-pi-uh’) dataset, a shared multi-modal (audio
and MIDI) database focusing on perceived emotion in pop piano music, to
facilitate research on various tasks related to music emotion. The dataset
contains 1,087 music clips from 387 songs and clip-level emotion labels
annotated by four dedicated annotators. Since the clips are not restricted to
one clip per song, they can also be used for song-level analysis. We present
the procedure for building the dataset, covering the song list curation, clip
selection, and emotion annotation processes. Moreover, we prototype use cases
on clip-level music emotion classification and emotion-based symbolic music
generation by training and evaluating corresponding models using the dataset.
The result demonstrates the potential of EMOPIA for being used in future
exploration on piano emotion-related MIR tasks.

##  1 Introduction

The affective aspect of music has been a major subject of research in the
field of music information retrieval (MIR), not only for music analysis and
labeling [1, 2, 3, 4, 5, 6, 7, 8, 9], but also for music generation or editing
[10, 11, 12, 13, 14]. Accordingly, there have been quite a few public music
datasets with emotion, as listed in Table 1. These datasets are different in
many ways, including the musical genres considered, data modality and data
size, and the way emotion is described.

With the growing interest in symbolic-domain music analysis and generation in
recent years of ISMIR [15, 16, 17, 18, 19], it is desirable to have an
emotion-labeled symbolic music dataset to add emotion-related elements to such
research. However, among the datasets listed in Table 1, only two provide MIDI
data, and they are both small in size. Moreover, the majority of the audio-
only datasets contain songs of multiple genres, making it hard to apply
automatic music transcription algorithms, which currently work better for
piano-only music [20, 21, 22, 23], to get MIDI-like data from the audio
recordings.

Name | Label type | Genre (or data source) | Size | Modality
---|---|---|---|---
Jamendo Moods [24] | adjectives | multiple genres | 18,486 | Audio
DEAM [25] | VA values | (from FMA [26], Jamendo, MedleyDB[27]) | 1,802 | Audio
EMO-Soundscapes [28] | VA values | (from FMA) | 1,213 | Audio
CCMED-WCMED [29] | VA values | classical (both Western & Chinese) | 800 | Audio
emoMusic [30] | VA values | pop, rock, classical, electronic | 744 | Audio
EMusic [31] | VA values | experimental, 8 others | 140 | Audio
MOODetector [6] | adjectives | multiple genres (AllMusic) | 193 | Audio$+$MIDI
VGMIDI [10] | valence | video game | 95 | MIDI
EMOPIA (ours) | Russell’s 4Q | pop (piano covers) | 1,078 | Audio$+$MIDI
Table 1: Comparison of some existing public emotion-labeled music datasets and
the proposed EMOPIA dataset.

To address this need, we propose a new emotion-labeled dataset comprising of
three main nice properties:

  * •

Single-instrument. We collect audio recordings of piano covers and creations
from YouTube, with fair to high audio and musical quality, and a diverse set
of playing styles and perceived emotions. Focusing on only piano music allows
for the use of piano transcription algorithms [20, 21], and facilitates
disentanglement of musical composition from variations in timbre, arrangement,
and other confounds seen in multi-instrument music.

  * •

Multi-modal. Both the audio and MIDI versions of the music pieces can be found
from the Internet (see Section 3.5 for details). The MIDI files are
automatically transcribed from the audio by a state-of-the-art model [21].

  * •

Clip-level annotation. The audio files downloaded from YouTube are full songs.
As different parts of a song may convey different emotions, the first four
authors of the paper manually and carefully pick emotion-consistent short
clips from each song and label the emotion of these clips using a four-class
taxonomy derived from the Russell’s valence-arousal model [32]. This leads to
clip-level emotion annotations for in total 1,087 clips from 387 songs (i.e.,
2.78 clips per song on average), with the number of clips per emotion class
fairly balanced.

Given these properties, EMOPIA has versatile use cases in MIR research. For
music labeling, EMOPIA can be used for clip-level music emotion recognition or
music emotion variation detection [2], in both the audio and symbolic domains.
For music generation, EMOPIA can be used for emotion-conditioned piano music
generation or style transfer, to create emotion-controllable new compositions,
or variations of existing pieces, again in both domains.

We present details of the dataset and the way we compile it in Section 3, and
report a computational analysis of the dataset in Section 4. Moreover, to
demonstrate the potential of the new dataset, we use it to train and evaluate
a few clip-level emotion classification models using both audio and MIDI data
in Section 5, and emotion-conditioned symbolic music generation models in
Section 6. The latter involves the use of a recurrent neural network (RNN)
model proposed by Ferreira _et al._ [10], and our own modification of a
Transformer-based model [33, 34, 35] that takes emotion as a conditioning
signal for generation.

We release the EMOPIA dataset at a Zenodo
repo.111https://zenodo.org/record/5090631 Source code implementing the
generation and classification models can be found on
GitHub.222https://github.com/annahung31/EMOPIA333https://github.com/SeungHeonDoh/EMOPIA_cls
Examples of generated pieces can be found at a demo
webpage.444https://annahung31.github.io/EMOPIA/

##  2 Related Work

Emotion Recognition in Symbolic Music. Symbolic music representations describe
music with the note, key, tempo, structure, chords, instruments. To understand
the relationship between music and emotion, various researchers have
investigated machine learning approaches with handcrafted features. Grekow _et
al._ [36] extract in total 63 harmony, rhythmic, dynamic features from 83
classic music MIDI files. Lin _et al._ [37] compare audio, lyric, and MIDI
features for music emotion recognition, finding that MIDI features lead to
higher performance in valence dimension. Panda _et al._ [6] also proposed
multi-model approaches, combining audio and MIDI features for emotion
recognition using a small dataset of 193 songs.

Emotion-conditioned Symbolic Music Generation. Only few work has started to
address this task recently. Ferreira _et al._ [10] compile a small dataset of
video game MIDI tracks with manual annotations of valence values, named VGMIDI
(cf. Table 1), and use it to train a long short term memory network (LSTM) in
tandem with a genetic algorithm (GA) based elite selection mechanism to
generate positive or negative music. Makris _et al._ [12] approach the same
task by using designated chord progression sequence in a sequence-to-sequence
architecture trained with the VGMIDI dataset. Zhao _et al._ [38] use LSTM to
generate music with four different emotions. Madhok _et al._ [13] use human
facial expressions as the condition to generate music. More recently, Tan &
Herremans demonstrate that their FaderNets [16] can achieve arousal-
conditioned symbolic music style transfer with a semi-supervised clustering
method that learns the relation between high-level features and low-level
representation. Their model modifies the emotion (specifically, only the
arousal) of a music piece, instead of generating new pieces from scratch.

Figure 1: Illustration of different token-based representation for symbolic
music: (a) MIDI-like [39], (b) REMI [34], and (d) CP [35] plus emotion token.
Sub-figure (c) is an intermediate representation of the CP one.

##  3 The EMOPIA Dataset

###  3.1 Song Selection and Segmentation

EMOPIA is a collection of 387 piano solo performances of popular music
segmented manually into 1,087 clips for emotion annotation. Two authors of the
paper curated the song list of the piano performances by scanning through
playlists on Spotify for its consistently high quality, then downloading the
recordings from YouTube. A song is included when it is played by a
professional conveying a clear emotion, and the recording has not been heavily
engineered during post-production. The genres of songs include Japanese anime,
Korean and Western pop song covers, movie soundtracks, and personal
compositions.

In an effort to extend the usefulness of the dataset for future research, at
the best of our ability, the songs are intentionally segmented (with the help
of the Sonic Visualizer [40]) only at cadential arrivals to make it an
_emotionally-consistent clip_ and a valid _musical phrase_ at the same time.
Accordingly, EMOPIA contains information for full songs, extracted phrases,
and emotion labels.

###  3.2 Emotion Annotation

Different emotion taxonomies have been adopted in the literature for emotion
annotation, with no standard so far [2]. For EMOPIA, we consider a simple
four-class taxonomy corresponding to the four quadrants of the Russell’s
famous Circumplex model of affect [32], which conceptualizes emotions in a
two-dimensional space defined by valence and arousal. The four classes are:
HVHA (high valence high arousal); HVLA (high valence low arousal); LVHA (low
valence high arousal); and LVLA (low valence low arousal). We refer to this
taxonomy as Russell’s 4Q.

As pointed out in the literature [41, 2, 3], various factors affect the
perceived emotion of music, including cultural background, musical training,
gender, etc. Consensus on the perception of emotion is challenging
accordingly. Therefore, the annotations were made only among the first four
authors, all coming from similar cultural backgrounds and collaborating
closely during the annotation campaign, to ensure mutual standards for high or
low valence/arousal. As it is time-consuming and laborious to choose clips
from a song and label the emotion, each song was taken care of by only one
annotator. Yet, cross validation of the annotations among the annotators was
made several times during the annotation campaign (which spans 2.5 months), to
ensure all annotators work on the same standard.

Table 2 shows the number of clips and the average length (in seconds) for each
class. The clips amount to approximately 11 hours’ worth of data.

Quadrant | # clips |  Avg. length (in sec / #tokens)
---|---|---
Q1 | 250 | 31.9 / 1,065
Q2 | 265 | 35.6 / 1,368
Q3 | 253 | 40.6 / 771
Q4 | 310 | 38.2 / 729
Table 2: The number of clips and their average length in seconds, or in the
number of REMI tokens, for each quadrant of the Russell’s model in the EMOPIA
dataset.

###  3.3 Transcription

We transcribe the selected clips automatically with the help of the high-
resolution piano transcription model proposed by Kong _et al._ [21], which is
open source and represents the state-of-the-art for this task. We have
manually checked the transcription result for a random set of clips and find
the accuracy in note pitch, velocity, and duration satisfactory. The
transcription might be fragmented and undesirable for cases such as when the
audio recording is engineered to have unnatural ambient effects; we drop such
songs from our collection. The model also transcribes pedal information, which
we include to EMOPIA but do not use in our experiments.

Figure 2: Violin plots of the distribution in (a) note density, (b) length,
and (c) velocity for clips from different classes. Figure 3: Histogram of the
keys (left / right: major / minor keys) for clips from different emotion
classes.

###  3.4 Pre-processing and Encoding

For building machine learning models that deal with symbolic data, we need a
data representation that can be used as input to the models. For example,
MusicVAE [42] adopts the _event-based_ representation that encodes a symbolic
music piece as a sequence of “event tokens” such as note-on and note-off,
while MuseGAN [43] employs a timestep-based, _piano roll_ -like
representation. Since there is no standard on the symbolic representation thus
far, we adopt the following event-based ones in our experiments. Specifically,
we use MIDI-like and REMI in Section 5 and CP in Section 6. See Figure 1 for
illustrations.

  * •

The MIDI-like representation [39] encodes information regarding a MIDI note
with a “note-on” token, a “note-off”, and a “velocity” token. Moreover, the
“time shift” token is used to indicate the relative time gap (in ms) between
two tokens.

  * •

REMI [34] considers a beat-based representation that instead uses “bar” and
“subbeat” tokens to represent the time information. A “bar” token signifies
the beginning of a new bar, and “subbeat” points to one of the constant number
of subbeat divisions in a bar. Additionally, REMI uses “tempo” tokens to
control the pace of the music, and replaces “note-off” with “note-duration”.
Table 2 also shows the average number of REMI tokens for clips in each emotion
class.

  * •

CP [35]. Both MIDI-like and REMI view events as individual tokens. In CP,
tokens belonging to the same _family_ are grouped into a _super token_ and
placed on the same timestep (see Figure 1(c)). CP considers by default three
families: _metrical_ , _note_ , and _end-of-sequence_. We additionally
consider the “emotion” tokens and make it a new family, as depicted in Figure
1(d). The _prepending_ approach is motivated by CTRL [44], a state-of-the-art
controllable text generation model in natural language processing (NLP) that
uses global tokens to affect some overall properties of a sequence.

###  3.5 Dataset Availability

In EMOPIA, each sample is accompanied with its corresponding metadata,
segmentation annotations, emotion annotation, and transcribed MIDI, which are
all available in the Zenodo repository. Moreover, we have added the MIDI data
to the MusPy library [18] to facilitate its usage. Due to copyright issues,
however, we can only share audio through YouTube links instead of sharing the
audio files directl; the availability of the songs are subject to the
copyright licenses in different countries and whether the owners will remove
them.

Label pair Different arousal Different valence Q1 vs. Q4 Q2 vs. Q3 Q1 vs. Q2
Q3 vs. Q4 JS Div. 0.236 0.250 0.434 0.280

Table 3: Jensen-Shannon divergence between the key histograms of a few emotion
quadrant pairs.

##  4 Dataset Analysis

The emotion that listeners perceive is determined by a wide array of composed
and performed features of music [45]. To observe the emotional correlate of
the musical attributes in EMOPIA, we extract various MIDI-based features and
examine the distributions over the four quadrants of emotion. We present here
the most discriminative features among many choices we examined in our
analysis.

Note Density, Length, and Velocity. The arousal of music can be easily
observed based on the frequency of note occurrences and their strength [46].
We measure them by note density, length and velocity. The note density is
defined as the number of notes per beat, and the note length is defined as the
average note length in beat unit. The note velocity is obtained directly from
MIDI. Figure 2 shows three violin plots of the three features. The general
trend shows that the high-arousal group (Q1, Q2) and low-arousal group (Q3,
Q4) are distinguished well in all three plots. The results of note density and
velocity are expected considering the nature of arousal. However, it is quite
interesting that the note lengths are generally longer in the low-arousal
group (Q3, Q4). Between the same-arousal quadrants, the differences are
subtle. In note density, Q2 has more dynamics than Q1, whereas Q3 is not
distinguishable from Q4. In note length, Q1 has slightly longer notes than
those of Q2, whereas Q3 is again not distinguishable from Q4. In velocity, Q2
have louder notes than those of Q1, and Q3 has slightly louder notes than Q4.

Key Distribution. The valence of music is often found to be related to the
major-minor tonality [7]. For simplicity, we measure the tonality from the
musical key. We extract the key information using the Krumhansl-Kessler
algorithm [47] in the MIDI ToolBox [48]. Figure 3 shows the key distributions
on 12 major-minor pitch classes for the four emotion quadrants. They assure
that the major-minor tonality is an important clue in distinguishing valence.
In the high valence group (Q1, Q4), the distribution is skewed to the left
(major), while, in the low valence group (Q2, Q3), the trend is the opposite.
We also measured the distance between the key distributions using the Jensen-
Shannon divergence, which has the property of symmetry. Table 3 summarizes the
pair-wise distances, indicating that the valence difference group has a larger
difference than the arousal difference group.

##  5 Music Emotion Recognition

We report our baseline research on both symbolic- and audio-domain emotion
recognition using EMOPIA, which defines the task as classifying a song into
four categories. The clips in EMOPIA are divided into train-validation-test
splits with the ratio of 7:2:1 in a stratified manner.

Model | 4Q | A | V
---|---|---|---
Logistic regression | .581 | .849 | .651
LSTM-Attn [49]$+$MIDI-like [39] | .684 | .882 | .833
LSTM-Attn [49]$+$REMI[34] | .615 | .890 | .746
Table 4: Symbolic-domain classification performance. Model | 4Q | A | V
---|---|---|---
Logistic regression | .523 | .919 | .558
Short-chunk ResNet [50] | .677 | .887 | .704
Table 5: Audio-domain classification performance.

Symbolic-domain Classification. We evaluate two methods: one is based on hand-
crafted features with a simple classifier, and the other is on a deep neural
network model. For the former, we use the analysis features in the previous
section and a logistic regression classifier as a baseline. Specifically, we
use average values of note density, note length, velocity, and represent the
key as a one-hot vector. For the latter, we use two different symbolic note
representation methods introduced in Section 3.4, MIDI-like [39] and REMI
[34]. For the learning model, we use the combination of bidirectional LSTM and
a self-attention module, or _LSTM-Attn_ for short, proposed originally for
sentiment classification in NLP [49]. The LSTM extracts temporal information
from the MIDI note events, while the self-attention module calculates
different weight vectors over the LSTM hidden states with multi-head
attentions. The weighted hidden states are finally used for classification.

Audio-domain Classification. We evaluate two audio-domain classification
methods in a similar manner to the symbolic-domain ones. In the first method,
we use an average of 20 dimensions of mel-frequency cepstral coefficient
(MFCC) vectors and a logistic regression classifier. In the second method, we
use the short-chunk ResNet following [50], which is composed of 7-layer CNNs
with residual connections. The output is summarized as a fixed 128-dimensional
vector through max pooling, which is followed by two fully connected layers
with the ReLU activation for classification. The input audio to the short-
chunk ResNet is 3-second excerpts represented as a log-scaled mel-spectrogram
with 128 bins with 1024-size FFT (Hanning window), and 512 size hop at 22,050
Hz sampling rate. We randomly sample three seconds of the audio chunk as an
input size to the classifier.

Evaluation. We calculate 4-way classification accuracy over the four different
emotion quadrants and 2-way classification accuracy over either arousal and
valence.  Tables 4 and 5 show the results in the symbolic and audio domains,
respectively. Except for arousal, we can see that the deep learning approaches
generally outperform the logistic regression classifiers using hand-crafted
features. In audio domain arousal classification, MFCC vectors averaging the
entire song sequence showed better performance than the deep learning approach
with 3-second input. It seems that wider input sequence has the strength in
emotion recognition. In both domains, valence classification is a more
difficult task compared to arousal classification. For valence classification,
MIDI-domain classifiers yield better result than audio-domain classifiers
(0.883 vs. 0.704). Among the two token representations, MIDI-like seems to
outperform REMI for valence classification.

Model | Objective metrics | Subjective metrics
---|---|---
PR | NPC | POLY | 4Q | A | V | Humanness | Richness | Overall
EMOPIA (i.e., real data) | 51.0 | 8.48 | 5.90 | — | — | — | — | — | —
LSTM$+$GA [10] | 59.1 | 9.27 | 3.39 | .238 | .500 | .498 | 2.59$\pm$1.16 | 2.74$\pm$1.12 | 2.60$\pm$1.07
CP Transformer [35] | 53.4 | 9.20 | 3.48 | .418 | .690 | .583 | 2.61$\pm$1.03 | 2.81$\pm$1.03 | 2.78$\pm$1.03
CP Transformer w/ pre-training | 49.6 | 8.54 | 4.40 | .403 | .643 | .590 | 3.31$\pm$1.18 | 3.22$\pm$1.23 | 3.26$\pm$1.15
Table 6: Performance comparison of the evaluated models for emotion-
conditioned symbolic music generation in _surface-level objective metrics_
(Pitch Range, Number of Pitch Classes used, and POLYphony; the closer to that
of the real data the better), _emotion-related objective metrics_ (4Q
classification, Arousal classification, Valence classification; the higher the
better), and _subjective_ metrics (all in 1–5; the higher the better); bold
font highlights the best result per metric.

##  6 Emotion-conditioned Generation

We build the Transformer and LSTM models for emotion-conditioned symbolic
music generation using EMPOIA. For the former, we adopt the Compound Word
Transformer [35], the state-of-the-art in unconditioned symbolic music
generation. We employ the CP$+$emotion representation presented in Section 3.4
as the data representation.

For the LSTM model, we consider the approach proposed by Ferreira _et al._
[10], which represents the state-of-the-art in emotion-conditioned music
generation. Our implementation follows that described in [10], with the
following differences: 1) train on EMOPIA rather than VGMIDI; 2) use 512
neurons instead of 4,096 due to our limited computational resource; 3) use the
same linear logistic regression layer for classification but we classify four
classes instead of two.

As the size of EMOPIA might not be big enough, we use additionally the
_AILabs1k7_ dataset compiled by Hsiao _et al._ [35] to pre-train the
Transformer. AILabs1k7 contains 1,748 samples and is also pop piano music, but
it does not contain emotion labels. Most of the clips in AILabs1k7 are longer
than EMOPIA, so to keep the consistency of the input sequence length, the
length of the token sequence is set to be 1,024. We pre-train the Transformer
with 1e–4 learning rate on AILabs1k7, take the checkpoint with negative log-
likelihood loss 0.30, and then fine-tune it on EMOPIA with 1e–5 learning rate.
During pre-training, the emotion token is always set to be “ignore,” while in
fine-tuning it is set to the emotion of that sample.

Evaluation. We use the following three sets of metrics.

  * •

Surface-level objective metrics. We use the following three metrics proposed
by Dong _et al._ [43] to evaluate whether the generated samples fit the
training data: pitch range (PR), number of unique pitch classes used (NPC),
and number of notes being played concurrently (POLY). We use MusPy [18] to
compute these metrics.

  * •

Emotion-related objective metrics. Since both REMI and CP adopt a beat-based
representation of music, we employ the LSTM-Attn$+$REMI emotion classifier
(cf. Section 5) here to quantify how well the generation result is influenced
by the emotion condition. We first use the generative model to generate 100
samples per class, and use the assigned label as the target class of the
sample. The trained classifier is then used to make prediction on the
generated samples. Similar to the classification task, apart from 4Q
classification, we also conduct 2-way classification of both Arousal and
Valence aspect.

  * •

Subjective metrics. As the classifiers are not 100% accurate, we also resort
to a user survey to evaluate the emotion-controllability of the models.
Specifically, we deploy an online survey to collect responses to the music
generated by different models. A subject has to listen to 12 random-generated
samples, one for each of the three models and each of the four emotion
classes, and rate them on a five-point Likert scale with respect to 1)
Valence: is the audio negative or positive; 2) Arousal: is low or high in
arousal; 3) Humanness: how well it sounds like a piece played by human; 4)
Richness: is the content interesting; and, 5) Overall musical quality. In
total 25 subjects participated in the survey.

Table 6 tabulates some of the results. We see that the CP Transformer with
(‘w/’) pre-training performs the best in most of the objective metrics and the
three subjective metrics listed here. Nevertheless, the scores of the CP
Transformer with pre-training in the three emotion-related objective metrics
are much lower than that reported in Table 4, suggesting that either the
generated pieces are not emotion-laden, or the generated pieces are too
dissimilar to the real pieces to the classifier.

Figure 4: The subjective emotional scores (in 1–5) for the generative results
when the target emotion is (a) high valence, (b) low valence, (c) high
arousal, (d) low arousal.

Figure 4 shows the human assessment of the emotion-controllability of the
models. To our surprise, while the CP Transformer with pre-training does not
score high in the emotion related objective metrics, the subjective test shows
that it can actually control the emotion of the generated pieces to a certain
extent, better than the two competing models. In particular, the valence of
the samples generated by the Transformer with pre-training has a median rating
of 4 when the goal is to generate positive-valence music (i.e., Figure 4(a)),
while the scores of the other two models are around 3. Moreover, the arousal
of the samples generated by the Transformer with pre-training has a median
rating of 4 when the goal is to generate high-arousal music (Figure 4(c)),
which is higher than that of the non pre-trained Transformer. This suggests
that the LSTM-Attn classifier employed for computing the emotion-related
objective metrics may not be reliable enough to predict the emotion perceived
by human, and that the Transformer with pre-training is actually effective in
controlling the emotion of the music it generates to certain extent. But, the
model seems not good enough for the cases of generating low-valence (i.e.,
negative) music, as shown in Figure 4(b).

##  7 Conclusion

In this paper, we have proposed a new public dataset EMOPIA, a medium-scale
emotion-labeled pop piano dataset. It is a multi-modal dataset that contains
both the audio files and MIDI transcriptions of piano-only music, along with
clip-level emotion annotations in four classes. We have also presented
prototypes of models for clip-level music emotion classification and emotion-
based symbolic music generation trained on this dataset, using a number of
state-of-the-art models in respective tasks. The result shows that we are able
to achieve high accuracy in both four-quadrant and valence-wise emotion
classification, and that our Transformer-based model is capable of generating
music with a given target emotion to a certain degree.

In the future, in-depth importance analysis can be conducted to figure out
features that are important for emotion classification, and to seek ways to
incorporate those features to the generation model. Many ideas can also be
tried to further improve the performance of emotion conditioning, e.g., the
Transformer-GAN approach [51].

We share not only the dataset itself but the code covering all our implemented
models in a GitHub repo. We hope that researchers will find this contribution
useful in future emotion-related MIR tasks.

## References

  * [1] Y. Kim, E. Schmidt, R. Migneco, B. Morton, P. Richardson, J. Scott, J. Speck, and D. Turnbull, “Music emotion recognition: A state of the art review,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2010.
  * [2] Y.-H. Yang and H. H. Chen, _Music Emotion Recognition_. CRC Press, 2011.
  * [3] J. S. Gómez-Cañón, E. Cano, P. Herrera, and E. Gómez, “Joyful for you and tender for us: the influence of individual characteristics and language on emotion labeling and classification,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [4] J. S. Gómez-Cañón, E. Cano, Y.-H. Yang, P. Herrera, and E. Gómez, “Let’s agree to disagree: Consensus entropy active learning for personalized music emotion recognition,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2021.
  * [5] J. de Berardinis, A. Cangelosi, and E. Coutinho, “The multiple voices of musical emotions: Source separation for improving music emotion recognition models and their interpretability,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [6] R. Panda, R. Malheiro, B. Rocha, A. Oliveira, and R. P. Paiva, “Multi-modal music emotion recognition: A new dataset, methodology and comparative analysis,” in _Proc. Int. Symposium on Computer Music Multidisciplinary Research_ , 2013.
  * [7] R. Panda, R. M. Malheiro, and R. P. Paiva, “Audio features for music emotion recognition: A survey,” _IEEE Trans. Affective Computing_ , pp. 1–20, 2020.
  * [8] S. Chowdhury, A. Vall, V. Haunschmid, and G. Widmer, “Towards explainable music emotion recognition: The route via mid-level features,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2019.
  * [9] C. Cancino-Chacón, S. Peter, S. Chowdhury, A. Aljanaki, and G. Widmer, “On the characterization of expressive performance in classical music: First results of the con espressione game,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [10] L. N. Ferreira and J. Whitehead, “Learning to generate music with sentiment,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2019.
  * [11] K. Chen, C.-I. Wang, T. Berg-Kirkpatrick, and S. Dubnov, “Music SketchNet: Controllable music generation via factorized representations of pitch and rhythm,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020\.
  * [12] D. Makris, K. R. Agres, and D. Herremans, “Generating lead sheets with affect: A novel conditional seq2seq framework,” in _Proc. Int. Joint Conf. Neural Networks_ , 2021.
  * [13] R. Madhok, S. Goel, and S. Garg, “SentiMozart: Music generation based on emotions,” in _Proc. Int. Conf. Agents and Artificial Intelligence_ , 2018\.
  * [14] A. M. Grimaud and T. Eerola, “EmoteControl: an interactive system for real-time control of emotional expression in music,” _Personal and Ubiquitous Computing_ , 2020.
  * [15] T.-P. Chen, S. Fukayama, M. Goto, and L. Su, “Chord Jazzification: Learning Jazz interpretations of chord symbols,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [16] H. H. TAN and D. Herremans, “Music Fadernets: Controllable music generation based on high-level features via low-level feature modelling,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [17] Z. Wang, K. Chen, J. Jiang, Y. Zhang, M. Xu, S. Dai, X. Gu, and G. Xia, “POP909: A pop-song dataset for music arrangement generation,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [18] H.-W. Dong, K. Chen, J. McAuley, and T. Berg-Kirkpatrick, “MusPy: A toolkit for symbolic music generation,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [19] A. McLeod, J. Owers, and K. Yoshii, “The MIDI Degradation Toolkit: Symbolic music augmentation and correction,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2020.
  * [20] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and Frames: Dual-objective piano transcription,” _arXiv preprint arXiv:1710.11153_ , 2018.
  * [21] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onsets and offsets times,” _arXiv preprint arXiv:2010.01815_ , 2020.
  * [22] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in _Proc. Int. Conf. Learning Representations_ , 2019.
  * [23] K. W. Cheuk, D. Herremans, and L. Su, “ReconVAT: A semi-supervised automatic music transcription framework for low-resource real-world data,” in _Proc. ACM Multimedia_ , 2021.
  * [24] D. Bognadov, A. Porter, P. Tovstogan, and M. Won, “Mediaeval 2019: Emotion and theme recognition in music using Jamendo,” in _MediaEval 2019 Workshop_.
  * [25] A. Alajanki, Y.-H. Yang, and M. Soleymani, “Benchmarking music emotion recognition systems,” _PLOS ONE_ , 2016.
  * [26] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2017.
  * [27] R. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. Bello, “MedleyDB: A multitrack dataset for annotation-intensive mir research,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2014.
  * [28] J. Fan, M. Thorogood, and P. Pasquier, “Emo-Soundscapes: A dataset for soundscape emotion recognition,” in _Proc. Int. Conf. Affective Computing and Intelligent Interaction_ , 2017.
  * [29] J. Fan, Y.-H. Yang, K. Dong, and P. Pasquier, “A comparative study of Western and Chinese classical music based on soundscape models,” in _Proc. Int. Conf. Acoustics, Speech, and Signal Processing_ , 2020.
  * [30] M. Soleymani, M. Caro, E. Schmidt, C.-Y. Sha, and Y.-H. Yang, “1000 songs for emotional analysis of music,” in _Proc. ACM Int. Workshop on Crowdsourcing for Multimedia_ , 2013.
  * [31] J. Fan, K. Tatar, M. Thorogood, and P. Pasquier, “Ranking-based emotion recognition for experimental music,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2017.
  * [32] J. Russell, “A circumplex model of affect,” _Journal of Personality and Social Psychology_ , vol. 39, pp. 1161–1178, December 1980.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. Advances in Neural Information Processing Systems_ , 2017.
  * [34] Y.-S. Huang and Y.-H. Yang, “Pop Music Transformer: Beat-based modeling and generation of expressive pop piano compositions,” in _Proc. ACM Multimedia_ , 2020.
  * [35] W.-Y. Hsiao, J.-Y. Liu, Y.-C. Yeh, and Y.-H. Yang, “Compound Word Transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in _Proc. AAAI Conf. Artificial Intelligence_ , 2021.
  * [36] J. Grekow and Z. W. Raś, “Detecting emotions in classical music from MIDI files,” in _Proc. Int. Symposium on Methodologies for Intelligent Systems_ , 2009.
  * [37] Y. Lin, X. Chen, and D. Yang, “Exploration of music emotion recognition based on MIDI,” in _Proc. Int. Society for Music Information Retrieval Conf._ , 2013.
  * [38] K. Zhao, S. Li, J. Cai, H. Wang, and J. Wang, “An emotional symbolic music generation system based on LSTM networks,” in _Proc. IEEE Information Technology, Networking, Electronic and Automation Control Conf._ , 2019.
  * [39] S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Simonyan, “This time with feeling: Learning expressive musical performance,” _Neural Computing and Applications_ , 2018.
  * [40] C. Cannam, C. Landone, and M. Sandler, “Sonic Visualiser: An open source application for viewing, analysing, and annotating music audio files,” in _Proc. ACM Multimedia_ , 2010, pp. 1467–1468, https://www.sonicvisualiser.org/.
  * [41] L.-L. Balkwill and W. Thompson, “A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues,” _Music Perception_ , vol. 17, pp. 43–64, October 1999.
  * [42] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” in _Proc. Int. Conf. Machine Learning_ , 2018.
  * [43] H.-W. Dong, W.-Y. Hsiao, L.-C. Yang, and Y.-H. Yang, “MuseGAN: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment,” in _Proc. AAAI Conf. Artificial Intelligence_ , 2018.
  * [44] N. S. Keskar _et al._ , “CTRL - a conditional Transformer language model for controllable generation,” _arXiv preprint arXiv:1909.05858_ , 2019\.
  * [45] P. Juslin and E. Lindstorm, “Musical expression of emotions: Modelling listeners’ judgements of composed and performed features,” _Music Analysis_ , vol. 29, pp. 334 – 364, 2011.
  * [46] S. R. Livingstone, R. Muhlberger, A. R. Brown, and W. F. Thompson, “Changing musical emotion: A computational rule system for modifying score and performance,” _Computer Music Journal_ , vol. 34, no. 1, pp. 41–64, 2010\.
  * [47] C. L. Krumhansl, _Cognitive foundations of musical pitch_. Oxford University Press, 2001.
  * [48] T. Eerola and P. Toiviainen, _MIDI Toolbox: MATLAB Tools for Music Research_. Jyväskylä, Finland: University of Jyväskylä, 2004. [Online]. Available: www.jyu.fi/musica/miditoolbox/
  * [49] Z. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, “A structured self-attentive sentence embedding,” in _Proc. Int. Conf. Learning Representations_ , 2017.
  * [50] M. Won, A. Ferraro, D. Bogdanov, and X. Serra, “Evaluation of CNN-based automatic music tagging models,” in _Proc. Sound and Music Conf._ , 2020\.
  * [51] A. Muhamed _et al._ , “Symbolic music generation with Transformer-GANs,” in _Proc. AAAI Conf. Artificial Intelligence_ , 2021\.


Null Angular Momentum and Weak KAM Solutions

Null Angular Momentum and Weak KAM Solutions
of the Newtonian $\boldsymbol{N}$-Body Problem

Boris A. PERCINO-FIGUEROA

B.A. Percino-Figueroa

Facultad de Ciencias en Física y Matemáticas, Universidad Autónoma de Chiapas,
México<EMAIL_ADDRESS>
Received November 09, 2016, in final form August 16, 2017; Published online
August 24, 2017

In [Arch. Ration. Mech. Anal. 213 (2014), 981–991] it has been proved that in
the Newtonian $N$-body problem, given a minimal central configuration $a$ and
an arbitrary configuration $x$, there exists a completely parabolic orbit
starting on $x$ and asymptotic to the homothetic parabolic motion of $a$,
furthermore such an orbit is a free time minimizer of the action functional.
In this article we extend this result in abundance of completely parabolic
motions by proving that under the same hypothesis it is possible to get that
the completely parabolic motion starting at $x$ has zero angular momentum. We
achieve this by characterizing the rotation invariant weak KAM solutions as
those defining a lamination on the configuration space by free time minimizers
with zero angular momentum.

$N$-body problem; angular momentum; free time minimizer; Hamilton–Jacobi
equation 37J15; 37J50; 70F10; 70H20

##  1 Introduction

###  1.1 Preliminaries

Let $E=\mathbb{R}^{d}$ be the $d$-dimensional Euclidean space, $d\geq 2$, and
consider the $N$-body problem with Newtonian potential function $U\colon
E^{N}\to{}]0,+\infty]$,

$\displaystyle U(x)=\sum_{i<j}\frac{m_{i}m_{j}}{r_{ij}},$

where $x=(r_{1},\ldots,r_{N})\in E^{N}$ is a configuration of $N$ points
having positive masses $m_{1},\ldots,m_{N}$ in $E$, and
$r_{ij}=|r_{i}-r_{j}|$. Here $|\cdot|$ denotes the Euclidean norm. We will
denote by $\|\cdot\|$ to the norm induced by the mass inner product given by

$\displaystyle x\cdot y=\sum_{i=1}^{N}m_{i}\langle r_{i},s_{i}\rangle,$

with $x=(r_{1},\ldots,r_{N})$, $y=(s_{1},\ldots,s_{N})\in E^{N}$, and
$\langle\cdot,\cdot\rangle$ is the Euclidean inner product. We also introduce
the moment of inertia:

$\displaystyle I(x)=\|x\|^{2}.$

In this section we introduce the variational setting of the problem. The
Lagrangian function $L\colon E^{2N}\to{}]0,\infty]$ is given by

$\displaystyle
L(x,v)=\frac{1}{2}I(v)+U(x)=\frac{1}{2}\sum_{i=1}^{N}m_{i}|v_{i}|^{2}+U(x)$

and the action of an absolutely continuous curve $\gamma\colon[a,b]\to E^{N}$
by

$\displaystyle A_{L}(\gamma)=\int_{a}^{b}L(\gamma(t),\dot{\gamma}(t)){\rm
d}t,$

so that the solutions of the problem are the critical points of the action
functional.

For two given configurations $x,y\in E^{N}$, we will consider minima taken
over the set $\mathcal{C}(x,y)$ of absolutely continuous curves binding $x$
and $y$ without any restriction on time,

$\displaystyle\mathcal{C}(x,y):=\bigcup_{\tau>0}\big{\\{}\gamma\colon[a,b]\to
E^{N}\text{ absolutely continuous
}b-a=\tau,\,\gamma(a)=x,\,\gamma(b)=y\big{\\}}.$

The Mañé critical action potential $\phi\colon E^{n}\times
E^{n}\to[0,+\infty)$ is defined as

$\displaystyle\phi(x,y):=\inf\\{A(\gamma)\,|\,\gamma\in\mathcal{C}(x,y)\\}.$

Here the infimum is achieved if and only if $x\neq y$, this is essentially due
to the lower semi-continuity of the action. Marchal’s theorem asserts that
minimizers avoid collisions in the interior of their interval of definition
[3, 6, 9].

Let us define the set of configurations without collision

$\displaystyle\Omega:=\big{\\{}x=(r_{1},\dots,r_{N})\in E^{N}\,|\,\text{if
}r_{i}=r_{j},\text{ then }i=j\big{\\}},$ (1)

and let $M:=m_{1}+\dots+m_{N}$ the total mass of the system.

Given a configuration $x=(r_{1},\dots,r_{N})\in E^{N}$, the center of mass of
$x$ is defined by

$\displaystyle G(x)=\frac{1}{M}\sum_{i=1}^{N}m_{i}r_{i},$

it is a standard fact that if $\gamma\colon J\to E^{N}$ is a minimizer of the
action, then $G(\gamma(t))$ has constant velocity.

######  Definition 1.1.

A free time minimizer defined on an interval $J\subset\mathbb{R}$ is an
absolutely continuous curve $\gamma\colon J\to E^{N}$ which satisfies
$A(\gamma|_{[a,b]})=\phi(\gamma(a),\gamma(b))$ for all compact subinterval
$[a,b]\subset J$.

An important example is given by A. Da Luz and E. Maderna in [5] where they
proved that if $a$ is a  _minimal configuration_ , i.e., a minimum of the
potential restricted to the sphere $I(x)=1$, then the parabolic homothetic
ejection with central configuration $a$ is a free time minimizer. It is not
known if there are other central configurations with this property. A. Da Luz
and E. Maderna also proved that free time minimizers cannot be defined in the
whole line. On the other hand it is also proved that this minimizers are
completely parabolic motions.

###  1.2 Weak KAM solutions

Weak KAM solutions of the Hamilton–Jacobi equation related to the problem are
useful tools to study free time minimizers. The Hamiltonian associated to the
problem is given by

$\displaystyle H(x,p)=\tfrac{1}{2}||p||_{*}^{2}-U(x),$

where $||p||_{*}=\sum\limits_{i=1}^{N}m_{i}^{-1}|p|$.

######  Definition 1.2.

A weak KAM solution of the Hamilton–Jacobi equation

$\displaystyle\|Du(x)\|_{*}^{2}=2U(x)$ (2)

is a function $u\colon E^{N}\to\mathbb{R}$ that satisfies the following
conditions:

  * •

$u$ is dominated, i.e., $u(y)-u(x)\leq\phi(x,y)$ for all $x,y\in E^{N}$,

  * •

for any $x\in E^{N}$ there is an absolutely continuous curve
$\alpha\colon[0,\infty)\to E^{N}$ such that $\alpha(0)=x$ and $\alpha$
calibrates $u$, i.e., $u(x)-u(\alpha(t))=A(\alpha|_{[0,t]})$ for any $t>0$.

Notice that calibrating curves of weak KAM solutions are indeed free time
minimizers. On the other hand, existence of weak KAM solution is proved in
[7], where solutions are characterized as fixed points of the so called
Lax–Oleinik semigroup. A variety of weak KAM solutions is also obtained by
means of Busemann functions used in Riemannian geometry and introduced in weak
KAM theory by G. Contreras [4] in the case of regular Hamiltonians; in [11] it
is proved the following proposition

######  Proposition 1.3.

Let $a$ a minimal central configuration with $\|a\|=1$, define $U(a)=U_{0}$
and $c:=\big{(}\frac{9}{2}U_{0}\big{)}^{\frac{1}{3}}$, consider the parabolic
homothetic ejection with central configuration $a$ given by
$\gamma_{0}(t)=ct^{\frac{2}{3}}a$. Then the Busemann function

$\displaystyle
u_{a}(x)=\lim_{t\to+\infty}[\phi(x,\gamma_{0}(t))-\phi(0,\gamma_{0}(t))]$ (3)

is a weak KAM solution of the Hamilton–Jacobi equation (2). Moreover, for any
$x\in E^{N}$ there is a curve $\alpha\colon[0,\infty)\to E^{N}$ with
$\alpha(0)=x$ that calibrates $u$ and

$\displaystyle\lim_{t\to+\infty}\big{\|}\alpha(t)t^{-\frac{2}{3}}-cx_{0}\big{\|}=0.$

A solution defined by identity (3), will be called _Busemann solution_. It is
an open problem to determine if there are central configurations, different
from minimal configurations, defining Busemann solutions.

On the other hand, due to the symmetries of the potential function, it is
interesting to determine if weak KAM solutions are invariant under this
symmetries. In the case of translation invariance, E. Maderna proved in [8]
that given a weak KAM solution $u$ of (2), then

$\displaystyle u(r_{1},\dots,r_{N})=u(r_{1}+r,\dots,r_{N}+r)$

for any configuration $x=(r_{1},\dots,r_{N})\in E^{N}$ and every $r\in E$. The
proof is achieved by showing that calibrating curves of weak KAM solutions
have constant center of mass.

An important question is to determine if weak KAM solutions are rotation
invariant, the main goal of this article is to study this problem. Notice
that, there are solutions which are not rotation invariant, Busemann solutions
given in (3) for instance. Therefore the problem is to give conditions so that
a weak KAM solution is rotation invariant; we achieve this goal by studying
the angular momentum for the calibrating curves of rotation invariant
solutions and characterizing invariant solutions as those where calibrating
curves have zero angular momentum. We obtain rotation invariant solutions by
setting

$\displaystyle\hat{u}_{a}=\inf_{R\in{\rm SO}(d)}u_{Ra}(x),$

where $a$ is a minimal central configuration and $u_{Ra}(x)$ is the Busemann
function associated to $Ra$.

###  1.3 Main theorems

We consider the diagonal group action on $E^{N}$ defined by the special
orthogonal group ${\rm SO}(d)$, more precisely, the rotation on $E^{N}$ by an
element $\theta\in{\rm SO}(d)$ is

$\displaystyle R_{\theta}\colon\ E^{N}\to E^{N},\qquad
x=(r_{1},\dots,r_{N})\mapsto(\theta r_{1},\dots,\theta r_{N}),$

where $\theta r_{i}$ is the usual group action of ${\rm SO}(d)$ on $E$.

The _Angular momentum_ is a first integral closely related to the action of
${\rm SO}(d)$ on $E^{N}$. If $x=(r_{1},\dots,r_{N})\in E^{N}$ and a vector
$v=(v_{1},\dots,v_{N})\in E^{N}$ the angular momentum $C(x,v)$ is defined as

$\displaystyle C(x,v)=\sum_{j=1}^{N}m_{j}r_{j}\wedge v_{j}.$

If $d=3$ the $\wedge$ product becomes the usual cross product in $E$. If
$d=2$, by identifying $\mathbb{R}^{2}$ with $\mathbb{C}$, if
$x,v\in\mathbb{C}$ then $r\wedge v=\operatorname{Im}(v\bar{r})$, and $r\wedge
v$ is a real number.

Let $u\colon E^{N}\to\mathbb{R}$ a continuous function, we say that $u$ is
_rotation invariant_ if for any $x\in E^{N}$ and any $\theta\in{\rm SO}(d)$ we
have

$\displaystyle u(R_{\theta}(x))=u(x).$

We have the following characterization of invariant weak KAM solutions of (2)
in terms of the angular momentum of their calibrating curves.

######  Theorem 1.4.

Let $u$ be a weak KAM solution of the Hamilton–Jacobi equation

$\displaystyle\|Du(x)\|_{*}^{2}=2U(x).$

Then $u$ is rotation invariant if and only if all of its calibrating curves
have zero angular momentum. That is to say, for any $\theta\in{\rm SO}(d)$ and
any $x\in E^{N}$, the identity

$\displaystyle u(x)=u(R_{\theta}x)$

holds if and only if for any $\gamma\colon[0,+\infty[{}\to E^{N}$ calibrating
$u$, we have

$\displaystyle C(\gamma(t),\dot{\gamma}(t))=0.$

We can give a more general result by considering $G$ a Lie group acting
properly on $E$. Thus we can consider the diagonal action $S\colon G\times
E^{N}\to E^{N}$ of $G$ on $E^{N}$, defined by

$\displaystyle S_{g}x=(gr_{1},gr_{2},\dots,gr_{N}),$

where $g\in G$, $x=(r_{1},\dots,r_{N})\in E^{N}$ and $gr$ is the action of $g$
on $r$. Let us denote by $\mathfrak{g}$ to the Lie algebra of $G$. We denote
by $[\,,\,]$ to the pairing between $\mathfrak{g}$ and $\mathfrak{g}^{*}$.

Notice that the action $S$ can be lifted to $E^{N}\times E^{N}$ by
$g(x,v)=(S_{g}x,T_{x}S_{g}v)$ where $T_{x}S_{g}$ is the differential of
$S_{g}$ at $x$. Assume that the Lagrangian is $G$-invariant, i.e., $g^{*}L=L$,
$g\in G$ and assume also that the action lifts to $E^{N}\times E^{N}$ by
isometries of the mass inner product.

Under such conditions, the group action defines an _equivariant momentum map_

$\displaystyle\mu\colon\ E^{N}\times E^{N}\to\mathfrak{g}^{*},$

given by

$\displaystyle[\mu(x,v),\xi]=v\cdot X_{\xi}(x),$ (4)

where $X_{\xi}(x)=\frac{{\rm d}}{{\rm d}t}\bigr{|}_{t=0}S_{\exp(t\xi)}x$ is
the infinitesimal generator of the one-parameter subgroup action on $E^{N}$,
associated to $\xi\in\mathfrak{g}$.

A continuous function $u\colon E^{N}\to\mathbb{R}$ is _$G$ -invariant_ if for
any $g\in G$ and any $x\in E^{N}$ we have

$\displaystyle u(S_{g}x)=u(x).$

In a similar way to Theorem 1.4 we can give a characterization to
$G$-invariant weak KAM solutions in terms of the equivariant momentum map.

######  Theorem 1.5.

Let $G$ a connected Lie group acting diagonally on $E^{N}$ and suppose that
the group action satisfies the assumptions above, and let $u$ be a weak KAM
solution of the Hamilton–Jacobi equation (2). Then $u$ is $G$-invariant if and
only if for any $\gamma\colon[0,+\infty[{}\to E^{N}$ that calibrates $u$, for
any $t>0$, we have

$\displaystyle\mu(\gamma(t),\dot{\gamma}(t))=0.$

##  2 Rotation invariance

Given $x\in E^{N}$, consider the orbit of $x$ under ${\rm SO}(d)$ given by

$\displaystyle M_{x}:=\\{R_{\theta}x\,|\,\theta\in{\rm SO}(d)\\},$

let us remind that

$\displaystyle T_{x}M_{x}=\\{Ax\,|\,A\in\mathfrak{so}(d)\\}.$

The key point in the proof of Theorem 1.4 is the Saari decomposition of the
velocities [3, 12]. Define

$\displaystyle\mathcal{H}_{x}:=\big{\\{}v\in E^{N}\,|\,C(x,v)=0\big{\\}}.$

Then $T_{x}M_{x}\perp\mathcal{H}_{x}$, with respect to the mass scalar product
and

$\displaystyle E^{N}=T_{x}M_{x}\oplus\mathcal{H}_{x}.$

In other words, if $v\in E^{N}$, then $v$ can be decomposed as

$\displaystyle v=v_{r}+v_{h},$

where $v_{r}\in T_{x}M_{x}$, $C(x,v_{h})=0$ and $v_{r}\cdot v_{h}=0$, moreover
the components $v_{r}$ and $v_{h}$ are uniquely determined by $v$. For
dimensions 2 and 3 this is a direct consequence of the properties of the cross
product, for dimensions $\geq 4$ it is due to the properties of the “wedge”
product.

On the other hand notice that if $u$ is a rotation invariant function and
$x\in E^{N}$, then $u$ is constant on $M_{x}$. Thus, if $x$ is a point of
differentiability of $u$, we have that

$\displaystyle T_{x}M_{x}\subset\ker{\rm d}_{x}u.$

######  Proof 2.1 (Proof of Theorem 1.4).

Let $u$ be a rotation invariant weak KAM solution and let $x\in E^{N}$,
consider a curve $\gamma\colon[0,+\infty[{}\to E^{N}$ calibrating $u$ and
starting at $x$. It is known that $u$ is differentiable at $\gamma(t)$ for any
$t>0$, we also know that

$\displaystyle{\rm d}_{\gamma(t)}u(w)=w\cdot\dot{\gamma}(t),$ (5)

for all $w\in E^{N}$.

On the other hand, by the previous remark we have that
$T_{\gamma(t)}M_{\gamma(t)}\subset\ker{\rm d}_{\gamma(t)}u$ and from (5) we
get $\dot{\gamma}(t)\perp T_{\gamma(t)}M_{\gamma(t)}$, therefore
$\dot{\gamma}(t)\in\mathcal{H}_{\gamma(t)}$, then

$\displaystyle C(\gamma(t),\dot{\gamma}(t))=0$

for all $t>0$.

Let us consider now a weak KAM solution $u$ such that all of its calibrating
curves have zero angular momentum. Let $x\in E^{N}$ and let $\theta\in{\rm
SO}(d)$. We will prove that $u(R_{\theta}x)=u(x)$.

Clearly if $R_{\theta}x=x$, the result follows trivially. Suppose that
$R_{\theta}x\neq x$, since $u$ is continuous and the set of collisionless
configurations $\Omega$, given in (1), is open, dense and rotation invariant,
we can assume $x\in\Omega$.

Since ${\rm SO}(d)$ is compact, $\exp\colon\mathfrak{so}(d)\to{\rm SO}(d)$ is
surjective, thus we can take $\omega\in\mathfrak{so}(d)$ such that
$\exp(\omega)=\theta$. Define the curve $\alpha\colon[0,1]\to E^{N}$ by
$\alpha(t)=R_{\exp(t\omega)}(x)$. Let $\varepsilon>0$ be small enough so that

$\displaystyle B:=\big{\\{}z\in E^{N}\,|\,\langle
z,\dot{\alpha}(0)\rangle=0,\,\|z-x\|<\varepsilon\big{\\}}\subset\Omega.$

Notice that the set $W:=\\{z\in E^{N}\,|\,R_{\theta}z\neq z\\}$ is open,
therefore we can also choose $\varepsilon>0$, smaller if necessary, so that
$B\subset W$.

We can assume that $\varepsilon$ is sufficiently small so that the map

$\displaystyle B\times[0,1]\to E^{N},\qquad(z,t)\mapsto R_{\exp(tw)}z$ (6)

is a diffeomorphism onto its image. Indeed, If
$\alpha(t)=R_{\exp(t\omega)}(x)\neq x$ for every $t\in(0,1]$, then the curve
$\alpha$ is an embedding and, by choosing $\varepsilon$ sufficiently small,
the map (6) is a diffeomorphism onto its image. Suppose, on the contrary, that
$\alpha(\tau)=x$ for some $\tau\in(0,1]$, then $\alpha$ is $\tau$-periodic.
Let $\tau$ be the minimal period of $\alpha$, then there exists $s\in(0,\tau)$
such that $\alpha(s)=R_{\exp(s\omega)}(x)=R_{\exp(\omega)}(x)$ and $\alpha$
has no self intersections in the interval $[0,s]$. Replacing $\omega$ by
$s\omega$, and choosing $\varepsilon$ sufficiently small we get as before that
(6) is a diffeomorphism onto its image.

Define

$\displaystyle C:=\big{\\{}R_{\exp(t\omega)}(z)\,|\,z\in
B,\,t\in[0,1]\big{\\}}$

and denote by $C^{\prime}$ to the set point in $C$ where $u$ is
differentiable. Since $u$ is dominated, $u$ is Lipschitz in $C$ and therefore
$C^{\prime}$ has total measure in $C$. Notice also that $C\subset\Omega$.

Let $y\in C^{\prime}$ and let $\gamma_{y}\colon[0,+\infty[{}\to E^{N}$ be a
calibrating curve such that $\gamma(0)=y$, then ${\rm
d}_{y}u(w)=w\cdot\dot{\gamma}_{y}(0)$ for any $w\in E^{N}$. From the
hypothesis $\dot{\gamma}_{y}(0)\in\mathcal{H}_{q}$, thus
$\dot{\gamma}_{y}(0)\perp T_{y}M_{y}$, thus $T_{y}M_{y}\subset\ker{\rm
d}_{y}u$ for any $y\in C^{\prime}$.

Let $f\colon B\times[0,1]\to\mathbb{R}$ the Lipschitz continuous function
given by

$\displaystyle f(z,t)=u\big{(}R_{\exp(t\omega)}(z)\big{)}-u(z).$

Since $T_{y}M_{y}\subset\ker{\rm d}_{y}u$, we get $\frac{\partial f}{\partial
t}=0$ almost everywhere, by Fubini theorem in $A\times[0,1]$, with $A$ any
open subset of $B$

$\displaystyle 0=\int_{A}\int_{[0,1]}\frac{\partial f}{\partial t}{\rm d}t{\rm
d}y=\int_{A}f(y,1){\rm d}y,$

therefore $f(y,1)=0$ for every $y\in B$, in particular we have

$\displaystyle u(R_{\theta}(x))=u(x).$ ∎

######  Remark 2.2.

Let us denote by $\mathcal{S}$ to the set of weak KAM solutions of (2) and
notice that if $\mathcal{U}\subset\mathcal{S}$ is such that
$\underset{u\in\mathcal{U}}{\inf}u(x)>-\infty$, then

$\displaystyle\tilde{u}(x)=\underset{u\in\mathcal{U}}{\inf}\\{u(x)\,|\,u\in\mathcal{U}\\}$

is in $\mathcal{S}$. This is due to the fact that weak KAM solutions are the
fixed points of the Lax–Oleinik semigroup [7].

######  Corollary 2.3.

Let $a$ be a minimal central configuration with $I(a)=1$. For any
$\theta\in{\rm SO}(d)$, let $u_{R_{\theta}a}$ be the Busemann solution
associated to the minimal central configuration $R_{\theta}a$. Then the
function

$\displaystyle\hat{u}_{a}\colon\
E^{N}\to\mathbb{R},\qquad\hat{u}_{a}(x):=\inf_{\theta\in{\rm
SO}(d)}u_{\theta,a}(x)$

is a rotation invariant weak KAM solution of the Hamilton–Jacobi equation.
Therefore the callibrating curves of $\hat{u}_{a}$ are free time minimizers
having zero angular momentum.

######  Proof 2.4.

Let $\mathcal{M}$ be the set of minimal central configurations with moment of
inertia one. Let $a\in\mathcal{M}$ and $\theta\in{\rm SO}(d)$ let

$\displaystyle u_{\theta,a}(x)=u_{a}\circ R_{\theta^{-1}}(x),$

it is not hard to see that $u_{\theta,a}$ is also a weak KAM solution,
furthermore, notice that $u_{R_{\theta}a}=u_{\theta,a}$, thus

$\displaystyle\hat{u}_{a}(x)=\inf_{\theta\in{\rm SO}(d)}u_{\theta,a}(x),$ (7)

and from the previous remark the function on the right is a weak KAM solution.

Due to (7), $\hat{u}_{a}$ is rotation invariant and from Theorem 1.4, these
solutions define laminations by free time minimizer with zero angular
momentum.

Given a minimal central configuration $a$, notice that the rotation invariant
weak KAM solution $\hat{u}_{a}$ given in the previous Corollary in uniquely
determined by $M_{a}$, the orbit of $a$ under ${\rm SO}(d)$, we call this
solution _invariant Busemann solution_ associated to $M_{a}$.

##  3 $\boldsymbol{G}$-invariance

Let $G$ be a connected Lie group acting on $E^{N}$ with the assumptions of
Section 1.3, let us notice that in this setting, due to (4) the equivariant
momentum map defines a Saari decomposition of the velocity (see [1, 2, 10]),
as follows.

For a fixed momentum value, $\mu(x,v)=\mu$, there are orthogonal vectors
$v_{\mathcal{H}}$ and $v_{\mathcal{V}}$ such that

$\displaystyle
v=v_{\mathcal{H}}+v_{\mathcal{V}},\qquad\mu(x,v_{\mathcal{V}})=\mu\qquad\text{and}\qquad\mu(x,v_{\mathcal{H}})=0.$

Let $x\in E^{N}$ and let $G_{x}$ be the orbit of $x$ under the $G$-action.
Consider the subspaces of $T_{x}E^{N}=E^{N}$

$\displaystyle\mathcal{H}{\rm or}_{x}=\big{\\{}v\in
E^{N}\,|\,\mu(x,v)=0\big{\\}}\qquad\text{and}\qquad T_{x}G_{x},$

then these subspaces are orthogonal with respect to the mass inner product and

$\displaystyle E^{N}=\mathcal{H}{\rm or}_{x}\oplus T_{x}G_{x}.$

Thus, any $v\in E^{N}$ can be uniquely decomposed as

$\displaystyle v=v_{\mathcal{H}}+v_{\mathcal{V}},$ (8)

where $v_{\mathcal{H}}\in\mathcal{H}{\rm or}_{x}$, $v_{\mathcal{V}}=X_{\xi}\in
T_{x}G_{x}$, and $\xi\in\mathfrak{g}$ is the a element such that
$\mu(x,v)=\mu(x,X_{\xi})$.

Finally notice that if $u$ is a $G$-invariant function and $x$ a point of
differentiability of $u$, then $T_{x}G_{x}\subset{\rm d}_{x}u$.

######  Proof 3.1 (Proof of Theorem 1.5).

The main difficulty is the surjectivity of the exponential map, nevertheless
it can be avoided as follows. Since $G$ is connected, it is well known that
for any $g\in G$, there exists $\xi_{1},\dots,\xi_{n}\in\mathfrak{g}$ such
that $g=\exp(\xi_{1})\cdots\exp(\xi_{n})$. Therefore, if we can prove that

$\displaystyle u\big{(}S_{\exp(\xi)}x\big{)}=u(x)$ (9)

for any $x\in E^{n}$ and any $\xi\in\mathfrak{g}$, we get that
$u(S_{g}x)=u(x)$ for any $x\in E^{n}$ and any $g\in G$. Given the Saari
decomposition of the velocities (8), the proof of (9) follows, as the one of
Theorem 1.4

We can apply Theorem 1.5 to any connected subgroup $G\subset{\rm SO}(d)$
getting that a solution is $G$-invariant if and only if the corresponding
component of the angular momentum of the calibrating curves, in the direction
of $\mathfrak{g}^{*}$ is null at any instant of the motion.

### Acknowledgements

The author acknowledges the referees for their valuable suggestions and
remarks that substantially improved this article. The author is grateful to
Ezequiel Maderna for his advice, to Héctor Sánchez Morgado for his suggestions
and to Eddaly Guerra Velasco for her support in the process of this research.

## References

  * [1] Abraham R., Marsden J.E., Raţiu T.S., Manifolds, tensor analysis, and applications, Global Analysis Pure and Applied: Series B, Vol. 2, Addison-Wesley Publishing Co., Reading, Mass., 1983.
  * [2] Arnol’d V.I. (Editor), Dynamical systems. VIII, Encyclopaedia of Mathematical Sciences, Vol. 39, Springer-Verlag, Berlin, 1993.
  * [3] Chenciner A., Action minimizing solutions of the Newtonian $n$-body problem: from homology to symmetry, in Proceedings of the International Congress of Mathematicians, Vol. III (Beijing, 2002), Higher Ed. Press, Beijing, 2002, 279–294.
  * [4] Contreras G., Action potential and weak KAM solutions, Calc. Var. Partial Differential Equations 13 (2001), 427–458.
  * [5] da Luz A., Maderna E., On the free time minimizers of the Newtonian $N$-body problem, Math. Proc. Cambridge Philos. Soc. 156 (2014), 209–227, arXiv:1301.7034.
  * [6] Ferrario D.L., Terracini S., On the existence of collisionless equivariant minimizers for the classical $n$-body problem, Invent. Math. 155 (2004), 305–362, math-ph/0302022.
  * [7] Maderna E., On weak KAM theory for $N$-body problems, Ergodic Theory Dynam. Systems 32 (2012), 1019–1041, arXiv:1502.06273.
  * [8] Maderna E., Translation invariance of weak KAM solutions of the Newtonian $N$-body problem, Proc. Amer. Math. Soc. 141 (2013), 2809–2816, arXiv:1105.4484.
  * [9] Marchal C., How the method of minimization of action avoids singularities, Celestial Mech. Dynam. Astronom. 83 (2002), 325–353.
  * [10] Meyer K.R., Offin D.C., Introduction to Hamiltonian dynamical systems and the $N$-body problem, Applied Mathematical Sciences, Vol. 90, 3rd ed., Springer, Cham, 2017.
  * [11] Percino B., Sánchez-Morgado H., Busemann functions for the $N$-body problem, Arch. Ration. Mech. Anal. 213 (2014), 981–991.
  * [12] Saari D.G., Symmetry in $n$-particle systems, in Hamiltonian Dynamical Systems (Boulder, CO, 1987), Contemp. Math., Vol. 81, Amer. Math. Soc., Providence, RI, 1988, 23–42.