$k$-algebra. Assume that $R^{2}\neq 0$. Then either the first or the second
conclusion of Theorem 5.7 hold for $(R;\mathcal{L}_{\it{alg}})$.

###### Proof.

By Item 2 of Lemma 5.11 there exists a nonzero finitely generated $k$-algebra
of scalars $L$ such that $(L;\mathcal{L}_{\it{alg}})$ (in particular
$(L;\mathcal{L}_{\it{ring}})$) is e-interpretable in
$(R;\mathcal{L}_{\it{alg}})$. This same lemma states that if $R^{2}$ is
nonzero then $L$ is nonzero as well. Now the result follows by Theorem 6.1 and
transitivity of e-interpretations. ∎

Similarly as done in Section 5, a combination of Theorem 6.1 and our main
result for bilinear maps (Theorem 3.9) yields the following analogue of Lemma
5.1. Note that the full power of the previous Theorem 6.1 is not exploited
here, since the aforementioned result for bilinear maps always gives finite-
dimensional $k$-algebras, hence of Krull dimension $0$ by Noether
normalization.

######  Lemma 6.4.

Let $f:A\times B\to C$ be a $k$-bilinear map between finite-dimensional
$k$-modules, with $\it{Im}(f)\neq 0$. Suppose that
$(A,B,C;f,\mathcal{L}_{\it{group}})$ is e-interpretable in a structure
$\mathcal{M}$. Then there exists a finite field extension $F$ of $k$ that is
e-interpretable in $\mathcal{M}$ in the language of rings (with constants),
and $\mathcal{D}(F;\mathcal{L}_{\it{ring}})$ is reducible to $\mathcal{M}$.

###### Proof.

By Theorem 3.9, there exists a nonzero finite-dimensional $k$-algebra of
scalars $R$ such that $(R;\mathcal{L}_{\it{ring}})$ is e-interpretable in
$(A,B,C;f,\mathcal{L}_{\it{group}})$, and thus in $\mathcal{M}$. Since $R$ is
finite-dimensional, the Krull dimension of $R$ is necessarily $0$ (by Noether
normalization), and the lemma now follows by Theorem 6.1. ∎

The following can be proved in the exact same way as Theorems 5.2, 5.7 and
Corollary 5.9, using Lemma 6.4 instead of Lemma 5.1. Below, $R$ may be non-
associative, non-commutative, and non-unitary. See Subsection 5.2 for an
explanation of the terminology used here.

######  Theorem 6.5.

Let $R$ be a finitely generated algebra over a field $k$. Suppose that $R$
satisfies one of the following:

  1. 1.

$R$ has finite dimension over $k$, and $R^{2}\neq 0$.

  2. 2.

$R$ is associative or right-normed-generated, and $(R/I_{n})^{2}\neq 0$ for
some $n\geq 1$.

  3. 3.

$R$ is a simply graded Lie algebra and $[R/I_{n},R/I_{n}]\neq 0$ for some
$n\geq 1$.

Then there exists a finite field extension $K$ of $k$ such that
$(K;\mathcal{L}_{\it{ring}})$ is e-interpretable in
$(R;\mathcal{L}_{\it{ring}})$, and $\mathcal{D}(K;\mathcal{L}_{\it{ring}})$ is
reducible to $\mathcal{D}(R;\mathcal{L}_{\it{ring}})$.

###### Proof.

The three-sorted structure $(R,R,R;\mathcal{L}_{\it{group}},\cdot)$ is
e-interpretable in $(R;\mathcal{L}_{\it{ring}})$, where as usual $\cdot$
denotes the ring multiplication of $R$. Hence Item 1 follows from Lemma 6.4.
If $R$ is associative or right-normed-generated, then
$(R/I_{n};\mathcal{L}_{\it{ring}})$ is e-interpretable in
$(R;\mathcal{L}_{\it{ring}})$ by Lemma 5.6, and so Item 2 follows by such
lemma and by Item 1 (and by transitivity of e-interpretations). Item 3 is a
particular case of Item 2, given that any simply graded Lie algebra is right-
normed-generated due to Lemma 5.8. ∎

Hence if $k$ is a field all of whose finite extensions have undecidable
Diophantine problem, then $\mathcal{D}(R;\mathcal{L}_{\it{ring}})$ is
undecidable. This occurs, for example, if $k$ is a global function field [8,
39].

## References

  * [1] E. S. Chibrikov.  A right normed basis for free Lie algebras and Lyndon-Shirshov words.  J. Algebra, 302(2):593–612, 2006.
  * [2] P. L. Clark.  Commutative algebra, 2015.  Available at http://math.uga.edu/~pete/integral.pdf.
  * [3] M. Davis, H. Putnam, and J. Robinson.  The decision problem for exponential Diophantine equations.  Annals of Mathematics, 74(3):425–436, 1961.
  * [4] J. Denef.  The Diophantine problem for polynomial rings and fields of rational functions.  Trans. Amer. Math. Soc. 242, pages 391–399, 1978.
  * [5] J. Denef.  The Diophantine problem for polynomial rings of positive characteristic.  In Maurice Boffa, Dirkvan Dalen, and Kenneth Mcaloon, editors, Logic Colloquium ’78, volume 97 of Studies in Logic and the Foundations of Mathematics, pages 131 – 145. Elsevier, 1979.
  * [6] J. Denef and L. Lipshitz.  Diophantine sets over some rings of algebraic integers.  Journal of the London Mathematical Society, s2-18(3):385–391, 1978\.
  * [7] D. Eisenbud.  Commutative Algebra: With a View Toward Algebraic Geometry.  Graduate Texts in Mathematics. Springer, 1995.
  * [8] K. Eisenträger.  Hilbert’s Tenth Problem for algebraic function fields of characteristic 2.  Pacific Journal of Mathematics, 210, 08 2002.
  * [9] K. Eisenträger.  Integrality at a prime for global fields and the perfect closure of global fields of characteristic p>2\.  Journal of Number Theory, 114(1):170 – 181, 2005.
  * [10] M. D. Fried and M. Jarden.  Field arithmetic.  Ergebnisse der Mathematik und ihrer Grenzgebiete. Springer-Verlag, 1986\.
  * [11] K. R. Goodearl and R. B. Warfield.  An Introduction to Noncommutative Noetherian Rings.  London Mathematical Society St. Cambridge University Press, 2004.
  * [12] W. Hodges.  Model theory, volume 42 of Encyclopedia of Mathematics and its Applications.  Cambridge University Press, Cambridge, 1993.
  * [13] C. Huneke and I. Swanson.  Integral Closure of Ideals, Rings, and Modules.  Number v. 13 in Integral closure of ideals, rings, and modules. Cambridge University Press, 2006.
  * [14] I. Kaplansky.  Commutative Rings.  Chicago lectures in mathematics. University of Chicago Press, 1974.
  * [15] O. Kharlampovich and A. G. Miasnikov.  Equations in algebras.  ArXiv e-prints, June 2016.
  * [16] O. Kharlampovich and A. G. Miasnikov.  What does a group algebra of a free group know about the group?  ArXiv e-prints, July 2016.
  * [17] O. Kharlampovich and A. G. Miasnikov.  Undecidability of equations in free Lie algebras.  ArXiv e-prints, August 2017.
  * [18] O. Kharlampovich and A. G. Miasnikov.  Undecidability of the first order theories of free non-commutative Lie algebras.  ArXiv e-prints, April 2017.
  * [19] O. Kharlampovich and A. G. Miasnikov.  Tarski-type problems for free associative algebras.  Journal of Algebra, 500:589 – 643, 2018.  Special Issue dedicated to Efim Zelmanov.
  * [20] K. H. Kim and F. W. Roush.  An approach to rational Diophantine undecidability.  In Proceedings of Asian Mathematical Conference, page 242–248, River Edge, NJ, 1990. World Sci. Publ.
  * [21] K. H. Kim and F. W. Roush.  Diophantine undecidability of c(t1, t2).  Journal of Algebra, 150(1):35 – 44, 1992.
  * [22] J. Koenigsmann.  Undecidability in Number Theory, pages 159–195.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2014.
  * [23] D. Marker.  Model Theory : An Introduction.  Graduate Texts in Mathematics. Springer New York, 2002.
  * [24] Y. V. Matijasevič.  The Diophantineness of enumerable sets.  Dokl. Akad. Nauk SSSR, 191:279–282, 1970.
  * [25] B. Mazur and K. Rubin.  Ranks of twists of elliptic curves and Hilbert’s Tenth Problem.  Inventiones mathematicae, 181(3):541–575, Sep 2010.
  * [26] A. G. Miasnikov.  Definable invariants of bilinear mappings.  Siberian Mathematical Journal, 31(1):89–99, Jan 1990.
  * [27] A. G. Miasnikov, F. Oger, and S. Sohrabi.  Elementary equivalence of rings with finitely generated additive groups.  Annals of Pure and Applied Logic, 2018.
  * [28] A. G. Miasnikov and M. Sohrabi.  Groups elementarily equivalent to a free nilpotent group of finite rank.  Annals of Pure and Applied Logic, 162(11):916 – 933, 2011.
  * [29] A. G. Miasnikov and M. Sohrabi.  Elementary coordinatization of finitely generated nilpotent groups.  ArXiv e-prints, November 2013.
  * [30] G. A. Noskov.  Elementary theory of a finitely generated commutative ring.  Mathematical notes of the Academy of Sciences of the USSR, 33(1):12–15, Jan 1983.
  * [31] T. Pheidas and K. Zahidi.  Undecidability of existential theories of rings and fields: A survey.  _Contemporary Mathematics, 270, 49-106_ , 2000.
  * [32] B. Poonen.  Using elliptic curves of rank one towards the undecidability of Hilbert’s Tenth Problem over rings of algebraic integers.  In C. Fieker and D.R. Kohel, editors, Algorithmic Number Theory, pages 33–42, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
  * [33] B. Poonen.  Hilbert’s tenth problem and mazur’s conjecture for large subrings of Q.  Journal of the American Mathematical Society, 16(4):981–990, 2003\.
  * [34] B. Poonen.  Hilbert’s Tenth Problem over rings of number-theoretic interest.  http://math.mit.edu/~poonen/papers/aws2003.pdf, February 2003.  _Notes for Arizona Winter School on "Number theory and logic"_.
  * [35] J. Robinson.  Definability and decision problems in arithmetic.  The Journal of Symbolic Logic, 14(2):98–114, 1949.
  * [36] J. Robinson.  The undecidability of algebraic rings and fields.  Proceedings of the American Mathematical Society, 10(6):950–957, 1959.
  * [37] V. A. Roman’kov.  Unsolvability of the endomorphic reducibility problem in free nilpotent groups and in free rings.  Algebra and Logic, 16(4):310–320, Jul 1977.
  * [38] H. N. Shapiro and A. Shlapentokh.  Diophantine relationships between algebraic number fields.  Communications on Pure and Applied Mathematics, 42(8):1113–1122, 1989.
  * [39] A. Shlapentokh.  Hilbert’s Tenth Problem for rings of algebraic functions in one variable over fields of constants of positive characteristic.  Transactions of the American Mathematical Society, 333(1):275–298, 1992.
  * [40] A. Shlapentokh.  Hilbert’s Tenth Problem for rings of algebraic functions of characteristic 0.  Journal of Number Theory, 40(2):218 – 236, 1992.
  * [41] A. Shlapentokh.  Diophantine relations between rings of S-integers of fields of algebraic functions in one variable over constant fields of positive characteristic.  The Journal of Symbolic Logic, 58(1):158–192, 1993.
  * [42] A. Shlapentokh.  Diophantine classes of holomorphy rings of global fields.  Journal of Algebra, 169(1):139 – 175, 1994.
  * [43] A. Shlapentokh.  Hilbert’s Tenth Problem: Diophantine Classes and Extensions to Global Fields.  New Mathematical Monographs. Cambridge University Press, 2007.
  * [44] H. Stichtenoth.  Algebraic Function Fields and Codes.  Springer Publishing Company, Incorporated, 2nd edition, 2008.
  * [45] G. van der Geer, B. J. J Moonen, and R. Schoof.  Number Fields and Function Fields – Two Parallel Worlds.  Progress in Mathematics. Birkhäuser, Boston, 2006.


# Keypoint Communities

Duncan Zauss, Sven Kreiss, Alexandre Alahi
EPFL VITA lab
CH-1015 Lausanne
<EMAIL_ADDRESS>
###### Abstract

We present a fast bottom-up method that jointly detects over 100 keypoints on
humans or objects, also referred to as human/object pose estimation. We model
all keypoints belonging to a human or an object –the pose– as a graph and
leverage insights from community detection to quantify the independence of
keypoints. We use a graph centrality measure to assign training weights to
different parts of a pose. Our proposed measure quantifies how tightly a
keypoint is connected to its neighborhood. Our experiments show that our
method outperforms all previous methods for human pose estimation with fine-
grained keypoint annotations on the face, the hands and the feet with a total
of 133 keypoints. We also show that our method generalizes to car poses.

##  1 Introduction

Recent large-scale datasets with fine-grained annotations of complex poses
present a new challenge for pose estimation methods. Beyond detecting a coarse
person bounding box and a small set of keypoints for large body joints, we now
have large datasets that include over 100 extra fine-grained keypoints in the
face, the hands and the feet. Resolving these fine details will allow us to
build robust representations of humans for downstream tasks like action
recognition [24, 2], and intent prediction [25, 31].

Training our current pose estimation algorithms on poses with mixed coarse and
fine keypoints presents a challenge as they assume a uniform importance of all
keypoints in a pose. We introduce a principled keypoint weighting method to
take into account the difference of the importance of coarse and fine-grained
keypoints. Figure 1 shows a complex person and car poses. For instance, the
person pose contains coarse keypoints like the hips and shoulders and fine-
grained keypoints like the ones along the eyebrows.

Figure 1:  Our proposed keypoint weighting method for complex poses yields
state-of-the-art results for whole body human pose estimation and for complex
car poses, whilst running at high frame rates.

In [12], Jin _et al_. share a large scale annotation for complex human body
poses and propose their method, ZoomNet, that set the state-of-the-art for
this type of complex human body pose. Their method first localizes a person
with their major keypoints and then estimates the areas of the hands and face.
On the estimated areas, it runs a separate head that is zoomed-in on that area
to determine fine-grained keypoint locations. In contrast, we propose a fast,
bottom-up method that directly estimates all keypoints in parallel. Our method
does not need predefined areas for fine-grained estimation and, therefore,
generalizes to any pose, like a fine-grained car pose. Such a car pose is
proposed in the ApolloCar3D dataset [35] and we show that our method
generalizes to this pose as well.

Complex, coarse and fine-grained poses present a challenge for current pose
estimation methods that assume a uniform distribution of keypoints across a
person or object. A cluster of fine-grained keypoints overly emphasizes that
region and focuses the neural network optimization on that area reducing the
importance of other regions that only have a single keypoint. We propose a
method that quantifies how tightly connected these keypoints are and that
rebalances the training weights such that all areas of a pose are equally well
connected to the rest of the pose. We introduce the details in Section 3.

Our contributions are (i) a method to weigh the importance of keypoints and
their connections in complex poses based on graph-based methods for community
detection, (ii) an efficient implementation for fine-grained human poses and
(iii) generalization from human poses to fine-grained car poses. We show the
impact of our contribution with state-of-the-art results on the challenging
COCO WholeBody dataset [12] and the ApolloCar3D dataset [35]. The software is
open source and publicly available
111https://github.com/DuncanZauss/Keypoint_Communities.

##  2 Related Work

There is an extensive literature on pose estimation. While many works have
focused on human pose estimation, there are recent works that extend the
method to animal pose estimation [23] and car pose estimation [35]. Recent
datasets include more keypoints that represent finer details on human and car
poses and are reviewed below.

#### Human Pose Estimation.

The recent release of the COCO WholeBody dataset [12] with 133 keypoints for a
single human pose presents new challenges for existing methods. The authors,
Jin _et al_., established baseline numbers of existing methods on their
dataset and proposed ZoomNet, a new neural network architecture that refines
regions with fine-grained annotations with dedicated networks.

In general, state-of-the-art methods for human pose estimation are based on
Convolutional Neural Networks [37, 10, 6, 27, 29, 40, 36, 38, 28, 16, 7].
There are two major approaches for pose estimation. Bottom-up methods estimate
each body joint first and then group them into poses. Top-down methods first
run a person detector to estimate person bounding boxes before estimating body
joint locations within each bounding box.

The first bottom-up methods were introduced, e.g., by Pishchulin _et al._ with
DeepCut [30]. They solve the keypoint association problem with an integer
linear program. In these early methods, the processing time for a single image
was of the order of hours. Newer methods introduced additional concepts to
reduce prediction time, _e.g._ , in Part Affinity Fields [6], Associative
Embedding [27], PersonLab [29] and multi-resolution networks with associate
embedding [7]. Composite Fields as introduced in PifPaf [17] predict more
precise associations than OpenPose’s Part Affinity Fields [6] and PersonLab’s
mid-range fields [29] which allows for particularly fast and greedy decoding
with high precision.

Figure 2:  Overview of our method. We obtain the average euclidean distance
for every connection in the pose graph from the training dataset. We then
create ego graphs of radius three for every vertex and compute the local
centrality for the ego vertex. The centrality is directly related to the
training weight of a joint. The training weight of a connection is obtained by
taking the max of the weights from the vertices of this connection.

#### Car Pose Estimation.

The new ApolloCar3D dataset [35] with its 66 keypoints for car pose estimators
presents similar challenges than the WholeBody dataset [12]. The authors [35]
presented baseline performance numbers using Convolutional Pose Machines (CPM)
[38] and also quantified the performance of human labelers on their dataset.

It is only recently that methods that were developed for human pose estimation
have been applied to other classes. Car poses provide finer detail for a car
than a 2D or 3D detection bounding box. While human pose estimation focuses on
the location of body joints within the human body, car poses annotate points
on the surface of the car.

One of the earlier works by Reddy  _et al_. proposes Occlusion-Net [32] that
highlights the issue of self-occlusion for these keypoints on the surface of
an object. As the car is viewed from different sides, the set of visible
keypoints changes drastically due to self-occlusion. Their work includes
extensive modeling with a 3D graph network and self-supervised training with
the CarFusion dataset [8] to predict 2D and 3D keypoints. In OpenPose [5], Cao
_et al_. show qualitative results for car pose estimation. Simple Baseline
[33] trains a top-down pose estimator on car annotations of the Pascal3D+
dataset [39]. Other works choose different representations for finer details
beyond bounding boxes. In GSNet [14], a car orientation in 3D space is
predicted along with a 3D shape estimate.

#### Keypoint weighting

In general, previous methods used uniform distributions to weigh the keypoints
in training for single networks or used separate networks for the different
fine-grained regions. We show that with our keypoint weighting method the
performance of single neural networks for predicting poses that contain fine-
grained and coarse features can be improved significantly.

##  3 Method

We need to devise a training procedure for poses that combine coarse keypoints
that localize large body parts (hips, shoulders, _etc_.) and fine-grained
keypoints like the outline of a hand. Individual fine-grained keypoints are
highly predictable from neighboring keypoints whereas coarse keypoints are
more independent. However, the importance of a keypoint is not only based on
its individual predictability, but also on how it contributes to a local group
of keypoints. In other words, every individual keypoint in a group of five
keypoints might have negligible importance, the group of five keypoints
together is still important.

(a)

(b)

Figure 3:  Visualization of proposed weighting for joints and connections. The
colors indicate the training weights. For the joints, also the circle radius
is proportional to the joint weight. In (a), all shortest paths are taken into
account. In (b), only shortest paths within a radius of three are taken into
account.

(a)

(b)

Figure 4:  Visualization of proposed weighting for the skeleton of an instance
from the ApolloCar3D dataset [35]. In (a), all shortest paths are taken into
account. In (b), only shortest paths within a radius of three are taken into
account. The dotted lines indicate connections on the left side of the car for
a clearer visualization.

###  3.1 Pose Estimation Architecture

Figure 2 provides an overview of our architecture. Our method is independent
of the particular choice of pose estimator, and could be used for any top-down
or bottom up pose estimation algorithm. We will use OpenPifPaf [17], which is
a bottom-up pose detector based on Composite Fields. A backbone in form of a
ResNet [11] or ShuffleNetV2 [21] processes single images to create a common
representation for the head networks. The head networks are the Composite
Intensity Fields (CIFs) and Composite Association Fields (CAFs) that are
$1\times 1$ convolutions followed by subpixel convolutions [34]. The heads are
trained to detect keypoints and to associate keypoints respectively. Per
keypoint type $k$ and for every location $i,j$ in the output field, the CIF
head predicts an intensity component $c^{i,j}_{k}$ to indicate a keypoint is
nearby, a two-dimensional vector component $v^{i,j}_{k}$ to precisely regress
to the keypoint location, the uncertainty of the location $b^{i,j}_{k}$ and a
scale component $s^{i,j}_{k}$ to estimate the size of a keypoint. The learnt
scale of the keypoint $s^{i,j}_{k}$ depends on the size of that specific joint
in the image and is used in the decoding step as the width of an unnormalized
Gaussian convolution to create high-resolution confidence fields. Similarly,
the CAF head also has an intensity component to indicate the vicinity of an
association between two keypoints, two vector components that regress to the
two keypoints instances to associate, and two scale components to estimate the
two keypoint sizes. The CIF loss with an extension to weigh all loss
components by keypoint type $k$ with $w_{k}$ is:

$\displaystyle\mathcal{L}_{\textrm{CIF}}$
$\displaystyle=\sum\limits_{k}w_{k}\Bigg{[}$
$\displaystyle\sum_{m_{k,c}}\textrm{BCE}(c,\hat{c})$ (3)
$\displaystyle+\;\sum_{m_{k,v}}\textrm{Laplace}(v,\hat{v},\hat{b})\;\;\;\;\;$
$\displaystyle+\;\sum_{m_{k,s}}\textrm{Laplace}\bigg{(}1,\frac{\hat{s}}{s},b_{s}\bigg{)}\;\;\;\Bigg{]}$

where $c$, $v$, $b$ and $s$ are components of the composite field with
suppressed indices ($k,i,j$) for the keypoint type and feature map location
and where symbols with a hat indicate predicted quantities. With $m_{k,c}$,
$m_{k,v}$ and $m_{k,s}$, we indicate keypoint specific masks over the feature
map. BCE is a binary cross entropy loss with Focal loss extension [19] and
Laplace is a linear regression loss for vector components that is attenuated
by a predicted $\hat{b}$ or a fixed $b_{s}$ [15, 18]. The probabilistic
interpretation of the loss function as the negative logarithm of a joint
likelihood function requires that the three components of the loss are equally
weighted with respect to each other. The CAF head is trained with an
equivalent loss with two vector components (3) and two scale components (3).
We now focus our attention on the blue branch of Figure 2 that determines the
training weights $w_{k}$ for the CIF and CAF heads.

###  3.2 Graph Centrality

We represent a pose as a graph $\\{V,E\\}$ with vertices $V$ representing each
keypoint, and edges $E$ representing the Euclidean distance in the image plane
between the keypoints. That Euclidean distance is estimated with an average
over all training annotations. The importance of a keypoint does not only
depend on its direct neighbors, but on its connectedness in the neighborhood.
Therefore, we consider graph centrality measures.

There exists an enormous number of graph centrality measures, for example
closeness centrality [1], eigenvector centrality [3], Katz centrality [13],
betweenness centrality [9] and harmonic centrality [22] among others.

Traditionally, centrality measures are used to determine the centrality or
importance of persons in social networks or more generally the importance of
nodes in complex graphs. Highly central nodes get assigned a high centrality
value. For our application highly central nodes are part of a community and
thus we use the inverse of the centrality.

We want to use our understanding of the particular problem do derive the best
metric for our use case. To train complex poses, we aim to make every keypoint
equally well connected to the rest of the pose. Most centrality measures are
based on shortest paths from a vertex to all other vertices. The
“connectedness” of a vertex is represented in the average length of the
shortest paths that originate at the vertex. For example, the ankle keypoint
is not very well connected and the average length of all shortest paths that
originate at the ankle is high as all the paths to the face and hand keypoints
are long. To rebalance our training such that all vertices are connected
equally well, we want to assign an equal weight to an average unit of distance
in the shortest paths.

In practice, we restrict the centrality computation to a neighborhood by
extracting an ego graph of radius three (the subgraph with all the vertices
around a particular vertex that can be reached in three steps) for every
vertex and computing the centrality for that vertex only within that subgraph.

The weighted length (weighted by Euclidean distance) of the shortest path
between two vertices $v_{1}$ and $v_{2}$ is $d(v_{1},v_{2})$. As we are
interested in equally weighting a unit length, the harmonic average is
appropriate. For every vertex $v_{i}$, we compute the harmonic average $h$ of
all the shortest paths originating at $v_{i}$:

$h(v_{i})=\left[\sum_{v_{j}\in
V\backslash\\{v_{i}\\}}\frac{1}{d(v_{i},v_{j})}\right]^{-1}$ (4)

where we can identify the harmonic centrality $H$ [22] in the square brackets,
leading to $h=H^{-1}$.

Numerically, the closeness centrality [1] and harmonic centrality are similar
and it might be helpful to interpret this weighting in terms of assigning a
high weight to keypoints with low “closeness”.

###  3.3 Training Weights

We use the graph centrality measure to derive training weights for keypoints
and their connections. The keypoint weights are obtained directly from the
centrality by normalizing $\sum_{i}h(v_{i})$ to the number of keypoints. This
normalization facilitates easier comparisons between the different weighting
methods.

We also need to obtain training weights for keypoint connections. Again, the
weight of the connection should not depend just on its own length, but also
take into account the structure of the local cluster of keypoints this edge is
a part of. Given we already have a principled method for the vertices, we
derive the weight $w_{ij}$ for the edge that connects vertices $v_{i}$ and
$v_{j}$ from $h$:

$w_{ij}\propto\max\big{(}h(v_{i}),h(v_{j})\big{)}\;\;\;.$ (5)

We normalize the sum of edge weights to the total number of edges.

The resulting weights for the human WholeBody pose [12] and the car pose [35]
are shown in Figures 3 and 4. We show two configurations of our method. One
where we use the entire pose to compute our graph centrality measure and one
where we use an ego graph of radius three. The WholeBody skeleton has clear
hierarchical clusters of keypoints in the hands and face and one level down in
the eyes and fingers. The ApolloCar3D skeleton is more uniformly distributed.
There are keypoint agglomerations in the area of the lights and number plates
both in the front and in the rear of the car. The keypoints in the roof are
the most separate from the other keypoints. In contrast to the COCO WholeBody
skeleton the communities are not as strongly separated. For the WholeBody
pose, the computed weights range from 0.21 to 5.15 and for ApolloCar3D from
0.57 to 1.63. Our method automatically determines the keypoint communities for
the WholeBody pose and produces highly varied training weights. Groups of
keypoints that are highly predictable from each other receive a lower weight.
Our method also successfully determines the more uniform distribution of
keypoints in the car pose and produces less varied training weights.

For any generic pose and training dataset, this method automatically creates
training weights for keypoints and associations in a principled way and we
show its effectiveness on challenging experiments in the next section.

##  4 Experiments

We conduct extensive experiments on complex poses to demonstrate the
effectiveness and efficiency of our method. We investigate a human pose with
17 COCO keypoints as the main skeleton that was then extended with an
additional 116 keypoints for fine-grained details in the face, the hands and
the feet. We also demonstrate that our method generalizes to a fine-grained
car pose with 66 keypoints.

#### Datasets.

For human pose estimation, we conduct experiments on the COCO WholeBody [12]
dataset. This dataset contains extra annotations on the 64,000 training and
5,000 validation images of COCO [20] for face, hands and feet. The full pose
contains 133 keypoints with 152 connections. There are about 130,000 instances
with annotations for the left hand, the right hand and the face. The body
annotations are taken from COCO [20] that contains about 250,000 instances.

For car pose estimation, we use the ApolloCar3D dataset [35]. It provides car
annotations with 66 keypoints and we assigned them 108 connections. The
dataset consists of 4283 training and 200 validation images with 52942 and
2674 annotated instances respectively. As cars are only visible from one side
and often partially occlude each other, only an average of 16.2 keypoints are
annotated per instance.

#### Evaluation.

We follow the evaluation method proposed in the COCO WholeBody [12] dataset
paper. It is based on keypoint-based average precision (AP) that was
popularized with the COCO keypoint task [20]. The evaluation weighs every one
of the 133 keypoints equally.

For the ApolloCar3D dataset, we report the detection rate which was proposed
by Song _et al_. [35]. A keypoint is counted as detected if the distance from
the prediction to the ground truth is less than 10 pixels. Additionally, we
also report the keypoint-based AP as it is already common for human pose
detection [20]. We compute AP based on object keypoint similarity with sigmas
of 0.05 for all car keypoints.

#### Implementation Details.

We extend OpenPifPaf [18] with an option to weigh the training of keypoint and
connection types. We populate the weights for the given pose in the generic
fashion described in Section 3.

We train models with ShuffleNetV2 [21] backbones that were pretrained without
weighting the MS COCO keypoint task. The head networks CIF (Composite
Intensity Field) and CAF (Composite Association Field) are single $1\times 1$
convolutions followed by a subpixel convolution [34]. The total stride after
the backbone is 16 and decreased to 8 in the head networks. We train for 100
epochs with a learning rate of 0.0001 with an SGD [4] optimizer with Nesterov
momentum [26] of 0.95 and a batch size of 16.

Figure 5:  Qualitative results from the COCO WholeBody validation set [12].
Our method resolves multiple persons per image and captures their facial
expressions and gestures like hailing a cab. The bottom-left image is
processed with human and car pose estimators.

Figure 6:  Qualitative results with images from the transport domain that we obtained from flickr. The two images in the bottom row were processed with our human and car pose estimation network. Method | WB | body | foot | face | hand
---|---|---|---|---|---
HRNet [36] | 43.2 | 65.9 | 31.4 | 52.3 | 30.0
ZoomNet [12] | 54.1 | 74.3 | 79.8 | 62.3 | 40.1
AE [27] | 27.4 | 40.5 | 7.7 | 47.7 | 34.1
OpenPose [6] | 33.8 | 56.3 | 53.2 | 48.2 | 19.8
Ours | 60.4 | 69.6 | 63.4 | 85.0 | 52.9
Table 1:  Average precision (AP) results in percent on the COCO WholeBody
dataset [12]. WB indicates evaluation on all 133 keypoints. The first two
methods are top-down methods and the lower three are bottom-up methods.
Reference numbers from [12].

#### Results on the COCO WholeBody dataset.

Quantitative results on the COCO WholeBody dataset [12] are shown in Table 1.
Our result is based on a single model that is evaluated for all (WB) or a
subset of the predicted keypoints. Our method outperforms previous methods and
achieves especially high precision on fine-grained regions such as the face or
hands. The “body” task is equivalent to the COCO keypoint task on the val set
[20]. Our method is based on OpenPifPaf [18] which only achieves 71.6% on the
COCO val set with a model trained on that specific task and we therefore did
not expect it to outperform ZoomNet [12]. Our method makes up for its lower
body AP with excellent results for face and hand AP and is nearly twice as
precise as any other bottom-up method.

Qualitative results are shown in Figure 6 and Figure 5. The additional
keypoints in the face can serve as a powerful representation from which human
emotions such as happiness and surprise but also attention and intent can be
derived. The fine and coarse-grained WholeBody pose can be used to predict
actions from images. For the example image on the bottom right of Figure 5, it
can be predicted that the person is eating while holding a cell phone and for
the person on the middle picture in the bottom row it can be predicted that
the person is surprised that a cat jumped on the table. Through adding fine-
grained hand keypoints the interaction between humans and objects can be
detected. In the transportation domain, the fine grained keypoints on the hand
region help to understand the intentions of pedestrians. For example in the
bottom left image of Figure 6 it is possible to detect that the pedestrian
wants to hail a cab and is not intending to cross the road even though she is
standing at the side of the road.

Figure 7:  Qualitative results from the ApolloCar3D validation set [35]. We
demonstrate excellent detection rates and spatial localizations of all the
visible car keypoints even at far distances.

#### Results on the ApolloCar3D dataset.

Our method is not specific to human poses and can be applied to any pose. To
demonstrate that our method generalizes, we apply it to the ApolloCar3D
dataset [35], where every car instance is annotated with up to 66 keypoints.
Our method achieves an average precision (AP) of 72.0% with all sigmas for the
computation of the object keypoint similarity set to 0.05. The previous work
[35] evaluates detection rate instead of AP and a comparison of our method
with their Convolutional Pose Machines [38] evaluation and human annotators is
shown in Table 2. We achieve a detection rate of 91.9% and thus outperform the
previous state-of-the-art from [35] which achieved a detection rate of 75.4%.
They also report the detection rate for human annotators at 92.4%. Our
proposed method reduces the gap to human level performance from 17.0% to just
0.5%.

Method | Detection rate[%]
---|---
Human annotators | 92.4
CPM [38] | 75.4
Ours | 91.9
Table 2:  Detection rate on the ApolloCar3D dataset [35]. The metrics for the
Convolutional Pose Machines and the human annotators are from [35].

We share qualitative results on the validation set of ApolloCar3D [35] in
Figure 7. We predict fine-grained keypoints for close-by and far-away cars.
Cameras for self-driving technology cover a wide angle and therefore have to
perceive small car instances even for cars at moderate distances. It is safety
relevant to determine the locations of break and indicator lights for
downstream tasks which we achieve with high precision.

Method | WB | body | foot | face | hand
---|---|---|---|---|---
Baseline | 55.3 | 60.8 | 54.2 | 86.2 | 50.6
Equal | -3.1% | -4.8% | -8.1% | +1.5% | +0.8%
Global | -1.2% | -2.5% | -1.1% | +0.8% | +2.6%
Crafted | -1.2% | -1.6% | -1.7% | +1.2% | -1.0%
Table 3:  Ablation studies. Average precision (AP) results in percent on the
COCO WholeBody val dataset [12]. We report the percentage gain of other
weighting methods in comparison to our baseline method. In addition to our
main method, we compare with applying no weighting (Equal), applying our
method to the global graph instead of ego graphs (Global) and using hand-
crafted weights (Crafted). All results are produced with a ShuffleNetV2k16
backbone. WB indicates an evaluation on all 133 keypoints.

#### Ablation Studies.

We study the effects of different loss weightings on the average precision
(AP). In Table 3 the results from training with different weighting schemes
are shown. For this ablation study we train for 50 epochs starting from a
model that was pretrained on the 133 keypoints without weighting. “Crafted”
means that the body and foot keypoints are weighted three times higher than
the rest of the keypoints. Hand-crafting the weights can be seen as choosing
133 additional hyperparameters which is generally infeasible and the
motivation for our method.

The weighting based on the local harmonic centrality, which is our baseline
method, achieves an AP of 55.3%, which is a significant improvement over the
unweighted training, which results in an AP of 53.6%. The weighting based on
the local harmonic centrality also achieves a higher AP than the weighting
based on the vanilla harmonic centrality and the hand-crafted weights. This
shows a) that the local influence between the keypoints is more important than
the global one and b) that using our proposed centrality measure is more
optimal than hand-crafting the weights.

Seed thr. | CAF thr. | AP | t [ms] | tdec [ms]
---|---|---|---|---
ZoomNet [12] | 54.1 | 175 | -
0.2 | 0.001 | 60.4 | 153 | 60
0.5 | 0.001 | 58.4 | 120 | 27
0.5 | 0.01 | 54.6 | 112 | 20
Table 4:  Ablation studies. Average precision (AP) results in percent on the
COCO WholeBody val dataset [12] and their associated prediction time for
different decoding methods. We show ZoomNet’s average precision and runtime
for better comparison with our method. Our neural network runs in 93ms on a
NVIDIA GTX 1080 Ti. The decoding starts with the joints where the confidence
exceeds the seed threshold. The vector and scale from a cell of a CAF field is
only used if the confidence is above the CAF threshold.

We study the effect of different parameter choices on the precision and
prediction time trade-off of our method. The runtime of our method is
influenced by the decoding algorithm. Since the WholeBody pose has 133 joints
and 152 associations the duration of the decoding is more prevalent than for
poses with a lower number of joints and associations. The first step in the
decoding process is to determine seed joints from which the decoding starts
and from which connections to the other joints will be created with the help
of the CAF fields. All joints that have a confidence that is higher than a
certain seed threshold will be used as seed joints. Increasing the seed
threshold will reduce the number of seeds and thus cause a faster decoding
process. However, with a higher seed threshold, some humans may not be
detected which can result in a lower average precision. Using a higher CAF
threshold generally increases the decoding speed in exchange for a lower
accuracy as fewer associations are considered for decoding. The quantitative
results that the variation of these parameters yield can be seen in Table 4.
With our standard decoder setting, we already achieve a higher AP than ZoomNet
[12] whilst being 22 ms faster. With a seed threshold of 0.5 and a CAF
threshold of 0.01 our model achieves an AP of 54.5 with a prediction time of
112 ms, which is an excellent trade-off between inference speed and precision.
Our ShuffleNetV2K16 backbone achieves an AP of 50.9 at a frame rate of 15.2
frames per second on a NVIDIA GTX 1080 Ti, which makes at suitable for most
real-time applications that require fine-grained pose estimation.

##  5 Conclusion

We have proposed a generic and principled method to train complex poses with
fine and coarse-grained details. Our experiments demonstrate our ability to
perceive detailed facial expressions and hand gestures and produce state-of-
the-art results on standard pose benchmarks for human and car poses. We have
shown that our method operates at state-of-the-art prediction speeds and we
have studied the trade-offs between accuracy and prediction speeds.

##  6 Acknowledgments

This project has received funding from the Initiative for Media Innovation
based at Media Center, EPFL, Lausanne, Switzerland.

## References

  * [1] Alex Bavelas.  Communication patterns in task-oriented groups.  The journal of the acoustical society of America, 22(6):725–730, 1950.
  * [2] Lorenzo Bertoni, Sven Kreiss, and Alexandre Alahi.  Perceiving humans: from monocular 3d localization to social distancing.  IEEE Transactions on Intelligent Transportation Systems, 2021.
  * [3] Phillip Bonacich.  Power and centrality: A family of measures.  American journal of sociology, 92(5):1170–1182, 1987.
  * [4] Léon Bottou.  Large-scale machine learning with stochastic gradient descent.  In Proceedings of COMPSTAT’2010, pages 177–186. Springer, 2010\.
  * [5] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A Sheikh.  Openpose: realtime multi-person 2d pose estimation using part affinity fields.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [6] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 7291–7299, 2017.
  * [7] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang.  Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5386–5395, 2020.
  * [8] N Dinesh Reddy, Minh Vo, and Srinivasa G Narasimhan.  Carfusion: Combining point tracking and part detection for dynamic 3d reconstruction of vehicles.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1906–1915, 2018.
  * [9] Linton C Freeman.  A set of measures of centrality based on betweenness.  Sociometry, pages 35–41, 1977.
  * [10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [12] Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo.  Whole-body human pose estimation in the wild.  In European Conference on Computer Vision (ECCV), 2020.
  * [13] Leo Katz.  A new status index derived from sociometric analysis.  Psychometrika, 18(1):39–43, 1953.
  * [14] Lei Ke, Shichao Li, Yanan Sun, Yu-Wing Tai, and Chi-Keung Tang.  Gsnet: Joint vehicle pose and shape reconstruction with geometrical and scene-aware supervision.  In European Conference on Computer Vision (ECCV), pages 515–532. Springer, 2020.
  * [15] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In Advances in neural information processing systems, pages 5574–5584, 2017.
  * [16] Muhammed Kocabas, Salih Karagoz, and Emre Akbas.  Multiposenet: Fast multi-person pose estimation using pose residual network.  In European Conference on Computer Vision (ECCV), pages 417–433, 2018.
  * [17] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi.  Pifpaf: Composite fields for human pose estimation.  In Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [18] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi.  OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association.  IEEE Transactions on Intelligent Transportation Systems, 2021.
  * [19] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision (ICCV), pages 2980–2988, 2017.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European Conference on Computer Vision (ECCV), pages 740–755. Springer, 2014.
  * [21] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In European Conference on Computer Vision (ECCV), pages 116–131, 2018.
  * [22] Massimo Marchiori and Vito Latora.  Harmony in the small-world.  Physica A: Statistical Mechanics and its Applications, 285(3-4):539–546, 2000.
  * [23] Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge.  Deeplabcut: markerless pose estimation of user-defined body parts with deep learning.  Technical report, Nature Publishing Group, 2018.
  * [24] Sina Mokhtarzadeh, Mina Ghadimi, Ahmad Nickabadi, and Alexandre Alahi.  Convolutional relational machine for group activity recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [25] Taylor Mordan, Matthieu Cord, Patrick Perez, and Alexandre Alahi.  Detecting 32 pedestrian attributes for autonomous vehicles.  IEEE Transactions on Intelligent Transportation Systems - under review, 2020.
  * [26] Yurrii Nesterov.  A method of solving a convex programming problem with convergence rate o(1/k2).  In Soviet Mathematics Doklady, volume 27, pages 372–376, 1983.
  * [27] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  In Advances in Neural Information Processing Systems, pages 2277–2287, 2017.
  * [28] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In European Conference on Computer Vision (ECCV), pages 483–499. Springer, 2016.
  * [29] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy.  Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model.  In European Conference on Computer Vision (ECCV), pages 269–286, 2018.
  * [30] Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter V Gehler, and Bernt Schiele.  Deepcut: Joint subset partition and labeling for multi person pose estimation.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4929–4937, 2016.
  * [31] Haziq Razali and Alexandre Alahi.  Pedestrian intention prediction: A convolutional bottom-up approach.  Transportation Research Part C, 2021.
  * [32] N Dinesh Reddy, Minh Vo, and Srinivasa G Narasimhan.  Occlusion-net: 2d/3d occluded keypoint localization using graph networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7326–7335, 2019.
  * [33] Héctor Corrales Sánchez, Antonio Hernández Martínez, Rubén Izquierdo Gonzalo, Noelia Hernández Parra, Ignacio Parra Alonso, and David Fernandez-Llorca.  Simple baseline for vehicle pose estimation: Experimental validation.  IEEE Access, 8:132539–132550, 2020.
  * [34] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874–1883, 2016.
  * [35] Xibin Song, Peng Wang, Dingfu Zhou, Rui Zhu, Chenye Guan, Yuchao Dai, Hao Su, Hongdong Li, and Ruigang Yang.  Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5452–5462, 2019.
  * [36] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5693–5703, 2019.
  * [37] Alexander Toshev and Christian Szegedy.  Deeppose: Human pose estimation via deep neural networks.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1653–1660, 2014.
  * [38] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh.  Convolutional pose machines.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4732, 2016.
  * [39] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In Proceeding of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 75–82. IEEE, 2014.
  * [40] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 466–481, 2018.