# Causality Diagrams using Hybrid Vector Clocks

Ishaan Lagwankar  Michigan State University,
East Lansing
MI  Kanishka Wijewardena  Michigan State University,
East Lansing
MI

###### Abstract

Causality in distributed systems is a concept that has long been explored and
numerous approaches have been made to use causality as a way to trace
distributed system execution. Traditional approaches usually used system
profiling and newer approaches profiled clocks of systems to detect failures
and construct timelines that caused those failures. Since the advent of
logical clocks, these profiles have become more and more accurate with ways to
characterize concurrency and distributions, with accurate diagrams for message
passing. Vector clocks addressed the shortcomings of using traditional logical
clocks, by storing information about other processes in the system as well.
Hybrid vector clocks are a novel approach to this concept where clocks need
not store all the process information. Rather, we store information of
processes within an acceptable skew of the focused process. This gives us an
efficient way of profiling with substantially reduced costs to the system.
Building on this idea, we propose the idea of building causal traces using
information generated from the hybrid vector clock. The hybrid vector clock
would provide us with a strong sense of concurrency and distribution, and we
theorize that all the information generated from the clock is sufficient to
develop a causal trace for debugging. We post-process and parse the clocks
generated from an execution trace to develop a swimlane on a web interface,
that traces the points of failure of a distributed system. We also provide an
API to reuse this concept for any generic distributed system framework.

###### Index Terms:

hybrid vector clocks, distributed systems, causality

##  I Introduction

Causality diagrams have long been used in distributed systems to produce
relationships between how messages are communicated and interpreted by
different processes in the system. They are often represented as swimlane
diagrams, where message passing is represented as transactions between
processes shown by horizontal axes, as depicted in figure 1.

Figure 1: Example Swimlane Diagram with 3 Processes

Distributed systems often rely on clock systems to ensure consistency across
replicas, debugging, and causal ordering of events to track execution. Clock
systems in the new age have come a long way since their inception, and the
strides taken in logical time to define causality in distributed systems.
Vector clocks allow finer control over this causality, where each process
independently tracks the clocks of the other processes in the system. This
proves particularly useful when clocks exhibit skew with respect to each
other. The skew described is a characteristic of a system when the frequency
of the quartz clock used to update the system clock changes over time, causing
clocks to drift apart from each other. This was the main motivation of
introducing logical time, but soon it was apparent that we would need physical
time to better explain execution traces and attribute them to external factors
due to real-world factors tampering with system execution. In the new age of
networking, the Network Time Protocol (NTP) was a key factor in ensuring
clocks remain within consistent bounds of each other, but this protocol proved
to be a large memory bottleneck as the number of bits the clocks would use
would be proportional to the number of processes in the system. This was a key
factor in introducing clocks that reduced the number of bits the clocks needed
to represent the causal consistency of a distributed system. One of the
earliest works on this was the Hybrid Logical Clock, which allowed lesser bits
to be stored while maintaining physical time on the system by rounding off
high precision clocks to lower precisions, due to the properties of the
distributed system that introduce latency. The latency induced allows us to
ignore the higher precision offered by protocols such as NTP while preserving
the happens-before relationship between messages.

We implement a version of the Hybrid Vector Clock [20], a clock system that
extends the idea of the HLC to vector clocks. In the HVC, we keep an array of
offsets to keep track of potential skews in the distributed system. The key
idea here is to store offsets rather than the large physical time, and for
that, we discretize our time into intervals, based on a value set by the
system administrator. For events between epochs, we implement a counter system
to count events happening within an interval to make sure all events are
tracked.

Through the clock’s logic, we can enforce causality if a clock of a process is
lesser than a clock of another process. If two clocks are equal, those events
are concurrent. If a process ${e}$ sends a message to another process ${f}$,
the clock of ${e}$ is guaranteed to be lower than process ${f}$. This provides
a way to enforce a partial order of the clock system. The concept of partial
ordering has been explored widely in the concept of logical clocks, but
through the HVC, we show that lesser information is sufficient to enforce the
ordering.

Using clocks to develop execution traces has been studied using older clock
systems, such as Lamport clocks, vector clocks, and so on. These use the more
primitive implementations of these clocks, where they store vast amounts of
information to generate traces. This adds a high overhead on the distributed
system, giving diminishing returns. A traditional vector clock, for example,
stores data about all processes. This scales as a factor of the number of
processes. We reduce this overhead to a subset of the number of processes,
which are only the processes perceived to not be drifted away from the process
point of view. We term this drift to be $\epsilon$, the clock skew. Using the
HVC, we need only store how the clocks are offset in time from each other, and
if the offset is too large, we need not need care about the time of that
particular process. We can show that these clocks still enforce the same
partial ordering, as any involvement from a skewed process would prompt the
clock to record that interaction. Simply put, we care about skewed processes
only when they take part in the distributed system’s working. If we have not
heard from a process, from the point of view of the clock, the process is
considered dead, and we need not store information about it. This idea is
particularly useful in an execution trace, as we can pinpoint processes not
performing in a distributed computation, while also characterizing failure in
those processes. This could also prove useful in determining what processes
have skewed clocks in time-dependent computations.

In this work, we propose a system to generate execution traces and visualize
them on an easy-to-use web interface. We provide controls to tweak how the
traces are shown and characterized. We also provide an API to integrate the
trace generation and visualization in the infrastructure it is implemented in.
The rest of this paper is organized as follows. We first give a background on
the advent of clocks and how distributed systems have changed using different
clock infrastructures. We then provide the recent developments in generating
execution traces from clock systems and motivate why our system would
outperform the state of the art. We then provide our design of the clock and
the proposed methodology to generate the execution traces.

Our contributions: We Implement the design of the Hybrid Vector Clock (HVC)
proposed in [20] and expose it as an API. We then build a visualization
infrastructure that utilizes the HVC infrastructure to show traces of
execution with controls to tweak the visualization to observe process behavior
during execution.

##  II Background

###  II-A Clocks in Distributed Systems

Logical clocks were proposed in 1978 by Leslie Lamport [1] to trace the
ordering of events in a distributed system. Vector time was designed
independently by multiple researchers [14][2][15], and they proposed the idea
of representing time in a distributed system as a set of n-dimensional non-
negative integer vectors. According to [19], Vector clocks are defined by
three properties: Isomorphism, Strong consistency, and Event Counting.
Isomorphism suggests that if two events $x$ and $y$ have timestamps $vh$ and
$vk$, respectively, then $x\longrightarrow y\Longleftrightarrow vh<vk$. Here,
$\longrightarrow$ implies a partial ordering between a set of events. Strong
consistency implies that by examining the vector timestamp of two events, we
can determine the causal relationship between the two events. Event counting
suggests that if $d$ is always 1 in the rule $R1$, then the $i^{th}$ component
of the vector clock at process $p_{i},vt_{i}[i]$, denotes the number of events
that have occurred at $p_{i}$ until that instant.

There have been several prior implementations of vector clocks including
Singhal-Kshemkalyani’s differential technique [16] and Fowler-Zwaenepoel
direct dependency technique [17]. While vector clocks are upper bounded by
$O(n)$ complexity in terms of both time and memory complexity, the different
implementations of the past have tried to reduce this complexity and generate
more efficient representations, with some success. Singham-Kshemkalyani’s
differential technique [16], relied on piggybacking using the last sent and
last update, without updating every vector clock. This method relies on the
assumption that even though the number of processes is large, only a few key
processes in a system would interact frequently by passing messages. A benefit
of this method is that it cuts down storage overhead at each process to
$O(n)$. However, this method doesn’t make a substantial contribution to
reducing the time complexity incurred when updating the vector clock, as it
relies on piggybacking to work.

Fowler-Zwaenepoel’s direct-dependency technique cuts down storage complexity
again by reducing the message size during transmission by transmitting only a
scalar value in the messages. Here, a process only maintains information
regarding direct dependencies on other processes. The downside of this method
is that it has a high computational overhead as it has to trace dependencies
and update the vector clock, especially in systems where a few key processes
may have a large number of events.

Clock synchronization using Network Time Protocol (NTP) uses the Offset Delay
Estimation method to ensure physical clocks are synchronized across the
internet. Clock offsets and delays are calculated, and timestamps are issued
between different machines within a system accordingly. The system then
attempts to establish a causal relationship by using the corrected timestamps.
This however, can be computationally expensive and is open to error, as the
delay estimation may not always be accurate and result in a violation of the
causal relationship between processes issued by different machines.

One existing limitation between vector clocks representing logical time and
physical clock synchronization is the difficulty in reconciling one with the
other. To overcome this challenge, Hybrid Logical clocks were introduced by
Kulkarni et al. [13] to capture the causality relationship of a logical clock
with the characteristics of a physical clock embedded into it. Another variant
of the hybrid clock is the Hybrid Vector Clock [18], [20], which, unlike the
Hybrid Logical Clock, is able to provide all possible/potential consistent
snapshots for a given time. For this experiment, we use the Hybrid Vector
Clock design presented in Yingchareonthawornchai et al. [20], as it provides
desirable characteristics to build our visualization framework.

###  II-B Visualizing Traces

Mattern [2] talks about how distributed systems use the concept of global
state to communicate information, and the need to characterize this global
state. They talk about how a process can only approximate the global view of
the system, and no process can have, at any given instant, a consistent view
of the global state. To verify a distributed system, the author provides a
comparison between three key approaches: simulating a synchronous distributed
system given an asynchronous system, simulating a common clock or simulating a
global state. They highlight the need of a vector clock system to provide a
consistent snapshot of the global state, as each process having a clock that
stores only its own state is not enough to describe the global state of the
computation.

PARAVER [3] uses the PVM message passing library to analyze traces generated
from a computation. PVM primarily uses parallel message passing, and PARAVER
analyses these parallel traces using data analytics and provides a graphical
description of the analysis. This was one of the earliest visualization works
on distributed systems simulated only for parallel traces. It used a parser to
parse through the logs of the PVM-generated traces and analysed CPU activity,
communications, and user events. This however required the addition of
functionality to PVM itself and was not generalized to any distributed system
interface. It also did not provide generic support and incurred a larger
overhead to profile system resources while computation went on.

VAMPIR [4] provided analysis of MPI programs by generating timeline traces by
profiling MPI applications. It used different visualization metrics to show
whether processes were still active or not. It also provided views of system
activities and aggregated statistics about the system itself. However, it was
made specifically only for MPI applications and added to the profiling
interfaces of MPI.

D3S [5] allowed developers to specify predicates on distributed properties of
the system. These predicates can vary depending on what consistency checks one
would require on the distributed system. They modeled the tracing as a
consistency checker, and generated traces of predicate evaluation. The
predicates are injected dynamically at compile time into the system and are
evaluated based on the customization provided by the user. However, we believe
this approach would add overhead to running the distributed computation due to
complex predicate checking.

Zinsight [6] provides hierarchies of tasks and provides aggregated metrics to
show timeline visualizations of events. It also provides users with changing
granularity of the metric they want to see with sequences of computations per
process.

Trumper et al. [7] present a dynamic analysis tool that uses boundary tracing
and post processing to analyse system behavior through a distributed
computation. These are task based visualizations, where tasks are mapped to
memory resources. But this may not always be the case where processes share
the same memory, as in the case of OpenMP based infrastructures.

Dapper [8] is Google’s tracing software for distributed systems where they
provided low overhead, application-level transparency and scalability. Dapper
uses annotations and spans to generate traces through RPCs. However, the
authors mention that Dapper cannot correctly point to causal history, as it
uses annotations in non-standard control primitives, that may mislead the
causality calculations. Our approach would overcome this, as causality is
enforced through a lattice of clocks, rather than the events itself.

Isaacs et al. [9] provide a comprehensive survey of different distributed
monitoring and tracing tools in the past decade, providing detailed
descriptions and categorizations based on task parallelism, causality
information and so on.

Isaacs, Bremer et al. [10] design a trace visualization system purely relying
on logical clocks, and then transposing those clocks back to real-time clocks
in the visualization. Processes are also clustered based on logical behavior.
However, this would incur more overhead than our solution, and may cause
conflicts in enforcing causality, due to the usage of a standard logical
clock.

Verdi [11] provides developers with choosing the fault system to diagnose, and
verifies the implementation of the system. This is a formal verification
system where it provides the developer with an idealized fault model, and once
this is verified, it applies the correctness to a more realistic fault model.

ShiViz [12] uses vector clocks in generating distributed system traces using
happens-before relationships. By using vector clocks, it provides a verifiable
and accurate notion of causality. However, since it uses traditional vector
clocks, it uses a higher complexity than our proposed model.

##  III Hybrid Vector Clocks

In this section, we provide the design of the clock we would be using for
generating causal traces. The hybrid vector clock (HVC) [20] stores offsets
instead of the timestamp themselves. This is very useful, as instead of
storing logical time, we can store the physical time of a particular process
and its offset with another process, providing the physical time of all
processes in the system. We ignore skewed processes that have drifted too far
away from the process’ point of view, using $\epsilon$, the clock drift.

Figure 2: HVC Class Diagram

Figure 2 shows the class diagram of the hybrid vector clock. The offsets store
the offset of all other clocks in the point of view of the process clock. The
counters store the counters of events occurring in the same epoch.

We enforce partial ordering of all processes in the system, such that they
form a lattice of clocks. The way we enforce the order is as follows:

Assume for a HVC of process $i$, there exists an array clockhvc:

$max\\_epoch+offset[i]+counter[i]=clock_{hvc}[i]$ (1)

Then, for two events ${e}$ and ${f}$, we have the following relation, where
$\rightarrow$ implies causality between ${e}$ and ${f}$:

$\displaystyle e\rightarrow f\leftrightarrow$ $\displaystyle\forall\ i\ in\
e.clock_{hvc},e.clock_{hvc}[i]\leq f.clock_{hvc}[i]$ $\displaystyle\exists\ k\
in\ e.clock_{hvc},e.clock_{hvc}[k]<f.clock_{hvc}[k]$

We have two key algorithms in the clock’s execution. The Send (Or Local event)
[Algorithm 1] algorithm specifies how a clock moves forward with respect to
the system clock. The Recv [Algorithm 2] specifies how an incoming clock from
a message updates the process’ clock. This allows each clock to independently
update itself, rather than receiving inputs from the infrastructure. This
provides a system with no deadlocks and allows asynchronous execution.

Data:

e: hybrid clock of event e phy_clock: physical clock of the process

Result: f: The next hybrid clock after one step

$new\\_max\\_epoch\leftarrow max(e.max\\_epoch,floor(phy\\_clock/e.interval))$

if  _$new\\_max\\_epoch==e.max\\_epoch$_ then

$f.counters\leftarrow e.counters$

$f.counters[f.pid]+=1$

else

$f.counters\leftarrow[0]*N$

for  _$idx\leftarrow 0$ to $N$_ do

// To be converted to O(number of non-nulls) algorithm

$f.offsets[idx]\leftarrow
min(e.offsets[idx]+new\\_max\\_epoch-e.max_{e}poch,\epsilon)$

$f.max\\_epoch=new\\_max\\_epoch$ $f.offsets[f.pid]=0$

return _f_

Algorithm 1 Send Event

Data:

m: hybrid clock of incoming message event m

e: hybrid clock of event f

phy_clock: physical clock of the process

Result: f: The next hybrid clock after one step of event f

$new\\_max\\_epoch\leftarrow
max(e.max\\_epoch,m.max\\_epoch,phy\\_clock/e.interval)$

if  _$new\\_max\\_epoch==e.max\\_epoch\ and\
new\\_max\\_epoch==m.max\\_epoch$_ then

for  _$pid\leftarrow 0$ to $N$_ do

// To be converted to O(number of non-nulls) algorithm

$f.counters[pid]\leftarrow max(e.counters[f.pid],m.counters[pid])$

$f.counters[f.pid]\ =\ max(e.counters[f.pid],m.counters[f.pid])\ +\ 1$

$f.max\\_epoch=new\\_max\\_epoch$

$f.offsets[f.pid]=0$

else

if  _$new\\_max\\_epoch==e.max\\_epoch$_ then

$f.counters[f.pid]+=1$

$f.offsets[m.pid]=floor(min(e.max\\_epoch-m.max\\_epoch,\epsilon))$

else

if  _$new\\_max\\_epoch==m.max\\_epoch$_ then

$f.offsets=m.offsets$ $f.counters=m.counters$ $f.counters[f.pid]+=1$
$f.max\\_epoch=m.max\\_epoch$

else

$f.advance(phy\\_clock)$

$f.offsets[f.pid]=0$

return _$f$_

Algorithm 2 Recv Event

Here, we illustrate the working of the HVC through some sample scenarios.

  * 1.1

[Send] Physical clock of process in the same epoch: In figure 3, the epoch the
system is in does not change. In this case, we increment the counter of the
process clock to show that an event has occurred.

  * 1.2

[Send] Physical clock of process in a different epoch: In figure 4, the epoch
the system is in changes. In this case, we update the max epoch of the clock.
We do not make any other changes as with respect to the process, the other
processes are leading/lagging by the same amount.

  * 2.1

[Recv] Message and process are in the same epoch: In figure 5, the epoch of
both the message and process are in the same epoch. In this case, we take the
max of all pairwise counters in the message and process HVCs. We then
increment the counter of the process in the clock.

  * 2.2

[Recv] Message is lagging with respect to the process: In figure 6, the epoch
of the message is lagging with respect to the process. In this case, we
increment the counter of the process clock, and set the offset of the process
that the message came from to the difference of the max epochs, or epsilon,
depending on which is smaller.

  * 2.3

[Recv] Message is leading with respect to the process: In figure 7 the
message’s max epoch is much higher than that of the process. We copy over the
offsets and counters to the process clock. Then we increment the counter, and
then set the offset of the process to 0 to indicate that the process has
caught up with the distributed system.

Figure 3: Advancing when the epoch does not change. Figure 4: Advancing when
the epoch changes. Figure 5: Merging when both the message and process are in
the same epoch. Figure 6: Merging when the message is lagging. Figure 7:
Merging when the message is ahead of the process.

In all cases, we can observe that partial ordering is maintained. If any
clocks come up with the same offsets, counters and max epoch, these events are
deemed to be concurrent.

##  IV Design and Implementation

###  IV-A Experimental Design

To create our Hybrid Vector Clock implementation we utilized a Python 3.10
test-bed with each HVC being called from a custom-built class in Python. Our
HVC class consisted of the following member variables:

  1. 1.

Interval: Integer (Representing the Epoch Interval)

  2. 2.

PID: Integer (Representing the Process ID)

  3. 3.

Epsilon: Integer (Representing Epsilon of the System)

  4. 4.

Max Epoch: Integer (Representing Epoch of the Process)

  5. 5.

Counters: HashMap (Representing Counters of the clock)

  6. 6.

Offsets: HashMap (Representing offsets of the clock)

Each clock is initialized with the Interval, PID and Epsilon input by the
user, with Max Epoch set to 0 and Counters and offsets set to empty. The
reason for this is that the epoch, counters and offsets are updated based on
the advance and merge events the processes are subjected to, and should not be
hard-coded into the clocks to represent the accurate output when enforcing
causality. We implemented unit-testing to verify that our outputs are
accurate, where the max epoch, counters and offsets of the output clock of
each merge and advance event was compared with the target clock to verify the
accuracy. This allowed us to ascertain the accuracy of our clock class
implementation to enforce causality from our input data.

###  IV-B Ordering Input Traces

We used the idea of the HVC to embed information about generating traces. We
introduced a new variable broken, that would indicate a failure in sending a
message or a system-level failure. This turned out to be the explicit failure
we would be graphing. Implicit failures on the other hand, such as node power
being lost mid computation would be tracked by polls. If a process stops
responding to polls, we deem that process lost and set the process’s broken
bit to true. Using this information, we built a parser that parses logs and
generates causal traces in physical time. This provides much better causal
tracing as logical time would not make sense to a system administrator. With
physical timestamps and partial ordering, we did a time-based analysis of
system outages, and attribute them to real-world events that may have
potentially impacted the system.

The process of sorting the time traces of the input data adhered to the
algorithm seen in Algorithm 3. The web interface was designed using the
chart.js library and a sample preview of the web interface can be seen in
Figure 8.

Data:

from_node: Process ID of Message Transmitter

to_node: Process ID of Message Receiver

sender_max_epoch: Epoch of Transmitter

sender_offsets: Offsets of Transmitter

sender_counters: Counters of Transmitter

output_max_epoch: Max epoch of output clock

output_offsets: Offsets of output clock

output_counters: Counters of output clock

output_epsilon: Epsilon of system

epoch_interval: Epoch Interval of system

Result: output: An ordering of all the clocks in a causal format

Read Input Data Input traces

Find send clock and output clock of input traces according to the following
formulae:
$send\\_time[i]=sender\\_max\\_epoch+sender\\_offsets[i]+sender\\_counters[i]$

$output\\_time[i]=output\\_max\\_epoch+output\\_offsets[i]+output\\_counters[i]$

Order traces based on send time and output time according to the following
formula:

if $send\\_time$ $a<send\\_time$ $b$ $\mathbf{then}$

a appears before b

if $send\\_time$ $a==send\\_time$ $b$ $\mathbf{then}$

if $output\\_time$ $a<output\\_time$ $b$ $\mathbf{then}$

a appears before b

output: list containing ordering

return _output_

Algorithm 3 Sorting Input Traces

###  IV-C Designing the Web interface

We designed a web interface that reads from the logs generated on the master
node. We assume for the sake of this project that all clocks write to the same
log base but provide a methodology that extends this approach to multiple
systems. The abstraction would read from the single source database, and then
construct timelines for each participating process. We designed the web
interface using the json library in the Python programming language. The API
has the following features and benefits:

Figure 8: Sample API example of output traces with swimlanes in visualization

  1. 1.

The API was capable of isolating a particular message and zooming in on a
visualization of a snapshot that isolated a subset of messages from a trace.
This subset included some success cases but also highlighted failure cases and
the reasons behind said failure. As can be seen in Figure 9, we can see some
instances of a failure cases that was isolated from the main figure.

Figure 9: Sample API example of failure diagnosing in a single trace

  2. 2.

The API is also dynamic in that it is capable of updating in real time. The
API is connected to the localhost of the user machine, so it can be generated
on an available port of the machine and then updated in real-time, based on
updates made to the input data traces.

##  V Results and Discussion

###  V-A Failure Diagnosing

We evaluated the degree to which our system was capable of diagnosing failure
and understanding causal violations in logic. As can be seen in Figure 9, this
is a sample instance of failure that was observed in Process 2. As can be seen
in each of the “Failure” messages, a dummy instance of failure was logged in
for three out of the eight messages originating to and from Process 2. By
observing the “dummy” messages, we could analyze the root cause of failure to
see where a violation in causal consistency took place.

###  V-B Memory and Time Complexity

Figure 10: Runtime Complexity and Space Complexity of our HVC implementation
compared to regular vector clock implementations

We plotted the time complexity of our HVC implementation against traditional
vector clock implementations and discovered that our HVC implementation,
despite being bounded by $O(n)$ for the worst case scenario, generally
outperformed a typical vector clock by a wide margin. In practical
applications, our HVC had a performance close to $O(n/2)$, which is a much
more efficient implementation than a Vector Clock with O(n) performance. We
also found that the space complexity of our HVC implementation was bounded by
$O(n/2)$. However, according to Kulkarni et al.[18], the memory complexity of
the HVC should be bounded by $O(\sqrt{n})$. This implies that there is more
work to be done, especially in representing zero counters or offsets as empty
in our hashmap representations. Therefore, hashmaps with empty counters and
offsets should automatically be deleted from the HVC, which does cause issues
during implementation as the counters and offsets would have to be re-
instantiated each time a non-zero value is obtained for a counter or an
offset, which means that it would be more convenient to simply represent the
counters and offsets as hashmaps with no existing values, and accept the space
complexity that comes with being marginally greater than the ideal space
complexity.

###  V-C Comparison against othe visualization schemes

While our visualization scheme is not as robust or detailed as ShiViz [12] or
D3S [5], our ability to diagnose failure by isolating a single trace and our
ability to visualize causality using swimlanes does indicate great promise in
our implementation. We are optimistic that in future work we would be able to
expand upon our API and refine it further, for instance to identify the full
history of a process and diagnose where a point of failure took place.

##  VI Conclusion and Future Work

We designed a visualization scheme for Hybrid Vector Clocks. We illustrated
why HVCs were an improvement in the state-of-the-art when representing logical
time in distributed systems, and then formulated a scheme that sorted input
traces and then visualized the clocks built out of these input traces to see
if our input traces enforced causality. Our visualization API was able to
isolate traces with points of failure and we were able to evaluate causes of
failure within a trace based on its history.

Future work done in this regard would involve expanding upon the API to make
it more robust. Improvements include being able to visualize a more in-depth
history of a trace, and a visually appealing API compared to the state-of-the-
art.

## References

  * [1] Leslie Lamport. 1978. Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21, 7 (July 1978), 558–565. https://doi.org/10.1145/359545.359563
  * [2] Mattern, Friedemann. Virtual time and global states of distributed systems. Univ., Department of Computer Science, 1988.
  * [3] Pillet, Vincent, Jesús Labarta, Toni Cortes, and Sergi Girona.“Paraver: A tool to visualize and analyze parallel code.” In Proceedings of WoTUG-18: transputer and occam developments, vol. 44, no. 1, pp. 17-31. 1995.
  * [4] Knüpfer, Andreas, Holger Brunst, Jens Doleschal, Matthias Jurenz, Matthias Lieber, Holger Mickler, Matthias S. Müller, and Wolfgang E. Nagel. “The vampir performance analysis tool-set.” In Tools for High Performance Computing: Proceedings of the 2nd International Workshop on Parallel Tools for High Performance Computing, July 2008, HLRS, Stuttgart, pp. 139-155. Springer Berlin Heidelberg, 2008.
  * [5] Liu, Xuezheng, Zhenyu Guo, Xi Wang, Feibo Chen, Xiaochen Lian, Jian Tang, Ming Wu, M. Frans Kaashoek, and Zheng Zhang. “D3S: Debugging deployed distributed systems.” In NSDI. 2008.
  * [6] De Pauw, Wim, and Steve Heisig. “Zinsight: A visual and analytic environment for exploring large event traces.” In Proceedings of the 5th international symposium on Software visualization, pp. 143-152. 2010.
  * [7] Trümper, Jonas, Johannes Bohnet, and Jürgen Döllner. “Understanding complex multithreaded software systems by using trace visualization.” In Proceedings of the 5th international symposium on Software visualization, pp. 133-142. 2010.
  * [8] Sigelman, Benjamin H., Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. “Dapper, a large-scale distributed systems tracing infrastructure.” (2010).
  * [9] Isaacs, Katherine E., Alfredo Giménez, Ilir Jusufi, Todd Gamblin, Abhinav Bhatele, Martin Schulz, Bernd Hamann, and Peer-Timo Bremer. “State of the Art of Performance Visualization.” EuroVis (STARs) (2014).
  * [10] Isaacs, Katherine E., Peer-Timo Bremer, Ilir Jusufi, Todd Gamblin, Abhinav Bhatele, Martin Schulz, and Bernd Hamann. “Combing the communication hairball: Visualizing parallel execution traces using logical time.” IEEE transactions on visualization and computer graphics 20, no. 12 (2014): 2349-2358.
  * [11] Wilcox, James R., Doug Woos, Pavel Panchekha, Zachary Tatlock, Xi Wang, Michael D. Ernst, and Thomas Anderson. “Verdi: a framework for implementing and formally verifying distributed systems.” In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 357-368. 2015.
  * [12] Beschastnikh, Ivan, Patty Wang, Yuriy Brun, and Michael D. Ernst. “Debugging distributed systems.” Communications of the ACM 59, no. 8 (2016): 32-37.
  * [13] Kulkarni, S.S., Demirbas, M., Madappa, D., Avva, B., Leone, M. (2014). Logical Physical Clocks. In: Aguilera, M.K., Querzoni, L., Shapiro, M. (eds) Principles of Distributed Systems. OPODIS 2014. Lecture Notes in Computer Science, vol 8878. Springer, Cham.
  * [14] C. Fidge, “Logical time in distributed computing systems,” in Computer, vol. 24, no. 8, pp. 28-33, Aug. 1991, doi: 10.1109/2.84874.
  * [15] Schmuck, Frank B.. “The Use of Efficient Broadcast Protocols in Asynchronous Distributed Systems.” (1988).
  * [16] Singhal, Mukesh and Ajay D. Kshemkalyani. “An Efficient Implementation of Vector Clocks.” Inf. Process. Lett. 43 (1992): 47-52.
  * [17] Fowler, Jerry and Willy Zwaenepoel. “Causal distributed breakpoints.” Proceedings.,10th International Conference on Distributed Computing Systems (1990): 134-141.
  * [18] M. Demirbas and S. Kulkarni, “Beyond truetime: Using augmented time for improving google spanner,” in Proc. 7th Workshop, Large-Scale Distrib. Syst. Middleware, 2013, pp. 1–5.
  * [19] Ajay D. Kshemkalyani and Mukesh Singhal. 2008. Distributed Computing: Principles, Algorithms, and Systems (1st. ed.). Cambridge University Press, USA.
  * [20] S. Yingchareonthawornchai, D. N. Nguyen, S. S. Kulkarni and M. Demirbas, ”Analysis of Bounds on Hybrid Vector Clocks,” in IEEE Transactions on Parallel and Distributed Systems, vol. 29, no. 9, pp. 1947-1960, 1 Sept. 2018, doi: 10.1109/TPDS.2018.2818700.


# Towards Simple and Accurate Human Pose Estimation with Stair Network

Chenru Jiang<EMAIL_ADDRESS>Kaizhu Huang
<EMAIL_ADDRESS>Shufei Zhang<EMAIL_ADDRESS>Xinheng Wang<EMAIL_ADDRESS>Jimin Xiao<EMAIL_ADDRESS>Zhenxing Niu<EMAIL_ADDRESS>Amir Hussain<EMAIL_ADDRESS>Department of Computer Science, University of Liverpool, Liverpool L69 7ZX,
U.K.  Department of Electrical and Electronic Engineering, Xi’an Jiaotong-
Liverpool University, Suzhou 215123, China.  Data Science Research Center,
Duke Kunshan University, No. 8 Duke Avenue, Kunshan, 215316, China.  School of
Computing, Edinburgh Napier University, Edinburgh, EH11 4BN, UK.  School of
Electronic Engineering, Xidian University, Xian, 710000, China.

###### Abstract

In this paper, we focus on tackling the precise keypoint coordinates
regression task. Most existing approaches adopt complicated networks with a
large number of parameters, leading to a heavy model with poor cost-
effectiveness in practice. To overcome this limitation, we develop a small yet
discrimicative model called STair Network, which can be simply stacked towards
an accurate multi-stage pose estimation system. Specifically, to reduce
computational cost, STair Network is composed of novel basic feature
extraction blocks which focus on promoting feature diversity and obtaining
rich local representations with fewer parameters, enabling a satisfactory
balance on efficiency and performance. To further improve the performance, we
introduce two mechanisms with negligible computational cost, focusing on
feature fusion and replenish. We demonstrate the effectiveness of the STair
Network on two standard datasets, e.g., 1-stage STair Network achieves a
higher accuracy than HRNet by 5.5% on COCO test dataset with 80% fewer
parameters and 68% fewer GFLOPs. 111The code is released at
https://github.com/ssr0512

###### keywords:

Stair Network, Human Pose Estimation, Feature Diversity

##  1 Introduction

Human pose estimation is one fundamental yet challenging task to estimate
precise human joint coordinates (eyes, ears, shoulders, elbows, wrists, knees,
etc.). It is essential for various high-level visual understanding tasks such
as human action recognition [1, 2, 3, 4], video surveillance [5], and tracking
[6, 7]. In recent years, there have been significant advances from single pose
estimation [8, 9, 10, 11, 12, 13, 14] to multiple pose estimation [15, 16, 17,
18, 19, 20]. These methods can be categorized into bottom-up [15, 17, 19, 20,
21, 22] and top-down [16, 23, 24, 25, 18, 26, 27] methods. Top-down methods
are gaining more popularity due to their higher accuracy.

Figure 1: Performance vs. parameter number of various methods on COCO test
data with 384$\times$288 input size. Method in red is 3-stage STair Network,
and * means the method adopts pre-training.

In existing top-down methods, multi-stage structures [28, 13, 29] and
knowledge distillation [29, 30] are commonly adopted for pose estimation.
However, two main drawbacks limit both the efficiency and the accuracy. One
defect is that single receptive field for local features is commonly adopted
in existing methods. Due to the insufficient local features extraction,
current network structures are usually deep and complicated for good
performance. Fig. 1 shows a number of existing popular methods which commonly
adopt a heavy model with a large number of parameters. Although multi-scale
module structure [31, 12] is commonly designed to aggregate information, the
feature diversity is still coarse and insufficient for regression tasks. The
top part of Fig. 2 illustrates a scenario that the single receptive field (the
smallest red rectangles) is deficient to distinguish background or different
torsos. For knowledge distillation, complicated teacher networks and iterative
training process are indispensable. The other defect is that the information
loss is inevitable in these structures. We observe that, for localization
tasks, the position error will be accumulated and enlarged after the iterative
down/upsampling process. In addition, these methods seldom consider the
important high-frequency texture representations [32].

Figure 2: The upper figure part illustrates the local features of multi-scale
keypoints, where the four red boxes on local features are the multiple
receptive fields of STair Cell. The lower figure part demonstrates the STair
Network structure.

To reduce computational cost and still achieve superior performance, we design
a novel basic feature extraction block, STair Cell (STC), to simultaneously
pursue advantageous feature diversity and high efficiency. In each block,
multiple receptive fields structure is introduced to promote local feature
diversity and aggregate rich local representations, enabling the block to
obtain stronger discriminative capability on multi-scale keypoints or
background (as illustrated multiple red rectangles in the top part of Fig. 2).
Meanwhile, a lightweight context attention is embedded to enlarge the features
discrimination. Fig. 3 visualizes the correlations within STC where multiple
receptive fields focus on aggregating different local features and bring rich
information difference. For efficiency, the block channel number is gradually
halved and depthwise separable mechanism is adopted to attain lower
computational cost.

Figure 3: Branch correlation of STC. Deeper color means stronger correlation.
C1, C2, C3 and C4 are corresponding to the four branches of STC in Fig. 4.
Better viewed in color.

To reduce computational cost and still achieve superior performance, we design
a novel basic feature extraction block, STair Cell (STC), to simultaneously
pursue advantageous feature diversity and high efficiency. In each block,
multiple receptive fields structure is introduced to promote local feature
diversity and aggregate rich local representations, enabling the block to
obtain stronger discriminative capability on multi-scale keypoints or
background (as illustrated multiple red rectangles in the top part of Fig. 2).
Meanwhile, a lightweight context attention is embedded to enlarge the features
discrimination. Fig. 3 visualizes the correlations within STC where multiple
receptive fields focus on aggregating different local features and bring rich
information difference. For efficiency, the block channel number is gradually
halved and depthwise separable mechanism is adopted to attain lower
computational cost.

We encapsulate STCs to build a simple multi-branch module termed STair Unit
(STU), which is illustrated as triangles in Fig. 2. We propose to tackle the
information loss problem from two aspects. Within each unit, down/upsampling
is not taken in the first branch such that the network can consistently keep
the high resolution feature maps to alleviate information loss. Outside each
unit, we design a lightweight structure to focus on feature re-usage and re-
exploitation among multiple units. Illustrated as squares in Fig. 2, such
structure can be readily attached after each unit for enhancing feature
utilization. Meanwhile, we try to reserve high-frequency texture
representations (illustrated as diamonds in Fig. 2), to supply these important
features at the last unit for precise localization.

STair Units can be simply stacked to form a multi-stage network for the
coarse-to-fine estimation. Significantly different from knowledge
distillation, by enhancing the feature extraction block while alleviating
information loss problem, the large teacher network is not needed in our
network. In a nutshell, our contributions are three-fold:

  1. 1.

To attain a low computational cost and accuracy framework, we propose a
lightweight yet effective basic feature extraction block to focus on
extracting more diverse local representations, enabling better capability to
localize different keypoints with less parameters.

  2. 2.

To reduce information loss during the training, the first branch within each
STC consistently keeps high resolution feature maps. Meanwhile, two efficient
mechanisms are proposed to connect STCs to enhance feature utilization and
replenish.

  3. 3.

STNet attains new SOTA performance with lower computational cost on standard
benchmarks. Specifically, even 1-stage network can boost the accuracy by 5.5%
over previous SOTA network [18] with only 20% parameters when evaluated on
COCO test datasets.

##  2 Related Work

#### Feature Extraction Unit

Residual block is a popular basic feature extraction unit and commonly adopted
in existing pose estimation methods [33, 34, 12, 16, 26, 18]. By the efficient
bottleneck design with the skip connection, the residual block can be utilized
to form much deeper and more complicated structures. Meanwhile, some novel
methods can also assist to improve the residual block capability. For
instance, SENet [35] embeds channel attention after each block to reweight
each feature map. SKNet [36] increases a convolution with the different kernel
size in the block, and adopts soft attention to select useful channels.
ResNeXt [37] applies group convolution [38] in order to obtain rich features
from different subspace. Currently, RSN [39] proposes to equally split
channels into 4 branches in one block, and increases the convolutions and
inner connections to fuse features gradually. However, RSN relies heavily on
complicated and advanced platforms MegDL, and the performance RSN_18 drops
significantly (3.2%) when Pytorch is adopted. As demonstrated in Sec. 3.1, the
basic feature extraction blocks of this paper consist of four sequential
convolutions, where the channel dimension is halved and receptive field is
doubled gradually to capture rich diverse local features while maintaining
high efficiency.

#### Multi-stage Learning Structure

Multi-stage structure proves suitable to attain accurate localization, which
has been widely used in various recent approaches [40, 18, 28, 13] for
refining predictions gradually. Hourglass [12] and fast pose [29] take the
same sub-module to form multi-stage networks. MSPN [40] follows the hourglass
design yet doubles the channel dimension gradually after each downsampling.
Among multiple stages, existing methods [41, 42, 29] focus on feature
attention, and intermediate supervision [10] to adjust and supervise the sub-
module learning process. Differently, as shown in Sec. 3.3, we propose a
multi-stage dual branch structure which mainly focus on feature re-usage and
re-exploitation among stages.

#### Low-frequency Structure Feature Fusion

In the traditional methods [39, 16, 18], multi-scale feature fusion is adopted
commonly to extract low-frequency semantic information. Hourglass network [12]
proposes a U-shape structure to attain multi-scale feature fusion within each
stage. Later works such as cascaded pyramid network [16] execute a multi-scale
fusion process between high-to-low and low-to-high structures. HRNet [18] sets
up four parallel branches and aggregates branch features iteratively.
Intuitively, current methods all work on low-frequency structure feature
fusion. However, high-frequency information is another critical factor [32]
for the precise localization tasks as it provides rich texture
representations. Unfortunately, high-frequency information is hardly reserved
when networks go deeper. Contrary to conventional approaches, in Sec. 3.4 we
develop an effective method to directly replenish abundant multi-scale high-
frequency representations.

##  3 Our Approach

In this section, we first detail our basic component STair Cell. After that,
we introduce the multi-scale STair Unit structure consisting of a number of
STCs that maintains high resolution feature maps from end to end. Next, we
will describe the Multi-stage Dual Branch Structure and STair Fusion mechanism
to further enhance the network.

###  3.1 STair Cell

#### STair Convolution

For each STair Cell, we propose a sequential stair-shape atrous convolution
structure to aggregate significant diverse local features for different scale
keypoints through multiple receptive field sizes. In contrast to traditional
convolutions, atrous convolution [43] helps to attain larger receptive fields
without increasing the computational cost. The kernel sizes of atrous
convolution layers are calculated by $K=k+(k-1)\times(R-1)$, where $K$ is the
equivalent kernel size, $k$ is the practical kernel size, and $R$ is the
dilation rate.

Figure 4: STair Cell structure. Purple rectangles indicate atrous convolution
processes with multiple dilation rates, gray rectangles describe the embedded
context attentions.

As shown in Fig. 4, one stair shape convolution structure contains four atrous
convolution layers with the different dilation rates ($R=1,2,3,4$). We notice
that the atrous convolution has an inherent gridding effect which leads to
information inconsistency [44]. To alleviate such gridding effect, we keep the
dilation rate as $1$ in the top base convolution layer, make it as the same as
the normal convolution. In STC, as the receptive field size is increased, the
channel number is reduced in half for the efficiency purpose. Concretely, the
total computational cost of four-branch structure are: $4T(T=k\times k\times
C_{in}\times C_{out}\times H\times W$), where the computational cost depends
multiplicatively on the number of input channels $C_{in}$, the number of
output channels $C_{out}$, the kernel size $k$, and the feature map size
$H\times W$. The channel halving strategy factorizes conventional mechanism
into four parts and the amount of parameters are:

$\displaystyle\frac{1}{2}T+\frac{1}{8}T+\frac{1}{32}T+\frac{1}{64}T=\frac{43}{64}T$
(1)

Thus, the channel halving strategy attains $4/(\frac{43}{64})\approx 6$ times
computation reduction than the conventional structure. Table 1 demonstrates
the computational cost comparison of channel halving strategy. More channels
might provide richer diverse features but hinder structure efficiency, which
is not pursued in this work. Particularly, we have investigated various
structures for STC which are shown in ablation parts. However, the current
structure demonstrates the best performance for aggregating diverse local
features with high efficiency.

Table 1: Results of STC with/without channel halving strategy. Method | pretrain | Input Size | Halve | #Params | GFLOPs | AP
---|---|---|---|---|---|---
1-stage | N |  256 $\times$ 192 | $\checkmark$ | 5.7M | 2.3 | 72.1
1-stage | N |  256 $\times$ 192 | $\times$ | 8.7M | 3.0 | 73.0

Table 2 lists the detail parameter configurations of STC and STU. Since the
inputs of each atrous convolution are based on the previous layers, the
receptive fields are gradually superimposed. Thus, the fourth branch of the
STC contains four receptive fields equivalently. In this regard, STC enables
to extract rich diverse local representations for the network. As demonstrated
in Fig. 3, the correlations between different STC branches ($Ci$ and $Cj$,
$i\neq j$) are low, meaning that multiple branches with various receptive
fields focus on aggregating different local features. Consequently, four-
branch STair Units can obtain more dense diverse features with STCs, as shown
in the second part of Table 2.

Table 2: Parameter Configuration of STair Cells and STair Units. M means that the feature map size is maintained, and H means the feature map size is halved by downsampling. STC Branch Index | Feature Map Size | Channel Number | Kernel Size
---|---|---|---
c = 1 | M | 16 | 3
c = 2 | M | 8 | 3, 5
c = 3 | M | 4 | 3, 5, 7
c = 4 | M | 4 | 3, 5, 7, 9
STU Branch Index | Feature Map Size | Channel Number | Kernel Size
b = 1 | M | 32 | 3, 5, 7, 9
b = 2 | H | 64 | 5, 7, 9, 11
b = 3 | H | 128 | 7, 9, 11, 13
b = 4 | H | 256 | 9, 11, 13, 15

To further reduce the parameters, we adapt the depthwise separable convolution
[45] to our STC, which factorizes a standard convolution into a depthwise
operation and a pointwise operation. First, each atrous convolution of STC
exploits depthwise operation to apply a single filter for each input channel.
Then, the pointwise operation applies a $1\times 1$ convolution to combine the
depthwise outputs. There is a significant difference on the amounts of network
parameters between traditional convolution and depthwise separable
convolution. Specifically, the total computational cost of a standard
convolution are: $T$, The depthwise separable convolution splits standard
convolution operation into two parts: $T/k^{2}+T/C_{out}$. After decomposing
convolution as a two-step process of filtering and combining, we can achieve
the parameter reduction as below:

$\displaystyle\frac{T/k^{2}+T/C_{out}}{T}=\frac{1}{k^{2}}+\frac{1}{C_{out}}$
(2)

After the stair-shape convolution, we concatenate four level features and pass
them to the lightweight context attention part.

#### Mix Attention

For the postprocessing of STC, we consider separately for different local
features so that the block attains a stronger feature diversity and fine-
grained recalibration. The proposed mix attention strategy is shown in the
right part of Fig. 4 which aims to reinforce the extracted diverse features
through both channel and spatial dimensions. For calculation efficiency, we
introduce soft reduction rate to the channel dimension attention, which is
dynamically enlarged as the channels of STU is increased. For spatial
dimension attention, STC applies average and max pooling methods to generate
only two masks for keeping low computational cost. Table 3 demonstrates that
the mechanism is effective to enhance feature diversity with negligible
computational cost. In addition, we maintain skip connection to support
gradients propagation of the network.

Table 3: Comparison of STC with/without mix attention. Method | pretrain | Input Size | #Params | GFLOPs | Mix Attention | AP
---|---|---|---|---|---|---
1-stage | N |  256 $\times$ 192 | 5.6M | 2.3 | $\times$ | 71.5
1-stage | N |  256 $\times$ 192 | 5.7M | 2.3 | $\checkmark$ | 72.1

###  3.2 STair Unit

STU can be stacked to compose a multi-stage learning mechanism as demonstrated
in Fig. 2. Inspired by HRNet [18], each STU adopts a multi-branch structure to
attain multi-scale feature fusion. Differently, with the superior STCs, the
basic extraction block number of four branches in each STU is: (4,3,2,1),
whilst that of the HRNet is: 4x(4,3,2,1). Thus, the computational cost is much
less than HRNet. Meanwhile, as seen in the bottom part of Table 2, we list the
detail kernel size range of the units. More receptive field of STU (3-15) can
obtain rich representations than HRNet (3-9). The inner structure of the units
is illustrated in Fig. 5 where the purple rectangles describe STCs and the
gray rectangles are the multi-scale feature fusion process. To reduce
information loss, the sizes of feature maps are reduced in half but branch
channels are doubled gradually from top to bottom, and the top branch
consistently maintains high resolution featrue map size from end to end. To
make the network to obtain rich multi-scale features for precise predictions,
we design the four-stage feature fusion in one unit. In feature fusion
process, each sub-branch iteratively aggregates the features which are
downsampled or upsampled from other parallel sub-branches.

Figure 5: Structure of each STair Unit. Purple and gray rectangles describe
respectively STair Cells and multi-scale feature fusion process.

###  3.3 Multi-stage Dual Branch Structure

For feature re-usage, skip connections are widely applied between network
stages [16, 34]. However, existing methods pay little attention to feature re-
exploration, since the parameters grow quadratically as the densely connected
path width increases linearly. In this section, we will explain the
limitations of ResNet [33] which focus on feature re-usage, and DenseNet [46]
which focus on feature re-exploitation first. We then present the Multi-stage
Dual Branch Structure (MDBS), which enjoys the benefits from both path
topologies for learning good representations.

#### Revisiting ResNet and DenseNet

Traditional convolution feed-forward networks utilize the output of $L^{th}$
layer as the input for (${L+1}$)th layer, which can be summarized as:
$X_{L}=H_{L}$($X_{L}-1$). ($X_{L}$ is the output of $L^{th}$ layer, and
$H_{L}$ are the composite operations such as batch normalization, rectified
linear unit, or convolution). ResNet proposes a skip connection which bypasses
the non-linear transformations with an identity function:

$\displaystyle X_{L}=H_{L}(X_{L-1})+X_{L-1}.$ (3)

The main contribution of ResNet is that the gradients can propagate directly
through the identity function to relieve gradient attenuation and reinforce
the feature re-usage in deeper layers. However, ResNet neglects the features
re-exploration and the summation process in Eq. (3) may impede the information
flow in the network. After that, Huang et al. [46] propose DenseNet, where the
skip connections are used to concatenate the inputs to the outputs instead of
adding operation in order to improve the information flow between layers.
Consequently, the $L_{th}$ layer will receive the feature maps of all
preceding layers as inputs:

$\displaystyle X_{L}=H_{L}(X_{0}||X_{1}||...||X_{L-1}).$ (4)

$X_{0}||X_{1}||...||X_{L-1}$ refers to the feature maps concatenation ($||$)
of layer $0,…,(L-1)^{th}$. Since the width of the densely connected path and
the cost of GPU memory linearly increase as the network goes deeper, building
a deeper and wider densenet is substantially restricted.

#### Multi-stage Dual Branch Structure

MDBS combines advantages of both path learning [33, 46, 47], able to reinforce
information re-usage and re-exploration among multiple STair Units. In each
stage of MDBS, we feed the unit outputs into a simple 1$\times$1 conv to
generate two branches: consolidation branch and excavation branch. Similar to
DenseNet, we set a constant $n$ as the growth rate in excavation branch, which
is half of the unit first branch channel number. The small number of $n$ helps
slow the increase on width of branch excavation and the GPU memory occupation.
The stage inputs, consolidation branch and excavation branch of MDBS can be
expressed as follows:

$\displaystyle C_{i}=H_{i}(X_{i}^{out})[0:n]+H_{i-1}(X_{i-1}^{out})[0:n],$ (5)
$\displaystyle
E_{i}=H_{i}(X_{i}^{out})[n:end]||...||H_{0}(X_{0}^{out})[n:end],$ (6)
$\displaystyle X_{i+1}^{in}=C_{i}||E_{i},$ (7)

$X_{i}^{out}$ are the MDBS outputs of stage $i$ and $X_{i+1}^{in}$ are the STU
inputs of the stage $i+1$. $H_{i}$ is the shared composite operations that the
outputs are equally separated into two parts ($[0:n],[n:end]$) for
consolidation and excavation branches. As exhibited in Eq. (5) and Eq. (6),
element-wise summation is applied on consolidation branch for features re-
usage, and concatenation operation is applied on excavation branch for the
features re-exploration. After that, as listed in Eq. (7), two branch features
are concatenated together as the inputs for the next STU. As shown in in Fig.
2, the method can be readily attached after each unit for enhancing feature
utilization.

###  3.4 STair Fusion

In the existing methods [39, 16, 18], multi-scale structures are adopted to
focus on low-frequency semantic feature fusion. However, the quantity of high-
frequency features is another critical factor for precise localization tasks
[32] as the features contain rich texture representations and better
discrimination ability. To this end, we present a STair Fusion (STF) mechanism
to replenish multi-scale high-frequency features to the network. As shown in
Fig. 6, STF adopts downsampling to generate multi-scale images. After that, we
apply lightweight transformation blocks at each scale to change channels for
matching multiple STU branches. The transformation blocks contain just four
layers which help to reserve more high-frequency texture features. The
reserved high-frequency representations are then concatenated with multi-scale
low-frequency features of STU respectively. For the fusion stage, we take
upsampling and deconvolution methods to gradually enlarge small-scale
concatenated feature maps. We then sum four-scale feature maps to attain
multi-scale fusion. For efficiency purpose, we apply a reduction factors on
the transformation blocks, which is similar to STC context attention.
Significantly, we attempt to modify this mechanism into a multi-stage design.
The multi-stage design combines high-frequency features with low-frequency
features after each STU, where the high-frequency information may still be
lost in the next unit, and the results of single STF is better. Therefore, we
adopt the single fusion at the end of the network.

Figure 6: STair Fusion structure. Purple rectangles are multi-scale original
images, and gray rectangles are transformation blocks for matching channel
with STU multiple branches.

##  4 Experiments

STair Network is evaluated on two standard human pose estimation datasets,
i.e. the COCO keypoint-2017 dataset [48] and MPII keypoint [49] dataset. We
follow the process of [24] to decode the predicted heatmaps. The Adam
optimizer [50] is used in training, and we develop STNet on 4 NVIDIA 2080 Ti
GPUs.

#### Results on COCO

This dataset contains over 200,000 images and 250,000 human instances labeled
with 17 keypoints. The evaluation metric of the dataset is based on Object
Keypoint Similarity (OKS). Standard Average Precision (AP) and Average Recall
(AR) are used as the metric. AP and AR denotes the mean value of 10 OKS
numbers (OKS = 0.50, 0.55, … , 0.95), AP50 denotes AP at OKS = 0.5, APM
denotes the AP for medium objects. For this dataset, we follow HRNet and adopt
the same human instance detector to provide human bounding boxes for both the
validation and test sets. Table 4 lists the results of STNet and other
competitive methods on the COCO test-dev dataset.The trunk branch channel
number of STNet is 32.

Table 4: Comparison results on the COCO test-dev dataset. #Params are total parameters of the networks, and GFLOPs are total computational cost of the methods. * means STC without channel halving strategy. Method | Backbone | Input Size | Pretrain | #Params | GFLOPs | AP |  AP50 |  AP75 |  APM |  APL | AR
---|---|---|---|---|---|---|---|---|---|---|---
Mask-RCNN [23] | ResNet_50_FPN | - | Y | - | - | 63.1 | 87.3 | 68.7 | 57.8 | 71.4 | -
G-RMI [25] | ResNet_101 |  353$\times$257 | Y | 42.6M | 57.0 | 64.9 | 85.5 | 71.3 | 62.3 | 70.0 | 69.7
HRNet | HRNet_w32 |  384$\times$288 | N | 28.5M | 16.0 | 67.0 | 85.4 | 74.3 | 64.8 | 73.4 | 78.1
IPR [51] | ResNet_101 |  256$\times$256 | Y | 45.0M | 11.0 | 67.8 | 88.2 | 74.8 | 63.9 | 74.0 | -
G-RMI + extra data [25] | ResNet_101 |  353$\times$257 | Y | 42.6M | 57.0 | 68.5 | 87.1 | 75.5 | 65.8 | 73.3 | 73.3
SimpleBaseline [26] | ResNet_152 |  384$\times$288 | N | 68.6M | 35.5 | 69.5 | 90.1 | 77.0 | 66.4 | 75.3 | 75.5
SimpleBaseline | ResNet_50 |  384$\times$288 | N | 34.0M | 20.2 | 70.4 | 90.7 | 77.5 | 66.7 | 76.9 | 75.8
SimpleBaseline | ResNet_101 |  384$\times$288 | N | 53.0M | 27.9 | 71.9 | 91.1 | 79.8 | 68.7 | 78.0 | 77.4
CPN [16] | Resnet_Inception |  384$\times$288 | Y | - | - | 72.1 | 91.4 | 80.0 | 68.7 | 77.2 | 78.5
RMPE [52] |  PyraNet [13] |  320$\times$256 | - | 28.1M | 26.7 | 72.3 | 89.2 | 79.1 | 68.0 | 78.6 | -
CFN [53] | - | - | Y | - | - | 72.6 | 86.1 | 69.7 | 78.3 | 64.1 | -
CPN (ensemble) | ResNet_Inception |  384$\times$288 | Y | - | - | 73.0 | 91.7 | 80.9 | 69.5 | 78.1 | 79.0
SimpleBaseline | ResNet_152 |  384$\times$288 | Y | 68.6M | 35.6 | 73.7 | 91.9 | 81.1 | 70.3 | 80.0 | 79.0
HRNet | HRNet_w32 |  384$\times$288 | Y | 28.5M | 16.0 | 74.9 | 92.5 | 82.8 | 71.3 | 80.9 | 80.1
HRNet | HRNet_w48 |  384$\times$288 | Y | 63.6M | 32.9 | 75.5 | 92.5 | 83.3 | 71.9 | 81.5 | 80.5
TokenPose [54] | L/D24 |  384$\times$288 | Y | 29.8M | 22.1 | 75.9 | 92.3 | 83.4 | 72.2 | 82.1 | 80.8
1-stage | STNet |  256$\times$192 | N | 5.7M | 2.3 | 71.4 | 91.0 | 79.0 | 68.1 | 77.2 | 76.8
1-stage | STNet |  384$\times$288 | N | 5.7M | 5.2 | 72.5 | 91.0 | 79.7 | 68.9 | 78.5 | 77.8
2-stage | STNet |  256$\times$192 | N | 8.5M | 2.9 | 73.4 | 91.7 | 81.1 | 70.2 | 78.9 | 78.7
2-stage | STNet |  384$\times$288 | N | 8.5M | 6.5 | 74.8 | 92.0 | 82.1 | 71.4 | 80.7 | 80.0
3-stage | STNet |  256$\times$192 | N | 11.3M | 3.5 | 74.1 | 91.8 | 81.8 | 71.0 | 79.7 | 79.4
3-stage | STNet |  384$\times$288 | N | 11.3M | 7.8 | 75.3 | 92.1 | 82.7 | 71.8 | 81.2 | 80.4
3-stage* | STNet |  384$\times$288 | N | 20.3M | 12.6 | 75.9 | 92.3 | 83.4 | 72.5 | 81.8 | 81.1

As observed, without pre-training, our proposed method achieves very
encouraging 75.3% AP score on the 3-stage backbone with the 384$\times$288
input size. This result is higher than all the comparison models that are even
pre-trained. Specifically, compared with HRNet_32, the parameters of STNet is
fewer over 60% (11.3M), and the GFLOPs is less over 50% (7.8). Compared with
ResNet_152 backbone of SimpleBaseline, the gain of 3-stage network is 1.6%
with 384$\times$288 input size, and the parameters of STNet is over 83% fewer,
and the reduction on GFLOPs is nearly 80%. It is also noted that, the
computational cost of STNet is the least among these popular methods. In
contrast to the existing methods, the least reduction on parameters and GFLOPs
achieved by 3-stage also attains 60% and 30% respectively. Significantly, with
the same 384$\times$288 input size, 1-stage STNet obtains 5.5% improvements
with 80% drop on parameters and 68% drop in GFLOPs when compared with no
pretrained HRNet_32. In addition, STNet can achieve higher accuracy (75.9%)
once channel halving strategy is abandoned. We further report the comparisons
results on the COCO validation set in Table 5.

Table 5: Comparison results on the COCO validation set. OHKM means online hard keypoints mining [16]. Pretain means the method is/isn’t pretrained on the ImageNet classification task. * means STC without channel halving strategy. Method | Backbone | Input Size | Pretrain | #Params | GFLOPs | AP |  AP50 |  AP75 |  APM |  APL | AR
---|---|---|---|---|---|---|---|---|---|---|---
Hourglass [12] | 8_Hourglass |  256$\times$192 | N | 25.1M | 14.3 | 66.9 | - | - | - | - | -
CPN [16] | ResNet_50 |  256$\times$192 | Y | 27.0M | 6.2 | 68.6 | - | - | - | - | -
HRNet | HRNet_w32 |  384$\times$288 | N | 28.5M | 16.0 | 69.0 | 84.7 | 75.8 | 66.2 | 77.4 | 79.0
CPN+OHKM | ResNet_50 |  256$\times$192 | Y | 27.0M | 6.2 | 69.4 | - | - | - | - | -
SimpleBaseline [26] | ResNet_152 |  384$\times$288 | N | 68.6M | 35.5 | 70.2 | 88.2 | 77.3 | 66.8 | 77.1 | 76.1
SimpleBaseline | ResNet_50 |  256$\times$192 | Y | 34.0M | 8.9 | 70.4 | 88.6 | 78.3 | 67.1 | 77.2 | 76.3
SimpleBaseline | ResNet_101 |  256$\times$192 | Y | 53.0M | 27.9 | 71.4 | 89.3 | 79.3 | 68.1 | 78.1 | 77.1
HRNet | HRNet_w32 |  256$\times$192 | N | 28.5M | 7.1 | 72.1 | 89.5 | 78.6 | 69.5 | 78.0 | 78.6
SimpleBaseline | ResNet_152 |  256$\times$192 | Y | 68.6M | 15.7 | 72.0 | 89.3 | 79.8 | 68.7 | 78.9 | 77.8
HRNet [18] | HRNet_w32 |  256$\times$192 | Y | 28.5M | 7.1 | 74.4 | 90.5 | 81.9 | 70.8 | 81.0 | 79.8
HRNet | HRNet_w32 |  384$\times$288 | Y | 28.5M | 16.0 | 75.8 | 90.6 | 82.7 | 71.9 | 82.8 | 81.0
HRNet | HRNet_w48 |  384$\times$288 | Y | 63.6M | 32.9 | 76.3 | 90.8 | 82.9 | 72.3 | 83.4 | 81.2
1-stage | STNet |  256$\times$192 | N | 5.7M | 2.3 | 72.1 | 89.1 | 79.2 | 68.7 | 78.7 | 77.6
1-stage | STNet |  384$\times$288 | N | 5.7M | 5.2 | 73.2 | 89.1 | 80.0 | 69.4 | 80.1 | 78.6
2-stage | STNet |  256$\times$192 | N | 8.5M | 2.9 | 73.9 | 89.7 | 80.8 | 70.4 | 80.6 | 79.1
2-stage | STNet |  384$\times$288 | N | 8.5M | 6.5 | 75.6 | 90.0 | 81.8 | 71.8 | 82.5 | 80.6
3-stage | STNet |  256$\times$192 | N | 11.3M | 3.5 | 74.8 | 89.8 | 81.5 | 71.5 | 81.3 | 79.9
3-stage | STNet |  384$\times$288 | N | 11.3M | 7.8 | 76.2 | 90.4 | 82.4 | 72.5 | 82.9 | 81.2
3-stage* | STNet |  384$\times$288 | N | 20.3M | 12.6 | 76.8 | 90.7 | 83.3 | 73.3 | 83.4 | 81.8

#### Results on MPII

This dataset contains 25,000 images with 40,000 human instances labeled with
16 keypoints. The evaluation metric of the dataset is the Head-normalized
Probability of Correct Keypoint (PCKh) score. For MPII testing, we adopt the
official testing strategy to use the provided human bounding boxes to estimate
joints. We follow the six-scale testing procedure in [41, 55, 13], and the
<EMAIL_ADDRESS>results are reported in Table 6. As the parameters and layer numbers
of three SimpleBaseline structures (ResNet_50,101,152) increase gradually,
their network capability becomes stronger. The results however show that
stacking single receptive field layers tends overfitting without time-
consuming pre-training. Table 6 shows the comparison results on MPII
validation dataset with<EMAIL_ADDRESS>As observed, STNet outperforms HRNet and
SimpleBaseline with much fewer parameters and GFLOPs. Compared to
SimpleBaseline (ResNet_152), the gains of three different STNet structures are
3.4%, 4.3% and 4.9% with 92%, 88% and 84% parameters drop.

Table 6: Comparison results on the MPII validation set.  Method | Backbone | Input Size | Pretrain | #Param | GFLOPs | Head | Shoulder | Elbow | Wrist | Hip | Knee | Ankle | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
SimpleBaseline [26] | ResNet_50 |  256$\times$256 | N | 34.0M | 12.0 | 96.5 | 94.8 | 87.7 | 82.2 | 87.9 | 82.3 | 77.1 | 87.6
SimpleBaseline | ResNet_101 |  256$\times$256 | N | 53.0M | 16.5 | 96.1 | 94.3 | 86.1 | 80.3 | 87.2 | 81.4 | 76.2 | 86.6
SimpleBaseline | ResNet_152 |  256$\times$256 | N | 68.6M | 21.0 | 95.7 | 93.7 | 84.9 | 77.9 | 85.5 | 79.2 | 74.3 | 85.2
HRNet [18] | HRNet_w32 |  256$\times$256 | N | 28.5M | 9.5 | 95.6 | 94.2 | 88.1 | 83.7 | 88.1 | 83.7 | 79.3 | 88.1
HRNet | HRNet_w48 |  256$\times$256 | N | 63.6M | 19.5 | 96.0 | 94.6 | 88.8 | 84.1 | 87.4 | 83.6 | 80.4 | 88.4
TokenPose [54] | L/D24 |  256$\times$256 | Y | 28.1M | - | 97.1 | 95.9 | 90.4 | 86.0 | 89.3 | 87.1 | 82.5 | 90.2
1-stage | STNet |  256$\times$256 | N | 5.7M | 3.1 | 96.7 | 94.9 | 89.0 | 83.6 | 88.4 | 84.4 | 79.7 | 88.6
2-stage | STNet |  256$\times$256 | N | 8.5M | 3.9 | 96.9 | 95.6 | 89.8 | 84.8 | 89.2 | 84.7 | 81.6 | 89.5
3-stage | STNet |  256$\times$256 | N | 11.3M | 4.6 | 97.0 | 95.9 | 90.2 | 85.2 | 89.6 | 86.4 | 83.1 | 90.1

###  4.1 Ablation Analysis

#### STC Structure Exploration

For the STC design, we explore a number of structures as illustrated in Fig.
7. Among multiple structures, though some have fewer parameters and lower
computational cost (Fig. 7 (B), (C)), they lead to very limited performance
practically. Meanwhile, we observe that channel separation manner is not
preferable to extract significant local features. On the one hand, the channel
separation manner leads to feature incoherence that the receptive fields of
the following branch are not based on the previous one. On the other hand, the
channel separation manner with the sum operation may cause some redundancies.
By contrast, the final structure adopted by STC (Fig. 7 (E)) attains the best
performance with limited parameters and GFLOPs.

Figure 7: Different structures investigated in our paper. Structure E is adopted in this work. Table 7: Comparison results of multiple STC structures with 1-stage network. Structure E is adopted in this work. Structure | pretrain | Input Size | AP |  AP50 |  AP75 |  APM |  APL | AR
---|---|---|---|---|---|---|---|---
(A) | N |  256 $\times$ 192 | 71.5 | 88.9 | 78.5 | 68.0 | 78.3 | 77.0
(B) | N |  256 $\times$ 192 | 70.6 | 88.7 | 77.9 | 67.0 | 77.5 | 76.3
(C) | N |  256 $\times$ 192 | 69.7 | 88.3 | 76.8 | 66.4 | 76.3 | 75.4
(D) | N |  256 $\times$ 192 | 71.5 | 88.7 | 78.8 | 67.8 | 78.4 | 77.0
(E) | N |  256 $\times$ 192 | 72.1 | 89.1 | 79.2 | 68.7 | 78.7 | 77.6

#### Ablation Analysis of STC

STC is proposed to focus on extracting multi-scale local representations
through aggregating the outputs from four atrous convolution layers with
different receptive fields. In this section, we take a closer comparison on
STC to evaluate network performance with the different branch numbers. Table 8
shows the results on COCO validation dataset with the $256\times 192$ input
size. As demonstrated in Table 8, the performance gain is obviously in
different stage networks, and the gain of accuracy becomes slight as we
increase branch number more than 4. For 6-branch design, the channel number of
the largest kernel size convolution is only 1 which brings a marginal
increase. Fig. 8 shows the 1-stage STNet inference speed of the network with
different branches on COCO validation dataset where the inference speed
gradually decreases without obvious accuracy improvement. As such, 4-branch
design is adopted in this work.

Table 8: Ablation analysis of STC with different branch numbers. $K$ means the kernel size. #Branch | Method | $K$ | AP |  |  AP50 |  AP75 |  APM |  APL | AR
---|---|---|---|---|---|---|---|---|---
c = 1 | 1-stage | 3 | 70.6 |  | 88.4 | 77.5 | 67.3 | 77.2 | 76.4
c = 2 | 1-stage | 3,5 | 71.6 |  | 89.1 | 78.8 | 68.0 | 78.4 | 77.1
c = 3 | 1-stage | 3,5,7 | 71.9 |  | 88.9 | 78.9 | 68.2 | 78.7 | 77.4
c = 4 | 1-stage | 3,5,7,9 | 72.1 |  1.5 $\uparrow$ | 89.1 | 79.2 | 68.7 | 78.7 | 77.6
c = 5 | 1-stage | 3,5,7,9,11 | 72.1 |  | 89.1 | 79.1 | 68.7 | 78.7 | 77.5
c = 6 | 1-stage | 3,5,7,9,11,13 | 72.2 |  | 89.3 | 78.8 | 68.9 | 78.6 | 77.6
c = 1 | 2-stage | 3 | 71.8 |  | 88.8 | 79.2 | 68.4 | 78.5 | 77.3
c = 4 | 2-stage | 3,5,7,9 | 73.9 |  2.1 $\uparrow$ | 89.7 | 80.8 | 70.4 | 80.6 | 79.1
c = 1 | 3-stage | 3 | 73.2 |  | 89.3 | 80.3 | 70.0 | 79.8 | 78.6
c = 4 | 3-stage | 3,5,7,9 | 74.8 |  1.6 $\uparrow$ | 89.8 | 81.5 | 71.5 | 81.3 | 79.9
Figure 8: Accuracy and Speed of STC vs. branch numbers.

#### Ablation Analysis of MDBS and STF

We now examine Multi-stage Dual Branch Structure, and STair Fusion on the COCO
validation dataset. For simplicity, we take 1-stage and 2-stage STNets with
the $256\times 192$ input size as the illustrative examples. Table 9 reports
the performance when we gradually apply MDBS and STF on the networks. As
observed, MDBS can lead to $0.6\%$ improvement than the baseline model on the
1-stage network. On the other hand, when STF is applied, compensation of the
high-frequency information can further increase AP by $1.4\%$. Similarly, MDBS
attains $1.7\%$ improvement and STF further achieves $0.6\%$ accuracy increase
on the 2-stage network.

Table 9: Ablation analysis of STNet with different components.  Method | pretrain | Input Size | #Params | GFLOPs | MDBS | STF | AP
---|---|---|---|---|---|---|---
1-stage | N |  256$\times$192 | 3.35M | 1.74 | $\times$ | $\times$ | 70.1
1-stage | N |  256$\times$192 | 3.37M | 1.79 | ✓ | $\times$ | 70.7
1-stage | N |  256$\times$192 | 5.74M | 2.32 | ✓ | ✓ | 72.1
2-stage | N |  256$\times$192 | 6.11M | 2.24 | $\times$ | $\times$ | 71.6
2-stage | N |  256$\times$192 | 6.17M | 2.36 | ✓ | $\times$ | 73.3
2-stage | N |  256$\times$192 | 8.53M | 2.89 | ✓ | ✓ | 73.9

#### Ablation Analysis of STU Number

As the basic block of the multi-stage structure, STU can be simply stacked for
adjusting the network capability. We perform comparison experiments on
multiple architectures from one to three stages with two kinds of input sizes
(256$\times$192 and 384$\times$288). We demonstrate the comparison results in
Table 10 where both the MDBS and STF mechanisms are applied in this section,
but pre-training process is not adopted. Table 10 demonstrates that the
network performance is consistently improved with the increase of unit number.

Table 10: Performance of STU with different unit numbers. Input Size | Pretrain | 1-stage | 2-stage | 3-stage
---|---|---|---|---
256$\times$192 | N | 72.1 | 73.9 | 74.8
384$\times$288 | N | 73.2 | 75.6 | 76.2

##  5 Visualization Analysis

Fig. 10 visualizes some successful prediction cases, and the red dotted
circles demonstrate some challenging scenes. The small and vague person
subjects (5th case of 1st row), serious self-occlusion (3rd and 5th cases of
2nd row), serious occlusion (5th case of 3rd row) and ambiguous post (1st and
2nd cases of 4th row) can be predicted successfully due to the powerful local
feature aggregation capability of STNet. In addition, Fig. 9 illustrates some
failure cases of our method where the red circles point out the error
predictions. The ambiguous occlusion and dark illumination are still
challenges which we will focus in the future works.

Figure 9: Several failure cases. Figure 10: Successful cases visualization.
The dotted circles denote some difficult scenes: small, vague, occlusion,
ambiguity.

##  6 Conclusion

In this work, we present a small yet effective multi-stage network for precise
keypoint localization. To reduce the computational cost while maintaining
superior performance, we propose a basic feature extraction block to focus on
aggregating more diverse local representations through adopting multiple
kernel sizes with fewer parameters. We alleviate the information loss problem
from two aspects. Within each STair Unit, we keep high resolution feature maps
to relieve feature loss. Outside the units, we develop a dual path structure
to enhance feature re-usage and re-exploitation with low computational cost.
Meanwhile, we design another mechanism to extract high-frequency texture
representations. We test the effectiveness of our method through evaluations
on standard pose estimation datasets, and the results demonstrate that the
STNet’s superiority with remarkable efficiency on parameters and GFLOPs.

## Acknowledgements

The work was partially supported by the following: National Natural Science
Foundation of China under no.61876155; Natural Science Foundation of Jiangsu
Province BE2020006-4; Key Program Special Fund in XJTLU under no. KSF-T-06,
KSF-E-26.

## References

  * [1] C. Wang, Y. Wang, A. L. Yuille, An approach to pose-based action recognition, in: CVPR, 2013.
  * [2] Z. Liang, X. Wang, R. Huang, L. Lin, An expressive deep model for human action parsing from a single image, in: ICME, 2014.
  * [3] J. Zhu, W. Zou, Z. Zhu, L. Xu, G. Huang, Action machine: Toward person-centric action recognition in videos, IEEE Signal Processing Letters (2019).
  * [4] C. Wang, X. Song, Robust head pose estimation via supervised manifold learning, Neural Networks 53 (2014) 15–25.
  * [5] P. Li, J. Zhang, Z. Zhu, Y. Li, L. Jiang, G. Huang, State-aware re-identification feature for multi-target multi-camera tracking, in: CVPR Workshops, 2019.
  * [6] B. Xiaohan Nie, C. Xiong, S.-C. Zhu, Joint action recognition and pose estimation from video, in: CVPR, 2015.
  * [7] N.-G. Cho, A. L. Yuille, S.-W. Lee, Adaptive occlusion state estimation for human pose tracking under self-occlusions, Pattern Recognition (2013).
  * [8] J. J. Tompson, A. Jain, Y. LeCun, C. Bregler, Joint training of a convolutional network and a graphical model for human pose estimation, in: NIPS, 2014.
  * [9] A. Toshev, C. Szegedy, Deeppose: Human pose estimation via deep neural networks, in: CVPR, 2014.
  * [10] S.-E. Wei, V. Ramakrishna, T. Kanade, Y. Sheikh, Convolutional pose machines, in: CVPR, 2016.
  * [11] M. Andriluka, S. Roth, B. Schiele, Pictorial structures revisited: People detection and articulated pose estimation, in: CVPR, 2009.
  * [12] A. Newell, K. Yang, J. Deng, Stacked hourglass networks for human pose estimation, in: ECCV, 2016.
  * [13] W. Yang, S. Li, W. Ouyang, H. Li, X. Wang, Learning feature pyramids for human pose estimation, in: CVPR, 2017.
  * [14] P. Witoonchart, P. Chongstitvatana, Application of structured support vector machine backpropagation to a convolutional neural network for human pose estimation, Neural Networks 92 (2017) 39–46.
  * [15] Z. Cao, T. Simon, S.-E. Wei, Y. Sheikh, Realtime multi-person 2d pose estimation using part affinity fields, in: CVPR, 2017.
  * [16] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, J. Sun, Cascaded pyramid network for multi-person pose estimation, in: CVPR, 2018.
  * [17] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele, Deepercut: A deeper, stronger, and faster multi-person pose estimation model, in: ECCV, 2016\.
  * [18] K. Sun, B. Xiao, D. Liu, J. Wang, Deep high-resolution representation learning for human pose estimation, in: CVPR, 2019.
  * [19] A. Newell, Z. Huang, J. Deng, Associative embedding: End-to-end learning for joint detection and grouping, in: NIPS, 2017.
  * [20] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, B. Schiele, Deepcut: Joint subset partition and labeling for multi person pose estimation, in: CVPR, 2016.
  * [21] G. Papandreou, T. Zhu, L.-C. Chen, S. Gidaris, J. Tompson, K. Murphy, Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model, in: ECCV, 2018.
  * [22] B. Cheng, B. Xiao, J. Wang, H. Shi, T. S. Huang, L. Zhang, Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation, in: CVPR, 2020.
  * [23] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: ICCV, 2017.
  * [24] J. Huang, Z. Zhu, F. Guo, G. Huang, The devil is in the details: Delving into unbiased data processing for human pose estimation, in: CVPR, 2020.
  * [25] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, K. Murphy, Towards accurate multi-person pose estimation in the wild, in: CVPR, 2017.
  * [26] B. Xiao, H. Wu, Y. Wei, Simple baselines for human pose estimation and tracking, in: ECCV, 2018.
  * [27] F. Zhang, X. Zhu, H. Dai, M. Ye, C. Zhu, Distribution-aware coordinate representation for human pose estimation, in: CVPR, 2020.
  * [28] H. Zhang, H. Ouyang, S. Liu, X. Qi, X. Shen, R. Yang, J. Jia, Human pose estimation with spatial contextual information, arXiv preprint arXiv:1901.01760 (2019).
  * [29] F. Zhang, X. Zhu, M. Ye, Fast human pose estimation, in: CVPR, 2019.
  * [30] Z. Li, J. Ye, M. Song, Y. Huang, Z. Pan, Online knowledge distillation for efficient pose estimation, in: CVPR, 2021.
  * [31] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: CVPR, 2017.
  * [32] H. Wang, X. Wu, Z. Huang, E. P. Xing, High-frequency component helps explain the generalization of convolutional neural networks, in: CVPR, 2020.
  * [33] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: CVPR, 2016.
  * [34] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, Mobilenetv2: Inverted residuals and linear bottlenecks, in: CVPR, 2018.
  * [35] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: CVPR, 2018.
  * [36] X. Li, W. Wang, X. Hu, J. Yang, Selective kernel networks, in: CVPR, 2019.
  * [37] S. Xie, R. Girshick, P. Dollár, Z. Tu, K. He, Aggregated residual transformations for deep neural networks, in: CVPR, 2017.
  * [38] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: NIPS, 2012.
  * [39] Y. Cai, Z. Wang, Z. Luo, B. Yin, A. Du, H. Wang, X. Zhou, E. Zhou, X. Zhang, J. Sun, Learning delicate local representations for multi-person pose estimation, arXiv preprint arXiv:2003.04030 (2020).
  * [40] W. Li, Z. Wang, B. Yin, Q. Peng, Y. Du, T. Xiao, G. Yu, H. Lu, Y. Wei, J. Sun, Rethinking on multi-stage networks for human pose estimation, arXiv preprint arXiv:1901.00148 (2019).
  * [41] X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, X. Wang, Multi-context attention for human pose estimation, in: CVPR, 2017.
  * [42] L. Ke, M.-C. Chang, H. Qi, S. Lyu, Multi-scale structure-aware network for human pose estimation, in: ECCV, 2018.
  * [43] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, TPAMI (2017).
  * [44] Y. Fang, Y. Li, X. Tu, T. Tan, X. Wang, Face completion with hybrid dilated convolution, Signal Processing: Image Communication (2020).
  * [45] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861 (2017).
  * [46] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks, in: CVPR, 2017.
  * [47] Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, J. Feng, Dual path networks, in: NIPS, 2017\.
  * [48] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: ECCV, 2014.
  * [49] M. Andriluka, L. Pishchulin, P. Gehler, B. Schiele, 2d human pose estimation: New benchmark and state of the art analysis, in: CVPR, 2014.
  * [50] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [51] X. Sun, B. Xiao, F. Wei, S. Liang, Y. Wei, Integral human pose regression, in: ECCV, 2018.
  * [52] H.-S. Fang, S. Xie, Y.-W. Tai, C. Lu, Rmpe: Regional multi-person pose estimation, in: ICCV, 2017.
  * [53] S. Huang, M. Gong, D. Tao, A coarse-fine network for keypoint localization, in: ICCV, 2017.
  * [54] Y. Li, S. Zhang, Z. Wang, S. Yang, W. Yang, S.-T. Xia, E. Zhou, Tokenpose: Learning keypoint tokens for human pose estimation (2021).
  * [55] W. Tang, P. Yu, Y. Wu, Deeply learned compositional models for human pose estimation, in: ECCV, 2018.