# An Iterative Approach to Data-Driven Inference for Decoding Oscillator
Network Structures

Shicheng Lia,†, Bharat Singhalb,†, and Jr-Shin Lib,c
† Both authors contribute equally and share the first authorship.a Department
of Mechanical Engineering and Materials Science, Washington University in St.
Louis, St. Louis, MO, USA. b Department of Electrical and Systems Engineering,
Washington University in St. Louis, St. Louis, MO, USA. c Division of Biology
and Biomedical Sciences, Washington University in St. Louis, St. Louis, MO,
63130, USA. Correspondence to<EMAIL_ADDRESS>work was supported by the
Air Force Office of Scientific Research under the award FA9550-21-1-0335.

###### Abstract

In complex networks, interactions between multiple agents give rise to an
array of intricate global dynamics, ranging from synchronization to cluster
formations. Decoding the connectivity structure as well as the types of
interactions from measurement data is the first step toward understanding
these intriguing behaviors. In this paper, we present a bilinear optimization
framework to infer both the connectivity and interaction functions of
oscillator networks with the identical class of coupling functions. We then
propose an iterative algorithm to solve the resulting bilinear problem and
illustrate its convergence. We validate our approach on both simulated and
noisy experimental datasets, where we demonstrate its effectiveness compared
with existing approaches.

##  I Introduction

Complex systems with multiple interacting agents are prevalent in nature and
human society on different scales [1]. The dynamic interplay exhibited by
these interconnected agents gives rise to a multitude of intricate network
functions, the understanding of which requires decoding the connectivity and
the specific nature of interactions, i.e., coupling functions, between the
agents. For example, elucidating the neuronal network in the suprachiasmatic
nucleus (SCN) is critical for our understanding of the mechanisms driving SCN
synchronization [2], while determining the changes in the brain network
structure is important to predict the onset of seizures [3]. In these
scenarios, time series recording from each agent (node) is accessible, and the
objective is to determine both the network structure and the nature of
interactions.

The aforementioned applications have led to the development of several network
inference techniques. The fundamental idea behind these methods is to learn
the network model, i.e., the network topology and coupling functions, that
explain the recorded time series data111It is important to note that the
information-theoretic measures such as correlation, or mutual-information, are
not considered as they determine the functional connectivity, not the
effective connectivity (see [4] for the difference between functional and
effective connectivity) . The primary difference arises from the variations in
the candidate network models and the approach used to identify the best-fit
model. For example, inference methods employing Granger causality typically
fit a vector autoregressive model to the time series, and the autoregression
coefficients are determined using classical likelihood ratio tests [4, 5].
Similarly, regression-based methods transform the network inference problem
into a least-squares (or LASSO [6]) problem by utilizing the orthonormal
function approximation [7, 8, 9].

In numerous experimental and physiological scenarios, nodes within the network
are interconnected through similar forms of interaction (coupling) functions.
For example, electrochemical oscillator networks have resistive coupling [10],
and neurons in the SCN communicate primarily through VIP and GABA signaling
[11]. These applications necessitate inference techniques that can utilize the
relationship between the network coupling functions of different nodes.
Indeed, the inability to harness these relationships would result in
suboptimal performance of inference techniques.

In this paper, employing this observation, we consider oscillator networks
with similar coupling functions and propose a bilinear optimization
formulation for the network inference problem. Specifically, we first
transform the network inference task into a bilinear optimization by
leveraging the Fourier decomposition of orthogonal functions. We then
introduce an iterative algorithm to solve the resulting bilinear problem and
substantiate its convergence properties. We compare our method to some of the
existing network inference techniques and show its effectiveness on networks
of Kuramoto and Stuart-Landau oscillators. We further validate our technique
on experimental data obtained from a network of nonlinear electronic
oscillators, illustrating its robustness against measurement noise.

##  II Background and Problem Formulation

In this section, we introduce the network model for coupled limit-cycle
oscillators and formulate the network inference problem.

###  II-A Background

We consider a network of $N$ limit-cycle oscillators such that the dynamics of
node $i$ comprises its own dynamics and the pair-wise coupling with other
nodes in the network, i.e.,

$\frac{d}{dt}X_{i}(t)=f_{i}(X_{i}(t))+\sum_{\begin{subarray}{c}j=1\\\ j\neq
i\end{subarray}}^{N}\epsilon_{ij}K(X_{i}(t),X_{j}(t)).$ (1)

Here, $X_{i}\in\mathbb{R}^{n}$ $(n\geq 2)$, $f_{i}$ are the state and self-
dynamics of oscillator $i$, $K$ is the coupling function, and $\epsilon_{ij}$
scales the strength of connection from node $j$ to $i$. If two nodes are
uncoupled then $\epsilon_{ij}=\epsilon_{ji}=0$. For the network (1), if the
coupling strengths are sufficiently weak then a reduced one-dimension model
capturing the phase dynamics of each oscillator can be obtained by leveraging
the phase-reduction theory (see ref. [12] for more details). As a result, the
phase dynamics of oscillator $i$ can be written as

$\dot{\varphi_{i}}(t)=\omega_{i}+\sum_{\begin{subarray}{c}j=1\\\ j\neq
i\end{subarray}}^{N}k_{ij}\alpha(\varphi_{j}(t)-\varphi_{i}(t))$ (2)

where $\varphi_{i},\omega_{i}\in\mathbb{R}$ are the phase and natural
frequency of the oscillator $i$, $\alpha$ is a $2\pi-$ periodic coupling
function, and $k_{ij}$ denote the strength of connection from node $j$ to node
$i$, if there is a connection, otherwise $k_{ij}=0$. It is important to note
that our objective is not to infer the coupling functions $K$ of network (1).
Instead, we aim to recover the phase coupling function $\alpha$ of (2) as
accurate inference of both the coupling function and the network connectivity
(represented by $\\{k_{ij}\\}$) is crucial to predict the synchronization
behavior and to control the network functionality, for instance, network
desynchronization.

###  II-B Network Inference Problem

Given the time series recordings from each node $\\{x_{i}(t_{k})\\}$, where
$i=1,\dots,N$ and $k=1,\dots,M+1$, and $x_{i}$ is the one-dimensional
observable of oscillator state $X_{i}$, we want to efficiently (with limited
measurements) infer the connectivity structure $\\{k_{ij}\\}$ and phase
coupling functions of the network.

##  III Theory and Algorithm

In this section, we present the bilinear optimization formulation of the
network inference problem and the iterative algorithm to solve the resulting
optimization problem.

To begin, we first obtain the phase trajectories of each oscillator from their
time series data. These phases can be determined using conventional phase
estimation algorithms such as peak-finding or wavelet transform [13]. The next
step is to infer the unknown parameters $\\{k_{ij}\\}$ and the function
$\alpha$ of phase model (2) by plugging in the estimated phases. To this end,
without loss of any generality, we approximate the $2\pi-$ periodic smooth
coupling function using truncated Fourier series, i.e.,

$\alpha(\Delta\varphi_{ij})\approx
a_{0}+\sum_{m=1}^{r}a_{m}\cos(m(\Delta\varphi_{ij}))+b_{m}\sin(m(\Delta\varphi_{ij})),$
(3)

where $\Delta\varphi_{ij}=\varphi_{j}-\varphi_{i}$, $r$ is the number of
Fourier harmonics that can be taken sufficiently large to approximate the
coupling function with arbitrary accuracy, and $\\{a_{m},b_{m}\\}$ are the
unknown Fourier coefficients. As a result of this Fourier decomposition, the
phase dynamics (3) can be expressed as

$\dot{\varphi_{i}}(t)\approx\bar{\omega}_{i}+\sum^{N}_{j=1\neq
i}k_{ij}\sum_{m=1}^{r}a_{m}\cos(m\Delta\varphi_{ij})+b_{m}\sin(m\Delta\varphi_{ij}),$
(4)

where $\bar{\omega}_{i}=\omega_{i}+\sum^{N}_{j=1,j\neq i}k_{ij}a_{0}$ is the
adjusted frequency. On a closer inspection, we observe that (4) is in a
bilinear matrix form

$\dot{\varphi_{i}}(t)=K_{i}^{T}Z_{i}(t)A+\epsilon_{i}(t),$ (5)

where $\epsilon_{i}(t)$ corresponds to the approximation error,
$K=[k_{i1},\dots,k_{iN},\bar{\omega}_{i}]^{T}\in\mathbb{R}^{N}$ contains the
incoming connections to node $i$,
$A=[a_{1},\dots,a_{r},b_{1},\dots,b_{r},1]^{T}\in\mathbb{R}^{2r+1}$ represents
the coupling function coefficients, and

$Z_{i}(t)=\left[\begin{smallmatrix}\cos(\Delta\varphi_{i1})&\dots&\cos(r\Delta\varphi_{i1})&\sin(\Delta\varphi_{i1})&\dots&\sin(r\Delta\varphi_{i1})&0\\\
\vdots&&\vdots&\vdots&&\vdots&\vdots\\\
\cos(\Delta\varphi_{iN})&\dots&\cos(r\Delta\varphi_{iN})&\sin(\Delta\varphi_{iN})&\dots&\sin(r\Delta\varphi_{iN})&0\\\
0&\dots&0&0&\dots&0&1\end{smallmatrix}\right].$

By combining (5) for all sample points $t_{1},\dots,t_{M+1}$, we obtain a set
of $M$ bilinear equations, where $\dot{\varphi_{i}}$ is obtained by the first-
order approximation of the derivative, i.e.,
$\dot{\varphi_{i}}(t_{k})=\frac{\varphi(t_{k+1})-\varphi(t_{k})}{t_{k+1}-t_{k}}$,
$k=1,\dots,M$. The network inference problem (for node $i$) can now be
formulated as an optimization problem where the estimates of the unknown
coefficient vectors $K_{i}$ and $A$ are determined by

$(\hat{K_{i}},\hat{A})=\arg\min\limits_{K_{i},A}\mathcal{L}_{i}(K_{i},A),$ (6)

where
$\mathcal{L}_{i}(K_{i},A)=\sum_{k=1}^{M}(\dot{\varphi_{i}}(t_{k})-K_{i}^{T}Z_{i}(t_{k})A)^{2}$.

We want to emphasize that our network inference formulation is different from
the existing works using orthonormal functional approximation [7, 8] as we
assume the coupling functions $\alpha$ to be identical and vary the strengths
of the connections. Specifically, existing works model the connection between
node $i$ and $j$ as $\alpha_{ij}(\Delta\varphi_{ij})$, different from our
formulation where we take it as $k_{ij}\alpha(\Delta\varphi_{ij})$ with
$k_{ij}$ being an unknown constant. As a result, the network inference problem
turns into a bilinear least-squares form, different from the linear least-
squares in existing literature. The rationale behind this model consideration,
as explained in section I, is to facilitate the efficient recovery of network
parameters by reducing the number of unknown parameters from $N^{2}\times 2r$
to $N^{2}+2r$.

###### Remark 1

Note that if $(\hat{K}_{i},\hat{A})$ is a solution of (6), so is
$(c\hat{K}_{i},\hat{A}/c)$ for any $c\neq 0$. To avoid this ambiguity, we
normalize $A$, i.e., $\|A\|_{2}=1$, and assume the $\hat{K}_{i}$ to have non-
negative elements as the coupling strength cannot be negative.

###  III-A Iterative Algorithm for Network Inference

In this section, we propose an iterative method to solve the bilinear
optimization problem (6) while following the non-negative $\hat{K}_{i}$
constraint. The principle idea of our algorithm is to solve (6) for each node
iteratively. In the following, we describe the network inference algorithm in
detail.

We start by arbitrarily initializing Fourier coefficients of node 1,
$\hat{A}_{1}^{(0)}$ such that $\|\hat{A}_{1}^{(0)}\|_{2}=1$ and then obtain
connectivity coefficients $\hat{K}^{(1)}_{1}$ after plugging in
$\hat{A}_{1}^{(0)}$ in (6). Specifically,

$\hat{K}^{(1)}_{1}=\arg\min\limits_{K_{1}\geq
0}\mathcal{L}_{1}(K_{1},\hat{A}_{1}^{(0)})=\arg\min\limits_{K_{1}\geq
0}\|Y_{1}-X_{1,A}^{(0)}K_{1}\|_{2}^{2},$

where

$Y_{1}=\begin{bmatrix}\dot{\varphi}_{1}(t_{1})\\\ \vdots\\\
\dot{\varphi}_{1}(t_{M})\end{bmatrix},\qquad
X_{1,A}^{(0)}=\begin{bmatrix}(Z_{1}(t_{1})A_{1}^{(0)})^{T}\\\ \vdots\\\
(Z_{1}(t_{M})A_{1}^{(0)})^{T}\end{bmatrix},$

which is a non-negative least-squares problem. Now the estimated
$\hat{K}^{(1)}_{1}$ are used to update the Fourier coefficients, i.e.,
$\hat{A}_{1}^{(1)}=\arg\min\limits_{A}\mathcal{L}_{1}(K_{1}^{(1)},A)$ (the
superscript denotes the iteration number). The updated Fourier coefficients
can be directly obtained as

$\hat{A}_{1}^{(1)}=({X_{1,K}^{(1)}}^{T}X_{1,K}^{(1)})^{-1}{X_{1,K}^{(1)}}^{T}Y_{1},$

where

$X_{1,K}^{(1)}=\begin{bmatrix}{K_{1}^{(1)}}^{T}Z_{1}(t_{1})\\\ \vdots\\\
{K_{1}^{(1)}}^{T}Z_{1}(t_{M})\end{bmatrix}.$

This process is repeated till the Fourier coefficient converges, i.e.,
$\hat{A}_{1}^{(I_{1})}-\hat{A}_{1}^{(I_{1}-1)}<\epsilon,$ where $\epsilon$ is
the user-defined threshold and $I_{1}$ is the number of iterations. The output
Fourier coefficients at $I_{1}$ iteration are then used as an initialization
to solve (6) for node 2. The principle idea behind this sequential
initialization is to leverage the fact that all nodes have identical Fourier
coefficients. This process of sequential initialization, i.e.,
$\hat{A}_{i}^{(I_{i})}=\hat{A}_{i+1}^{(0)},i=1,\dots,N-1$ is executed for all
nodes.

Input: Normalized $A_{1}^{(0)}$, $\epsilon$, and $\Lambda$

Output: $\hat{A}$, $\hat{K}_{i}$ for $i=1,\dots,N$

Data: $\varphi_{i}(t_{k})$ and $\dot{\varphi}_{i}(t_{k})$ for $k=1,\dots,M$

$\lambda=1$

// $\lambda$: network iteration number

1 while  _$\lambda <\Lambda$_ do

$i,j=1$

// $i,j$: node and iteration number

2 $\hat{K}^{(j)}_{i}=\arg\min\limits_{K_{i}\geq
0}\mathcal{L}_{i}(K_{i},\hat{A}_{i}^{(j-1)})$

3 $\hat{A}^{(j)}_{i}=\arg\min\limits_{A}\mathcal{L}_{i}(\hat{K}^{(j)}_{i},A)$

4 Normalize $\hat{A}^{(j)}_{i}$

5 if  _$\|\hat{A}^{(j)}_{i}-\hat{A}^{(j-1)}_{i}\|_{2}\leq\epsilon$_ then

6 if  _$i==N$_ then

// Solve (6) for node 1.

7 $\lambda=\lambda+1$ and $I_{N}=j$

8 $\hat{A}_{1}^{(0)}=\hat{A}_{N}^{(j)}$ and go to step 2.

9 else

// Solve (6) for the next node.

10 $I_{i}=j$, $i=i+1$, and $j=1$

11 $\hat{A}_{i}^{(0)}=\hat{A}_{i-1}^{(I_{i-1})}$ and go to step 4.

12

13 else

14 $j=j+1$ and go to step 4.

15

16Estimate $\hat{A}=\frac{1}{N}\sum_{i=1}^{N}A_{i}^{(I_{i})}$

Estimate $\hat{K}_{i}=\arg\min\limits_{K_{i}\geq
0}\mathcal{L}_{i}(K_{i},\hat{A})\quad i=1,\dots,N.$

Algorithm 1 Estimation of $A$ and $K_{i}$, $i=1,\dots,N$.

Note that since we start from the random initialization, i.e.,
$\hat{A}_{1}^{(0)}$, the Fourier coefficients for the first few nodes might
deviate from the true Fourier coefficients. To resolve this initial random
initialization issue, the converged Fourier coefficients for the last node,
$\hat{A}_{N}^{(I_{N})}$, are then assigned as the initialization of the first
node, $\hat{A}_{1}^{(0)}$, and the whole process is repeated for all the nodes
again. This looping process, i.e., $\hat{A}_{N}^{(I_{N})}=\hat{A}_{1}^{(0)}$
is performed till we have performed a prescribed number of network iterations
$\Lambda$ (see Algorithm 1).

Finally, to ensure all nodes have identical Fourier coefficients, we take

$\hat{A}=\frac{1}{N}\sum_{j=1}^{N}\hat{A}_{i}^{(I_{i})}.$ (7)

This averaging also removes the estimation errors in different
$\hat{A}_{i}^{(I_{i})}$. Now that $A$ has been estimated, we determine
$\hat{K}_{i},i=1,\dots,N$, by solving the non-negative least-squares problem

$\hat{K}_{i}=\arg\min\limits_{K_{i}\geq 0}\mathcal{L}_{i}(K_{i},\hat{A}).$

This whole procedure is outlined in Algorithm 1.

###### Remark 2

It has been shown that the iterative procedure to solve (6) converges to a
stationary point (see [14]), given sufficiently large data points. Moreover,
under particular conditions, the algorithm converges in only one step.
However, different initialization might yield a different stationary point.
The proposed sequential initialization resolves this bottleneck, as observed
in our numerical studies.

###  III-B Convergence Analysis

In this section, we present a numerical analysis of the convergence of the
proposed algorithm. To keep the analysis informative, we consider a network of
$5$ Kuramoto oscillators, i.e., $N=5$, with oscillator $i$ following the phase
dynamics

$\dot{\theta}_{i}=\omega_{i}+\sum_{j=1}^{N}k_{ij}\sin{(\theta_{j}-\theta_{i})},$
(8)

where $\theta_{i}$ and $\omega_{i}$ are the phase and nature frequency;
$k_{ij}$ is the strength of the sinusoidally coupled connection with node $j$.
The natural frequencies of the oscillators are uniformly distributed in
$[3,5]$ rad/s and $k_{ij}\in[0.1,0.3]$, if node $i$ and $j$ are connected. We
consider $10$ such random networks and for each network, time series data for
$5$ cycles, i.e., $(T=2\pi\/)$ with a total of $200$ sampling points is
generated. We then apply our inference algorithm for each network, where the
coupling function is approximated using two Fourier harmonics, i.e., $r=2$.
The value of the algorithm parameters $\epsilon$ and $\Lambda$ are set as
$10^{-5}$ and $2$ and the Fourier coefficients are initialized to
$\hat{A}_{1}^{(0)}=(0.25,0.25,0.25,0.25)^{\prime}$. The error in the estimated
Fourier coefficients for each node $\|\hat{A}_{i}^{(I_{i})}-A\|_{2}$ is
plotted in Figure 1, where the true Fourier coefficients
$A=(0,0,1,0)^{\prime}$. The blue (black) line corresponds to the error in the
Fourier coefficients when $\lambda=1$ $(\lambda=2)$. We find that in the
second round of network iteration ($\lambda=2$), the error drops to zero for
all $10$ networks, i.e., the Fourier coefficients converge to their true value
The average number of iterations for each node remains low. Specifically, For
the first network iteration
$\\{I_{1},I_{2},I_{3},I_{4},I_{5}\\}=\\{75,42,26,32,17\\}$, and for the second
network iteration $\\{13,3,12,6,3\\}$.

Figure 1: Numerical convergence analysis of the proposed algorithm. The blue
(black) line denotes the mean error in the Fourier coefficients in the first
(second) network iteration. Error bars represent the first and third
quartiles.

.

The presented numerical analysis highlights the importance of our sequential
initialization scheme, which results in the convergence of $\hat{A}$ to its
true value. In addition, the number of iterations per node decreases as
$\lambda$ increases.

##  IV Numerical Simulations and Analysis

In this section, we illustrate the ability of our algorithm to infer large
oscillator networks efficiently compared to the commonly used inference
techniques. To this end, we consider the networks of Kuramoto and Stuart-Landu
oscillators, which are widely used to model the oscillatory behavior of
complex systems. For the comparison purpose, we use ICON [7] and sparse
regression which involves applying $l_{1}$ penalty on the unknown coefficients
and has been shown effective for the inference of dynamic topology [15, 16].
To begin, we describe the performance evaluation metrics that we use to
analyze the accuracy of inferred networks.

###  IV-A Performance Evaluation Metric

We use two different criteria for the comparative analysis of our technique:
(i) estimation error in the coupling function measured by
$2\int_{0}^{2\pi}\left\|\alpha(\Delta\theta)-\hat{\alpha}(\Delta\theta)\right\|_{2}^{2}d\Delta\theta=\|A-\hat{A}\|_{2}^{2}$,
and (ii) ability of the algorithm in recognizing true and false connections
measured by area under the receiver operating characteristic curve (ROC
curve). The ROC curve plots the variation of the true positive rate (TPR) with
the false positive rate (FPR) as the classification threshold is varied [9]. A
greater area under the ROC curve (AUC score) means that the inference
algorithm is better at classifying true and false connections. The quantities
TPR and FPR measure the proportion of correctly identified links and the
proportion of links that were missed by the inference algorithm (see ref. [17]
for more details).

###  IV-B Kuramoto Oscillator Network

We consider a network of 100 Kuramoto oscillators $(N=100)$ where the dynamics
of each node are described by $\eqref{eq:Kuramoto_oscillators}$. The natural
frequencies of the oscillators are uniformly distributed in $[3,5]$ rad/s, and
the coupling strengths $k_{ij}\in[0.05,0.1]$. We generate $10$ such networks
with Erdős-Rényi [18] topology, and for each network, we simulate the network
dynamics for $5$ times. In each simulation, the network starts from random
initial conditions, and the data is collected for $10$ cycles, i.e., $T=4\pi$,
with a total of 1000 sampled data points.

Now, we apply the proposed technique along with ICON and sparse regression.
The error threshold $\epsilon$ is set as $10^{-5}$ and the vector
$\hat{A}_{1}^{0}$ is initialized as a unit vector with identical coefficients.
The number of Fourier approximation terms $r$ is kept as 2 for all three
methods. Figure 2(a) shows the box plots of the estimated AUC scores for all
10 networks. The proposed method gets an AUC score of one for all the
networks. The sparse regression gets the second-best AUC score with a mean of
0.88; while ICON recovered networks have a mean AUC score of 0.60. In addition
to accurately determining the network connectivity structure, our technique
predicts the coupling function precisely; while the sparse regression and ICON
coupling function do not reflect the true coupling dynamics (Figure 2(b)). The
errors in the coupling function estimation $\|A-\hat{A}\|_{2}^{2}$ are in
$[1.8\times 10^{-21},1.3,\times 10^{-11}]$, $[0.9,1.1]$, and $[0.8,1.6]$ for
the proposed method, sparse regression, and ICON, respectively. Note that we
take the average of all $\alpha_{ij}(\Delta\theta)$, $i,j=1,\dots,N$, for the
sparse regression and ICON as the individual coupling function between the
nodes can be different.

The convergence results of the estimated Fourier coefficients to the true
Fourier coefficients for a randomly selected network out of the 10 candidate
networks are shown in Figure 3. The blue (red) line denotes the error in the
Fourier coefficients for the first (second) network iteration. The average
number of iterations per node also remains low ($\approx 3$).

Figure 2: Comparison of the proposed technique with other inference methods on
a network of Kuramoto oscillators. (a) Box plot illustrating the estimated AUC
scores for 10 randomly generated networks of 100 nodes. M1: Proposed
technique, M2: sparse regression, and M3: ICON. (b) Mean estimated coupling
function of the candidate networks using M1 (blue line), M2 (black), and M3
(red), where the dotted magenta line denotes the true coupling function. The
shaded region denotes the variation in the estimated coupling functions across
different networks.

.

We also examine the effect of network topology on inference accuracy by
considering networks of Small-world [19] and BA Scale-free [20] topologies, in
addition to the Erdős-Rényi topology of Figure 1. We generate 10 networks (100
nodes) for each topology with similar model parameters to Figure 1 and do the
comparative analysis. We find that our algorithm performs consistently for
both network topologies; while the inference accuracy of sparse regression
degrades for small-world topology (Figure 4). This demonstrates the robustness
of our approach to the underlying network structure.

Figure 3: Convergence behavior of the proposed algorithm for a 100-node
Kuramoto network. The blue (red) line denotes the error in the estimated
Fourier coefficients in the first (second) network iteration.

.

Figure 4: Effect of network topology on the accuracy of network
reconstruction. The first, second, and third barplot of each section
correspond to the proposed method, sparse regression, and ICON, respectively.

###  IV-C Stuart Landau (SL) Oscillator Network

So far, we demonstrated the performance of our technique on a coupled phase
model (Kuramoto network). Now, we consider the SL oscillator model where the
phases are not directly available but are estimated from the measured time
series data. The SL oscillator is widely employed to model complex oscillatory
behavior in a variety of applications, ranging from semiconductor lasers to
pacemaker cells in the heart [21]. A network of $N$ SL oscillators is
described by

$\displaystyle\dot{x}_{i}(t)=ax_{i}-\omega
y_{i}-(x_{i}^{2}+y_{i}^{2})x_{i}+\epsilon_{ij}\sum_{j=1}^{N}(x_{j}-x_{i}),$
$\displaystyle\dot{y}_{i}(t)=\omega
x_{i}+ay_{i}-(x_{i}^{2}+y_{i}^{2})y_{i}+\epsilon_{ij}\sum_{j=1}^{N}(y_{j}-y_{i}),$

where $x_{i}(t)$ and $y_{i}(t)$ are the states of oscillator $i$, and
$\epsilon_{ij}\in[0.05,0.1]$ denotes the strength of the connection between
node $i$ and $j$ if it exists, otherwise $\epsilon_{ij}=0$. The model
parameters $a$ and $\omega$ determine the amplitude and frequency of
oscillation, respectively. We consider $10$ such networks of $100$ nodes
($a=1$, $\omega=1$), and for each network, we generate data by stimulating the
model $50$ times from random initial conditions. In each simulation, we
collect the data for $7$ cycles with a sampling rate of $200$ points per
cycle. Note that the data is generated from multiple simulations to avoid
network synchronization [8]. This is because inference of topology becomes
infeasible when the network is synchronized since the observed phase
difference becomes constant and no longer provides any additional information
($Z_{i}(t)$ will not change with time during synchronization).

Figure 5: Comparative analysis of the proposed technique with ICON and sparse
regression on a network of SL oscillators. (a) Time series data displaying the
synchronization behavior of the network. The red and blue lines denote the
states $x_{i}(t)$ and $y_{i}(t)$ for $i=1,\dots,100$. (b) Box plot
illustrating the estimated AUC scores for 10 randomly generated networks of
100 nodes. M1: Proposed technique, M2: sparse regression, and M3: ICON. (c)
Mean of the estimated coupling functions using M1 (blue line), M2 (black), and
M3 (red), where the dotted magenta line denotes the true coupling function.
The shaded region denotes the variation in the coupling functions across
different networks.

.

After generating the data for each network, we estimate the phase from the
states $x_{i}(t)$ and $y_{i}(t)$ by
$\varphi_{i}(t)=\arctan(y_{i}(t)/x_{i}(t))$ [22] and apply the proposed
inference method ($\epsilon=10^{-5},\Lambda=2,r=2,\text{ and }\hat{A}_{1}^{0}$
is initialized as a unit vector with identical elements) along with ICON and
sparse regression. The results are shown in Figure 5, where the panel b
displays the bar plots of the estimated AUC scores. The proposed method
outperforms both sparse regression and ICON; the mean AUC scores for our
algorithm, sparse regression, and ICON are 0.99, 0.93, and 0.85, respectively.
The errors in the coupling function estimation are also significantly lower
for our method. Specifically, $\|A-\hat{A}\|_{2}^{2}\in[6\times
10^{-4},7\times 10^{-3}]$ for our technique compared with $[0.8,1.2]$ for
sparse regression and $[0.6,1.6]$ for ICON. The true phase coupling function
of the SL oscillator was estimated using phase-reduction theory [22].

##  V Experimental Validation

In this section, we consider experimental data obtained from a network of 28
Rössler electronic oscillators and evaluate the robustness of our technique to
measurement noise - an uncontrollable element in an experimental setting. This
experimental dataset contains time series recordings from 20 distinct network
topologies. Notably, for each network topology, data was recorded under
various coupling strengths between the oscillators, with three separate time
series acquired for each combination of coupling strength and network
configuration [23].

We first obtain the phases of the oscillators from the measured data using the
peak-finding method [13]. The principle idea behind the peak-finding technique
is to assign a phase $2\pi k$ at time $t_{k}$, the $k^{th}$ peak of the time
series data, and then estimate the phase at time $t$, ($t_{k}<t<t_{k+1}$)
using linear interpolation. After obtaining the phase data, we apply Algorithm
1 ($\epsilon=10^{-5}$, $\Lambda=2$, $r=1$, and $\hat{A}_{1}^{(0)}$ is a unit
vector of identical elements) along with ICON and sparse regression to all
$20$ network topologies for two coupling strengths $k_{ij}=0.01$ and
$k_{ij}=0.1$. The estimated AUC scores are all these network combinations are
shown in Figure 4, where the proposed technique outperforms both ICON and
sparse regression for all the networks even with the noisy data.

Figure 6: Experimental validation on a network of electronic oscillators.
Panels (a) and (b) depict the estimated AUC scores for all 20 networks using
the proposed approach (blue), ICON (red), and sparse regression (black) for
$k_{ij}=0.01$ and $k_{ij}=0.1$, respectively.

##  VI Conclusions

In this paper, we present a bilinear optimization formulation of the network
inference problem by considering the network to have similar coupling
functions between the nodes. We then propose an iterative algorithm with
sequential initialization with convergence guarantees. The proposed approach
is illustrated using the phase model description of oscillator networks;
however, the presented ideas can readily be extended to a network of non-
oscillatory agents. In addition, the idea of sequential initialization can
also be extended to other optimization tasks where solving a system of
optimization problems having identical (or similar) solutions is required.

## References

  * [1] S. H. Strogatz, “Exploring complex networks,” nature, vol. 410, no. 6825, pp. 268–276, 2001.
  * [2] J. H. Abel, K. Meeker, D. Granados-Fuentes, P. C. St. John, T. J. Wang, B. B. Bales, F. J. Doyle III, E. D. Herzog, and L. R. Petzold, “Functional network inference of the suprachiasmatic nucleus,” Proceedings of the National Academy of Sciences, vol. 113, no. 16, pp. 4512–4517, 2016.
  * [3] W. Bomela, S. Wang, C.-A. Chou, and J.-S. Li, “Real-time inference and detection of disruptive eeg networks for epileptic seizures,” Scientific Reports, vol. 10, no. 1, p. 8653, 2020.
  * [4] K. J. Friston, “Functional and effective connectivity: a review,” Brain connectivity, vol. 1, no. 1, pp. 13–36, 2011.
  * [5] J. Lu, B. Dumitrascu, I. C. McDowell, B. Jo, A. Barrera, L. K. Hong, S. M. Leichter, T. E. Reddy, and B. E. Engelhardt, “Causal network inference from gene transcriptional time-series response to glucocorticoids,” PLoS computational biology, vol. 17, no. 1, p. e1008223, 2021.
  * [6] R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 58, no. 1, pp. 267–288, 1996.
  * [7] S. Wang, E. D. Herzog, I. Z. Kiss, W. J. Schwartz, G. Bloch, M. Sebek, D. Granados-Fuentes, L. Wang, and J.-S. Li, “Inferring dynamic topology for decoding spatiotemporal structures in complex heterogeneous networks,” Proceedings of the National Academy of Sciences, vol. 115, no. 37, pp. 9300–9305, 2018.
  * [8] S. G. Shandilya and M. Timme, “Inferring network topology from complex dynamics,” New Journal of Physics, vol. 13, no. 1, p. 013004, 2011.
  * [9] B. Singhal, M. Vu, S. Zeng, and J.-S. Li, “Data-efficient inference of nonlinear oscillator networks,” in IFAC world congress, pp. 10829–10834, 2023.
  * [10] I. Z. Kiss, Y. Zhai, and J. L. Hudson, “Emerging coherence in a population of chemical oscillators,” Science, vol. 296, no. 5573, pp. 1676–1678, 2002\.
  * [11] S. J. Aton, C. S. Colwell, A. J. Harmar, J. Waschek, and E. D. Herzog, “Vasoactive intestinal polypeptide mediates circadian rhythmicity and synchrony in mammalian clock neurons,” Nature neuroscience, vol. 8, no. 4, pp. 476–483, 2005.
  * [12] M. A. Schwemmer and T. J. Lewis, “The theory of weakly coupled oscillators,” Phase response curves in neuroscience: theory, experiment, and analysis, pp. 3–31, 2012.
  * [13] N. Mitrou, A. Laurin, T. Dick, and J. Inskip, “A peak detection method for identifying phase in physiological signals,” Biomedical Signal Processing and Control, vol. 31, pp. 452–462, 2017.
  * [14] E.-W. Bai and D. Li, “Convergence of the iterative hammerstein system identification algorithm,” IEEE Transactions on automatic control, vol. 49, no. 11, pp. 1929–1940, 2004.
  * [15] M. Timme and J. Casadiego, “Revealing networks from dynamics: an introduction,” Journal of Physics A: Mathematical and Theoretical, vol. 47, no. 34, p. 343001, 2014.
  * [16] D. Napoletani and T. D. Sauer, “Reconstructing the topology of sparsely connected dynamical networks,” Physical Review E, vol. 77, no. 2, p. 026103, 2008.
  * [17] J. A. Hanley and B. J. McNeil, “The meaning and use of the area under a receiver operating characteristic (roc) curve.,” Radiology, vol. 143, no. 1, pp. 29–36, 1982.
  * [18] P. E.-A. Rényi, “On the evolution of random graphs,” Publ. Math. Inst. Hung. Acad. Sci. A, vol. 5, pp. 17–61, 1960.
  * [19] D. J. Watts and S. H. Strogatz, “Collective dynamics of a small-world networks,” nature, vol. 393, no. 6684, p. 440, 1998.
  * [20] R. Albert and A.-L. Barabási, “Statistical mechanics of complex networks,” Rev. Mod. Phys., vol. 74, no. 1, p. 47, 2002.
  * [21] J. Wang and W. Zou, “Collective behaviors of mean-field coupled stuart–landau limit-cycle oscillators under additional repulsive links,” Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 31, no. 7, 2021.
  * [22] H. Nakao, “Phase reduction approach to synchronisation of nonlinear oscillators,” Contemporary Physics, vol. 57, no. 2, pp. 188–214, 2016\.
  * [23] V. Vera-Ávila, R. Sevilla-Escoboza, A. Lozano-Sánchez, R. Rivera-Durón, and J. M. Buldú, “Experimental datasets of networks of nonlinear oscillators: Structure and dynamics during the path to synchronization,” Data in brief, vol. 28, p. 105012, 2020.


# Anchors Based Method for Fingertips Position Estimation from a Monocular RGB
Image using Deep Neural Network

Purnendu Mishra, and Kishor Sarawadekar

###### Abstract

In Virtual, augmented, and mixed reality, the use of hand gestures is
increasingly becoming popular to reduce the difference between the virtual and
real world. The precise location of the fingertip is essential/crucial for a
seamless experience. Much of the research work is based on using depth
information for the estimation of the fingertips position. However, most of
the work using RGB images for fingertips detection is limited to a single
finger. The detection of multiple fingertips from a single RGB image is very
challenging due to various factors. In this paper, we propose a deep neural
network (DNN) based methodology to estimate the fingertips position. We
christened this methodology as an Anchor based Fingertips Position Estimation
(ABFPE), and it is a two-step process. The fingertips location is estimated
using regression by computing the difference in the location of a fingertip
from the nearest anchor point. The proposed framework performs the best with
limited dependence on hand detection results. In our experiments on the SCUT-
Ego-Gesture dataset, we achieved the fingertips detection error of 2.3552
pixels on a video frame with a resolution of $640\times 480$ and about
$92.98\%$ of test images have average pixel errors of five pixels.

###### Index Terms:

Egocentric vision, Computer vision, Deep Neural Network, monocular RGB image,
Fingertip detection, Human-computer interaction

##  I Introduction

Human pose estimation is essential for many computer-vision applications such
as sign language recognition, augmented reality, virtual reality, human-
computer interaction, etc. In these systems, it is vital to accurate detect
and track hand joints as well as fingertips. As compared to other hand joints,
it is more difficult to localize the fingertips due to their higher
flexibility, occlusion, etc. In this work we have focused on estimation of
fingertips position in monocular RGB images.

The standard RGB cameras are popular, and high-resolution 2D cameras are
available at relatively lower prices. The usage of a standard RGB camera in a
gestural interaction system is rare. The methods available for estimation of
fingertips position either restricted to estimation of single finger [1, 2, 3]
or lacks accuracy. In our proposal, we used standard 2D camera to estimate the
position of multiple fingertips which is a crucial step in any hand gesture-
based machine interaction system.

Successful estimation of fingertips position can have its application in a
wide variety of domains. In several augmented reality (AR) applications, hands
are used to interact and manipulate the virtual content. Buchmann et al. [4]
demonstrated usage hand gestures in the AR urban planning application. Thumb
and index fingers were used to perform operations like pointing, grabbing,
etc. Liu et al. [5] presented an approach to grasp various objects in a
robotic application by integrating the position and orientation information of
thumb and wrist from a human. P. Mistry and Prattie Maes[6] developed a
gestural interface that permits a user to interact with augmented information
using marked fingertips. Billinghurst et al. [7] extracted fingertips position
using a depth camera and implemented numerous gesture-based interactions with
virtual content in three dimensions. A finger-based writing-in-the-air system
using Kinect is developed by Zhang et al. [8]. Recently, Choi et al. [9]
proposed a depth-based fingertip recognition method for interactive
projectors. Some examples demonstrating the use of fingertips detection are
shown in Fig. 1.

Figure 1: An illustration showing use of fingertips to interact with various
object in virtual/augmented reality environment.
(Source:https://www.wikipedia.org/).

The challenges with fingertips detection in monocular RGB images are complex
background conditions, varying illuminations, image resolutions. Moreover, the
hand occupies only a small region of an image, and it becomes challenging to
segment the hand area within a given RGB image [10, 11]. Hence, it becomes
difficult to estimate position of fingertips directly. We have tried to
address some of these problems in this work. We utilized abstract features
learning capability of Convolutional Neural Networks (CNNs) and have followed
the two-step approach for multiple fingertips detection. First step is to
detect the hand area using a deep neural network (DNN) based object detection
model. Having obtained the hand region, in the second step a few fixed (known)
points called ‘anchors’ are placed on this segmented image. With the help of
these anchors, nearest fingertips to them are identified and then their
positions are estimated. The contribution of this paper are summarized as
follows:

  * •

We propose an anchor-based approach for the estimation of fingertips position
in a multi-gesture scenario where different combination of fingers are used to
make a hand pose. A fingertip is detected by finding the nearest anchor point
and then estimating the offset from that anchor location.

  * •

We demonstrated the effectiveness of anchor-based method for estimation of
fingertip position on single RGB image. The use of anchor locations have
assisted in identifying fingertip location of multiple fingers present in the
given image and reducing the detection error.

The organization of the rest of the paper is as follows. A brief overview of
the literature available on the estimation of fingertip position is given in
Section II. The proposed methodology is discussed in Section III. The details
of the experiment are given in Section IV. In Section V experimental results
are discussed, and finally, Section VI concludes the paper.

##  II Related Works

In this section, we discuss research work available in the field of fingertips
position estimation. Research work available in this area can be categorized
as non-vision based methods and vision-based methods. The non-vision methods
use hand gloves to locate fingertips position. Sensors mounted on fingers are
used to track the movement of fingers. The information received from sensors
is used for interactive operation in a virtual environment. Buchmann et al.
[4] used a glove as an input device to interact with the 3D graphics. The hand
is rendered in the virtual urban planning workspace, and operations like
grabbing, pointing, pressing, and navigation are performed. In related work,
Dorfmuller-Ulhaas et al. [12] and Zhang et al. [13] accessed fingertips
position using data gloves. The rewarding point of data gloves method is the
improved accuracy of the system. However, hand gloves restrict hand movement,
and it is inconvenient for users too. Due to these limitations, the
researchers have focused on vision-based methods.

Vision-based fingertips detection methods involve the use of unique markers.
The fingertips are either marked with paint or colored tapes. It is easy to
track these markers and estimate the position of the fingertips. Nakamura et
al. [14] tracked fingertips by placing an LED light source on them. In the
Sixth Sense device [6], the user’s fingers are taped with multiple color
markers. The system tracks a specific color to identify the finger, and the
user interacts with the projected contents. However, it is very inconvenient
for the user to put a marker every time. Hence, more research is needed in
vision-based fingertips detection without causing any inconvenience to the
user.

Figure 2: The proposed framework for the estimation of fingertip(s) position.

The process of estimation of fingertips position in the appearance-based
method, without using any special markers, in general, is divided into two
steps: hand area detection and locating fingertips. Hand region localization
is the first step in any hand-based application, and it affects the
performance of the subsequent procedures. The commonly used method for
achieving hand localization is skin color segmentation as color differences
between the human skin tone and the background is distinguishable. The skin
color-based methods utilize different color spaces for hand localization.
YCbCr color space is used by Lee et al. [15], and G. Wu et al. [16] for hand
region detection. The color-based models face the challenge of color
similarity, lighting variations, and background complexity [10, 8]. Depth-
based hand segmentation is unaffected by these issues. Several methods adopted
this technique [9, 17, 8, 18, 19, 11] for hand-detection. However, drawbacks
of this method are - noise affects the depth information. It has a small
range, and it is ineffective in outdoor conditions [8, 20, 21]. Recently CNN
based models are extensively used for object detection and localization. Y.
Huang et al. [2] and Mukherjee et al. [3] used Faster-RCNN [22] to localize
the hand in the RGB image in their respective work. The work of Y. Huang et
al. [1], Mukherjee et al. [3], and W. Wu et al. [20] rely on CNN based object
detector to obtain bounding boxes around the hand region. In the proposed
work, we rely on the YOLOv3 (You Only Look Once) model [23] for hand area
segmentation.

The next step after the hand localization is to estimate the fingertips
position. The conventional methods are K curvature method, fingertip template
matching, focus distance method, contour finding [24, 25, 26, 15, 16]. Wu et
al. [16] proposed average centroid distance (ACD) method and Mukherjee et al.
[3] proposed the distance weighted curvature entropy method. These methods are
dependent on the robustness of skin segmentation process. The method proposed
by Mukherjee et al. [3] works only for a single finger. Some researchers have
started exploiting complex feature extraction capabilities of CNN for
fingertips position estimation. Y. Huang et al. [2, 1], use CNN to find
fingertips location. But, their model is only capable of locating the
fingertip of a single finger. A method using CNN, which relies on heat-map to
locate multiple fingertips, is proposed by Wu et al. [20]. Although this
method is effective in locating multiple fingertips, it lacks accuracy, and
sometimes the heat-map prediction has a few false-positives. We have proposed
an improvement over the method in [20] called as Fingertips Position
Estimation with Fingers Identification (FPEFI) [27] to estimate the fingertips
position. It uses regression combined with multi-label classification to
locate the fingertips in an RGB image. The FPEFI method needs hand gesture
information to detect the fingertips. In this work, we remove that constraint
and estimate the fingertips position.

##  III The Proposed Method

The framework of the proposed methodology for fingertip(s) detection is shown
in Fig. 2. The process followed in the proposed work is similar to our
previous work for fingertip detection in [27]. The step of hand detection is
same in the current and the previous work. The difference is in the
methodology of fingertips detection. For hand area segmentation we use a DNN
based object detection method. The hand area is extracted from the input image
using the bounding box coordinates obtained from the DNN model. The image thus
obtained is used as input to the fingertip(s) detection module.

The methodology of fingertips detection in [27], we used direct regression to
estimate the fingertip location on cropped image. In addition, we performed
multi-label classification to identify the extended finger used in hand pose.
We estimated fingertips position by combining these outputs. Different from
method in [27], here we propose to estimate the fingertips position with the
help fixed anchor points. The process is named as Anchor based Fingertips
Position Estimation (ABFPE). The location of a fingertip is calculated as an
offset from the nearest anchor point. The anchor point has advantage of
predicting multiple fingertip location with varying hand pose and a fingertip
location is obtained with improved accuracy then method present in [27].

The process of hand detection and the fingertip(s) position estimation is
described below.

(a) Gesture two

(b) Gesture three

(c) Gesture four

(d) Gesture five

Figure 3:  The scatter plot showing the distribution of fingertips location
for various gestures made using single hand. The data of fingertip(s) location
is from SCUT-Ego-Gesture [20] dataset. Figure 4: Instead of using regression
to directly estimate the positions of fingertips (marked with red square
boxes), we estimate the error ‘d’ of a fingertip from the nearest anchors
point (marked with black dot).

###  III-A Hand Segmentation

A hand occupies relatively small region in an image. Also, factors-like
complex background, illumination variations, presence of human skin like
objects in the scene, camera characteristics, etc. affect the hand
segmentation process. Since the process of fingertip(s) position estimation,
gesture recognition, hand pose estimation, etc. depends on the robustness of a
hand detection process, the algorithm used to detect hand must be robust.
Recently, DNN based object detection methods have shown their robustness to
multiple object detection in an image. Some of the popularly used DNN
architectures for object detection are Faster-RCNN [22], YOLO [28] and SSD
(Single Shot multi-box Detector) [29]. The performance of these multiple
object detection architectures is significantly better in comparison to the
other methods used for the hand segmentation process. Moreover, these methods
work extremely fast. In the proposed work, we re-trained YOLOv3 [23] on SCUT-
Ego-Gesture dataset [20] for hand detection. In order to compare the
performance of YOLOv3 for hand area detection, three different CNN
architectures were trained on the aforesaid dataset. The first architecture is
proposed by Y. Huang et al. [1] for hand segmentation. The architecture called
“DeepFinger” uses regression technique, and it generates bounding box
coordinates for the detected hand area. The other two architecture uses
ResNet50 model [30] for feature extraction. We replaced the classification
layer in the original ResNet50 model with a linear layer comprising of four
output nodes for bounding box regression. The only difference between the two
ResNet50 based models is that the first model uses direct regression to
generate the bounding box around the detected hand area while the second the
model uses a prior (a rectangular box of dimension $200\times 300$). The
outputs of the second model are offset to the height and the width of the
prior. The aforesaid second method uses a similar concept as being used in
YOLO [28] and SSD [29]. The performance comparison of various methods is
detailed in Section V.

Figure 5: The architecture used for estimation of fingertips position. The
backbone model can be any state-of-the-art model (e.g. VGG16 [31], ResNet
[30], etc.).

###  III-B Fingertips Detection

The output of hand detection module are bounding box coordinates enclosing the
hand area. These bounding box tightly enclose the hand area in the input
image. And the fingertips of all the extended fingers lies near the edges of
these bounding box. To verify this hypothesis, we analyzed the distribution of
fingertip(s) location from the SCUT-Ego-Gesture [20] dataset. It is observed
from the scatter plot of the fingertips distribution in the bounding box
region that the fingertips are clustered near the edges of the bounding box
enclosing the hand. The scatter plot of fingertips distribution for gesture
two, three, four, and five are shown in Fig. 3. Majority of fingertips lie
near the edges of the bounding box, as shown in Fig. 3.

Based on this observation, we can search for fingertip(s) along the edges of
the bounding box instead of searching it in the entire image. The proposed
methodology for the fingertip(s) position estimation is explained with the
help of Fig. 4. Gesture five is posed in this figure. The hand area is cropped
from the original image using the bounding box coordinates. The cropped RGB
image is then resized to $h\times w$ dimension. In the proposed work we use
224 as values for $h$ and $w$. The ground truth fingertips are marked with red
squares. Next, some points are fixed at regular intervals, along the
circumference of the circle but on the edges of the image. We call these
points “anchors.” The black points in Fig. 4 represent these anchor points.
The position of a fingertip is estimated by identifying the nearest anchor to
the fingertip. Then distance between the anchor and the finger is estimated.
For example, as shown in Fig. 4, the Pinky finger is at distance of d from the
nearest anchor. Further, we estimate the value of d w.r.t. the nearest anchor
and obtain position of the fingertip of the Pinky finger.

Suppose $X_{k}\in\mathbb{R}^{2}$ where $k\in[1,5]$ represents the location of
fingertip in the given RGB image. And the anchors point are represented as
$Z_{n}\in\mathbb{R}^{2}$ where $n\in[1,N]$. Here $N$ is the number of anchor
points. For a fingertip location $X_{i}$ the value of $d$ from the nearest
anchor point $Z_{j}$ can be calculated as

$\color[rgb]{0,0,0}d=\|Z_{j}-X_{i}\|$ (1)

To identify the nearest anchor point and estimate the value for $d$ we have
have used CNN. The former is treated as classification problem and for later
case we use regression technique. During training of CNN, the anchor position
is one-hot encoded.

The architecture to implement the proposed method for the estimation of
fingertips position is shown in Fig. 5. The network design is very simple and
the details of architecture are given as follows. The network has a backbone
model that extracts features from the RGB image. We experimented with three
different state-of-the-art DNN models as the backbone model. These models were
VGG16 [31], ResNet50 [30] and MobileNetV2 [32]. These models were initialized
with pre-trained weights on the ImageNet dataset [33]. The backbone model has
an output stride of 32. The feature map obtained from the backbone model is
passed through a series of convolutional layers and its spatial resolution is
reduced to $1\times 1$. This feature map will then acts as input to two sub-
networks. The first sub-networks estimate the offset value for each finger and
has a convolutional layer with 10 filters which is reshaped to an array of
shape (5,2). This array stores the difference in the values of $x$ and $y$
coordinates for each finger w.r.t. the nearest anchor coordinate. The second
sub-network identifies out of the $N$ anchors, which is nearest to a
particular finger in the input image. We considered $N=24$ in our experiment.
This value is empirically chosen. The output of the second network is an
$(5,N)$ array where each row identifies nearest anchor for respective
fingertip location.

##  IV Experiments

###  IV-A Dataset

We experimented with three different publicly available datasets to validate
the proposed theory. These are SCUT-Ego-Gesture dataset [20], SCUT-Ego-Finger
dataset [1, 34] and HGR [35, 36, 37] dataset. We have selected single hand
gesture images from the SCUT-Ego-Gesture dataset. The dataset has 40,636 RGB
images of resolution $640\times 480$ in the egocentric view. The dataset has
normalized coordinates for the bounding box, the fingertips, and the finger
joints for the outstretched fingers. We selected 32,508 images for training
and 8,128 images for testing from the dataset.

The performance of the proposed fingertips position estimation algorithm has
been tested on the SCUT-Ego-Finger dataset and HGR dataset. The SCUT-Ego-
Finger contains hand images in an egocentric view with only the index finger
of the right hand in the extended position. The proposed model is fine-tuned
on 17,128 images, and 68,513 images are used to test the performance of the
model. Similarly, the proposed model is fine-tuned and tested on 927 and 309
images for the HGR dataset, respectively. The HGR dataset provide annotation
for 27 points of a hand. During our experiments, we selected only those
annotations in which extended fingers are used to pose a gesture. All
aforesaid datasets contain images with illumination variation, complex
background, skin-like objects in the background, making fingertips position
estimation process very challenging.

###  IV-B Data augmentation

The proposed method for estimation of fingertips position does not take into
account the hand pose, and it is robust to hand rotation. Therefore, the
network is trained by augmenting the dataset by flipping hand randomly both
vertically and horizontally. Random rotation of image is also performed.

###  IV-C Loss Function

The network performs two operations. First, it finds the nearest anchor point
corresponding to a fingertip, and the second operation is to estimate the
error between the position of the anchor point and the fingertip. Hence, the
optimization of the network is achieved by utilizing two different loss
functions. The loss function used during training of the proposed network is
defined as

$loss=L_{classification}+L_{regression}.$ (2)

Here, $L_{classification}$ is cross-entropy and $L_{regression}$ is Huber loss
[38]. The $L_{classification}$ loss is used for estimating the anchor nearest
to a fingertip. During training, the ground truth anchor positions were one-
hot encoded. The $L_{regression}$ loss is used to minimize the error between
estimated position of a fingertip with respect to the nearest estimated anchor
point.

###  IV-D Optimization

The complete network is trained end-to-end. The network’s loss is optimized
using Nesterov’s accelerated stochastic gradient descent algorithm. The
momentum value is set to 0.9. The network is trained using a polynomial
learning rate policy with a warm restart [39]. It converges faster in
comparison to “step” learning rate annealing policy, and the network yields
better performance for the same number of iterations. The “poly” learning rate
policy is defined as

$lr=lr_{0}*\left(1-\frac{i}{M}\right)^{power}$ (3)

where $lr_{0}$ is the initial learning rate, $i$ is the current iteration
value and $M$ is the maximum number of iterations during training. The value
of power is set to 0.9. The warm restart takes place at a fraction of $25\%$
of the total number of epochs. The value of the initial learning rate and
batch size was $10^{-2}$ and 64, respectively. The value of the learning rate
at the restart was $6.5\times 10^{-3}$. The network was trained for 100
epochs. The model was implemented using Keras [40] with Tensorflow backend.
All the experiments were performed on Nvidia GeForce GTX 1080 GPU with 8 GB of
VRAM.

Figure 6: Some instances of hand region detection using the YOLOv3 [23] model.
TABLE I: Performance on hand detection on SCUT-Ego-Gesture [20] dataset.

|  | Architectures | IoU
---|---|---|---
- |  | DeepFinger [1] | 0.7693
modified | direct regression | ResNet50 | 0.8061
using prior | ResNet50 | 0.8406
retrained |  | YOLOv3 [23] | 0.9046

TABLE II: Comparative results of the proposed method for various performance
metrics on SCUT-Ego-Gesture [20] dataset with three different state-of-the-art
DNN architectures as backbone.

| Backbone
---
model
Image Size |  | Avg. Pixel
---
error (px)
| Threshold
---
(px)
Precision | Recall | $F_{1}score$ |  | Time
---
(ms)
VGG16 [31] | $640\times 480$ | 9.2573 | 10 | 0.9385 | 0.8705 | 0.8947 | 15.27
15 | 0.9738 | 0.9396 | 0.9518 | 15.26
ResNet50 [30] | 2.3552 | 10 | 0.9921 | 0.9824 | 0.9859 | 16.38
15 | 0.9976 | 0.9920 | 0.9940 | 16.44
MobileNetv2 [32] | 3.4483 | 10 | 0.9875 | 0.9716 | 0.9772 | 10.77
15 | 0.9954 | 0.9892 | 0.9914 | 10.83

TABLE III: Comparative results for various performance metrics on SCUT-Ego-
Gesture [20] dataset.

Algorithm | Backbone Model | Probability |  | Actual
---
fingertips
| Detected
---
fingertips
| Avg. Pixel
---
error (px)
Precision | Recall | $F_{1}score$ |  | Time
---
(ms)
SpatialNet [41] | - | 0.2 | 18554 | 18351 | 6.9388 | 0.9509 | 0.9452 | 0.9422 | 40.894
- | 0.5 | 16748 | 9.8587 | 0.9343 | 0.8816 | 0.8991 | 40.895
YOLSE [20] | - | 0.2 | 19229 | 8.6793 | 0.9617 | 0.9852 | 0.9702 | 9.018
- | 0.5 | 18386 | 10.9832 | 0.9394 | 0.9746 | 0.9521 | 9.067
FPEFI [27] | MobileNetV2 | - | 18541 | 6.1527 | 0.9711 | 0.9914 | 0.9785 | 9.072
ResNet50 | - | 18544 | 4.8224 | 0.9977 | 0.9925 | 0.9944 | 17.453
Proposed (ABFPE) | VGG16 |  | 18426 | 9.2573 | 0.9738 | 0.9396 | 0.9518 | 15.265
MobileNetV2 | - | 18548 | 3.4483 | 0.9954 | 0.9892 | 0.9914 | 10.829
ResNet50 | - | 18544 | 2.3552 | 0.9976 | 0.9920 | 0.9940 | 16.443

TABLE IV: Comparative results for various performance metrics on HGR [37]
dataset.

Algorithm | Backbone Model | Probability |  | Actual
---
fingertips
| Detected
---
fingertips
| Avg. Pixel
---
error (px)
Precision | Recall | $F_{1}score$ |  | Time
---
(ms)
SpatialNet [41] | - | 0.2 | 927 | 943 | 18.8543 | 0.9164 | 0.9290 | 0.9108 | 38.235
YOLSE [20] | - | 0.2 | 992 | 28.1352 | 0.8411 | 0.8613 | 0.8267 | 9.047
FPEFI [27] | MobileNetV2 | - | 874 | 26.6788 | 0.8614 | 0.6582 | 0.7195 | 8.756
ResNet50 | - | 891 | 22.4235 | 0.9352 | 0.7715 | 0.8240 | 13.115
Proposed (ABFPE) | VGG16 | - | 915 | 14.0069 | 0.9595 | 0.8734 | 0.9026 | 14.076
MobileNetV2 | - | 930 | 12.9178 | 0.9636 | 0.9055 | 0.9220 | 10.776
ResNet50 | - | 925 | 8.8267 | 0.9836 | 0.9104 | 0.9342 | 16.245

(a) SpatialNet [41]

(b) YOLSE [20]

(c) FPEFI [27]

(d) The proposed method (ABFPE)

Figure 7: Some samples results of fingertips detection results on SCUT-Ego-
Gesture [20] dataset.

(a) SpatialNet [41]

(b) YOLSE [20]

(c) FPEFI [27]

(d) The proposed method (ABFPE)

Figure 8: Some samples results of fingertips detection results on HGR [37]
dataset.

##  V Results and Discussion

###  V-A Hand detection

In general, the process of the estimation of fingertips position or the
estimation of hand pose is carried out by first segmenting the hand region.
Then the next step is locating fingertips. Similar approach is followed in the
proposed work to estimate fingertips position from a single RGB image. The
framework of the proposed methodology is shown in Fig. 2. The first step of
hand region segmentation is performed through DNN based object detection
model. In last few years, there has been significant advancement in object
detection using deep learning techniques. These techniques are fast and
robust. We adopted YOLOv3 [23] model for hand region segmentation. The model
is retrained on the SCUT-Ego-Gesture dataset [20] for hand detection. In the
current work, only a single hand region is being detected. The performance of
YOLOv3 [23] is compared with ground-truth bounding box data given in the SCUT-
Ego-Gesture dataset. The performance is compared using the Jaccard similarity
index (also known as Intersection over union (IoU)). It is defined as

$IoU=\frac{S_{p}\cap S_{g}}{S_{p}\cup S_{g}}$ (4)

where $S_{p}$ is the area under the predicted bounding box and $S_{g}$ is the
area under the ground truth bounding box. The hand region segmentation is also
performed with the DNN model proposed by Y. Huang et al. [1]. We have also
designed two ResNet50 [42] based architectures to perform hand region
segmentation. In the first model, hand region segmentation is performed with a
direct regression technique, while in the second model, bounding box
coordinates are obtained relative to a bounding box (called prior) of the
fixed aspect ratio. The performance of each model used for hand region
segmentation is furnished in Table I. The YOLOv3 model has an IoU of $0.9046$
on the test set of the SCUT-Ego-Gesture dataset.

The main advantage of YOLOv3 [23] over other hand detector used in [1, 2, 20,
3] is its robustness and speed in the detection of hand region. The only
requirement of the fingertip detection module in the proposed work is that the
cropped image should contain all the visible extend fingers present in the
input image. And that is the only dependence of the the proposed method for
fingertip detection process on hand detection module. The YOLOv3 [20] is
robust enough in locating the hand region from the input image as validated
with our experiments (Table I). Therefore, the effect of hand detection
process on fingertip detection process is negligible. Some instances of hand
region detection using YOLOv3 architecture are presented in Fig. 6.

###  V-B Fingertips detection

The next step after the hand region segmentation process is the estimation of
the fingertips position. $Precision$, $recall$, and $f_{1}score$ have been
used to measure the performance of the proposed algorithm. These metrics are
defined in terms of true positive (TP), false-positive (FP), and false-
negative (FN), and are given as

$precison(p)=\frac{TP}{TP+FP}$ (5)

$recall(r)=\frac{TP}{TP+FN}$ (6)

$f_{1}score=\frac{2*p*r}{p+r}.$ (7)

A detected fingertip point is considered as true positive, if it lies in a
circle with radius $\delta$ (a threshold value) with center at the ground-
truth. If the predicted fingertip point does not match with any of the ground-
truth fingertips, it is treated as false positive. In case fingertip is not
predicted or the predicted fingertip lies outside the circle, the ground-truth
fingertip is considered as a false negative. In our experiments, we set the
value for $\delta$ either 10 or 15 pixels. The performance comparison for
three different backbone models used in our experiments for two different
thresholds ($\delta$) values is presented in Table II. When ResNet50 [30] is
used as a backbone to extract features from the RGB image, the model has the
least average pixel error of 2.3552 pixels and the best $f_{1}score$ of
$0.9940$ (threshold $\delta$ is equal to 15 pixels). The fastest processing
time is achieved with MobileNetv2 backbone [32], but it is at the cost of a
minimal decrease in the performance of the model.

TABLE V: Comparative results for various performance metrics on SCUT-Ego-
Finger [1] dataset.

Algorithm | Backbone Model |  | Actual
---
fingertips
| Detected
---
fingertips
| Avg. Pixel
---
error (px)
YOLSE [20] | - | 68513 | 66789 | 14.7556
Pointing Gesture [2] | - | 68513 | 6.5066
DeepFinger [1] | - | 68513 | 6.5445
FPEFI [27] | MobileNetV2 | 68513 | 6.5292
ResNet50 | 68513 | 5.8138
Proposed (ABFPE) | VGG16 | 68511 | 5.3934
MobileNetV2 | 68510 | 5.1846
ResNet50 | 68490 | 5.1334

(a) SCUT-Ego-Gesture dataset [20]

(b) SCUT- Ego-Finger dataset [1]

(c) HGR dataset [37]

Figure 9: Comparison of the cumulative distribution curves of the proposed
methodology with other existing algorithms on different datasets (a) SCUT-Ego-
Gesture dataset [20], (b) SCUT-Ego-Finger dataset [1], and (c) HGR dataset
[37].

The performance of the proposed algorithm has been compared with SpatialNet
[41], YOLSE [20], and FPEFI (Fingertips Position Estimation with Fingers
Identification) [27]. The YOLSE and the FPEFI methods present a solution to
estimate the position of multiple fingertips on a single RGB image on publicly
available datasets. While the SpatialNet [41] was designed to estimate
position of seven human body joints for 2D pose estimation using heatmap
regression. The results furnished in Table III and Table IV provide comparison
of the proposed technique with SpatialNet [41], YOLSE [20], and FPEFI [27] on
SCUT-Ego-Gesture dataset [20] and HGR datasets [37], respectively. These
experiments are performed with the threshold value ($\delta$) equal to 15. In
these datasets, the proposed algorithm (ABFPE) has the least average pixel
error as compared to SpatialNet, YOLSE, and FPEFI. The improvement in the
estimation of fingertips position is also evident from the qualitative results
presented in Fig. 7 and Fig. 8 on SCUT-Ego-Gesture dataset [20] and HGR
dataset [37], respectively. The predicted fingertips location is nearest to
the ground-truth location in the proposed method. The prediction of SpatialNet
and YOLSE has more false negatives or multiple predictions for a ground-truth
fingertip location, as shown in first and second row of Fig. 8, respectively.

The performance of the proposed method is also tested on SCUT-Ego-Finger [1]
dataset. This dataset comprises of hand images in the egocentric view with
only index finger in the extended position. The performance comparison results
are furnished in Table V. The proposed model has a success rate of $99.96\%$,
whereas the YOLSE model [20] can detect $97.48\%$ of fingertips on the total
number of the test images.

The performance of the proposed methodology has also been compared in terms of
cumulative distribution error (CDE). The CDE calculates the percentages of
images that lie under thresholds. The cumulative distribution curve for each
of the three datasets has been shown in Fig. 9. About $92.98\%$ of the test
images from the SCUT-Ego-Gesture dataset [20] have the average pixel error of
five pixels or less, which demonstrates the robustness of the proposed method.

We also tested the effect of image rotation on the performance of the proposed
model and observed that the performance of the proposed algorithm remains
unchanged, as shown in Fig. 10. The use of anchor points combined with the
capability of DNN is useful in estimating the fingertips position of multiple
fingers from a single RGB image. Due to the small network size, the proposed
model can perform fingertips estimation in real-time on the GPU enabled
system.

Figure 10: There is no effect on performance of the proposed model due to the
rotation of the input image.

##  VI Conclusion

The algorithm proposed in this paper is focused on the estimation of
fingertips position from the single RGB image. With only a small dependence on
the hand detection process, the proposed methodology is robust in the
estimation of fingertips position of all the visible fingers used in posing a
gesture. In comparison to the heatmap-based process, the proposed method has
performed well in fingertips detection. The image background factor,
illumination variations, skin color, have a minimum effect on the robustness
of the proposed technique. Further, the proposed method is rotation invariant.
In proposed work, only fingertip(s) location of the extended fingers used
while posing a gesture are estimated. In the future, the proposed work will be
extended to the estimate fingertip positions of all the visible fingers in an
image (extended or non-extended) as well as the detection of other hand-
landmark joints to define the hand pose.

## References

  * [1] Y. Huang, X. Liu, L. Jin, and X. Zhang, “DeepFinger: A Cascade Convolutional Neuron Network Approach to Finger Key Point Detection in Egocentric Vision with Mobile Camera,” in _2015 IEEE International Conference on Systems, Man, and Cybernetics_ , Oct 2015, pp. 2944–2949.
  * [2] Y. Huang, X. Liu, X. Zhang, and L. Jin, “A Pointing Gesture Based Egocentric Interaction System: Dataset, Approach and Application,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , June 2016, pp. 370–377.
  * [3] S. Mukherjee, S. A. Ahmed, D. P. Dogra, S. Kar, and P. P. Roy, “Fingertip detection and tracking for recognition of air-writing in videos,” _Expert Systems with Applications_ , vol. 136, pp. 217–229, 2019.
  * [4] V. Buchmann, S. Violich, M. Billinghurst, and A. Cockburn, “FingARtips: Gesture based direct manipulation in Augmented Reality,” in _Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia_. ACM, 2004, pp. 212–221.
  * [5] C. Liu, B. Fang, F. Sun, X. Li, and W. Huang, “Learning to Grasp Familiar Objects Based on Experience and Objects’ Shape Affordance,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 49, no. 12, pp. 2710–2723, Dec 2019.
  * [6] P. Mistry and P. Maes, “SixthSense: A Wearable Gestural Interface,” in _ACM SIGGRAPH ASIA 2009 Sketches_ , ser. SIGGRAPH ASIA ’09. New York, NY, USA: ACM, 2009, pp. 11:1–11:1. [Online]. Available: http://doi.acm.org/10.1145/1667146.1667160
  * [7] M. Billinghurst, T. Piumsomboon, and H. Bai, “Hands in Space: Gesture Interaction with Augmented-Reality Interfaces,” _IEEE Computer Graphics and Applications_ , vol. 34, no. 1, pp. 77–80, Jan 2014.
  * [8] X. Zhang, Z. Ye, L. Jin, Z. Feng, and S. Xu, “A new writing experience: Finger writing in the air using a Kinect sensor,” _IEEE MultiMedia_ , vol. 20, no. 4, pp. 85–93, 2013.
  * [9] O. Choi, Y. Son, H. Lim, and S. C. Ahn, “Co-Recognition of Multiple Fingertips for Tabletop Human-Projector Interaction,” _IEEE Trans. Multimedia_ , vol. 21, no. 6, pp. 1487–1498, June 2019.
  * [10] G. Modanwal and K. Sarawadekar, “A robust wrist point detection algorithm using geometric features,” _Pattern Recognition Letters_ , vol. 110, pp. 72–78, 2018.
  * [11] G. Modanwal, P. Mishra, and K. Sarawadekar, “A Robust Algorithm for Hand-Forearm Segmentation,” in _Proceedings of the 2018 International Conference on Image and Graphics Processing_. ACM, 2018, pp. 102–105.
  * [12] K. Dorfmuller-Ulhaas and D. Schmalstieg, “Finger tracking for interaction in augmented environments,” in _Proceedings IEEE and ACM International Symposium on Augmented Reality_ , Oct 2001, pp. 55–64.
  * [13] R. Zhang and X. Zhang, “Interaction method based on data glove in virtual environment,” _Computer Engineering_ , vol. 12, 2005.
  * [14] T. Nakamura, S. Takahashi, and J. Tanaka, “Double-Crossing: A New Interaction Technique for Hand Gesture Interfaces,” in _Asia-Pacific Conference on Computer Human Interaction_. Springer, 2008, pp. 292–300.
  * [15] Lae-Kyoung Lee, Su-Yong An, and Se-Young Oh, “Robust fingertip extraction with improved skin color segmentation for finger gesture recognition in Human-robot interaction,” in _2012 IEEE Congress on Evolutionary Computation_ , June 2012, pp. 1–7.
  * [16] G. Wu and W. Kang, “Robust Fingertip Detection in a Complex Environment,” _IEEE Trans. Multimedia_ , vol. 18, no. 6, pp. 978–987, June 2016.
  * [17] H. Liang, J. Yuan, and D. Thalmann, “Parsing the Hand in Depth Images,” _IEEE Trans. Multimedia_ , vol. 16, no. 5, pp. 1241–1253, Aug 2014\.
  * [18] Z. Ren, J. Yuan, J. Meng, and Z. Zhang, “Robust Part-Based Hand Gesture Recognition Using Kinect Sensor,” _IEEE Trans. Multimedia_ , vol. 15, no. 5, pp. 1110–1120, Aug 2013.
  * [19] C. Wang, Z. Liu, and S. Chan, “Superpixel-Based Hand Gesture Recognition With Kinect Depth Camera,” _IEEE Trans. on Multimedia_ , vol. 17, no. 1, pp. 29–39, Jan 2015.
  * [20] W. Wu, C. Li, Z. Cheng, X. Zhang, and L. Jin, “YOLSE: Egocentric Fingertip Detection from Single RGB Images,” in _2017 IEEE International Conference on Computer Vision Workshops (ICCVW)_ , Oct 2017, pp. 623–630.
  * [21] Y. Wang, B. Zhang, and C. Peng, “SRHandNet: Real-Time 2D Hand Pose Estimation With Simultaneous Region Localization,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 2977–2986, 2020.
  * [22] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [23] J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [24] M. Baldauf, S. Zambanini, P. Fröhlich, and P. Reichl, “Markerless Visual Fingertip Detection for Natural Mobile Device Interaction,” in _Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services_ , ser. MobileHCI ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 539–544.
  * [25] D. Lee and S. Lee, “Vision-Based Finger Action Recognition by Angle Detection and Contour Analysis,” _ETRI journal_ , vol. 33, no. 3, pp. 415–422, 2011\.
  * [26] A. Kulshreshth, C. Zorn, and J. J. LaViola, “Poster: Real-time markerless kinect based finger tracking and hand gesture recognition for HCI,” in _2013 IEEE Symposium on 3D User Interfaces (3DUI)_. IEEE, 2013, pp. 187–188.
  * [27] P. Mishra and K. Sarawadekar, “Fingertips Detection in Egocentric Video Frames using Deep Neural Networks,” in _2019 International Conference on Image and Vision Computing New Zealand (IVCNZ)_ , Dec 2019, pp. 1–6.
  * [28] J. Redmon and A. Farhadi, “Yolo9000: Better, Faster, Stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [29] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _CoRR_ , vol. abs/1409.1556, 2014.
  * [32] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4510–4520.
  * [33] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [34] X. Liu, Y. Huang, X. Zhang, and L. Jin, “Fingertip in the eye: A cascaded cnn pipeline for the real-time fingertip detection in egocentric videos,” _arXiv preprint arXiv:1511.02282_ , 2015.
  * [35] M. Kawulok, J. Kawulok, J. Nalepa, and B. Smolka, “Self-adaptive algorithm for segmenting skin regions,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2014, no. 1, p. 170, 2014.
  * [36] J. Nalepa and M. Kawulok, “Fast and accurate hand shape classification,” in _International conference: beyond databases, architectures and structures_. Springer, 2014, pp. 364–373.
  * [37] Grzejszczak, Tomasz and Kawulok, Michal and Galuszka, Adam, “Hand landmarks detection and localization in color images,” _Multimedia Tools and Applications_ , vol. 75, no. 23, pp. 16 363–16 387, 2016.
  * [38] P. J. Huber, “Robust Estimation of a Location Parameter,” in _Breakthroughs in statistics_. Springer, 1992, pp. 492–518.
  * [39] P. Mishra and K. Sarawadekar, “Polynomial Learning Rate Policy with Warm Restart for Deep Neural Network,” in _TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON)_ , Oct 2019, pp. 2087–2092.
  * [40] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [41] T. Pfister, J. Charles, and A. Zisserman, “Flowing ConvNets for Human Pose Estimation in Videos,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , Dec 2015, pp. 1913–1921.
  * [42] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _Computer Vision – ECCV 2016_ , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 630–645.