# Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based
Motion Recognition

Benjia Zhou1,2, Pichao Wang2,
Jun Wan1,3, Yanyan Liang1, Fan Wang2, Du Zhang1, Zhen Lei3, Hao Li2, Rong Jin2
1Macau University of Science and Technology 2Alibaba Group
3National Laboratory of Pattern Recognition, Institute of Automation, Chinese
Academy of Sciences Work done during an internship at Alibaba
Group.Corresponding author, project lead.Corresponding author.

###### Abstract

Decoupling spatiotemporal representation refers to decomposing the spatial and
temporal features into dimension-independent factors. Although previous RGB-D-
based motion recognition methods have achieved promising performance through
the tightly coupled multi-modal spatiotemporal representation, they still
suffer from (i) optimization difficulty under small data setting due to the
tightly spatiotemporal-entangled modeling; (ii) information redundancy as it
usually contains lots of marginal information that is weakly relevant to
classification; and (iii) low interaction between multi-modal spatiotemporal
information caused by insufficient late fusion. To alleviate these drawbacks,
we propose to decouple and recouple spatiotemporal representation for RGB-D-
based motion recognition. Specifically, we disentangle the task of learning
spatiotemporal representation into 3 sub-tasks: (1) Learning high-quality and
dimension independent features through a decoupled spatial and temporal
modeling network. (2) Recoupling the decoupled representation to establish
stronger space-time dependency. (3) Introducing a Cross-modal Adaptive
Posterior Fusion (CAPF) mechanism to capture cross-modal spatiotemporal
information from RGB-D data. Seamless combination of these novel designs forms
a robust spatialtemporal representation and achieves better performance than
state-of-the-art methods on four public motion datasets. Our code is available
at https://github.com/damo-cv/MotionRGBD.

Figure 1: Illustration of the proposed multi-modal spatiotemporal
representation learning framework. The RGB-D-based motion recognition can be
described as spatiotemporal information decoupling modeling, compact
representation recoupling learning, and cross-modal representation interactive
learning. Wherein DSN and DTN represent decoupled spatial and temporal
representation learning networks, respectively;

##  1 Introduction

Recently, the CNN and RNN based RGB-D motion recognition methods greatly
improve the performance of recognition on both gesture [34, 59, 1, 55, 45] and
action [50, 18, 47, 7] through fully exploring the color and depth cues.
Meanwhile, inspired by the transformer scaling success in vision tasks,
Transformer-based methods [27, 10] also achieve surprising results on RGB-D-
based motion recognition by introducing the cross-attention module for multi-
modality fusion.

Figure 2: Illustration of proposed decoupling and recoupling saptiotemporal
representation learning network. The whole network mainly consists of a
decoupled spatial and temporal representation learning networks namely DSN and
DTN, as well as a spatiotemporal recoupling module (RCM). The FRP indicates a
fast regional positioning module; SMS and TMS indicate the space- and time-
centric multi-scale Inception Module respectively. $\bigodot$, $\bigotimes$
and $\bigoplus$ indicate element-wise product, 1D convolution and element-wise
add operation respectively.

Although these works make great progress, we find they are still problematic
in the following three aspects. (i) Optimization difficulty exists in the case
of limited RGB-D data due to the tightly spatiotmporal entangled modelling.
(ii) Redundant information is hard to deal with in the entangled space-time
space. To address the above two issues, some decoupled networks (_i.e_., 2D
CNN+LSTM/Transformer[43, 21]) are proposed to learn the spatiotemporal
independent representation. However, we argue that these methods are not
conducive to compact representation as they somewhat weaken or even destroy
the original spatiotemporal coupling structure. Considering that a certain
number of human action classes have strong correlations between time and
space, the recoupling process after spatiotemporal decoupling is still
necessary. (iii) Insufficient interaction occurs between multi-modal
spatiotemporal information. Several works [59, 61] adopt independent branches
for unimodal spatiotemporal representations learning followed by late fusion,
resulting in insufficient cross-modal information communication. Thus, it is
still a challenge to explore high quality multi-modal spatiotemporal features.

Given the aforementioned concerns, as illustrated in Figure 1, we introduce a
new method of multi-modal spatiotemporal representation learning for RGB-D-
based motion recognition. It mainly consists of a decoupled spatial
representation learning network (DSN), a decoupled temporal representation
learning network (DTN) and a cross-modal adaptive posterior fusion module
(CAPF). For each unimodal branch, as shown in Figure 2, we propose a
decoupling and recoupling spatiotemporal feature learning method, wherein a
spatiotemporal recoupling module (RCM) is designed as a bonding of DSN and
DTN. RCM acts as feature selection for DSN and knowledge integration for DTN.
The entire framework can be decomposed into 3 steps: (1) Spatiotemporal
decoupling learning. In the DSN, the video clips are first fed into stack of
inception-based spatial multi-scale features learning (SMS) modules to extract
hierarchical spatial features. Meanwhile, they are also input into a bypath
network, called fast regional positioning module (FRP), to generate visual
guidance map, which guides the network to focus on local important areas in
the video frame. Then the integrated spatial features from SMS and FRP are fed
into RCM for feature selection. After that, we sample several sub-sequences at
different frame rates from the enhanced spatial features as input to the DTN.
The DTN is configured as a multi-branch structure with an inception-based
temporal multi-scale layer (TMS) and multiple Transformer blocks for
hierarchical local fine-grained and global coarse-grained temporal feature
learning. (2) Spatiotemporal recoupling learning. To rebuild the space-time
interdependence, a self-distillation-based recoupling strategy is developed.
As shown by the red dashed line in Figure 2, the recoupling method is designed
as an inner loop optimization mechanism to distill the inter-frame
correlations from time domain into the space domain, to enhance the quality of
the spatial features via RCM. (3) Cross-modal interactive learning. For multi-
modal representation learning from RGB-D data, as shown in Figure 1, we
propose an interactive cross-modal spatiotemporal representation learning
method. Specifically, the cross-modal spatial features derived from unimodal
branches firstly interact at the spatial level and are mapped to a joint
spatial representation. Then it is separately integrated with the two unimodal
spatial feature streams through the residual structure. After that, the two
spatial feature streams are input into their respective temporal modeling
networks to capture temporal features. Similar to the spatial feature
interaction, a joint temporal representation can also be obtained through
interaction at the temporal level. Combined with the joint temporal
representation, the two temporal feature streams are fed into the CAPF, which
is based on a multi-loss joint optimization mechanism, to conduct deep multi-
modal representation fusion.

Through the above design, our method not only effectively achieves the
spatiotemporal information decoupling and recoupling learning within each
modality, but also realizes the deep communication and fusion of multi-modal
spatiotemporal information. The proposed method achieves state-of-the-art
performance on four public RGB+D gesture/action datasets, namely NvGesture
[33], Chalearn IsoGD [44], THU-READ [42], and NTU-RGBD [35].

##  2 Related Work

###  2.1 Motion Recognition based on RGB-D Data

Human motion recognition is one of the most important topics in the field of
human-centered research. With the availability of low-cost RGB-D sensors, RGB-
D-based motion recognition has attracted extensive attention. To effectively
encode the robust multi-modal spatiotemporal information for motion
recognition, Zhu _et al_. [60] presents a pyramidal-like 3D convolutional
network structure for spatiotemporal representation extraction and fusion.
Kong _et al_. [22] propose to compress and project the RGB-D data into a
shared space to learn cross-modal features for effective action recognition.
Yu _et al_. [55] employ NAS to search for modal-related network structures and
optimal multi-modal information transmission path for RGB-D data. Different
from the modal-separated multi-branch networks, scene flow is adopted in [48]
for compact RGB-D representation learning. Wang _et al_. [50] propose to use a
single network c-ConvNet for multi-modal spatiotemporal representation
learning and aggregation. Unlike previous methods that interact with the
cross-modal information on coupled spatiotemporal information, we focus on the
interaction of multi-modal features in two independent dimensions of space and
time.

###  2.2 Decoupled Spatiotemporal Feature Learning

Considering the importance of decoupled spatiotemporal feature learning in
sequence, Shi _et al_. [39] present a decoupled spatiotemporal attention
network (DSTA-Net) for skeleton-based action recognition. Liu _et al_. [29]
present a decoupled spatiotemporal Transformer (DSTT) architecture to improve
video inpainting tasks. They disentangle attention learning into 2 sub-tasks:
(1) attending the temporal object movements on different frames at same
spatial locations; (2) attending similar background textures on same frame of
all spatial positions. He _et al_. [15] propose an effective spatiotemporal
network StNet, which employs separated channel-wise and temporal-wise
convolution operations for decoupled local and global representation learning.
Zhang _et al_. [58] present a hierarchically decoupled spatiotemporal
contrastive learning method for self-supervised video representation learning.
They capture the spatial and temporal features by decoupling the learning
objective into two contrastive sub-tasks, and perform it hierarchically to
encourage multi-scale understanding. In contrast, in this work, we target to
learn recoupled features due to strong correlations between time and space for
some human actions. Thus, a distillation-based recoupling process is
introduced on the decoupled spatiotemporal features.

##  3 Proposed Method

In this paper, we assume the spatiotemporal representation can be decomposed
into two sub-domains: the spatial domain that correlates with the visual
information, and the temporal domain that describes the time-related concept.
Based on this, for unimodal spatiotemporal representation learning, we first
decouple the spatiotemporal modeling process to learn domain-independent
representations (Sec. 3.1). Then a recoupling method is introduced based on
the inner loop optimization mechanism during the training stage (Sec. 3.2), to
strengthen the spatiotemporal connection. For multi-modal features interactive
learning, we first separately integrate the cross-modal spatiotemporal
information into the spatial and temporal domains, and then employ an adaptive
posterior fusion mechanism to further fuse the multi-modal features (Sec.3.3).

###  3.1 Decoupling Spatiotemporal Representation

####  3.1.1 Decoupled Spatial Representation Learning (DSN)

As shown in Figure 2, the DSN is composed of fast regional positioning module
(FRP) and stack of inception-based spatial multi-scale features learning (SMS)
modules. Let $[I_{1},I_{2},\dots,I_{T}]$ denote the input with length $T$
sampled from the video. It is fed into the SMS and FRP modules in parallel to
capture the hierarchical spatial features and generate visual guidance maps.

Figure 3: Overview of the proposed fast regional positioning module FRP. In
each sliding window, the FRP module locates important areas in the last frame
according to the successive video frames before it.

SMS Module. The SMS module is composed of a space-centric 3D Inception
Module111The size of the convolution kernel in the spatial dimension is
$1\times 1$. [3] and a Max Pooling layer. It extracts the multi-scale features
of $m$-th frame at the first layer by:

$f_{m}^{l}=\textrm{Maxpool}(\mathcal{C}_{S-Inc}(I_{m},W)),\quad s.t.\quad l=1$
(1)

where $l=1$ means it is the first layer of the DSN network;
$\mathcal{C}_{S-Inc}(\cdot,W)$ indicates the Inception Module with learnable
parameter matrix $W$; Maxpool is the Max Pooling layer. Meanwhile, to guide
each SMS module to focus on local important areas in the image, a visual
guidance map is embedded in parallel to it to further enhance its visual
perception, which is the proposed FRP module.

FRP Module. The FRP module is designed to generate the visual guidance map to
rapidly locate important areas in an image for motion recognition. The visual
guidance map is obtained directly by combining static and dynamic guidance
maps. Specifically, as shown in Figure 3, within a sliding window from the
$n$-th to the $m$-th frame, we first calculate the dynamic image $DI(n,m)$
[59]:

$\begin{split}DI(n,m)=&DI(n-1,m-1)+(m-n)\times\\\
&(I_{n-1}+I_{m})-2\sum\limits_{l=n}^{m-1}I_{l},\quad s.t.\quad m>n\end{split}$
(2)

where $DI(n-1,m-1)$ represents the dynamic image from the previous sliding
window. Then the dynamic guidance map on $m$-th frame can be obtained by:

$D_{m}=\delta(DI(n,m))\times\lambda$ (3)

where $\delta$ and $\lambda$ indicate activation function and signal
amplification factor respectively, herein we use GELU [16] and $\lambda=2$ in
all of experiments. However, we find that $D_{m}$ is sensitive to lighting as
it only considers the motion information between multiple frames. To mitigate
this problem, we introduce the static guidance map $S_{m}$, which can be
defined as:

$\begin{split}&S_{m}=\textrm{dilate}(\textrm{erode}(\hat{D}_{\rm{m}}),(k\times
k)))\\\
&\hat{D}_{\rm{m}}=\begin{cases}D_{\rm{m,(i,j)}}&D_{\rm{m,(i,j)}}>=D_{\rm{m,mean}}\\\
0&otherwise\end{cases}\end{split}$ (4)

where $\textrm{dilate}(\textrm{erode}(\cdot,(k\times k)))$ indicates the
dilation and erosion operations with a kernel size of $k\times k$ in binary
mathematical morphology. $\hat{D}_{\rm{m}}$ is an attention matrix, in which
all elements except for these higher than the mean value are set to zero, as
we empirically observe that the response value at the region affected more by
lighting is generally below average. After that, combining the Eq.3 and 4, the
visual guidance map of $m$-th frame at the $l$-th layer of the network can be
derived by:

$G^{l}_{m}=\textrm{Maxpool}((D_{m}+S_{m})\times S_{m})$ (5)

where the Max Pooling operation is used to scale the size of the current
guidance map $G^{l}_{m}$ to match the feature map $f^{l}_{m}$. In addition, we
also perform the normalization and alignment operations for the guidance map
$G^{l}_{m}$, which have been discussed in detail in the supplementary
material. Finally, the spatial features $O\in\mathbb{R}^{T\times d}$ with
length $T$ and dimension $d$ output from the DSN can be simply formulated as:

$\begin{split}&O=[O^{l}_{1},O^{l}_{2},\dots,O^{l}_{T}],\quad\forall
l=1,2,\dots,M\\\ &s.t.\quad O^{l}_{m}=(f^{l}_{m}\odot G_{m}^{l})\otimes
f^{l}_{m}+f^{l}_{m}\end{split}$ (6)

where $M$ represents the number of total network layers used in DSN; $\odot$
and $\otimes$ represent element-wise product and 1D convolution operations
respectively. It is then enhanced via RCM module for decoupled temporal
representation learning.

Figure 4: Overview of the proposed recoupling module RCM.

####  3.1.2 Decoupled Temporal Representation Learning (DTN)

The temporal representation learning network DTN takes the enhanced spatial
features $\hat{O}\in\mathbb{R}^{T\times d/2}$ (formal definition in Sec.3.2)
as the input. As shown in Figure 2, the DTN is configured as a multi-branch
and two-stage structure to progressively learn the hierarchical temporal
representation at local fine-grained level and global coarse-grained level.
Specifically, for a single sub-branch $k$, we first sample a sub-sequence of
features $\hat{O}_{k}\in\mathbb{R}^{T_{n}\times d/2}$ of length $T_{n}$ from
$\hat{O}$ by a discrete sampling strategy:

$\small\hat{O}_{k}=\\{\hat{O}_{\tau}|\tau=\mathcal{R}[\lceil\frac{T}{T_{n}}\rceil\times
t-1,\lceil\frac{T}{T_{n}}\rceil\times t],t=1,2,\dots,T_{n}\\}$ (7)

where $\mathcal{R}$ is a random number generator. $\hat{O}_{k}$ serves as the
input of temporal multi-scale features learning module (TMS).

TMS Module. The TMS is composed of a time-centric 3D Inception Module222The
size of the convolution kernel in the time dimension is 1.
$\mathcal{C}_{T-Inc}(\cdot,W_{k})$ and Max Pooling layer. We only use one
layer TMS to capture the local fine-grained spatial features
$\hat{O}^{L}_{k}$:

$\hat{O}^{L}_{k}=\textrm{Maxpool}(\mathcal{C}_{T-Inc}(\hat{O}_{k},W_{k}))$ (8)

After that, $\hat{O}^{L}_{k}$ is fed into stack of Transformer blocks to learn
the coarse-grained temporal representation.

Transformer block. To reduce the redundant marginal information in captured
temporal features, we utilize a Transformer structure based on $k$-NN multi-
head self-attention layer [51]. Thus the modeling process of each Transformer
block can be formulated as:

$\hat{O}^{G}_{k}=\textrm{MLP}(\textrm{LN}(\textrm{MSA}_{kNN}(\hat{O}^{L}_{k})))+\hat{O}^{G-1}_{k}$
(9)

where $\hat{O}^{G-1}_{k}$ indicates the output feature of the previous layer;
$\textrm{MSA}_{kNN}(\cdot)$ indicates the $k$-NN multi-head self-attention
layer and LN represents layer normalization. Furthermore, to avoid overfitting
to one of the sub-branches, we introduce a temperature parameter $\tau$ to
control the sharpness of the output distribution of each sub-branch and impose
a constraint loss on it. Therefore, the output of the DTN network can be
formulated as:

$\small O^{CLS}=\sum_{k=1}^{K}{\textrm{MLP}(O^{CLS}_{k})/\tau},\forall
k=1,2,3,\dots,K\\\ $ (10)

where $O^{CLS}$ is the class token vector embedded in the Transformer block
and $K$ indicates the number of sub-branches. $\tau$ follows a cosine schedule
from 0.04 to 0.07 during the training. We analytically demonstrate in Sec.5.1
that the tactics of $k$-NN Attention, sharpness and multi-loss can bring
performance gains for the proposed network.

Based on spatial features $O$ from DSN and temporal features $O^{CLS}$ from
DTN, we propose a recoupling strategy to strengthen the space-time connection
during training, because the spatiotemporal decoupled learning method weakens
the original coupled structure.

###  3.2 Recoupling Spatiotemporal Representation

The recoupling strategy is developed to rebuild the space-time
interdependence, which reversely applies the distilled inter-frame
correlations from time domain into the space domain through a self-
distillation-based inner loop optimization mechanism during the training.
Specifically, the spatial feature stream $O\in\mathbb{R}^{T\times d}$ derived
from the spatial features learning network DSN is first linearly mapped into a
low-dimensional space. Then the mapped features $\bar{O}\in\mathbb{R}^{T\times
d/2}$ is transposed and fed into the recoupling module RCM.

RCM module. The RCM module is configured as dual pathway to enhance the
spatial features from the X (feature dimension) and Y (sequence dimension)
directions, as shown in Figure 4. For X direction (intra-frame), inspired by
self-attention [8], we utilize a set of learnable matrices:
$W_{Q}\in\mathbb{R}^{T\times T}$ and $W_{K}\in\mathbb{R}^{T\times T}$ to
calculate the attention map $A_{X}$.

$\begin{split}&Q=\bar{O}^{T}W_{Q},\quad K=\bar{O}^{T}W_{K},\\\
&A_{X}=\delta(\textrm{GAP}(\frac{QK^{T}}{\sqrt{d}})),\quad
A_{X}\in\mathbb{R}^{1\times d/2}\end{split}$ (11)

where $Q$ and $K$ denote the queries and keys, $\delta$ is the Sigmoid
activation function, and GAP indicates the adaptive global average pooling
operation. The attention map $A_{X}$ describes the correlation of the intra-
frame in the sequence. For Y direction (inter-frame), the $\bar{O}$ is first
compressed along the channel dimension through GAP operation to obtain a
feature vector with a dimension of $1\times T$. Then it passes through an MLP
block with two hidden layers to obtain a weight embedding $W_{embed}$.

$W_{embed}=\textrm{MLP}(\textrm{GAP}(\bar{O}^{T})),\quad
W_{embed}\in\mathbb{R}^{1\times T}$ (12)

Finally, it is combined with a Sigmoid activation function to describe the
correlation of the inter-frame in the sequence.

$A_{Y}=\delta(W_{embed}),\quad A_{Y}\in\mathbb{R}^{1\times T}$ (13)

However, the MLP layer cannot effectively learn the inter-frame correlation
from the captured spatial features $\bar{O}$. Therefore, we employ a self-
distillation loss function $\mathcal{C}_{d}$ to introduce additional
supervision for the MLP block to distill inter-sequence correlation knowledge
from temporal domain into $W_{embed}$. The distillation process can be
formulated as:

$\begin{split}&\mathcal{C}_{d}=\mathcal{L}_{d}(CLS,\mathcal{F}(W_{embed})),CLS=\sum_{k=1}^{K}{O^{CLS}_{k}}\\\
&s.t.\quad\mathcal{L}_{d}(x,y)=\frac{1}{N_{B}}\sum_{i=1}^{N_{B}}{KL(x_{i}/\mathcal{T}-y_{i}/\mathcal{T})}\end{split}$
(14)

where $\mathcal{T}$ is the distillation temperature parameter, $N_{B}$ is the
batch size, $KL$ indicates Kullback-Leibler divergence [17] and $\mathcal{F}$
indicates linear mapping function. Combining Eq.11 and 13, the attention map
for spatial feature enhancement can be derived as:

$\begin{split}A_{XY}=\sum_{i=1}^{I}{\sum_{j=1}^{J}{A^{T}_{X,i}\times
A_{Y,j}}},A_{XY}\in\mathbb{R}^{T\times d/2}\end{split}$ (15)

where $I$ and $J$ indicate the element index in $A_{X}$ and $A_{Y}$
respectively. It is then applied to the raw spatial feature stream $O$ by the
element-wise product operation:

$\hat{O}=O\odot A_{XY}$ (16)

where $\hat{O}$ represents the enhanced spatial feature, which is used for
temporal representation learning.

Figure 5: Adaptive fusion module CAPF. $\mathcal{L}_{C}$, $\mathcal{L}_{B}$,
$\mathcal{L}_{M}$ indicate cross entropy, binary cross entropy and mean square
error loss functions, respectively; $traget$ represents ground-truth; E and D
indicate the Encoder and Decoder respectively.

###  3.3 Cross-modal Interactive Learning

For the RGB-D multi-modal spatiotemporal features separately extracted from
the two network branches, we propose to interact them at the spatial and
temporal dimensions respectively at first, as shown in Figure 1. Take the RGB
modality as an example, the two spatial features from the RGB-D modalities are
first interacted through the MLP layer to generate a joint spatial
representation. It is then integrated with raw spatial features of the RGB
modality by a residual structure:

$\ddot{O}^{S}_{R}=\textrm{LN}(\textrm{MLP}([\hat{O}_{R}^{S}||\hat{O}_{D}^{S}]))+O_{R}^{S}$
(17)

where $\hat{O}^{S}_{R}$ and $\hat{O}^{S}_{D}$ denote the enhanced spatial
features of RGB-D modalities respectively; And $||$ represents concatenation.
Similar to the interactive way of spatial features, after obtaining a joint
temporal representation $\ddot{O}^{T}\in\mathbb{R}^{1\times d/2}$, it is fed
into the cross-modal adaptive posterior fusion (CAPF) module for deep multi-
modal features fusion.

CAPF module. As shown in Figure 5, the CAPF module contains an Encoder and
Decoder, which are composed of MLP blocks with multiple hidden layers. The
$\ddot{O}^{T}$ is first fed into the Encoder to generate the encoded embedding
$\ddot{O}^{E}\in\mathbb{R}^{1\times d/4}$ which can be used for
classification. Then $\ddot{O}^{E}$ passes through the Decoder to obtain a
decoded embedding $\ddot{O}^{D}\in\mathbb{R}^{1\times d/2}$ which can be used
to supervise the Encoder. Therefore, the final classification score of the
entire network can be obtained by:

$\small
C=\textrm{Argmax}(f^{1}(O^{CLS}_{R})+f^{2}(O^{CLS}_{D})+f^{3}(\ddot{O}^{E}))$
(18)

where $f^{*}$ denotes the linear classifier and Argmax is used to take the
index of the maximum value in score vector. Meanwhile, considering the
challenge of optimizing the MLP layer, the entire network is trained by a
multi-loss collaborative optimization strategy as shown in Figure 5.

##  4 Experiments

###  4.1 Implementation Details

The proposed method is implemented with Pytorch. The input sequences are
spatially resized to $256\times 256$, and then randomly cropped into
$224\times 224$ during training. The inputs are cropped into $224\times 224$
around the center during inference. We employ SGD as the optimizer with the
weight decay of 0.0003 and momentum of 0.9. The learning rate is linearly
ramped up to 0.01 during the first 3 epochs, and then decayed with a cosine
schedule [30]. The training lasts for 100 epochs. The data augmentation only
includes random clipping and rotation. Similar to [55], all of our experiments
except NTU-RGBD are pre-trained on 20BN Jester V1
dataset333https://www.kaggle.com/toxicmender/20bn-jester. Moreover, three sub-
branches are configured in DTN. The number of spatial and temporal feature
learning blocks in DSN and DTN are $M=6$ and $N=6$, respectively. We refer to
this setting as the basic configuration of our network, unless otherwise
specified.

###  4.2 Comparison with State-of-the-art Methods

The proposed method achieves state-of-the-art performance on four gesture and
action datasets. It is noted that we only list some of the state-of-the-art
methods for comparison, and more comparison results can be found in the
supplementary material.

Method | Modality | Accuracy(%)
---|---|---
Transformer [11] | RGB | 76.50
MTUT [1] | RGB | 81.33
NAS [55] | RGB | 83.61
Ours | RGB | 89.58
Transformer [11] | Depth | 83.00
MTUT [1] | Depth | 84.85
NAS [55] | Depth | 86.10
Ours | Depth | 90.62
Transformer [11] | RGB+Depth | 84.60
MTUT [1] | RGB+Depth | 85.48
MMTM [20] | RGB+Depth | 86.31
MMTM [20] | RGB-D+Flow | 86.93
PointLSTM [32] | point clouds | 87.90
NAS [55] | RGB+Depth | 88.38
human | RGB+Depth | 88.40
Ours(Multiplication) | RGB+Depth | 90.89
Ours(Addition) | RGB+Depth | 91.10
Ours(CAPF) | RGB+Depth | 91.70
Table 1: Comparison of the state-of-the-art methods on the NvGesture.

####  4.2.1 NvGesture Dataset

The NvGesture [33] dataset focuses on human-car interaction. It in total
contains 1532 dynamic gesture videos (1050 for training and 482 for testing)
in 25 classes. As can be seen in Table 1, the proposed method significantly
boosts performance (RGB: $\uparrow 5.97\%$, depth: $\uparrow 4.52\%$ and
RGB-D: $\uparrow 3.3\%$) on this dataset for both single- and multi-modal
configuration, which demonstrates its generalization ability in the field of
human-computer interaction and small dataset. This might be because
spatiotemporal decoupled modeling method prevents overfitting of the network
to some extent.

####  4.2.2 THU-READ Dataset

The THU-READ [42] dataset consists 1920 videos with 40 different actions
performed by 8 subjects. This dataset is challenging due to small-scale and
background noise. For a fair comparison with other state-of-the-art methods,
we follow the released leave-one-split-out cross validation protocol utilized
in [27]. As illustrated in Table 2, our method exceeds the state-of-the-art
results and achieves the best average accuracy (87.40%) in this protocol,
which further demonstrates that our method is also robust to complicated
background. We conjecture that this is mainly attributed to the proposed FRP
module.

Method | Modality | Accuracy(%)
---|---|---
SlowFast [13] | RGB | 69.58
NAS [55] | RGB | 71.25
Trear [27] | RGB | 80.42
Ours | RGB | 81.25
SlowFast [13] | Depth | 68.75
NAS [55] | Depth | 69.58
Trear [27] | Depth | 76.04
Ours | Depth | 77.92
SlowFast [13] | RGB+Depth | 76.25
NAS [55] | RGB+Depth | 78.38
Trear [27] | RGB+Depth | 84.90
Ours(Multiplication) | RGB+Depth | 86.10
Ours(Addition) | RGB+Depth | 86.25
Ours(CAPF) | RGB+Depth | 87.04
Table 2: Comparison of the state-of-the-art methods on the THU-READ.

####  4.2.3 IsoGD Dataset

The Chalearn IsoGD [44] dataset contains 47,933 RGB-D gesture videos divided
into 249 kinds of gestures and is performed by 21 individuals. It is a much
harder dataset because (1) it covers gestures in multiple fields and different
motion scales from subtle fingertip movements to large arm swings, and (2)
many gestures have a high similarity. However, as shown in Table 3, our method
also performs well on this dataset, possibly due to the hierarchical and
compact features learned from the multi-scale network with the recoupling
structure that can capture subtle differences from similar gestures.

Method | Modality | Accuracy(%)
---|---|---
3DDSN [9] | RGB | 46.08
AttentionLSTM [61] | RGB | 57.42
NAS [55] | RGB | 58.88
Ours | RGB | 60.87
AttentionLSTM [61] | Depth | 54.18
3DDSN [9] | Depth | 54.95
NAS [55] | Depth | 55.68
Ours | Depth | 60.17
AttentionLSTM [61] | RGB+Depth | 61.05
NAS [55] | RGB+Depth | 65.54
Ours(Multiplication) | RGB+Depth | 66.71
Ours(Addition) | RGB+Depth | 66.68
Ours(CAPF) | RGB+Depth | 66.79
Table 3: Comparison of the state-of-the-art methods on the Chalearn IsoGD.

####  4.2.4 NTU-RGBD Dataset

The NTU RGB-D [35] is a large-scale human action dataset, which contains more
than 56,000 multi-view videos of 60 actions performed by 40 subjects. This
dataset is challenging due to large intra-class and viewpoint variations. As
the skeleton information is available, many recent works tend to perform 2D/3D
skeleton-based action recognition on this dataset since skeleton inherently
highlights the key information of human body, whilst being robust to various
illuminations and complex backgrounds. However, the generalization ability and
robustness of skeleton-based methods are limited. In this paper, we only use
the modalities of color and depth for action recognition. As shown in Table 4,
we achieve the state-of-the-art performance on both released protocols: Cross-
view (CV) and Cross-subject (CS).

Method | Modality | CS(%) | CV(%)
---|---|---|---
Directed-GNN [37] | Skeleton | 89.9 | 96.1
Shift-GCN [6] | Skeleton | 90.7 | 96.5
DC-GCN+ADG [5] | Skeleton | 90.8 | 96.6
CTR-GCN [4] | Skeleton | 92.4 | 96.8
Chained Multi-stream [62] | RGB | 80.8 | -
SLTEP [2] | Depth | 58.2 | -
DynamicMaps+CNN [47] | Depth | 87.1 | 84.2
DSSCA-SSLM [36] | RGB+Depth | 74.9 | -
Cooperative CNN [49] | RGB+Depth | 86.4 | 89.1
Deep Bilinear [18] | RGB-D+Skeleton | 85.4 | 90.7
P4Transformer [12] | point | 90.2 | 96.4
Ours | RGB | 90.3 | 95.4
Ours | Depth | 92.7 | 96.2
Ours(Multiplication) | RGB+Depth | 93.6 | 96.6
Ours(Addition) | RGB+Depth | 93.9 | 96.7
Ours(CAPF) | RGB+Depth | 94.2 | 97.3

Table 4: Comparison of the state-of-the-art methods on the NTU-RGBD.

Comparing with the skeleton-based state-of-the-art method CTR-GCN [4], the
proposed method achieves about 2% improvement on CS protocol and 0.5% on CV
protocol. The performance of depth modality can be on par or even better,
which further demonstrates the robustness of our method to noisy background
and its strong motion perception abilities.

##  5 Ablation Study

NvGesture and THU-READ(CS2) are employed for the ablation study. All of
experiments are conducted on RGB data modality except Sec.5.3. We refer the
reader to supplementary material for more ablation studies.

###  5.1 Impacts of Embedded Components

In Table 5, we show the impacts by introducing different components on the
proposed network. First, we observe that in the absence of FRP module, the
network does not work well either on NvGeture or THU-READ datasets, which
shows that the early guidance of the network to focus on some local
significant regions is beneficial to prevent the model from being trapped into
the local optimum. We also demonstrate its robustness to lighting by
visualization in supplementary material. In addition, imposing a constraint to
each branch of the network can prevent the model from overfitting to one of
the branches. Sharpening of the output distribution can encourage each sub-
branch to learn more discriminative features. Finally, removing some of
redundant information through $k$-NN Attention in the temporal representation
can also bring certain performance gains.

###  5.2 Effect of Recoupling Representation Learning

Figure 6: The ablation study of the recoupling strategy. (a) The effect of
self-distillation-based recoupling learning on network performance. (b) The
effect of distillation temperature on network performance.

As shown in Figure 6 (a), recoupling learning can boost the performance (Nv:
$\uparrow 3\%$, THU: $\uparrow 2\%$) and make the network converge rapidly.
This is because distilling the knowledge from the temporal domain to enhance
the spatial representation can help the network focus more on the informative
features in the early stage of training. The idea behind it may be that the
inner loop optimization mechanism based on self-distillation can help the
network deviate from the local optima as soon as possible and move toward the
global optimal solution. Figure 6 (b) shows the influence of distillation
temperature $\mathcal{T}$ on network performance. We find that an appropriate
temperature can boost the performance of the network. During the training
stage, we set distillation temperature $\mathcal{T}$ as 0.4 and 0.5 for
gesture and action datasets respectively.

FRP | Multi-loss | Sharpness |  $k$-NN Attention | Accuracy(%)
---|---|---|---|---
Nv | THU
$\times$ | $\times$ | $\times$ | $\times$ | 85.21 | 75.24
✓ | $\times$ | $\times$ | $\times$ | 86.67 | 78.33
✓ | ✓ | $\times$ | $\times$ | 87.08 | 80.53
✓ | ✓ | ✓ | $\times$ | 89.13 | 80.91
✓ | ✓ | ✓ | ✓ | 89.58 | 81.67

Table 5: Impacts of some introduced components. The Multi-loss means that we
impose constraint loss on each sub-branch in DTN.

###  5.3 Cross-modal Information Interaction

As shown in Table 6, the communication of spatial or temporal information can
boost the performance, wherein the information exchange at the spatial level
alone can also bring performance gains (Nv: $\uparrow 0.2\%$, THU: $\uparrow
0.9\%$), which reflects that the interaction of cross-domain knowledge can
help learn discriminative information. The last experiment shows that the
joint effective cross-domain information interaction at the spatial and
temporal levels can bring the largest performance gains (Nv: $\uparrow 0.6\%$,
THU: $\uparrow 3.7\%$), verifying that multi-modal feature learning can
benefit from spatiotemporal independent information transformation.

Dataset | Baseline (Add) | CmSI | CmTI | CmSTI
---|---|---|---|---
NvGesture | 91.10 | 91.32 | 91.53 | 91.70
THU-READ | 86.35 | 87.34 | 88.75 | 90.00
Table 6: Effects of cross-modal spatiotemporal information interaction. CmSI:
Cross-modal Spatial information Interaction only. CmTI: Cross-modal Temporal
information Interaction only. CmSTI: Cross-modal spatiotemporal information
Interaction.

##  6 Conclusion

We propose a method for unimodal decoupling and recoupling learning as well as
a cross-modal interactive learning and fusion. Among them, we think that
spatiotemporal recoupling learning is more important because it can lower the
optimization difficulty, especially under settings of small-scale datasets. In
addition, we demonstrate that guiding the network to focus on the local
important areas helps boost the performance. Finally, we find that interacting
cross-modal spatiotemporal information at the spatial and temporal levels,
respectively, can encourage the network to extract and fuses multi-modal
spatiotemporal features.

##  7 Acknowledgment

This work was supported by Alibaba Group through Alibaba Research Intern
Program, the Science and Technology Development Fund of Macau (0008/2019/A1,
0010/2019/AFJ, 0025/2019/AKP, 0004/2020/A1, 0070/2020/AMJ), Key R&D Programme
of Guangdong Provincial (2019B010148001), the National Key Research and
Development Plan under Grant (2021YFE0205700), the External cooperation key
project of Chinese Academy Sciences (173211KYSB20200002), the Chinese National
Natural Science Foundation Projects (61876179, 61961160704).

## References

  * [1] Mahdi Abavisani, Hamid Reza Vaezi Joze, and Vishal M. Patel.  Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [2] Ji Xiaopeng A B C, Cheng Jun A B C, Tao Dapeng D, Wu Xinyu A B C, and Feng Wei A B C.  The spatial laplacian and temporal energy pyramid representation for human action recognition using depth sequences - sciencedirect.  Knowledge-Based Systems, 122(C):64–74, 2017.
  * [3] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [4] Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu.  Channel-wise topology refinement graph convolution for skeleton-based action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13359–13368, 2021.
  * [5] Ke Cheng, Yifan Zhang, Congqi Cao, Lei Shi, Jian Cheng, and Hanqing Lu.  Decoupling gcn with dropgraph module for skeleton-based action recognition.  In Computer Vision–ECCV 2020: 16th European Conference, pages 536–553. Springer, 2020.
  * [6] K. Cheng, Y. Zhang, X. He, W. Chen, and H. Lu.  Skeleton-based action recognition with shift graph convolutional network.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [7] Alban Main De Boissiere and Rita Noumeir.  Infrared and 3d skeleton feature fusion for rgb-d action recognition.  IEEE Access, 8:168297–168308, 2020.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [9] Jiali Duan, Jun Wan, Shuai Zhou, Xiaoyuan Guo, and Stan Li.  A unified framework for multi-modal isolated gesture recognition.  ACM Transactions on Multimedia Computing, Communications, and Applications, 14:1–16, 02 2018.
  * [10] Andrea D’Eusanio, Alessandro Simoni, Stefano Pini, Guido Borghi, Roberto Vezzani, and Rita Cucchiara.  A transformer-based network for dynamic hand gesture recognition.  In 2020 International Conference on 3D Vision (3DV), pages 623–632, 2020.
  * [11] Andrea D’Eusanio, Alessandro Simoni, Stefano Pini, Guido Borghi, Roberto Vezzani, and Rita Cucchiara.  A transformer-based network for dynamic hand gesture recognition.  In 2020 International Conference on 3D Vision (3DV), pages 623–632. IEEE, 2020.
  * [12] Hehe Fan, Yi Yang, and Mohan Kankanhalli.  Point 4d transformer networks for spatio-temporal modeling in point cloud videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14204–14213, June 2021.
  * [13] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [14] Vikram Gupta, Sai Kumar Dwivedi, Rishabh Dabral, and Arjun Jain.  Progression modelling for online and early gesture detection.  In 2019 International Conference on 3D Vision (3DV), pages 289–297, 2019.
  * [15] Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, and Shilei Wen.  Stnet: Local and global spatial-temporal modeling for action recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8401–8408, 2019.
  * [16] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [18] Jian-Fang Hu, Wei-Shi Zheng, Jiahui Pan, Jianhuang Lai, and Jianguo Zhang.  Deep bilinear learning for rgb-d action recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [19] Ting-Kuei Hu, Yen-Yu Lin, and Pi-Cheng Hsiu.  Learning adaptive hidden layers for mobile gesture recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [20] Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L. Iuzzolino, and Kazuhito Koishida.  Mmtm: Multimodal transfer module for cnn fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [21] M Esat Kalfaoglu, Sinan Kalkan, and A Aydin Alatan.  Late temporal modeling in 3d cnn architectures with bert for action recognition.  In European Conference on Computer Vision, pages 731–747. Springer, 2020.
  * [22] Yu Kong and Yun Fu.  Bilinear heterogeneous information machine for rgb-d action recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1054–1062, 2015.
  * [23] Okan Köpüklü, Ahmet Gunduz, Neslihan Kose, and Gerhard Rigoll.  Real-time hand gesture detection and classification using convolutional neural networks.  In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–8. IEEE, 2019.
  * [24] Bin Li, Xi Li, Zhongfei Zhang, and Fei Wu.  Spatio-temporal graph routing for skeleton-based action recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8561–8568, 2019.
  * [25] Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu.  Skeleton-based action recognition with convolutional neural networks.  In 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 597–600. IEEE, 2017.
  * [26] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian.  Actional-structural graph convolutional networks for skeleton-based action recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3595–3603, 2019.
  * [27] Xiangyu Li, Yonghong Hou, Pichao Wang, Zhimin Gao, Mingliang Xu, and Wanqing Li.  Trear: Transformer-based rgb-d egocentric action recognition.  IEEE Transactions on Cognitive and Developmental Systems, 2021.
  * [28] Yunan Li, Qiguang Miao, Kuan Tian, Yingying Fan, Xin Xu, Rui Li, and Jianfeng Song.  Large-scale gesture recognition with a fusion of rgb-d data based on saliency theory and c3d model.  IEEE Transactions on Circuits and Systems for Video Technology, 28(10):2956–2964, 2017.
  * [29] Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Decoupled spatial-temporal transformer for video inpainting.  arXiv preprint arXiv:2104.06637, 2021.
  * [30] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [31] Qiguang Miao, Yunan Li, Wanli Ouyang, Zhenxin Ma, Xin Xu, Weikang Shi, and Xiaochun Cao.  Multimodal gesture recognition based on the resc3d network.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 3047–3055, 2017.
  * [32] Yuecong Min, Yanxiao Zhang, Xiujuan Chai, and Xilin Chen.  An efficient pointlstm for point clouds based gesture recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5761–5770, 2020.
  * [33] Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz.  Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4207–4215, 2016.
  * [34] Xuan Son Nguyen, Luc Brun, Olivier Lézoray, and Sébastien Bougleux.  A neural network based on spd manifold learning for skeleton-based hand gesture recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12036–12045, 2019.
  * [35] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016.
  * [36] Amir Shahroudy, Tian-Tsong Ng, Yihong Gong, and Gang Wang.  Deep multimodal feature analysis for action recognition in rgb+d videos.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5):1045–1058, 2018.
  * [37] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu.  Skeleton-based action recognition with directed graph neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7912–7921, 2019.
  * [38] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu.  Two-stream adaptive graph convolutional networks for skeleton-based action recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12026–12035, 2019.
  * [39] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu.  Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [40] Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan.  An attention enhanced graph convolutional lstm network for skeleton-based action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1227–1236, 2019.
  * [41] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [42] Yansong Tang, Zian Wang, Jiwen Lu, Jianjiang Feng, and Jie Zhou.  Multi-stream deep neural networks for rgb-d egocentric action recognition.  IEEE Transactions on Circuits and Systems for Video Technology, 29(10):3001–3015, 2018.
  * [43] Jun Wan, Guodong Guo, and Stan Z Li.  Explore efficient local features from rgb-d data for one-shot learning gesture recognition.  IEEE transactions on pattern analysis and machine intelligence, 38(8):1626–1639, 2015.
  * [44] Jun Wan, Yibing Zhao, Shuai Zhou, Isabelle Guyon, Sergio Escalera, and Stan Z Li.  Chalearn looking at people rgb-d isolated and continuous datasets for gesture recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 56–64, 2016.
  * [45] Huogen Wang, Pichao Wang, Zhanjie Song, and Wanqing Li.  Large-scale multimodal gesture recognition using heterogeneous networks.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [46] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In European conference on computer vision, pages 20–36. Springer, 2016.
  * [47] P. Wang, W. Li, Z. Gao, C. Tang, and P. O. Ogunbona.  Depth pooling based large-scale 3-d action recognition with convolutional neural networks.  IEEE Transactions on Multimedia, 2018.
  * [48] Pichao Wang, Wanqing Li, Zhimin Gao, Yuyao Zhang, Chang Tang, and Philip Ogunbona.  Scene flow to action map: A new representation for rgb-d based action recognition with convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595–604, 2017.
  * [49] P. Wang, W. Li, J. Wan, P. Ogunbona, and X. Liu.  Cooperative training of deep aggregation networks for rgb-d action recognition.  2017\.
  * [50] Pichao Wang, Wanqing Li, Jun Wan, Philip Ogunbona, and Xinwang Liu.  Cooperative training of deep aggregation networks for rgb-d action recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [51] Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Wen Xie, Hao Li, and Rong Jin.  Kvt: k-nn attention for boosting vision transformers.  arXiv preprint arXiv:2106.00515, 2021.
  * [52] Yu-Hui Wen, Lin Gao, Hongbo Fu, Fang-Lue Zhang, and Shihong Xia.  Graph cnns with motif and variable temporal block for skeleton-based action recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8989–8996, 2019.
  * [53] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In Thirty-second AAAI conference on artificial intelligence, 2018\.
  * [54] Xiaodong Yang, Pavlo Molchanov, and Jan Kautz.  Making convolutional networks recurrent for visual sequence learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [55] Zitong Yu, Benjia Zhou, Jun Wan, Pichao Wang, Haoyu Chen, Xin Liu, Stan Z Li, and Guoying Zhao.  Searching multi-rate and multi-modal temporal enhanced networks for gesture recognition.  IEEE Transactions on Image Processing, 2021.
  * [56] Liang Zhang, Guangming Zhu, Lin Mei, Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Bennamoun.  Attention in convolutional lstm for gesture recognition.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 1957–1966, 2018.
  * [57] Liang Zhang, Guangming Zhu, Peiyi Shen, Juan Song, Syed Afaq Shah, and Mohammed Bennamoun.  Learning spatiotemporal features using 3dcnn and convolutional lstm for gesture recognition.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 3120–3128, 2017.
  * [58] Zehua Zhang and David Crandall.  Hierarchically decoupled spatial-temporal contrast for self-supervised video representation learning.  arXiv preprint arXiv:2011.11261, 2020.
  * [59] Benjia Zhou, Yunan Li, and Jun Wan.  Regional attention with architecture-rebuilt 3d network for rgb-d gesture recognition.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3563–3571, May 2021.
  * [60] Guangming Zhu, Liang Zhang, Lin Mei, Jie Shao, Juan Song, and Peiyi Shen.  Large-scale isolated gesture recognition using pyramidal 3d convolutional networks.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 19–24. IEEE, 2016.
  * [61] Guangming Zhu, Liang Zhang, Lu Yang, Lin Mei, Syed Afaq Ali Shah, Mohammed Bennamoun, and Peiyi Shen.  Redundancy and attention in convolutional lstm for gesture recognition.  IEEE transactions on neural networks and learning systems, 31(4):1323–1335, 2019.
  * [62] Mohammadreza Zolfaghari, Gabriel L Oliveira, Nima Sedaghat, and Thomas Brox.  Chained multi-stream networks exploiting pose, motion, and appearance for action classification and detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2904–2913, 2017.

##  8 Supplementary Material

###  8.1 Supplementary for FRP Module

Normalization for Visual Guidance Map. To further improve the numerical
stability, we do the normalization for generated visual guidance maps
$G^{l}_{m}$ (Eq.5 in main manuscript) as:

$G_{\rm{m,norm}}^{l}=\frac{G^{l}_{m}-G^{l}_{\rm{m,min}}}{G^{l}_{\rm{m,max}}-G^{l}_{\rm{m,min}}}$
(19)

where $G^{l}_{\rm{m,min}}$ and $G^{l}_{\rm{m,max}}$ represent the maximum and
minimum values in $G^{l}_{m}$, respectively; And $G_{\rm{m,norm}}^{l}$
represents the normalized visual guidance map.

Visual Guidance Map Alignment. To align the generated visual guidance maps
with the input sequence, we shift it backwards along the time dimension by
$m-n$ units, and the guidance map of the previous $m-n$ frames is filled with
the zeros matrix. Therefore, the final visual guidance map can be formulated
as:

$\begin{split}&\hat{G}^{l}_{\rm{norm}}=[G^{l}_{1,\rm{norm}},\dots,G^{l}_{T,\rm{norm}}],\forall
l=1,2,\dots,M\\\ &s.t.\quad
G_{\rm{t,norm}}^{l}=\begin{cases}G_{\rm{t-(m-n),norm}}^{l}&t>m-n\\\
0&otherwise\end{cases}\end{split}$ (20)

where $\hat{G}^{l}_{\rm{norm}}$ represents the aligned visual guidance map
with the input sequence. It then integrates with spatial feature stream
captured by the spatial multi-scale features learning module (SMS) and serves
as the input to next layer of the network.

###  8.2 Structure of the SMS and TMS Modules

As shown in Figure 7, the spatial and temporal multi-scale features learning
module SMS and TMS are based on the inception structure. And a Max Pooling
operation is embedded behind them to aggregate features with high correlation
to reduce information redundancy.

###  8.3 Loss Function

(a) SMS Module

(b) TMS Module

Figure 7: The structure of the spatial multi-scale features learning module
SMS and the temporal multi-scale features learning module TMS.

For training the unimodal network, inspired by [55], we configurate three sub-
branches in the decoupled temporal representation learning network DTN, and
each sub-branch imposes a constraint loss with weight coefficient of $\gamma$.
In addition, we also introduce two additional constraint losses with weight
coefficients of $1-\gamma$ and 1.0, to constrain the summation of three sub-
branches and output of the RCM module. So the overall loss for unimodal
network training is the sum of all of those losses, and can be denoted as:

$\begin{split}\mathcal{L}^{overall}_{\rm{uni}}=&\gamma\mathcal{L}^{\mathcal{S}_{1}}_{C}+\gamma\mathcal{L}^{\mathcal{S}_{2}}_{C}+\gamma\mathcal{L}^{\mathcal{S}_{3}}_{C}+\\\
&(1-\gamma)\mathcal{L}^{\mathcal{S}_{all}}_{C}+\mathcal{L}_{D}\end{split}$
(21)

where $\mathcal{S}_{1}$, $\mathcal{S}_{2}$ and $\mathcal{S}_{3}$ represent the
output of the three sub-branch respectively;
$\mathcal{S}_{all}=\mathcal{S}_{1}+\mathcal{S}_{2}+\mathcal{S}_{3}$; and
$\mathcal{L}_{C}$ and $\mathcal{L}_{D}$ represent classification loss and
distillation loss, respectively. For training the multi-modal network, we
introduce a multi-loss collaborative optimization strategy, which can be
denoted as:

$\begin{split}\mathcal{L}^{overall}_{\rm{multi}}=&\mathcal{L}^{\mathcal{S}_{R}}_{C}+\mathcal{L}^{\mathcal{S}_{D}}_{C}+\mathcal{L}^{\mathcal{S}_{R}}_{B}+\mathcal{L}^{\mathcal{S}_{D}}_{B}+\\\
&\mathcal{L}^{\mathcal{S}_{R}}_{M}+\mathcal{L}^{\mathcal{S}_{D}}_{M}+\mathcal{L}^{\mathcal{S}_{R}}_{D}+\mathcal{L}^{\mathcal{S}_{D}}_{D}\end{split}$
(22)

where $\mathcal{S}_{R}$ and $\mathcal{S}_{D}$ represent the output of the
color and depth network branches, respectively; and $\mathcal{L}_{B}$ and
$\mathcal{L}_{M}$ represent binary cross entropy loss and mean square error
loss. It is note that we assign a weight coefficient of 1.0 to all losses.

###  8.4 Ablation Study

###  8.5 Impact of Sliding Window Size

Size | 2 | 4 | 6 | 8 | 10 | 12
---|---|---|---|---|---|---
NvGesture | 87.5 | 87.7 | 88.1 | 88.8 | 89.6 | 88.2
THU-READ | 80.4 | 80.8 | 80.8 | 81.2 | 81.7 | 80.6
Table 7: The effect of the sliding window size.

In Table 7, we set different sliding window sizes in the FRP module to study
how it affects network performance. We observe that the performance gradually
improves as we increase the size of the window. However, when the size reaches
12, the performance of the network degrades instead. We conjecture that this
may be because the response range in the dynamic guidance map has increased,
and as a result, the value of some noise regions has also been amplified
simultaneously.

(a) Dynamic Guidance Map.

(b) Static Guidance Map.

(c) Visual Guidance Map.

Figure 8: Visualization of the generated visual guidance map. (a) The dynamic
guidance map defined with $D_{m}$ in the main manuscript. (b) The static
guidance map defined with $S_{m}$ in the main manuscript. (c) The visual
guidance map defined with $G_{m}$ in the main manuscript. Note that the deeper
the color, the greater the weight.

###  8.6 Study for the Robustness of Illumination

As shown in Figure 8 (a), the dynamic guidance map $D_{m}$ is inevitably
influenced by illumination as it is driven by dynamic images. To address this
issue, we introduce the static guidance map $S_{m}$, as shown in Figure 8 (b),
it can not only enhance the response value of important areas in the image,
but also significant alleviate the effects of lighting. After combining the
dynamic guidance map and static guidance map, the final visual guidance map,
as shown in Figure 8 (c), can effectively highlight the important areas in the
image.

###  8.7 Impact of Local and Global Modeling in DTN

(a) Global Modeling.

(b) Local and Global Modeling.

Figure 9: Visualization of the Class Activation Map (CAM). (a) The activation
response of global coarse-grained temporal information modeling. (b) The
activation response of the joint modeling with local fine-grained as well as
global coarse-grained temporal information.

Temporal features learning based on global contextual information is vital for
sequence. However, we find that solely utilizing the Transformer network for
global contextual information modeling in the sequence is hard to generate
effective motion descriptors, especially hard to capture the local subtle
movement information as shown in Figure 9 (a). To alleviate this drawback, we
introduce an inception-based temporal multi-scale features learning network
(TMS) for local fine-grained temporal representation learning. It first
captures local hierarchical temporal features, and then aggregates neighboring
features with high correlation. After that, we feed them into stack of
Transformer blocks to progressively learn the global temporal representation.
As shown in Figure 9 (b), after modeling temporal information at a local fine-
grained level and global coarse-grained level, the local and global motion
perception abilities of the network have been significantly enhanced.

###  8.8 Study for Feature Enhancement Attention

Figure 10: Visualization of the attention map for spatial feature enhancement
generated by RCM module.

Figure 10 visualizes the attention map $A_{XY}$ (Eq.15 in main manuscript)
generated by the spatiotemporal recoupling module (RCM), which shows that it
can selectively activate several important neuron from X and Y directions in
captured spatial features. In addition, we can obviously find that attention
map $A_{XY}$ mainly guides the network to focus on the intermediate frame,
which just shows that these intermediate frames contain most of the important
information of a sequence.

###  8.9 Frame Rate Study for Sub-branch

In this ablation, we configure different frame rates for each sub-branch to
understand its impact on DTN. We only fine-tune the DSN sub-network and
compare models trained for 100 epochs. As shown in Table 8, the experiment
result confirms that (1) configuring different frame rates for each sub-branch
can boost the performance, which demonstrates that motion recognition benefits
from multi-scale temporal features. And (2) setting a smaller or larger frame
rate for DTN results in a decrease in performance, we conjecture that the
former may be caused by the loss of important information, and the latter may
be caused by temporal information redundancy.

Small Transf. |  Medium Transf. |  Large Transf. | Nv | THU
---|---|---|---|---
16 | 16 | 16 | 79.88 | 77.08
8 | 16 | 24 | 80.63 | 77.92
16 | 32 | 48 | 81.46 | 78.75
16 | 48 | 80 | 81.30 | 77.92
Table 8: The impact of different frame rates for each sub-branch in DTN.
“Transf.” means Transformer network.

###  8.10 More Comparisons

In this section, we compare with other methods not listed in the main
manuscript. Table 9 lists some other methods on the action datasets namely
THU-READ and NTU-RGBD. And Table 10 lists some other methods on the gesture
datasets namely NvGesture and Chalearn IsoGD.

THU-READ Dataset
---
Method | Modality | Accuracy(%)
Appearance Stream [41] | RGB | 41.90
TSN [46] | RGB | 73.85
Ours | RGB | 81.25
Depth Stream [41] | Depth | 34.06
TSN [46] | Depth | 65.00
Ours | Depth | 77.92
MDNN [42] | RGB+Flow+Depth | 62.92
TSN [46] | RGB+Flow | 78.23
TSN [46] | RGB+Flow+Depth | 81.67
Ours(CAPF) | RGB+Depth | 87.04
NTU-RGBD Dataset
Method | Modality | CS(%) | CV(%)
CNN+Motion+Trans [25] | Skeleton | 83.2 | 88.8
ST-GCN [53] | Skeleton | 81.5 | 88.3
Motif+VTDB [52] | Skeleton | 84.2 | 90.2
STGR-GCN [24] | Skeleton | 86.9 | 92.3
AS-GCN [26] | Skeleton | 86.8 | 94.2
Adaptive GCN [38] | Skeleton | 88.5 | 95.1
AGC-LSTM [40] | Skeleton | 89.2 | 95.0
MMTM [20] | RGB+Pose | 91.9 | -
Ours | RGB | 90.3 | 95.4
Ours | Depth | 92.7 | 96.2
Ours(CAPF) | RGB+Depth | 94.2 | 97.3

Table 9: Comparison with other methods on action datasets. Method | Modality | Accuracy(%)
---|---|---
NvGesture Dataset
GPM [14] | RGB | 75.90
PreRNN [54] | RGB | 76.50
ResNeXt-101 [23] | RGB | 78.63
Ours | RGB | 89.58
ResNeXt-101 [23] | Depth | 83.82
PreRNN [54] | Depth | 84.40
GPM [14] | Depth | 85.50
Ours | Depth | 90.62
PreRNN [54] | RGB+Depth | 85.00
GPM [14] | RGB+Depth | 86.10
Ours(CAPF) | RGB+Depth | 91.70
Chalearn IsoGD Dataset
c-ConvNet [50] | RGB | 36.60
C3D-gesture [28] | RGB | 37.28
AHL [19] | RGB | 44.88
ResC3D [31] | RGB | 45.07
3DCNN+LSTM [57] | RGB | 51.31
attention+LSTM [56] | RGB | 55.98
Ours | RGB | 60.87
c-ConvNet [50] | Depth | 40.08
C3D-gesture [28] | Depth | 40.49
ResC3D [31] | Depth | 48.44
AHL [19] | Depth | 48.96
3DCNN+LSTM [57] | Depth | 49.81
attention+LSTM [56] | Depth | 53.28
Ours | Depth | 60.17
c-ConvNet [50] | RGB+Depth | 44.80
AHL [19] | RGB+Depth | 54.14
3DCNN+LSTM [57] | RGB+Depth | 55.29
Ours(CAPF) | RGB+Depth | 66.79
Table 10: Comparison with other methods on gesture datasets.


# A Functional Form for Liquid Scintillator Pulse Shapes

F.Q.L. Friesen  C.R. Howell  Department of Physics, Duke University and the
Triangle Universities Nuclear Laboratory (TUNL). Durham, North Carolina 27708,
USA

###### Abstract

Digitization of detector signals enables analysis of the original waveform to
extract timing, particle identification, and energy deposition information.
Here we present the use of analytical functions based on sigmoids to model and
fit such pulse shapes from liquid organic scintillators, though the method
should also be applicable to other detector systems. Neutron and gamma
interactions in NE213 detectors were digitized from the phototube anode and
fit using a sigmoid-based function. The acuity of the fit in extracting timing
information and performing neutron-gamma pulse-shape discrimination are
presented and discussed.

###### keywords:

neutron , scintillator , pulse-shape discrimination , sigmoid , NE213 , BC501A

††journal: Nuclear Instruments and Methods in Physics Research Section A

##  1 Introduction

Organic liquid scintillators are the dominant technology used in experiments
requiring fast neutron detection. A wide variety of scintillators have been
developed offering different light output characteristics. A common feature is
a rapid rise in amplitude, followed by a slow exponential return to baseline.
The decay tail may consist of multiple exponential components depending on the
formulation of the liquid and particle being detected. For example, NE213 and
the equivalent formulations of EJ301 and BC501A, are designed for fast neutron
counting with good pulse-shape discrimination [1] (PSD) capability for
distinguishing gamma from neutron interactions in the detector [2]. The
sensitivity of the decay time of the emitted light to the type of particle
interaction in the liquid is due to the dependence on the mechanism of
excitation. Gamma rays generally scatter from electrons, which then
preferentially cause the liquid to fluoresce. Neutrons generally scatter from
protons in the organic molecules, and the resulting excitations produce
phosphorescence in addition to fluorescence [3]. The lifetime associated with
phosphorescence is typically orders of magnitude larger, and pairs of the
corresponding triplet states recombine with each other leading to a final
delayed de-excitation by fluorescence [4].

It is desirable to have a functional form to generically represent neutron and
gamma pulse shapes. In addition to describing the decay, the function should
accurately represent the rising edge of the pulse. It should have few free
parameters, each with a clear physical interpretation. Uncertainties in the
quantities of interest are then directly evaluated by the fit. With these
features, it is possible to perform a fit on a given waveform, alter a
parameter in a physically interpretable way, and regenerate a different but
realistic signal. In this way, neutron and gamma waveforms with arbitrary
timing can be quickly generated for simulating events in the same data format
produced by the experimental apparatus. This ultimately enables processing of
simulation output and measurements using identical code.

A frequently used expression for the pulse shape has the form
$L=A(e^{-\theta(t-t_{0})}-e^{-\tau_{f}(t-t_{0})})+B(e^{-\theta(t-t_{0})}-e^{-\tau_{s}(t-t_{0})})$
[5]. While the number of free parameters is reasonable, obtaining reliable
convergence during the fit process can be challenging. This difficulty
increases if the location of the peak in time isn’t well constrained. The
initial (concave up) rise from baseline is also not modeled by this
expression. In sufficiently fast scintillators, this initial rise can be non-
negligible. In this work we examine the use of sigmoids for describing
detector pulse shapes. They capture the exponential behavior with the benefit
of remaining bounded. The proposed function is the product of several sigmoids
representing the key features of the detector signals, e.g., the rise and
decay times.

The widespread use of digitizers, supported by cheap commodity storage, have
made it common practice to keep detailed records of every critical signal in
an experiment with nanosecond precision. With this capability, PSD analysis no
longer requires dedicated hardware, e.g., using fixed width integration
windows [2]. More detailed signal analyses using fits becomes possible.

Figure 1: Example raw waveforms from neutron and photon interactions with
similar amplitudes. Labels for the time constants are positioned to indicate
the regions dominated by each variable. The key difference between the
interactions is the prevalence of the slow decay component $\tau_{s}$ as
parameterized by $R$.

The traditional, time-tested PSD analysis method uses the charge comparison
technique by computing the ratio of charge collected from the detector across
two different fixed windows which correspond to the total charge in the pulse
and the charge in the fast component (i.e., the total charge minus the charge
in the tail)[1] [6]. A drawback of this method is that it can be difficult to
find a pair of windows which work robustly over a wide range of pulse heights.
Even though decay times are independent of amplitude, the pulse height of a
signal determines how quickly the tail will become comparable to noise. This
means that large signals are best characterized with longer windows; the
opposite is true for small signals. This can lead to undesirable trade-offs in
experiments when the detector threshold is low and neutrons and gammas need to
be reliably distinguished across a large dynamic range.

This work was performed in support of ${}^{3}He(\gamma,pn)$ measurements at
TUNL. The functions are primarily used to generate digitizer-like data from
the output of GEANT4 simulations. The functions can also be used in signal
analysis by using them to fit the raw digitizer data. However, the use of
these functions in data analysis would come at the expense of computation time
relative to other methods.

##  2 Experimental Setup

Neutron detector efficiency calibration data were obtained using the
${}^{3}H(p,n)$ reaction at a polar angle of 136.4∘ relative to the incident
proton beam axis. The digitized PMT signals were fitted using the proposed
function. A proton beam was incident on a tritiated titanium foil target,
producing neutrons at 0.8 MeV, 1.0 MeV, 1.2 MeV, 1.4 MeV, and 1.6 MeV. The 1.6
MeV data are presented here, as they provide the largest range of pulse
heights. The backward angle was chosen by requiring a higher proton beam
energy than used for forward angle neutron production to reduce the energy
spread of the neutrons.

Figure 2: The red solid line shows a representative neutron pulse shape as
constructed from constituent sigmoids with $R=0.914$. The slow decay
contribution is non-negligible.

The detectors used in this study were filled with BC501A and coupled to
Hamamatsu R1250 photomultiplier tubes. The interior of the liquid scintillator
cells were cylindrical with diameter 12.7 cm and depth of 5.1 cm. The PMT
anodes were connected directly to sis3316 digitizers. The digitizers have 12
bit resolution with a 4 ns sampling period. Waveforms were recorded for 240 ns
such that the first 76 ns consisted of baseline as shown in Fig.1. The full
duration of sampled data was used for fitting.

##  3 Pulse-Shape Analysis

For BC501A, the relative weighting of the importance of the two decay rates is
a free parameter in the fit. This ratio $R$ is the fraction of decay light
output produced by the fast mechanism. The expectation is that long decay
times should be much less prominent in gamma events than neutron events. There
are 6 independent parameters to fit for a PMT signal from a scintillation
event in BC501A. The first 3 are fixed for a given PMT and scintillator
combination, the remaining 3 are fit to each individual waveform (see Fig.1):

  * 1.

$\tau_{r}$ = Rise time ($\sim$1 ns)

  * 2.

$\tau_{f}$ = Fast decay time ($\sim$5 ns) representing fluorescence

  * 3.

$\tau_{s}$ = Slow decay time ($\sim$50 ns) representing return from
phosphorescence

  * 4.

$R$ = Relative weighting of decay times $0\leq R\leq 1$, e.g. $R$=1 means that
the amplitude of the slow decay component is zero

  * 5.

$A$ = Pulse amplitude

  * 6.

$t_{0}$ = Pulse arrival time

$y(t)=Af(t)[Rg(t)+(1-R)h(t)]$ (1)

where

$f(t)=\frac{1}{e^{-\frac{t-t_{0}}{\tau_{r}}}+1}$ (2)

$g(t)=1-\frac{1}{e^{-\frac{t-t_{0}}{\tau_{f}}}+1}=\frac{1}{e^{\frac{t-t_{0}}{\tau_{f}}}+1}$
(3)

$h(t)=1-\frac{1}{e^{-\frac{t-t_{0}}{\tau_{s}}}+1}=\frac{1}{e^{\frac{t-t_{0}}{\tau_{s}}}+1}$
(4)

The constituent functions of Eq.1 are standard logistic functions [7]. Those
representing the decay of the pulse have been subtracted from unity. Eq.2
describes the early exponential rise from baseline at the start of the pulse
and part of the round-off as the amplitude approaches its maximum. Note that
there is no time offset between functions $f$, $g$, and $h$. This means that
when the rise time is much smaller than the decay times, then function f has
negligible effect on the tail.

Example waveforms are shown in Fig.1 with labels indicating the features
determined by each parameter. Waveforms were prepared for the fit by first
ensuring that the baseline level averages to 0. The subtraction was performed
on an event-by-event basis. The curve fit function from the scipy.optimize [8]
python library was used to perform the fit with bounded variables. Examples of
the functions described by Eqs.1-4 are shown in Fig.2 for experimentally
determined parameter values. The following bounds (in ns) were used to help
the fit converge quickly:

$0\leq R\leq 1\quad A\geq 0\quad 0\leq\tau_{r}\leq 6\quad 0\leq\tau_{f}\leq
24\quad 40\leq\tau_{s}\leq 100\quad$ (5)

Figure 3: The $R$ parameter plotted vs. waveform pulse height with $\tau_{f}$
and $\tau_{s}$ held constant during the fits to template generated samples.
The lower R values are more neutron-like. The lower pulse height threshold
corresponds to 60 keVee ($1/8$ 137Cs edge). This histogram was accumulated
with an incident neutron energy of 1.6 MeV.

The waveforms taken from data were divided into three categories: templates,
general samples, and template generated samples. The templates were those with
a pulse height equivalent to the 137Cs edge or larger. Traditional PSD works
very well in this region, so gamma pulses and neutron pulses can be
unambiguously tagged by type before being fit. The general samples were drawn
from the same pool of waveforms as the templates, but with a much lower pulse
height cut ($1/8$ of the 137Cs edge). General sample waveforms were used to
evaluate the timing capability of the fit. Template generated samples were
constructed from templates by scaling down a template waveform amplitude to
the obtain the desired pulse height, and then adding Gaussian noise to
reproduce the original detector noise level (the pulse height spectrum was set
to duplicate that of the general samples). The template generated samples were
used to evaluate PSD capability.

The templates were fit individually using the bounds shown in Eq.5. The mean
$\tau_{f}$ and $\tau_{s}$ values from neutron template fits were used to fix
these parameters for all subsequent fits (all $\gamma$ waveforms, general
samples, and template generated samples). The parameters $R$, $A$, $\tau_{r}$
and $t_{0}$ were left free. The $R$ can be interpreted as a PSD parameter
(Fig.3). The PSD capability of $R$ is improved if $\tau_{r}$ is kept as a free
parameter in the fit. Representative pulse shapes for neutron and photon
events were generated as shown in Fig.4.

Figure 4: Example of normalized mean gamma and neutron pulse shapes obtained
from fits to template waveforms. Figure 5: Histogram of fit (PSD) parameter R
from a pool of template generated waveforms. The upper lines are for a neutron
pulse height cut of 1/8 Cs, the lower lines are at 1/4 Cs. The left
distributions show known neutron inputs. The right distributions show known
gamma inputs. The gaussian fits used to calculate the FOM at $1/8$ 137Cs are
also shown.

Finally, the template scaling method provides a means to test PSD performance.
Templates with known identities are scaled down and added to background noise
to simulate lower pulse height events. Pools of realistic waveforms with low
pulse-heights can be produced from templates with known identities. Using
these generated events, the pulse identification capability can be
quantitatively evaluated. PSD results for two sets of waveforms at different
pulse height thresholds are shown in Fig.5. The $R$ value from the fit is
compared to a traditional PSD calculation in Fig.6. The traditional PSD
parameter was calculated as the quotient of two baseline-subtracted waveform
integrals. They extend to 20 and 4 samples past the leading edge, respectively
(i.e., 172 ns and 108 ns in Fig.1). There is good agreement over the
classification of the vast majority of waveforms.

Table 1: Neutron/$\gamma$ discrimination capability as a function of pulse height threshold. The discriminator threshold for $R$ was set to 0.98 for this analysis. Increasing it reduces rejection of true neutron events at the expense of mis-identifying more $\gamma$ events as neutrons. The uncertainties are statistical. Threshold | Correctly | Correctly |  $\\#$ n above |  $\\#$ $\gamma$ above
---|---|---|---|---
(MeVee) | Identified n | Identified $\gamma$ | Threshold | Threshold
0.0598 | 0.997$\pm$0.006 | 0.998$\pm$0.006 | 6.0x104 | 6.0x104
0.12 | 0.997$\pm$0.008 | 0.998$\pm$0.009 | 3.4x104 | 4.3x104
0.239 | 1.00$\pm$0.01 | 1.00$\pm$0.03 | 1.1x104 | 2.5x104
0.478 | 1.00$\pm$0.03 | 1.0$\pm$0.1 | 2.1x103 | 1.2x104

The result of fixing the discrimination threshold on $R$ at 0.98 is shown in
table 1. The table was generated from a pool of 1.2 x105 tagged waveforms
(generated by template scaling) with a 1/8 Cs (60 keVee) pulse height
threshold limit, where half are neutrons and the other half are photons. At
this threshold, 2 $\gamma$-ray signals leak through the PSD cut for every 1000
photons detected. For every 1000 incident neutrons, 3 are misidentified as
photons. The reduction in total number of neutron and gamma events as the
threshold is increased corresponds to the reduced detection efficiency. The
values and uncertainties of parameters obtained from fitting the template
signals are given in Table 2. A figure of merit (FOM) value was also
calculated for the $R$ discrimination parameter by fitting the neutron and
gamma peaks with gaussian functions and using the definition
FOM$=(\mu_{\gamma}-\mu_{n})/(\sigma_{\gamma}+\sigma_{n})$ where $\sigma$
refers to the FWHM of each gaussian [9]. The traditional charge comparison PSD
method had FOM values of 1.34 and 1.43 for pulse height thesholds of 1/8 and
1/4 Cs, respectively. The FOM values for the $R$ fit method were 1.39 and 1.52
calculated on the same data.

Table 2: Mean values and standard deviation of the parameters obtained by fitting templates. The pool consisted of 12,000 waveforms with an equal number of neutron and photon events. The uncertainty in $t_{0}$ is given as an estimate of the time resolution of this method because the absolute value of $t_{0}$ depends on the source of the particle. The $\tau_{f}$ and $\tau_{s}$ values for photons were constrained to be the same as the mean values obtained from neutron template fits. Therefore, these fixed parameters do not have uncertainties for photons. Parameter | Neutrons |  $\gamma$-rays
---|---|---
$\tau_{r}$ (ns) | 1.46 $\pm$ 0.29 | 1.37 $\pm$ 0.26
$\tau_{f}$ (ns) | 5.45 $\pm$ 0.63 | 5.45
$\tau_{s}$ (ns) | 52 $\pm$ 11 | 52
$R$ | 0.949 $\pm$ 0.035 | 0.992 $\pm$ 0.050
$t_{0}$ (ns) |  $\pm$ 0.17 |  $\pm$ 0.11
Figure 6: The PSD parameter calculated by the window integration method
plotted against the $R$ fit parameter for the same data. The peak in the upper
right hand corner corresponds to $\gamma$ events.

##  4 Timing Analysis

The $t_{0}$ fit parameter was compared with a software constant fraction
discriminator [10] (CFD) value computed with a trapezoidal filter. The data
sampling period was 4 ns. The peak and gap times were both 2 samples long (8
ns), and the time was interpolated on the falling edge of the filter at $1/2$
of the maximum output. The relationship between the two, shown in Fig.7,
indicates that the raw $t_{0}$ value from the fit closely follows the CFD
value independent of event type or amplitude. Slices taken either
perpendicular or parallel to the CFD axis in Fig.7 corresponding to $85.6\leq
t_{0}\leq 89.6$ ns (100 to 104 ns on the CFD axis) were used to compare timing
resolution. These are experimental data, so different arrival times have
different physical interpretations, and different slices should not be
directly combined. The standard deviation of the points in the vertical slices
from the fit is 0.33 ns within the range, compared to a standard deviation of
0.35 ns as determined with the CFD method (horizontal slices passing through
the same points).

Figure 7: Trapezoidal filter CFD vs $t_{0}$ from the sigmoid-based fit applied
to general samples. The pulse height threshold corresponds to 60 keVee ($1/8$
137Cs edge). Slices passing through the bin with the most counts (102.6 ns,
88.1 ns) were used to compare timing resolution.

##  5 Conclusions

A sigmoid-based functional form was developed for representing pulse shapes
from liquid scintillation detectors. The rise time, amplitude, time offset,
and decay components are represented in a physically interpretable way.
Contributions from different decay times can be reliably identified by the
fit, thereby enabling PSD analysis of each waveform. The bounded nature of
sigmoids simplifies construction of plausible functions, and the function
presented here can be further generalized to allow for more decay components
as needed. The function parametrized in Eqs.1-4 can be used to generate
signals to realistically represent pulse shapes from liquid scintillator
detectors. Waveforms associated with BC501A can be generated using values
given in table 2. Other sigmoids could also be explored for constructing the
fit function.

## References

  * [1] J. Adams, G. White, A versatile pulse shape discriminator for charged particle separation and its application to fast neutron time-of-flight spectroscopy, Nuclear Instruments and Methods 156 (3) (1978) 459 – 476.
  * [2] A. Comrie, A. Buffler, F. Smit, H. Wörtche, Digital neutron/gamma discrimination with an organic scintillator at energies between 1MeV and 100MeV, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 772 (2015) 43 – 49.
  * [3] J. Birks, The theory and practice of scintillation counting, International series of monographs on electronics and instrumentation 27.
  * [4] P. Schuster, E. Brubaker, Investigating the anisotropic scintillation response in anthracene through neutron, gamma-ray, and muon measurements, IEEE Transactions on Nuclear Science 63 (3) (2016) 1942–1954.
  * [5] S. Marrone, D. Cano-Ott, N. Colonna, C. Domingo, F. Gramegna, E. Gonzalez, F. Gunsing, M. Heil, F. Käppeler, P. Mastinu, P. Milazzo, T. Papaevangelou, P. Pavlopoulos, R. Plag, R. Reifarth, G. Tagliente, J. Tain, K. Wisshak, Pulse shape analysis of liquid scintillators for neutron studies, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 490 (1) (2002) 299 – 307.
  * [6] J. Heltsley, L. Brandon, A. Galonsky, L. Heilbronn, B. Remington, S. Langer, A. V. Molen, J. Yurkon, J. Kasagi, Particle identification via pulse-shape discrimination with a charge-integrating adc, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 263 (2) (1988) 441 – 445.
  * [7] D. H. von Seggern, CRC Standard Curves and Surfaces with Mathematica, Chapman and Hall/CRC, 2016.
  * [8] E. Jones, et al., SciPy: Open source scientific tools for Python (2001).
URL http://www.scipy.org/

  * [9] S. Bose, M. Chatterjee, B. Sinha, R. Bhattacharyya, Neutron-gamma discrimination based on leading edge shape measurement, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 270 (2) (1988) 487 – 491.
  * [10] M. Nakhostin, Signal Processing for Radiation Detectors, John Wiley and Sons, 2017\.