# Facial Tic Detection in Untrimmed Videos of Tourette Syndrome Patients

Yutao Tang1, Benjamín Béjar2, Joey K.-Y. Essoe3, Joseph F. McGuire3 and René
Vidal1  1 Mathematical Institute for Data Science and Department of Biomedical
Engineering, Johns Hopkins University, USA  2 Swiss Data Science Center, Paul
Scherrer Institut, Switzerland  3 Department of Psychiatry and Behavioral
Sciences, Johns Hopkins University School of Medicine, USA  {ytang67,
<EMAIL_ADDRESS><EMAIL_ADDRESS>{essoe<EMAIL_ADDRESS>
###### Abstract

Tourette Syndrome (TS) is a behavioral disorder that onsets in childhood and
is characterized by the expression of involuntary movements and sounds
commonly referred to as tics. Behavioral therapy is the first-line treatment
for patients with TS, and it helps patients raise awareness about tic
occurrence as well as develop tic inhibition strategies. However, the limited
availability of therapists and the difficulties for in-home follow up work
limits its effectiveness. An automatic tic detection system that is easy to
deploy could alleviate the difficulties of home-therapy by providing feedback
to the patients while exercising tic awareness. In this work, we propose a
novel architecture (T-Net) for automatic tic detection and classification from
untrimmed videos. T-Net combines temporal detection and segmentation and
operates on features that are interpretable to a clinician. We compare T-Net
to several state-of-the-art systems working on deep features extracted from
the raw videos and T-Net achieves comparable performance in terms of average
precision while relying on interpretable features needed in clinical practice.

##  I Introduction

Tourette Syndrome (TS) is a neuropsychiatric condition characterized by
involuntary movements (motor tics) and vocalizations (vocal tics) [1], which
can cause pain, injury, stress, or serious social impairment [2, 3]. Tics
usually start in the facial areas with eye movements being the most common
tics [4]. Although pharmacotherapies are available, behavioral therapy is
universally recommended as a first-line treatment because of its efficacy and
low adverse effects [5, 6, 7].

In behavioral therapy, therapeutic improvement depends on patients learning
behavioral skills to reduce tic expression. The two central skills are the
development of tic awareness and the implementation of behavioral strategies
that inhibit tic expression contingent upon such awareness. To master these
skills, patients are assigned “homework” practices between therapy sessions.
However, as many patients with TS have limited tic awareness, a support
partner (e.g., a parent) is required during homework practices to help
patients identify tic occurrences. Unfortunately, limitations in the
availability and tic identification accuracy of the support partners are often
barriers to homework practices. Since the success of homework practices
predicts therapeutic improvements and treatment response [8] to behavioral
therapy, tools that address these barriers hold considerable promise.

A possible solution to this problem would be to have an easy-to-deploy
automatic system that is able to detect and inform patients about the
occurrences of tics. Such a system would lower the barriers for homework
practice and bear the promise of improving therapy outcomes. A few studies
considered the problem of automatic tic detection [9, 10, 11]. [9] used
support vector machines to classify tics based on segmented temporal windows
of deep brain simulation signals. [10] used LeNet-5 to classify tics based on
segmented temporal windows of wireless channel information signals. [11] used
LSTM to classify tics based on well-trimmed temporal windows of deep features
extracted from the videos of patients using a convolutional neural network.

Despite the outstanding contributions of these studies, they have two main
drawbacks. First, most studies rely on data that requires a complex device
setup, which is hard to implement in common clinical settings or at the
patients’ home. Second, all prior studies performed tic classification on
well-localized temporal windows, which ignores long-range temporal relevancy
and limits the accuracy of the identification of tic occurrences. To overcome
these shortcomings and promote the accessibility and accuracy of the tool, we
propose to use videos recorded by a smartphone and perform temporal tic
detection in untrimmed videos. Besides, as facial motor tics are very common
among TS patients, we focus our attention on developing a tool for facial tic
detection.

Facial tic detection from videos is a particular instance of activity
detection. Activity detection is a significant and challenging task in the
field of video analysis and has practical applications in video
recommendation, smart surveillance, robotics, and beyond. Benefiting from the
increased computational power and the vast success of deep learning, as well
as the availability of large-scale public datasets with thousands of annotated
videos, many successful temporal action detection methods [12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22] have been proposed. For example, [14] developed a
boundary-sensitive approach, Boundary-Matching Network (BMN), for obtaining
candidate action proposals. [21] proposed Temporal Context Aggregation Network
(TCANet) that uses both local and global temporal context to progressively
obtain accurate action boundaries and reliable probability predictions. [22]
combined many image-level data augmentations to design a data augmentation
one-stage temporal action detector (DaoTAD).

However, tic detection poses several challenges that limit the use of state-
of-the-art methods. First, due to patient privacy and confidentiality,
publicly available videos are infrequent and never annotated, while most deep
learning methods require large datasets for training. Second, tics are
extremely variable in duration and heterogeneous in expression across and
within patients despite sharing the same category. For example, an “eye blink”
tic may vary in duration, repetition, and forcefulness across patients, or
even vary week-to-week within the same person. Third, the lack of
interpretability presents another barrier in applying current deep learning
techniques to tic detection as patients and clinicians may not be able to
interpret results in the context of behavioral therapy.

In this paper, we address the problem of tic detection in untrimmed videos
obtained from a smartphone. Specifically, we first establish baseline systems
based on BMN [14], DaoTAD [22] and TCANet [21] using deep features extracted
from the raw videos. Then, to make our system interpretable to potential
users, we use facial Action Unit (AU) intensities as features [23]. AUs are
localized facial movements, such as “upper lip raiser” and “nose wrinkler”,
representing the fundamental actions of individual facial muscles or muscle
groups. AU intensities are clinically meaningful features as clinicians
identify facial motor tics through the forcefulness, abruptness, and location
of patient movements. However, directly extending the baseline systems to use
AU intensities as input features degrade the performance. To reduce this drop
of performance, we propose a novel Tic detection network (T-Net) that uses
interpretable AU intensities as input features and achieves comparable
performance to BMN and better performance than DaoTAD and TCANet, using deep
features. As shown in Fig. 1, the proposed T-Net mainly consists of two
branches. The _temporal segmentation branch_ is an encoder-decoder
architecture that takes a sequence of AU intensities as input and predicts a
tic probability at each frame. Hence, some post-processing is needed to
convert segmentation probabilities into detection probabilities. The _temporal
detection branch_ directly produces temporal windows with associated detection
probabilities via an encoder-decoder architecture. However, the detection
probabilities tend to be noisier than the segmentation probabilities. T-Net
obtains the best of both worlds by fusing both segmentation and detection
probabilities, while at the same time mitigating the challenges of applying
state-of-the-art action detection methods to tic detection. Specifically, the
need for large training sets is mitigated by using a compact architecture with
a small number of layers as well as various data augmentation techniques.
Then, T-Net obtains precise boundaries for tics of variable duration by using
k-means clustering to get more suitable anchor sizes and applying an
additional regression loss, the efficient intersection over union (EIoU) loss
[24], to penalize imprecise boundaries. Finally, T-Net produces interpretable
detections by using AU intensities as features.

In summary, our contributions are three-fold:

  1. 1.

We study a novel application of temporal action detection, detection of facial
tics in untrimmed videos, which requires extending extant methods to the small
training data regime.

  2. 2.

We propose a novel one-stage temporal detector, T-Net, that uses temporal
segmentation probabilities to refine detection probabilities for temporal tic
detection.

  3. 3.

We demonstrate that using clinically meaningful AU intensities as input
features maintains good performance, in addition to gaining interpretability
of the detection results.

Figure 1: Structure of T-Net (shared). “En0” represents pre-residual
convolutional layer, and “En1” through “En4” denote the four residual stages
of the temporal ResNet-18. Segmentation branch: “Seg1” through “Seg3”
represent the T-UNet3+ decoder layers. Detection branch: “Det1” through “Det4”
represent the T-RetinaNet decoder layers (T-FPN). Score Fusion Head: 1D
RoIPooling was performed on the output of “Seg1” to create probability
vectors, guided by the windows predicted from the detection branch. After
that, these probability vectors were concatenated with the probability
predictions from the detection branch. The concatenations were then processed
by an MLP, generating a refined probability for each window.

##  II Related work

Our proposed T-Net architecture for tic detection integrates both temporal
detection and temporal segmentation approaches, which we briefly review in
this section.

###  II-A Temporal Action Detection

The goal of temporal action detection is to find instances of an action in a
video. Most temporal action detection methods draw inspiration from the object
detection literature, where the goal is to find instances of an object in a 2D
image. Object detectors can be classified into two main categories depending
on the way they operate. On the one hand, two-stage detectors such as [25, 26,
27, 28] follow the propose-then-classify scheme in which a number of candidate
bounding boxes are first generated through a proposal generation network and
subsequently classified. On the other hand, one-stage detectors like [29, 30,
31, 32] bypass the proposal generation stage and directly detect objects.
While two-stage detectors are usually more accurate, one-stage detectors are
generally more efficient.

Temporal action detection methods follow a similar approach to object
detection methods, except that 2D bounding boxes are replaced by temporal
windows. Specifically, two-stage methods, such as [13, 12, 14, 15, 33, 34, 16,
17], learn a class-agnostic temporal action proposal generator and the
generated proposals are then fed to an action classifier to produce the final
detection results. Meanwhile, one-stage methods, such as [18, 19, 20, 35, 36],
jointly learn the action proposal and classification networks, thus they are
more efficient. Since our goal is to develop a compact model that can be
trained from limited data, the detection branch of our T-Net uses a one-stage
temporal action detector adapted from RetinaNet [29].

###  II-B Temporal Action Segmentation

The goal of temporal action segmentation is to predict an action label for
each frame in a video. This requires capturing short- and long-range
dependencies between frames. Popular models to capture such temporal
dependencies include hidden Markov models [37, 38, 39], conditional random
fields [40, 41], recurrent neural networks [42], and temporal convolutional
networks [43, 44, 45]. However, the computational complexity of such
approaches during training and inference limits their applicability to short-
range dependencies, which can result in oversegmentation. Multi-stage temporal
convolutional networks [46, 47, 48] alleviate this issue by using multiple
stages of refinement. However, such networks have a very large number of
parameters and thus they cannot achieve good performance with limited data.
Temporal semantic segmentation networks [49, 50] alleviate this problem by
using multi-scale encoder-decoder architectures, which can also capture long-
range interactions, and thus have the potential to produce more accurate
estimate of the probabilities of a tic at the frame level. Therefore, in this
work we use a temporal semantic segmentation network as the segmentation
branch of our T-Net. Specifically, we adapted UNet3+ [51], which was
originally proposed for medical image segmentation. UNet3+ uses full-scale
skip connections to capture full-scale semantics for the segmentation of
organs that appear at different scales. Therefore, the temporal UNet3+ also
captures full-scale details, which is extremely beneficial for tic
segmentation as tics have highly variable duration.

###  II-C Automatic Tic Detection

Very few studies have addressed the automatic tic detection problem from video
data. For instance, [52] adaptively thresholded signals from triaxial
accelerometers placed on the patients’ trunk and achieved automatic tic
detection by classifying 2-second non-overlapping windows of the recorded
signals, [9] used deep brain stimulation (DBS) – an invasive neuro-modulatory
therapy – to collect and classify segmented temporal windows of DBS signals
using support vector machines, and [10] used LeNet-5 to classify segmented
temporal windows of wireless channel information (WCI) signals. One of the few
studies that considers video data for tic detection in well-trimmed videos is
[11], which used an unsupervised learning framework similar to SimCLR [53] to
extract robust video features from segmented one-second video clips and then
fed these features to an LSTM for classification. However, despite the
outstanding contributions of these studies, they are difficult to translate
into clinical practice for two main reasons: On the one hand, non-video
signals are difficult to obtain in everyday clinical settings or in patients’
homes. In fact, most patients with TS would consider DBS as a last resort. On
the other hand, all prior studies [52, 9, 10, 11] focus on detecting tics in
well-trimmed signals or video clips. This limits the translational value
because many tics would span between two or more windows. To the best of our
knowledge, we are the first to address the problem of tic detection in
untrimmed videos obtained from ubiquitous devices such as smartphones, which
maximizes the translational value of our work.

##  III Methods

Let $X=\\{x_{l}\\}_{l=1}^{L}$ denote an ${L}$ frame-long untrimmed video,
where $x_{l}$ is the $l$-th frame. We denote detection annotations for $X$ as
$Y=\\{y_{n}\\}_{n=1}^{N}$, where $y_{n}=(t_{n}^{S},t_{n}^{E})$ is a tic
instance with starting frame $t_{n}^{S}$ and ending frame $t_{n}^{E}$. For the
segmentation task, we convert $Y$ to a sequence of labels
$S=\\{s_{l}\\}_{l=1}^{L}$, where $s_{l}=1$ (tic) if
$l\in[t_{n}^{S},t_{n}^{E}]$ and $s_{l}=0$ (non-tic) otherwise.

###  III-A Feature Encoding

We adopt the two-stream style network TSN [54] to extract deep features
directly from raw videos. This network consists of two convolutional streams
which process the RGB and optical flow data, respectively, followed by global
average pooling. Concatenating the output of the two streams, we get the deep
features $f_{l}\in\mathbb{R}^{C^{v}}$ of frame $x_{l}$, where $C^{v}=3,072$ is
the dimension of the features. Alternatively, we also consider interpretable
features, AU intensities, extracted from the videos using OpenFace [55].
OpenFace extracts intensities for 17 AUs and thus we get the features
$f_{l}\in\mathbb{R}^{C^{a}}$ of frame $x_{l}$, where $C^{a}=17$ is the
dimension of the AU intensities. Therefore, given an untrimmed video $X$, we
can extract a feature sequence $F=\\{f_{l}\\}_{l=1}^{L}\in\mathbb{R}^{C\times
L}$ where $C\in\\{C^{v},C^{a}\\}$.

###  III-B Data Augmentation

Since untrimmed videos are of variable temporal length and are usually too
large to fit into memory for computation, we divide a video of length $L$ into
$W$ video clips of fixed length $T$ using an overlapping sliding window
strategy. Here, we use a long observation window with $T=416$ frames to obtain
$W$ feature sequences $\\{F_{w}\\}_{w=1}^{W}$, where
$F_{w}=\\{f_{t}\\}_{t=1}^{T}\in\mathbb{R}^{C\times T}$. We also apply random
Gaussian noise ($\mu=0,\sigma=0.01$) with a chance of 50% to the input
features during training stage.

Figure 2: Structure of T-RetinaNet. The system builds a 4-layer T-FPN on top
of the temporal ResNet-18, followed by a two-branch convolutional head
performing classification and regression.

###  III-C T-Net

We propose a novel one-stage temporal detector, Tic-Net (T-Net) that
integrates Temporal RetinaNet (T-RetinaNet) and temporal UNet3+ (T-UNet3+)
together to perform temporal tic detection. T-Net includes two branches
(detection branch and segmentation branch) that share the same encoder and a
score fusion head that heuristically combines the probability predictions from
the two branches to generate refined probabilities for the final detections
(T-Net structure is shown in Fig. 1).

####  III-C1 Detection Branch

We adapted RetinaNet [29] to a temporal action detector, T-RetinaNet, for
detecting tics (structure shown in Fig. 2). We use temporal ResNet-18
(T-ResNet-18) as the shared encoder and build a 4-stage temporal Feature
Pyramid Network [56] (T-FPN) on top of it as the decoder of the detection
branch. In the T-FPN, The output of each ResNet residual stage is input to the
corresponding stage of T-FPN to compute pyramid outputs. Then, a two-branch
one-layer convolutional head is applied to the outputs of T-FPN, performing
tic classification and temporal boundary window regression respectively. In
summary, the detection branch $\mathcal{B}_{D}$ takes the features sequence
$F_{w}$ as input and outputs detection windows with corresponding
probabilities, namely

$\mathcal{B}_{D}(F_{w})=D_{w}=\\{b_{i},c_{i}\\}_{i=1}^{N_{D}}\;,\ \
b_{i}=(\widehat{t_{i}^{S}},\widehat{t_{i}^{E}})\enspace,$ (1)

where $b_{i}$ denotes the $i$-th detection window with $\widehat{t_{i}^{S}}$
and $\widehat{t_{i}^{E}}$ being the starting and ending frame respectively,
and $c_{i}$ represents the probability of $b_{i}$ being a tic instance.

####  III-C2 Segmentation Branch

We adapted UNet3+ [51] to T-UNet3+ for temporal tic segmentation. T-UNet3+
utilizes the full-scale information by incorporating both the features maps
from the same- and smaller-scale stages in the encoder, and the outputs from
the larger-scale stages in the decoder. We build a 3-stage T-UNet3+ on top of
the first three stages of T-ResNet-18 (shared encoder), namely the pre-
residual stage and the first two ResNet residual stages. The segmentation
branch $\mathcal{B}_{S}$ also takes the feature sequence $F_{w}$ as input and
outputs the probability prediction for each frame at each stage, namely

$\mathcal{B}_{S}(F_{w})=\\{P_{w}^{k}\\}_{k=1}^{K}\;,\ \
P_{w}^{k}=\\{p_{j}^{k}\\}_{j=1}^{T}\enspace,$ (2)

where $K=3$ denotes the number of stages in T-UNet3+, and $p_{j}^{k}$
represents the probability of the $j$-th frame to be a tic in the $k$-th stage
of the T-UNet3+ decoder.

####  III-C3 Score Fusion Head

In the score fusion head, we combine the probability predictions from both
branches heuristically to obtain refined probabilities. Specifically, we first
perform an 1D RoIPooling on the output of the lowest-scale stage of the
T-UNet3+ decoder, i.e. $P_{w}^{1}$, according to $\\{b_{i}\\}_{i=1}^{N_{D}}$
to extract a probability vector $V_{i}$ for each detection window $b_{i}$.
This probability vector is then concatenated with $c_{i}$, the probability
prediction from the detection branch. After this, the concatenation is
processed by a multilayer perceptron (MLP) to get a refined probability
$\widetilde{c}_{i}$ to form the refined detections $\widetilde{D}_{w}$.
Specifically, this can be expressed as

$\displaystyle\widetilde{c}_{i}=\text{MLP}(V_{i}\bigoplus c_{i}),\
V_{i}=\phi(b_{i},P_{w}^{1}),\ i=1,\ldots,N_{D}\enspace,$ (3)
$\displaystyle\mathcal{B}_{H}(D_{w},P_{w}^{1})=\widetilde{D}_{w}=\\{b_{i},\widetilde{c}_{i}\\}_{i=1}^{N_{D}}\;,\
\ b_{i}=(\widehat{t_{i}^{S}},\widehat{t_{i}^{E}})\enspace,$ (4)

where $\bigoplus$ denotes concatenation and $\phi$ denotes the 1D RoIPooling
operation.

Anchors: Following YOLO9000 [31], we use k-means clustering to generate proper
anchor sizes. Anchors are matched to ground-truth annotations (positives) if
their efficient intersection over union (EIoU) is greater than $0.3$ or to
background (negatives) if their EIoU is less than $-0.4$. EIoU [24] is a
refined version of IoU as it considers not only the IoU, but also the central
points distance and lengths difference among anchors and ground-truth
annotations. Specifically, we can denote an anchor as

$A=(t_{A}^{S},t_{A}^{E}),\quad m_{A}=(t_{A}^{S}+t_{A}^{E})/2,\quad
d_{A}=t_{A}^{E}-t_{A}^{S}\enspace,$ (5)

where $t_{A}^{S}$ and $t_{A}^{E}$ represent the starting and ending frame of
this anchor, respectively, $m_{A}$ represents the central point of this
anchor, and $d_{A}$ represents the length. Similarly, we can also denote a
ground-truth annotation $G=(t_{G}^{S},t_{G}^{E})$ with central point $m_{G}$
and length $d_{G}$, and the smallest enclosing window
$C=(t_{C}^{S},t_{C}^{E})$ where $t_{C}^{S}=\min\\{t_{A}^{S},t_{G}^{S}\\}$ and
$t_{C}^{E}=\max\\{t_{A}^{E},t_{G}^{E}\\}$, with central point $m_{C}$ and
length $d_{C}$. Finally, the EIoU between the anchor and the annotation can be
expressed as

$\text{EIoU}=\text{IoU}-\frac{(m_{A}-m_{G})^{2}}{d_{C}^{2}}-\frac{(d_{A}-d_{G})^{2}}{d_{C}^{2}}\enspace.$
(6)

Training: For the detection branch, we first perform online hard negative
mining as used in [30] with a positive-to-negative ratio of $1:3$ to address
the imbalance problem. Then, we apply binary cross-entropy (bce) loss for
classification to all the positives and negatives and $\text{SmoothL}_{1}$
(sl) loss for regression to all the positives. The implementation of these
losses is the same as in [30]. Additionally, we apply EIoU loss [24], namely
$1-\text{EIoU}$, for regression as well, which encourages the detections to be
as similar as possible to the ground-truth annotations. The total loss for the
detection branch can be expressed as the weighted sum of the classification
loss and the regression losses, namely

$L_{w}^{D}=\frac{1}{N_{p}}\sum\nolimits_{i}L_{w,{\text{bce}}}(\widetilde{c}_{i},\widetilde{y}_{i})+\gamma(L_{w,\text{sl}}+\lambda
L_{w,\text{EIoU}})(b_{i},y_{i})\;,$ (7)

where $\widetilde{y}_{i}=1$ if the anchor for generating $b_{i}$ was matched
to a ground-truth (positive) otherwise $\widetilde{y}_{i}=0$, and
$y_{i}=(t_{i}^{S},t_{i}^{E})$ denotes the ground-truth annotation. $N_{p}$ is
the number of positives. For the hyperparameters, we set $\gamma=2$ and
$\lambda=1.5$.

For the segmentation branch, we adopt focal loss (fl) [29] and Dice loss and
use deep supervisions on the predicted probability sequences from all stages
in T-UNet3+, namely $\\{P_{w}^{k}\\}_{k=1}^{K}$, following [51]. The loss for
the segmentation branch can be expressed as

$L_{w}^{S}=\frac{1}{T}(L_{w,\text{fl}}+L_{w,\text{Dice}})(\\{P_{w}^{k}\\}_{k=1}^{K},\\{s_{j}\\}_{j=1}^{T})\enspace.$
(8)

Finally, the total loss for training T-Net can be expressed as the weighted
sum of the detection branch loss, the segmentation branch loss, and $L_{2}$
regularization term,

$L_{w}=L_{w}^{D}+\alpha L_{w}^{S}+\beta L_{2}(\Theta)\enspace,$ (9)

where we set $\alpha=3$ and $\beta=0.001$.

To enlarge the model capacity, we also allow independent encoders for
detection and segmentation branch separately, which is referred to as T-Net
(independent) while the shared-encoder version is T-Net (shared).

Inference: Detections from all pyramidal levels are merged with a probability
threshold of 0.2 and NMS using EIoU with a threshold of 0.2 is applied to
yield final detections.

##  IV Experiments

###  IV-A Data Acquisition and Annotation Protocol

Facial motor tics of one 12-year-old male diagnosed with TS were analyzed in
this paper. We recorded 12 15-minute videos across four visits for this
participant. In each visit, he performed three activities. In Activity 1, the
participant sat in a chair and did not engage in any intentional movement. In
Activity 2, the participant was engaged in a movement-based activity (e.g.,
completing schoolwork). In Activity 3, the participant was engaged in
conversation with an experimenter about tics. All videos were recorded using a
smart phone with 30 FPS. Thereafter, temporal boundary windows were annotated
using open-source software VIA3 [57]. Each video was first labelled by two
trained annotators. Then the two sets of annotations were checked by a senior
researcher, who resolved any inconsistency.

###  IV-B Cross-validation Strategies

In total, 2631 tic instances were annotated from the entire 12 videos. Mean
and median duration of annotations were 41.15 and 32 frames, respectively, and
99.96% of the annotations were shorter than 416 frames. Experiments were
performed following two cross-validation strategies: (i) Leave One Session Out
(LOSO): one out of four sessions (three out of twelve videos) were held for
testing, and (ii) Leave One Video Out (LOVO): one out of twelve videos was
held for testing.

Features | Methods |<EMAIL_ADDRESS>
---|---|---
|  | LOSO | LOVO
Deep Features | BMN [14] | 40.32 | 45.87
Deep Features | DaoTAD [22] | 35.52 | 38.29
Deep Features | TCANet [21] | 39.40 | 40.30
AU intensities | BMN [14] | 33.09 | 38.26
AU intensities | DaoTAD [22] | 10.19 | 3.29
AU intensities | TCANet [21] | 16.42 | 30.02
AU intensities | T-Net (detection branch only) | 16.33 | 25.08
AU intensities | T-Net (shared) | 35.92 | 41.84
AU intensities | T-Net (independent) | 37.30 | 43.73

TABLE I: Detection results: AP at IoU threshold of 0.5 and probability
threshold of 0.2. The metrics are averaged across splits for each cross-
validation strategy.

###  IV-C Detection Results

Table I presents the metric, average precision (AP), for both cross-validation
strategies. The metric is calculated at the IoU threshold of 0.5 and
probability threshold of 0.2, i.e. only detections with predicted probability
(of this detection being a tic) greater than 0.2 will be considered and the
tic is considered detected if the IoU between the detection and the ground-
truth annotation is larger than 0.5. We evaluate the baseline systems based on
BMN [14], DaoTAD [22] and TCANet [21] that use deep features extracted from
raw videos and they achieve the AP of 45.87, 38.29 and 40.30 respectively in
LOVO. This shows that we successfully established workable systems for
temporal tic detection in untrimmed videos. Moreover, LOVO results show
significant improvements as compared to LOSO results, which indicates getting
more data could be beneficial to promote detection performance.

Besides, when comparing T-Net with baseline methods, T-Net that operates on AU
intensities outperforms all three baseline methods using AU intensities as
input features. For comparison with the baseline methods using deep features,
T-Net achieves comparable performance (AP of 43.73) to BMN (AP of 45.87) and
better performance than DaoTAD (AP of 38.29) and TCANet (AP of 40.30) in LOVO.
Therefore, these results show the effectiveness of our proposed T-Net which
can improve the interpretability through replacing deep features with AU
intensities without significantly affecting the performance.

Moreover, the grey-shaded rows in Table I present an ablation analysis where
we compare the usefulness of adding the segmentation branch and enlarging
model capacity in T-Net. Having the additional segmentation branch in T-Net
(shared) brings around 19% and 16% increase in LOSO and LOVO respectively,
compared to using the detection branch only. We also investigated the results
and found the refinement of probability significantly reduces many high-
probability false positives and therefore improves the performance. This shows
the refinement of probability is crucial and effective for T-Net. Besides,
enlarging the model capacity by allowing independent encoders for detection
and segmentation branch, namely T-Net (independent) brings around 2% increase
compared to T-Net (shared), which implies the detection branch and the
segmentation branch require different type of features and by having different
encoders, branch-specific instead of general features can be extracted.

###  IV-D Interpretability of AU intensities

In clinical practice, clinicians identify motor tics by evaluating the
forcefulness, abruptness, and location of patient movements. This includes a
combination of the intensity of muscle activations, identity and combination
of the muscle involved, and the temporal relationships between these
activations. As AUs represent the basic movements of facial muscles or muscle
groups, AUs capture the spatial aspect of the features clinicians use to
identify tics. Since AUs represent a meaningful analog to how tics are
identified by clinicians, models using AU intensities as features would in
turn be more interpretable by humans, unlike visual features extracted by deep
neural networks.

Fig. 3 illustrates the interpretability of AU intensities. Here, heatmaps show
the normalized AU intensities of six actions. The y-axis lists the 17 AUs
extracted using OpenFace (e.g., “inner brow raiser”, “upper lid raiser”, “chin
raiser”). The x-axis denotes time frames and the color represents the
normalized AU intensities (brighter colors indicate stronger intensity). When
the participant sat quietly (non-tic behavior, top row left), “Blink” was the
only AU moderately activated. When participant talked (non-tic behavior, top
row middle), AUs related to lips and mouth were activated whereas the other
AUs showed low intensity. The other four maps correspond to specific tics of
this participant. During his “Head jerking tic”, the participant would
suddenly twist his head, and the heatmap shows nearly all AUs were activated,
with the highest intensities observed for AUs involving the jaw, mouth, and
lips. Likewise, AUs involving the lips were strongly activated during the
participant’s “Lip biting tic” (upper lip pressing down on the lower lip) and
“Teeth biting lip tic” (teeth biting the lower lip). Finally, during the “Wink
eye tic” (closing one eye forcefully), the AU “(Eye)Lid Tightener” was
activated and the AU “Blink” exhibited higher intensity than that during the
“sitting quietly” activity, in which the same AU was only moderately
activated.

Fig. 4 illustrates two examples showing how the T-Net captures the patterns of
AU intensities using 1D-Grad-CAM. Grad-CAM was designed to visually explain
classifiers’ decisions, which highlights spatially important regions in 2D
images. It uses the gradients to weigh feature maps which are then combined to
get the class activation map (CAM). In this study, we adapted it to 1D domain
(1D-Grad-CAM) to map important time frames to the detector’s classification
decision (i.e. whether a temporal window contains a tic). The top row in Fig.
4 shows heatmap of normalized AU intensities for an observation window where
AUs are listed in the y-axis and the x-axis represents time frames. The bottom
row shows the visualization of 1D-Grad-CAM results. In the left example, we
have a detection from 85.40 to 126.86 frame which contained a “Wink eye tic”.
From the figure, the CAM is focused on the area around the detection which
highlights the strongly activated AU “(Eye)Lid Tightener” marked by a green
box on the heatmap. In the right example, a detection starting from 0.00 to
26.34 frame contained a “Lip biting tic” where CAM also shows focusing on the
area and lips and mouth related AUs are activated (marked by green boxes).
Thus, the AU intensities themselves are interpretable and our T-Net detector
captures the interpretable patterns of the AU intensities which subsequently
brings more clinical interpretability to our T-Net.

Figure 3: Normalized AU intensities heatmap for example actions. Figure 4:
Top: normalized AU intensities heatmap. Bottom: 1D-Grad-CAM visualization.
Brighter means more informative.

##  V Conclusion

In this work, we were the first to develop a system, T-Net that combined
segmentation probabilities to refine detection probabilities, for automatic
facial tic detection in untrimmed videos. T-Net helped lift the barriers to
the behavioral therapy by promoting accessibility and providing accurate tic
detections. T-Net was easy-to-implement because the videos used were recorded
by a ubiquitous device, a smartphone. T-Net was also shown to be accurate as
it achieved comparable performance to baseline methods which use deep
features. Finally, T-Net was interpretable as it used interpretable AU
intensities as input features and captured the interpretable patterns of AU
intensities.

## Acknowledgment

This work has been supported by Discovery Challenge Award, Johns Hopkins
Discovery Fund Program.

## References

  * [1] M. H. Bloch and J. F. Leckman, “Clinical course of tourette syndrome,” _Journal of psychosomatic research_ , vol. 67, no. 6, pp. 497–501, 2009.
  * [2] C. A. Conelea, D. W. Woods, S. H. Zinner, C. Budman, T. Murphy, L. D. Scahill, S. N. Compton, and J. Walkup, “Exploring the impact of chronic tic disorders on youth: results from the tourette syndrome impact survey,” _Child Psychiatry & Human Development_, vol. 42, no. 2, pp. 219–242, 2011.
  * [3] C. A. Conelea, D. W. Woods, S. H. Zinner, C. L. Budman, T. K. Murphy, L. D. Scahill, S. N. Compton, and J. T. Walkup, “The impact of tourette syndrome in adults: results from the tourette syndrome impact survey,” _Community mental health journal_ , vol. 49, no. 1, pp. 110–120, 2013.
  * [4] J. F. McGuire, E. Nyirabahizi, K. Kircanski, J. Piacentini, A. L. Peterson, D. W. Woods, S. Wilhelm, J. T. Walkup, and L. Scahill, “A cluster analysis of tic symptoms in children and adults with tourette syndrome: clinical correlates and treatment outcome,” _Psychiatry research_ , vol. 210, no. 3, pp. 1198–1204, 2013.
  * [5] J. K.-Y. Essoe, M. A. Grados, H. S. Singer, N. S. Myers, and J. F. McGuire, “Evidence-based treatment of tourette’s disorder and chronic tic disorders,” _Expert review of neurotherapeutics_ , vol. 19, no. 11, pp. 1103–1115, 2019.
  * [6] H. Weisman, I. A. Qureshi, J. F. Leckman, L. Scahill, and M. H. Bloch, “Systematic review: pharmacological treatment of tic disorders–efficacy of antipsychotic and alpha-2 adrenergic agonist agents,” _Neuroscience & Biobehavioral Reviews_, vol. 37, no. 6, pp. 1162–1171, 2013.
  * [7] T. Pringsheim, Y. Holler-Managan, M. S. Okun, J. Jankovic, J. Piacentini, A. E. Cavanna, D. Martino, K. Müller-Vahl, D. W. Woods, M. Robinson _et al._ , “Comprehensive systematic review summary: treatment of tics in people with tourette syndrome and chronic tic disorders,” _Neurology_ , vol. 92, no. 19, pp. 907–915, 2019.
  * [8] J. K.-Y. Essoe, E. J. Ricketts, K. A. Ramsey, J. Piacentini, D. W. Woods, A. L. Peterson, L. Scahill, S. Wilhelm, J. T. Walkup, and J. F. McGuire, “Homework adherence predicts therapeutic improvement from behavior therapy in tourette’s disorder,” _Behaviour Research and Therapy_ , vol. 140, p. 103844, 2021.
  * [9] J. B. Shute, M. S. Okun, E. Opri, R. Molina, P. J. Rossi, D. Martinez-Ramirez, K. D. Foote, and A. Gunduz, “Thalamocortical network activity enables chronic tic detection in humans with tourette syndrome,” _NeuroImage: Clinical_ , vol. 12, pp. 165–172, 2016.
  * [10] A. Barua, C. Dong, and X. Yang, “A deep learning approach for detecting tic disorder using wireless channel information,” _Transactions on Emerging Telecommunications Technologies_ , vol. 32, no. 7, p. e3964, 2021.
  * [11] J. Wu, T. Zhou, Y. Guo, Y. Tian, Y. Lou, H. Ru, J. Feng, and J. Li, “Tic detection in tourette syndrome patients based on unsupervised visual feature learning,” _Journal of Healthcare Engineering_ , vol. 2021, 2021.
  * [12] R. Zeng, W. Huang, M. Tan, Y. Rong, P. Zhao, J. Huang, and C. Gan, “Graph convolutional networks for temporal action localization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7094–7103.
  * [13] Z. Shou, D. Wang, and S.-F. Chang, “Temporal action localization in untrimmed videos via multi-stage cnns,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1049–1058.
  * [14] T. Lin, X. Liu, X. Li, E. Ding, and S. Wen, “Bmn: Boundary-matching network for temporal action proposal generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3889–3898.
  * [15] T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang, “Bsn: Boundary sensitive network for temporal action proposal generation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 3–19.
  * [16] Y. Liu, L. Ma, Y. Zhang, W. Liu, and S.-F. Chang, “Multi-granularity generator for temporal action proposal,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3604–3613.
  * [17] C. Lin, J. Li, Y. Wang, Y. Tai, D. Luo, Z. Cui, C. Wang, J. Li, F. Huang, and R. Ji, “Fast learning of temporal action proposal via dense boundary generator,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 11 499–11 506.
  * [18] T. Lin, X. Zhao, and Z. Shou, “Single shot temporal action detection,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 988–996.
  * [19] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin, “Temporal action detection with structured segment networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2914–2923.
  * [20] D. Zhang, X. Dai, X. Wang, and Y.-F. Wang, “S3d: single shot multi-span detector via fully 3d convolutional networks,” _arXiv preprint arXiv:1807.08069_ , 2018.
  * [21] Z. Qing, H. Su, W. Gan, D. Wang, W. Wu, X. Wang, Y. Qiao, J. Yan, C. Gao, and N. Sang, “Temporal context aggregation network for temporal action proposal refinement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 485–494.
  * [22] C. Wang, H. Cai, Y. Zou, and Y. Xiong, “Rgb stream is enough for temporal action detection,” _arXiv preprint arXiv:2107.04362_ , 2021.
  * [23] P. Ekman and W. V. Friesen, “Facial action coding system,” _Environmental Psychology & Nonverbal Behavior_, 1978.
  * [24] Y.-F. Zhang, W. Ren, Z. Zhang, Z. Jia, L. Wang, and T. Tan, “Focal and efficient iou loss for accurate bounding box regression,” _arXiv preprint arXiv:2101.08158_ , 2021.
  * [25] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [26] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, pp. 91–99, 2015.
  * [27] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [28] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6154–6162.
  * [29] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [30] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [31] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [32] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 781–10 790.
  * [33] J. Gao, Z. Yang, K. Chen, C. Sun, and R. Nevatia, “Turn tap: Temporal unit regression network for temporal action proposals,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 3628–3636.
  * [34] J. Gao, K. Chen, and R. Nevatia, “Ctap: Complementary temporal action proposal generation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 68–83.
  * [35] M. Xu, C. Zhao, D. S. Rojas, A. Thabet, and B. Ghanem, “G-tad: Sub-graph localization for temporal action detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 156–10 165.
  * [36] Q. Liu and Z. Wang, “Progressive boundary refinement network for temporal action detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 11 612–11 619.
  * [37] L. Tao, E. Elhamifar, S. Khudanpur, G. D. Hager, and R. Vidal, “Sparse hidden markov models for surgical gesture classification and skill evaluation,” in _International conference on information processing in computer-assisted interventions_. Springer, 2012, pp. 167–177.
  * [38] H. Kuehne, J. Gall, and T. Serre, “An end-to-end generative framework for video segmentation and recognition,” in _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2016, pp. 1–8.
  * [39] H. Kuehne, A. Richard, and J. Gall, “A hybrid rnn-hmm approach for weakly supervised temporal action segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 4, pp. 765–779, 2018.
  * [40] C. Lea, R. Vidal, and G. D. Hager, “Learning convolutional action primitives for fine-grained action recognition,” in _2016 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2016, pp. 1642–1649.
  * [41] E. Mavroudi, D. Bhaskara, S. Sefati, H. Ali, and R. Vidal, “End-to-end fine-grained action segmentation and recognition using conditional random field models and discriminative sparse coding,” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2018, pp. 1558–1567.
  * [42] B. Singh, T. K. Marks, M. Jones, O. Tuzel, and M. Shao, “A multi-stream bi-directional recurrent neural network for fine-grained action detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1961–1970.
  * [43] C. Lea, A. Reiter, R. Vidal, and G. D. Hager, “Segmental spatiotemporal cnns for fine-grained action segmentation,” in _European Conference on Computer Vision_. Springer, 2016, pp. 36–52.
  * [44] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 156–165.
  * [45] P. Lei and S. Todorovic, “Temporal deformable residual networks for action segmentation in videos,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6742–6751.
  * [46] Y. A. Farha and J. Gall, “Ms-tcn: Multi-stage temporal convolutional network for action segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3575–3584.
  * [47] S.-J. Li, Y. AbuFarha, Y. Liu, M.-M. Cheng, and J. Gall, “Ms-tcn++: Multi-stage temporal convolutional network for action segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * [48] Z. Wang, Z. Gao, L. Wang, Z. Li, and G. Wu, “Boundary-aware cascade networks for temporal action segmentation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 34–51.
  * [49] F. Wang, Y. Song, J. Zhang, J. Han, and D. Huang, “Temporal unet: Sample level human action recognition using wifi,” _arXiv preprint arXiv:1904.11953_ , 2019.
  * [50] Y. Y. Luktuke and A. Hoover, “Segmentation and recognition of eating gestures from wrist motion using deep learning,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 1368–1373.
  * [51] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu, “Unet 3+: A full-scale connected unet for medical image segmentation,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 1055–1059.
  * [52] M. Bernabei, G. Andreoni, M. O. M. Garcia, L. Piccini, F. Aletti, M. Sassi, D. Servello, M. Porta, and E. Preatoni, “Automatic detection of tic activity in the tourette syndrome,” in _2010 Annual International Conference of the IEEE Engineering in Medicine and Biology_. IEEE, 2010, pp. 422–425.
  * [53] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [54] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _European conference on computer vision_. Springer, 2016, pp. 20–36.
  * [55] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 2018, pp. 59–66.
  * [56] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [57] A. Dutta and A. Zisserman, “The VIA annotation software for images, audio and video,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , ser. MM ’19. New York, NY, USA: ACM, 2019. [Online]. Available: https://doi.org/10.1145/3343031.3350535