# Recognizing Dynamic Scenes
with Deep Dual Descriptor
based on Key Frames and Key Segments

###### Abstract

Recognizing dynamic scenes is one of the fundamental problems in scene
understanding, which categorizes moving scenes such as a forest fire,
landslide, or avalanche. While existing methods focus on reliable capturing of
static and dynamic information, few works have explored frame selection from a
dynamic scene sequence. In this paper, we propose dynamic scene recognition
using a deep dual descriptor based on ‘key frames’ and ‘key segments.’ Key
frames that reflect the feature distribution of the sequence with a small
number are used for capturing salient static appearances. Key segments, which
are captured from the area around each key frame, provide an additional
discriminative power by dynamic patterns within short time intervals. To this
end, two types of transferred convolutional neural network features are used
in our approach. A fully connected layer is used to select the key frames and
key segments, while the convolutional layer is used to describe them. We
conducted experiments using public datasets as well as a new dataset comprised
of 23 dynamic scene classes with 10 videos per class. The evaluation results
demonstrated the state-of-the-art performance of the proposed method.

Index Terms—  dynamic scene classification, key frame, key segment,
convolutional neural networks, transfer learning, deep dual descriptor (D3)

##  1 INTRODUCTION

Dynamic scene recognition is an increasingly important problem in identifying
a moving scene, such as a forest fire, waterfall, or avalanche [1]. A major
challenge in dynamic scene recognition is the reliable capturing of spatial
and temporal information from dynamic scenes. Over the past decade,
considerable efforts [2, 3, 4, 5, 6] have been devoted to these issues. The
various approaches can be divided into two categories: simultaneous modeling
and separate modeling. In simultaneous modeling, numerous spatiotemporal
descriptors are constructed. Each one describes a small 3D (x-y-t) cuboid. In
separate modeling, the spatial and temporal features are separately modeled
then combined, i.e., late fusion.

Early investigations of simultaneous modeling have involved linear dynamical
systems (LDS) [7], which analyze video sequences in the Stiefel manifold. This
approach consists of a hidden-state process that encodes motions from the
video as well as the appearance of each frame, which is conditioned on the
current hidden state. Several subsequent approaches [2, 3, 4] considered the
LDS model to represent dynamic texture that closely relates to a dynamic
scene. However, these approaches have not shown promising results in dynamic
scene recognition because the first-order Markov property or linearity
assumption of LDS is restrictive in dynamic scenes [5].

To better represent dynamic scenes, Derpanis et al. [6] introduced multi-scale
orientation features using 3D Gaussian third-derivative filters. The bag of
features (BoF) scheme [8] was additionally applied to represent several
spatiotemporal patches in dynamic scenes [9, 10]. Encouraged by the promising
results of convolutional neural networks (CNNs) [11, 12, 13], Tran et al. [14]
recently proposed a convolutional three-dimensional (C3D) architecture that is
a spatiotemporal version of CNN. The performances of CNN-based features, i.e.,
C3D, were shown to be significantly better than those of hand-crafted features
in public dynamic scene datasets.

On the other hand, separate modeling approaches were advanced with the
introduction of the methods in [5]. To capture spatial information, the
authors first extracted GIST features [15] from whole frames in the sequence.
For temporal information, chaotic features [5] are calculated and then
combined with spatial GIST features for video-level representation. Meanwhile,
Yang et al. [16] suggested an ensemble scheme for aggregating spatial and
temporal features from dynamic scenes. Tremendous spatial features, such as
local binary pattern (LBP) [17], Gabor [18], and GIST [15], are extracted from
each frame in the sequence, and naive dynamic features [7] are used for
temporal information. Furthermore, Qi et al. [19] suggested a transferred CNN
feature (TCoF)-based approach. A spatial feature is extracted for a chosen
number of frames by a pre-trained CNN. To capture spatial information, each
frame is used as the input of the pre-trained CNN, while the differences
between two adjacent frames are used as the inputs to capture temporal
information.

Overall, recent experiments have shown that separate modeling, including
tremendous hand-crafted spatial features [16] or transferred CNN features
[19], outperform methods involving simultaneous modeling [6, 9, 10, 20, 21].
Nevertheless, we believe that performance can still be improved of separate
modeling of dynamic scenes in two aspects as outlined below.

  1. $\bullet$

Frame selection: There are several frame selection strategies for capturing
the static appearances of dynamic scenes. Shroff et al. [5] used whole frames
in the sequence, which is a time-consuming approach that produces redundant
information. Recently, randomly selected frames [16] or partially consecutive
frames [19], e.g., the first $\frac{N}{8}$ frames, were used to represent
dynamic scenes. Although these approaches address computational time and
redundancy issues, they do not well reflect the characteristics of a given
sequence.

  1. $\bullet$

Temporal modeling: In addition to the static appearance, temporal information
may provide additional discriminant features. In the TCoF-based method [16],
the difference of two adjacent frames is used as input for the pre-trained CNN
model for temporal modeling. However, the information between only two
adjacent images is considered insufficient for capturing irregular motions of
dynamic scenes.

Fig. 1:  Dynamic scene representation using key frames and key segments. A key
frame is used for capturing the salient static appearance of a dynamic scene.
A key segment, which is captured from the area around each key frame, provides
crucial discriminative power with dynamic patterns.

To address the above issues, we propose a dynamic scene recognition method
based on the ‘key frame’ and ‘key segment.’ The main concept of the proposed
method is illustrated in Fig. 1. A key frame solves the limitation of
conventional frame selection, while a key segment, which is captured from the
area around each key frame, efficiently provides temporal information. To this
end, two types of transferred CNN features are used in our approach. A fully
connected (FC) layer, which reflects the overall structure of each scene, is
used to select the key frame and key segment. On the other hand, a
convolutional (Conv) layer, which considers the local texture of each scene,
is used to describe the key frame and key segment.

To validate the effectiveness of the proposed method, we conducted extensive
experiments on both dynamic scene datasets—Maryland [5] and Yupenn [6]—and
dynamic texture datasets—Dyntex-Alpha, Dyntex-Beta, and Dyntex-Gamma [22].
Currently, there exist only two benchmark datasets for dynamic scenes; thus,
we constructed a dataset comprised of 23 dynamic scene classes with ten videos
per class by combining the Maryland and Yupenn datasets.

The main contributions of this study are summarized as follows: (i) We propose
a CNN-based dynamic scene representation method based on key frames and key
segments, which capture characteristics of a dynamic scene sequence using only
a small number of frames. (ii) We investigated various factors that influence
the performance of our approach, such as the frame selection strategy, number
of frames, and layers for feature description. (iii) Owing to the lack of
public dynamic scene datasets, we constructed a rich dynamic scene dataset of
23 classes with 10 videos per class.

Fig. 2:  Outline of the proposed method. Two main streams exist using the Conv
layer after key feature selection via the FC layer: a key frame description
for the static appearance, and a key segment description for the dynamic
pattern.

##  2 Deep Dual Descriptor

Fig. 2 depicts a flow diagram of the proposed method. Given a dynamic scene
video, we first extract features from each frame via the FC layer of the pre-
trained CNN. Once all FC-features are obtained, we find key features that
reflect the feature distribution of the sequence using a small number. We then
select the key frames and key segments according to the key features. To
describe the key frame and key segment, we use the Conv layer as a feature
extractor. Static features of key frames and dynamic features of key segments
are then generated by codebook learning. Finally, we combine the static and
dynamic features to construct our deep dual descriptor (D3).

###  2.1 Key Frame and Key Segment Selection

Because each key segment is selected based on the temporal location of the key
frame, we first must select key frames. Unlike traditional frame selection
approaches, the proposed frame selection process reflects the feature
distribution of a dynamic scene sequence. Inspired by partitioning around
medoids (PAM) [23], we first find key features that well reflect the feature
distribution of whole frames. Then, each frame corresponding to each key
feature is determined as a key frame. Our approach minimizes the sum of
pairwise dissimilarities between features and represents the cluster center by
existing features. It is, therefore, robust to noisy or outlier frames.

Fig. 3:  The main concept of key frame and key segment selection. Given
features from whole frames, the initial key features are randomly determined.
A swap process is then performed iteratively to find optimal key features.
Finally, the key frame and key segment are selected according to the key
feature.

The main concept of the proposed method is illustrated in Fig. 3. Given a
dynamic scene sequence, we first extract spatial features from each frame via
the FC layer. The circles in the figure depict corresponding FC features. Once
all features are obtained, we select the initial random features, as shown in
the left side of the middle row of Fig. 3. The non-key features are then
clustered based on the closest key features. Once the initial clusters are
generated based on the key features, each key feature is swapped with a non-
key feature that belongs to its cluster. During the swap process, we verify
the change of our objective function (1), where $K$ denotes the key feature
set and $C_{i}$ indicates the non-key feature set belonging to the $i$-th
cluster.

$\begin{split}J=\sum_{i}\sum_{j}\left\lVert
f_{i}-f_{j}\right\rVert_{2}^{2}\>\>\>\\\ \forall i:f_{i}\in K\quad\forall
j:f_{j}\in C_{i}\end{split}$ (1)

In the case of a swap that makes the objective function (i.e., the sum of the
dissimilarities between the key features and non-key features) smaller than
before swapping, the swap is accepted. Conversely, the swap that makes an
objective function larger is rejected. If the swap is accepted, non-key
features are re-clustered. This swapping process is repeated multiple times
until objective function (1) does not change in a new iteration. Overall, this
approach obtains the optimized key feature set that reflects the feature
distribution of all frames.

In key feature selection, calculating the pairwise dissimilarity over all
features requires a considerable amount of computation because the number of
FC features in a sequence is usually more than 700, and the dimension of each
feature is 4,096. Therefore, to find key features, we only perform the swap
process in a random subset of FC features. This procedure is repeated multiple
times and clustering with the lowest objective function is retained. Finally,
the frames corresponding to the key features are determined as key frames. In
addition, the frames around each key frame within a short time interval are
chosen as key segments.

###  2.2 Feature Description

Feature description for dynamic scenes consists of three main steps: low-level
feature extraction, feature aggregation for static and dynamic features, and
late fusion.

####  2.2.1 Low-level Feature Description

In feature description, there are two separate streams, each with a different
purpose: a static appearance from a key frame, and a dynamic pattern from a
key segment. The first step for describing a static appearance is feature
description for each key frame using the Conv layer. An output of the Conv
layer is a 7$\times$7$\times$512-dimensional vector. We handle this Conv-
feature as densely extracted 49 (7$\times$7) local features of 512 dimensions.
It is assumed that the number of key frames is $R$. From $R$ key frames, we
consequently obtain 49 $\times R$ static features termed as
$S_{Conv}=\\{c_{1}^{1},c_{1}^{2}\cdots c_{1}^{49}\cdots
c_{R}^{1},c_{R}^{2}\cdots c_{R}^{49}\\}$.

For temporal modeling, rather than using temporal information of all frames,
as in [5, 16], only a dynamic pattern within a short time interval is
considered in our approach. Specifically, we use central moments of Conv
features extracted from each frame in a key segment as dynamic features:

$\begin{split}t_{i}^{j}=\frac{1}{\tau+1}\sum_{w}{\left({}_{w}c_{i}^{j}-\mu_{i}^{j}\right)}^{2}\quad
where\>\>\mu_{i}^{j}=\frac{1}{\tau+1}\sum_{w}{{}_{w}c_{i}^{j}}\quad\\\
i=1,2\cdots R\quad j=1,2\cdots 49\quad
w=-\frac{\tau}{2}\cdots{-1},\>0,\>1\cdots\frac{\tau}{2}\quad\end{split}$ (2)

where $i$ refers to the frame index, $j$ indicates the index of the local
region, and $w$ denotes the relative position from the middle frame of the key
segment, i.e., the key frame. For instance, ${}_{1}c_{6}^{2}$ denotes the
second Conv feature extracted from the seventh frame. The temporal size of the
key segment, which is usually below eight in our approach, is denoted by
$\tau$. As a result, we obtain $49\times R$ dynamic features,
$T_{Conv}=\\{t_{1}^{1},t_{1}^{2}\cdots t_{1}^{49}\cdots
t_{R}^{1},t_{R}^{2}\cdots t_{R}^{49}\\}$.

####  2.2.2 Feature Aggregation and Late Fusion

Once low-level static feature $S_{Conv}$ and dynamic feature $T_{Conv}$ are
calculated from a video sequence, we respectively aggregate these features
into a single feature vector. Three well-known feature aggregation techniques
based on codebook learning are applied in our approach: bag of features (BoF)
[8], vector of locally aggregated descriptors (VLAD) [24], and Fisher vectors
(FV) [25].

Here, we describe the details of feature aggregation in the case of
$S_{Conv}$. Nonetheless, all procedures are likewise applied to $T_{Conv}$.
Assume that a codebook with $k$ visual words,
$C=[v_{1},v_{2}\;\;\ldots\;\;v_{k}]$, has been learned from the training set,
and $S_{Conv}$ is given from a query video. For convenience, because the frame
index within a sequence is meaningless in our feature aggregation, we change
the notation of static features from $S_{Conv}=\\{c_{1}^{1},c_{1}^{2}\cdots
c_{1}^{49}\cdots c_{R}^{1},c_{R}^{2}\cdots c_{R}^{49}\\}$ to
$S_{Conv}=\\{c_{1},c_{2}\cdots c_{49\times R}\\}$.

According to the method of assigning each visual word to the given local
features, feature aggregation approaches are divided into two types: hard
assignment and soft assignment. BoF and VLAD representations are based on the
hard assignment. Once codebook $C$ and feature set $S_{Conv}$ are given, each
feature in a BoF representation is first assigned to the closest visual word.
The histogram of each visual word is then used as a feature vector. VLAD is
similar to BoF; however, instead of using a histogram, the differences between
each vector and the nearest visual word are accumulated as follows:

$\begin{split}\Psi_{j}=\sum_{\forall
i:NN(c_{i})=j}c_{i}-v_{j}\qquad\qquad\quad\\\
where\>NN(c_{i})=\underset{j}{\operatorname{argmin}}\left\lVert
c_{i}-v_{j}\right\rVert_{2}\quad\quad\>\>\\\ i=1,2\>\ldots\>49\times R\qquad
j=1,2\>\ldots\>k\quad\quad\end{split}$ (3)

Only a single visual word is assigned to local features; therefore, the
relevance of the local features to other visual words can be lost in BoF and
VLAD. Following the work in [26], we use the soft-assignment FV, which encodes
both first- and second-order statistics between each local feature and a
Gaussian mixture model (GMM):

$\begin{split}\qquad\Phi_{j}^{(1)}=\frac{1}{49R\sqrt{w_{k}}}\sum_{i=1}^{49R}\alpha_{i}(j)\left({\frac{c_{i}-\mu_{j}}{\sigma_{j}}}\right)\qquad\>\>\\\
\Phi_{j}^{(2)}=\frac{1}{49R\sqrt{2w_{k}}}\sum_{i=1}^{49R}\alpha_{i}(j)\left({\frac{\left(c_{i}-\mu_{j}\right)}{\sigma_{j}^{2}}^{2}-1}\right)\quad\\\
i=1,2\>\ldots\>49\times R\qquad j=1,2\>\ldots\>k\qquad\quad\;\end{split}$ (4)

where $w_{j}$, $\mu_{j}$, and $\sigma_{j}$ denote the mixture weights, means,
and diagonal covariances of the $j$-th Gaussian, respectively. Moreover,
$\alpha_{i}(j)$ is the soft assignment weight of the $i$-th feature $c_{i}$ to
the $j$-th Gaussian. From results of comparative experiments, FV showed the
best performance among feature aggregation approaches. Thus, we used FV
representation in the remainder of our experiments.

We furthermore applied feature aggregation to dynamic features with the same
approach that was used for the static features. Finally, we concatenate the FV
from static features and FV from dynamic features and denote the result as the
deep dual descriptor. In the remainder of this paper, we denote FV from static
features as D3s, FV from dynamic features as D3d, and combined version (static
+ dynamic) as D3.

##  3 Experimental Results

In Section 3.1, we briefly describe the datasets used in our experiments. We
then present details of the pre-trained CNN architecture in Section 3.2.
Various parameters and schemes that influence the recognition rate are
described in Section 3.3. Finally, our comparative evaluation of the
performance of the proposed method with that of state-of-the-art methods is
provided in Section 3.4.

###  3.1 Dataset

The Maryland dataset [5] contains 13 classes of natural scenes with 10 videos
per class. On average, the length of these videos is 617 frames, and the size
of each frame is 308$\times$417 pixels. The items in this dataset were
collected from various public websites, such as YouTube. Thus, the videos in
Maryland have significant camera motion and large variations in illuminations,
viewpoints, and scales. Similar to the Maryland dataset, the Yupenn dataset
[6] includes variations in viewpoints, scales, and illuminations; however, it
does not include camera-introduced motion. This latter dataset consists of 14
dynamic scene classes with 30 videos per class. The average resolution of the
videos is 250$\times$370 pixels, and the average frame length is 145. Example
frames from the Maryland and Yupenn datasets are shown in Fig. 6.

While static scene datasets have been extensively introduced [15, 27, 28, 26,
29], only two public datasets exist for dynamic scenes. To address this lack,
we constructed a dataset that includes 23 dynamic scene classes with 10 videos
per class by combining the Maryland and Yupenn datasets. Because the number of
sequences in each class differs between Maryland (10) and Yupenn (14), we
selected 10 random sequences per each class in both datasets. For those
classes that exist in both datasets, such as “Forest Fire,” “Fountain,” and
“Waterfall”, five random sequences were extracted from each dataset and
combined. We additionally incorporated “Smooth Traffic” in Maryland and
“Highway” in Yupenn because they have very similar appearances and motion. The
whole dataset can be downloaded at our online repository
(https://goo.gl/bGPnbu).

###  3.2 Pre-trained CNN Architecture

Fig. 4:  Architecture of the VGG-VD model. Among the model layers, the FC7
layer is used to select key frames and key segments, while the Conv5 layer is
used to describe the local textures of the key frames and key segments.

Throughout the overall process, we used a pre-trained CNN model, VGG-VD with
16 layers [30], which was trained on the ImageNet Large Scale Visual
Recognition Challenge (ILSVRC) [31]. This network consisted of five Conv
layers containing successive $3\times 3$ convolution filters; a stack of Conv
layers was followed by three FC layers, as shown in Fig. 4. Among them, two
different types of layers were used in our approach. The FC7 layer was used to
select key frames and key segments from a dynamic scene sequence. On the other
hand, the Conv5 layer was used to describe local textures of key frames and
key segments. The FC7 layer was a 4,096-dimensional vector, which indicates a
single global feature with 4,096 dimensions. The output of the Conv5 layer was
7$\times$7$\times$512, which yielded 49 (7$\times$7) local features with 512
dimensions for each frame. For further details, please refer to [30].

###  3.3 Key Factors

In this section, we present a comparative analysis of the various factors—the
frame selection strategy, number of frames, layer for feature description,
codebook size, and combining of spatial and temporal features—that influence
the performance of the proposed method. Analytical experiments were performed
on the challenging Maryland dataset, following the leave-one-out cross-
validation approach described in [5]. For classification, a support vector
machine (SVM) [32], where parameter C was fixed to 100, was used. We
additionally evaluated key factors in the Maryland–Yupenn dataset following
the experimental setup in the Maryland dataset.

####  3.3.1 Frame Selection Strategy

When processing the spatial features from a video in a frame-by-frame manner,
a large difference of performance could be caused with respect to frame
selection strategy. We compared the proposed method with previous approaches
on the same setting. For a fair comparison, we used only static features from
the key frames, i.e., D3s with FV aggregation wherein codebook size was 128.

The baseline frame selection strategies were divided into two types: fixed
frame number and varying frame number. In the comparative experiments for the
fixed frame number setting, we evaluated the random sampling introduced in
[16]. We furthermore applied adaptive key frame extraction based on clustering
[33] (CHC in the table). In this approach, all the frames were grouped by a
$k$-means algorithm, and the closest frame to the cluster centroid was
selected as the key frame.

The comparison results with the fixed frame number approaches are presented in
Table 1. From the table, the tendency of performance improvement is evident
with the increase of the number of frames in most cases. Interestingly, the
CHC approach outperforms the random sampling except when only one frame is
used. This result indicates that one key frame, which was obtained by
$k$-means clustering, may not well represent a dynamic scene sequence. In
contrast, the proposed method shows a relatively promising result with only
one frame. Overall, the results from the Maryland dataset (Table 1) and those
from the Maryland–Yupenn dataset (Table 2) support the superiority of the
proposed method.

Table 1: Classification rate (%) of the methods using a fixed frame number on the Maryland dataset Selection | Frame Number
---|---
1 | 5 | 10 | 15
Random | 73.08 | 80.77 | 85.38 | 86.15
CHC | 70.77 | 84.62 | 86.92 | 84.62
D3s | 75.38 | 84.62 | 89.23 | 90.77
Table 2: Classification rate (%) of the methods using a fixed frame number on the Maryland–Yupenn dataset Selection | Frame Number
---|---
1 | 5 | 10 | 15
Random | 78.70 | 88.26 | 86.09 | 87.39
CHC | 81.30 | 89.13 | 89.57 | 89.57
D3s | 82.17 | 90.87 | 91.74 | 91.74
Table 3:  Comparison results of the methods using a varying frame number on the Maryland dataset Selection | Parameter | Accuracy (%)
---|---|---
Consecutive | 1 | 74.62
N/8 | 80.77
N/4 | 83.08
N/2 | 84.62
N | 88.46
GHD | $\mu-\sigma$ | 86.15
$\mu$ | 87.69
$\mu+\sigma$ | 87.69
Thumbnail | 10 s | 76.15
D3s | 15 frames | 90.77

In addition, we compared the proposed approach with several frame selection
approaches based on a varying frame number. The results from the Maryland
dataset are presented in Table 3. As a baseline, we first evaluated a
partially consecutive sampling of [19] (Consecutive in the table). A second
approach is a histogram-difference-based approach (GHD in the table). The
histogram differences between adjacent gray images were obtained for all
frames and their mean and standard deviations were used as a threshold to
remove outlier frames. One standard deviation subtracted from the
mean($\mu-\sigma$), the mean ($\mu$), and one standard deviation plus the mean
($\mu+\sigma$) were used in our experiments. On average, the number of frames
was 3.76%, 68.79%, and 90.59% of the total number of frames, respectively. We
additionally applied video-thumbnail extraction of FFMPEG, which extracted the
most representative frame per 10 s (Thumbnail in the table). From the table,
it is apparent that the proposed method outperformed the conventional sampling
strategies and key frame selection.

####  3.3.2 Description Layer and Codebook Size

While features via the FC layer well describe the overall structure of the
scene, the features via the Conv layer that we used considered the local
textures of the scene. In this section, we compare the results between the FC
layer and Conv layer in terms of the description layer. In both cases, key
frame selection was performed by the FC layer, and FV was used for feature
aggregation. The performance against codebook size used in FV aggregation was
also given in the experiments. We used the $k$-means++ algorithm [34] for the
codebook generation, where the termination condition occurred after 1,000
iterations or when the positions of new centroids did not change in the new
iteration.

From Table 4, it is clear that the Conv layer outperforms that of the FC layer
with respect to the number of frames and codebook size. It is evident that a
different tendency exists with the increase of codebook size. The performance
gradually decreases when using the FC layer, whereas the performance increases
when using the Conv layer. The best performance in this experiment is 90.77%
when the number of frames is 15 and the codebook size is 128.

Table 4:  Classification rate (%) of D3s with respect to the number of frames, feature description layer, and codebook size on the Maryland dataset Frame | Layer | Codebook Size
---|---|---
32 | 64 | 128 | 256 | 512
1 | FC | 66.15 | 69.23 | 68.46 | 68.46 | 68.46
Conv | 73.85 | 74.62 | 75.38 | 76.92 | 80.00
5 | FC | 81.54 | 82.31 | 75.38 | 72.31 | 66.92
Conv | 82.31 | 82.31 | 84.62 | 86.15 | 89.23
10 | FC | 82.31 | 80.00 | 82.31 | 76.92 | 72.31
Conv | 84.62 | 86.15 | 89.23 | 89.23 | 89.23
15 | FC | 83.85 | 79.23 | 82.31 | 69.23 | 76.15
Conv | 82.31 | 83.85 | 90.77 | 90.00 | 88.46

####  3.3.3 Spatial and Temporal Features

Fig. 5: Classification rate of the individual descriptor using different
feature aggregations and segment sizes on the Maryland dataset.

To describe a dynamic scene, we calculated the static appearance of key
frames, D3s, and dynamic patterns in key segments, D3d. We then combined D3s
and D3d for video descriptor D3. Fig. 5 shows the performance of individual
descriptors using BoF, VLAD, and FV feature aggregation. The baseline
(represented by a dotted horizontal line) is the result of using the static
feature alone. We can see that the combined feature gives better performance
than using the static feature or dynamic feature alone. In this experiment, we
confirmed 92.31% of D3 with FV, which is the state-of-the-art performance in
the Maryland dataset.

###  3.4 Comparison with State-of-the-Art Methods

In this section, we compare the proposed method with state-of-the-art methods
on the public dynamic scene datasets. We additionally evaluate the proposed
method on the newly compiled Maryland–Yupenn dataset. Finally, we show
comparison results on dynamic texture datasets to demonstrate the
generalization ability of the proposed method.

####  3.4.1 Dynamic Scene Dataset

For a fair comparison with previous studies on Maryland and Yupenn, we
followed the leave-one-out evaluation protocol and reported classification
results using SVM. Table 5 shows the comparison results, which can be divided
into hand-crafted features [20, 21, 9, 16, 10] and a CNN feature based on deep
learning [35, 14, 19]. In the experiments, several hand-crafted features, such
as bags of spacetime energies (BoSE), ensemble SVM (E-SVM), and dynamically
pooled complementary features (DPCF), show good performance in the stabilized
Yupenn dataset. However, they show poor performance compared to the CNN-based
methods, including ours, in the non-stabilized Maryland dataset.

Table 5:  Comparison results using different methods in the Maryland and Yupenn datasets Method | Accuracy (%)
---|---
Maryland | Yupenn
SFA [20] | 74.60 | 85.00
CSO [21] | 67.69 | 85.95
BoSE [9] | 77.69 | 96.19
E-SVM [16] | 78.77 | 96.43
DPCF [10] | 80.00 | 98.81
Imagenet [35] | 87.70 | 96.70
C3D [14] | 87.70 | 98.10
st-TCoF [19] | 88.46 | 99.05
D3 | 92.31 | 99.05
Table 6:  Comparison results between TcoF-based approaches and the proposed method in the Maryland and Yupenn datasets  |  Method | Accuracy (%)
---|---|---
| Maryland | Yupenn
Static |  s-TCoF [19] | 88.46 | 97.14
D3s | 90.77 | 97.62
Dynamic |  t-TCoF [19] | 66.15 | 97.86
D3d | 90.77 | 98.33
Static + |  st-TCoF [19] | 88.46 | 99.05
Dynamic |  D3 | 92.31 | 99.05
Fig. 6: Sample frames from each class in Maryland, Yupenn, and Maryland-Yupenn
datasets

In considering separate modeling of dynamic scenes and the transferred CNN
model, TCoF methods, including s-TCoF, t-TCoF, and st-TCoF, are most closely
related to our D3s, D3d, and D3 respectively. We performed direct comparative
experiments of TCoF methods and our approach. As shown in Table 6, TCoF
methods show promising results except for t-TCoF in Maryland dataset. This
result may be because of their temporal encoding based on differences between
two adjacent frames, which motivated this study. In contrast, our D3d based on
key segments shows good performance. We can also see dynamic feature D3d
provides complementary information when combined with static feature D3s.
Overall, D3 shows the best result among the methods in both Maryland and
Yupenn datasets.

Table 7:  Classification rate (%) with respect to different methods and feature aggregation on the Maryland–Yupenn dataset Method | Feature aggregation
---|---
BoF | VLAD | FV
D3s | 82.61 | 81.74 | 90.43
D3d | 80.87 | 84.78 | 91.30
D3 | 86.96 | 87.83 | 92.61

The results from the Maryland–Yupenn dataset are also presented in Table 7. We
followed an experimental setup similar to the one in the Maryland and Yupenn
datasets. As expected, D3 outperformed D3s and D3d with all feature
aggregation approaches. The best results of each descriptor are D3s (90.43%),
D3d (91.30%), and D3 (92.61%), respectively, which can be baselines in further
experiments in the Maryland–Yupenn dataset.

####  3.4.2 Dynamic Texture Dataset

To validate the effectiveness of the proposed method, we also evaluated the
experiments on dynamic textures datasets: Dyntex-Alpha, Dyntex-Beta, and
Dyntex-Gamma [36]. The Dyntex-Alpha dataset is composed of 60 DT sequences
that are equally divided into three categories, and the Dyntex-Beta consists
of 162 sequences with 10 classes. The Dyntex-Gamma, which is the most
challenging set, is composed of 264 sequences grouped into 10 classes. Sample
frames of each dataset are given in Fig. 7. Table 8 shows overall comparison
results in each dynamic texture dataset. We can see that the proposed D3
outperforms the previous works in all dynamic texture datasets.

Table 8: Comparison results using different methods with Dyntex-Alpha, Dyntex-Beta, and Dyntex-Gamma datasets Method | Accuracy(%)
---|---
Dyntex-Alpha | Dyntex-Beta | Dyntex-Gamma
OTF [37] | 83.61 | 73.22 | 72.53
WMF [38] | 84.83 | 75.21 | 73.32
DFS [39] | 85.24 | 76.93 | 74.82
LA [40] | 89.60 | 80.90 | 79.90
s-TCoF [19] | 100.00 | 100.00 | 98.11
t-TCoF [19] | 98.33 | 97.53 | 95.45
st-TCoF [19] | 100.00 | 100.00 | 98.11
D3s | 100.00 | 98.77 | 97.35
D3d | 100.00 | 98.77 | 98.11
D3 | 100.00 | 100.00 | 98.11
Fig. 7: Sample frames from each class in Dyntex-Alpha, Dyntex-Beta, and
Dyntex-Gamma datasets

##  4 Conclusion

In this study, we have proposed a new dynamic scene representation based on
‘key frame’ and ‘key segment.’ Given a sequence, considering feature
distribution obtained by FC layer, we first select key frames and key
segments. With Conv layer, static features are then extracted from key frames
while dynamic features are calculated from key segments. Finally, static and
dynamic features are aggregated by FV separately, followed by late fusion,
which constructs a deep dual descriptor (D3). We validated our approach by
outperforming the conventional frame selection strategies. From the
experiments, we also confirmed that our D3 using Conv layer outperforms that
of FC layer. Results using three dynamic scene datasets and three dynamic
texture datasets further demonstrate the superiority of the proposed method
compared to state-of-the-art methods. In our approach, we utilize only part of
the deep learning architecture by using transferred CNN feature with
conventional feature aggregation. We plan to expand our approach to full CNN
architecture in which the feature aggregation and decision were jointly
performed on the CNN architecture.

## References

  * [1] Liang Du and Haibin Ling,  “Dynamic scene classification using redundant spatial scenelets,”  IEEE Transactions on Cybernetics, vol. PP, pp. 1, 2015.
  * [2] Antoni B Chan and Nuno Vasconcelos,  “Classifying video with kernel dynamic textures,”  in 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2007, pp. 1–6.
  * [3] Bernard Ghanem and Narendra Ahuja,  “Maximum margin distance learning for dynamic texture recognition,”  in European Conference on Computer Vision. Springer, 2010, pp. 223–236.
  * [4] Gui-Song Xia, Sira Ferradans, Gabriel Peyré, and Jean-François Aujol,  “Compact representations of stationary dynamic textures,”  in 2012 19th IEEE International Conference on Image Processing. IEEE, 2012, pp. 2993–2996.
  * [5] Nitesh Shroff, Pavan Turaga, and Rama Chellappa,  “Moving vistas: Exploiting motion for describing scenes,”  in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 1911–1918.
  * [6] Konstantinos G Derpanis, Matthieu Lecce, Kostas Daniilidis, and Richard P Wildes,  “Dynamic scene understanding: The role of orientation features in space and time in scene classification,”  in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 1306–1313.
  * [7] Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto,  “Dynamic textures,”  International Journal of Computer Vision, vol. 51, no. 2, pp. 91–109, 2003.
  * [8] Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray,  “Visual categorization with bags of keypoints,”  in Workshop on statistical learning in computer vision, ECCV. Prague, 2004, vol. 1, pp. 1–2.
  * [9] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes,  “Bags of spacetime energies for dynamic scene recognition,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2681–2688.
  * [10] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes,  “Dynamic scene recognition with complementary spatiotemporal features,”  2016\.
  * [11] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  in Advances in neural information processing systems, 2012, pp. 1097–1105.
  * [12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik,  “Rich feature hierarchies for accurate object detection and semantic segmentation,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
  * [13] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu,  “3d convolutional neural networks for human action recognition,”  IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 1, pp. 221–231, 2013.
  * [14] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri,  “Learning spatiotemporal features with 3d convolutional networks,”  in 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, 2015, pp. 4489–4497.
  * [15] Aude Oliva and Antonio Torralba,  “Modeling the shape of the scene: A holistic representation of the spatial envelope,”  International journal of computer vision, vol. 42, no. 3, pp. 145–175, 2001.
  * [16] Feng Yang, Gui-Song Xia, Gang Liu, Liangpei Zhang, and Xin Huang,  “Dynamic texture recognition by aggregating spatial and temporal features via ensemble svms,”  Neurocomputing, vol. 173, pp. 1310–1321, 2016.
  * [17] Timo Ojala, Matti Pietikäinen, and Topi Mäenpää,  “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,”  Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 24, no. 7, pp. 971–987, 2002.
  * [18] Simona E Grigorescu, Nicolai Petkov, and Peter Kruizinga,  “Comparison of texture features based on gabor filters,”  IEEE Transactions on Image processing, vol. 11, no. 10, pp. 1160–1167, 2002.
  * [19] Xianbiao Qi, Chun-Guang Li, Guoying Zhao, Xiaopeng Hong, and Matti Pietikäinen,  “Dynamic texture and scene classification by transferring deep image features,”  Neurocomputing, vol. 171, pp. 1230–1241, 2016.
  * [20] Christian Theriault, Nicolas Thome, and Matthieu Cord,  “Dynamic scene classification: Learning motion descriptors with slow features analysis,”  in Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013, pp. 2603–2610.
  * [21] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes,  “Spacetime forests with complementary features for dynamic scene recognition.,”  in BMVC, 2013.
  * [22] Renaud Péteri, Sándor Fazekas, and Mark J Huiskes,  “Dyntex: A comprehensive database of dynamic textures,”  Pattern Recognition Letters, vol. 31, no. 12, pp. 1627–1632, 2010\.
  * [23] Leonard Kaufman and Peter J Rousseeuw,  Finding groups in data: an introduction to cluster analysis, vol. 344,  John Wiley & Sons, 2009.
  * [24] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez,  “Aggregating local descriptors into a compact image representation,”  in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 3304–3311.
  * [25] Florent Perronnin, Jorge Sánchez, and Thomas Mensink,  “Improving the fisher kernel for large-scale image classification,”  in European conference on computer vision. Springer, 2010, pp. 143–156.
  * [26] Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi,  “Deep filter banks for texture recognition and segmentation,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3828–3836.
  * [27] Ariadna Quattoni and Antonio Torralba,  “Recognizing indoor scenes,”  in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009, pp. 413–420.
  * [28] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba,  “Sun database: Large-scale scene recognition from abbey to zoo,”  in Computer vision and pattern recognition (CVPR), 2010 IEEE conference on. IEEE, 2010, pp. 3485–3492.
  * [29] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao,  “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,”  arXiv preprint arXiv:1506.03365, 2015.
  * [30] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,  “Imagenet large scale visual recognition challenge,”  International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [32] Corinna Cortes and Vladimir Vapnik,  “Support-vector networks,”  Machine learning, vol. 20, no. 3, pp. 273–297, 1995.
  * [33] Yueting Zhuang, Yong Rui, Thomas S Huang, and Sharad Mehrotra,  “Adaptive key frame extraction using unsupervised clustering,”  in Image Processing, 1998. ICIP 98. Proceedings. 1998 International Conference on. IEEE, 1998, vol. 1, pp. 866–870.
  * [34] David Arthur and Sergei Vassilvitskii,  “k-means++: The advantages of careful seeding,”  in Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. Society for Industrial and Applied Mathematics, 2007, pp. 1027–1035.
  * [35] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell,  “Caffe: Convolutional architecture for fast feature embedding,”  in Proceedings of the 22nd ACM international conference on Multimedia. ACM, 2014, pp. 675–678.
  * [36] Sloven Dubois, Renaud Péteri, and Michel Ménard,  “Characterization and recognition of dynamic textures based on the 2d+ t curvelet transform,”  Signal, Image and Video Processing, vol. 9, no. 4, pp. 819–830, 2015.
  * [37] Yong Xu, Sibin Huang, Hui Ji, and Cornelia Fermüller,  “Scale-space texture description on sift-like textons,”  Computer Vision and Image Understanding, vol. 116, no. 9, pp. 999–1013, 2012.
  * [38] Hui Ji, Xiong Yang, Haibin Ling, and Yong Xu,  “Wavelet domain multifractal analysis for static and dynamic texture classification,”  IEEE Transactions on Image Processing, vol. 22, no. 1, pp. 286–299, 2013.
  * [39] Yong Xu, Yuhui Quan, Zhuming Zhang, Haibin Ling, and Hui Ji,  “Classifying dynamic textures via spatiotemporal fractal analysis,”  Pattern Recognition, 2015.
  * [40] Yuping Sun, Yong Xu, and Yuhui Quan,  “Characterizing dynamic textures with space-time lacunarity analysis,”  in Multimedia and Expo (ICME), 2015 IEEE International Conference on. IEEE, 2015, pp. 1–6.