# A Dynamic Spatial-temporal Attention Network for Early Anticipation of
Traffic Accidents

Muhammad Monjurul Karim, Yu Li, Ruwen Qin1, Member, Zhaozheng Yin, Senior
Member Muhammad Monjurul Karim, Yu Li, and Ruwen Qin are with the Department
of Civil Engineering, Stony Brook University, Stony Brook, NY 11794,
USA.Zhaozheng Yin is with the Department of Biomedical Informatics, Department
of Computer Science, and AI Institute, Stony Brook University, Stony Brook, NY
11794, USA.1 Corresponding author: Ruwen Qin, email:
<EMAIL_ADDRESS>received MM DD, YYYY; revised MM DD, YYYY.

###### Abstract

The rapid advancement of sensor technologies and artificial intelligence are
creating new opportunities for traffic safety enhancement. Dashboard cameras
(dashcams) have been widely deployed on both human driving vehicles and
automated driving vehicles. A computational intelligence model that can
accurately and promptly predict accidents from the dashcam video will enhance
the preparedness for accident prevention. The spatial-temporal interaction of
traffic agents is complex. Visual cues for predicting a future accident are
embedded deeply in dashcam video data. Therefore, the early anticipation of
traffic accidents remains a challenge. Inspired by the attention behavior of
humans in visually perceiving accident risks, this paper proposes a Dynamic
Spatial-Temporal Attention (DSTA) network for the early accident anticipation
from dashcam videos. The DSTA-network learns to select discriminative temporal
segments of a video sequence with a Dynamic Temporal Attention (DTA) module.
It also learns to focus on the informative spatial regions of frames with a
Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is
trained jointly with the attention modules to predict the probability of a
future accident. The evaluation of the DSTA-network on two benchmark datasets
confirms that it has exceeded the state-of-the-art performance. A thorough
ablation study that assesses the DSTA-network at the component level reveals
how the network achieves such performance. Furthermore, this paper proposes a
method to fuse the prediction scores from two complementary models and
verifies its effectiveness in further boosting the performance of early
accident anticipation.

###### Index Terms:

Dynamic temporal attention, dynamic spatial attention, model fusion,
autonomous vehicle, human-inspired AI

##  I Introduction

Autonomous driving has drawn increasing attention and made significant
achievements in recent years [1, 2]. While autonomous driving provides
convenience to people and addresses emerging needs from industry, it raises
concerns about traffic accidents. From 2014 to September 1, 2021, 337
autonomous vehicle collisions were reported in California, USA [3]. Moreover,
according to the 2018 global status report on road safety from World Health
Organization, about 1.35 million people are killed in traffic accidents every
year [4]. Accident anticipation is a desired safety-enhancement capability for
not only autonomous vehicles but also countless human driving vehicles.
Successful anticipation of accidents from the widely deployed dashcams even
just a few seconds ahead would effectively increase the situational awareness
of human drivers, Advanced Driver Assistance Systems (ADAS), and autonomous
vehicles to trigger a higher level of preparedness for accident prevention.

Many causal factors contribute to traffic accidents [5], including but not
limited to human factors, environmental conditions, road and traffic
characteristics, temporal-spatial factors, and vehicle types [6]. Detecting
accident causal factors can help improve the awareness of accident risks and
develop methods to mitigate their negative impacts on safety. Dashcams capture
many of these causal factors in the form of video data. Computational methods
that can translate the easily obtained dashcam video data into the perception
of accident risks are largely desired [7].

Some pioneering studies of computer vision-based accident anticipation use
conventional recurrent neural networks to capture causal factors by
distributing soft attention to agents in the traffic scene [8, 9, 10, 11].
Dashcam videos contain not only relevant information but irrelevant
information. Without explicitly considering the spatial and temporal
importance of traffic agents and the driving context along with their dynamic
changes, it is impossible to effectively learn the video representation for
anticipating accidents.

Humans can turn their attention to risky regions in the driving scene. They
use their peripheral vision to assess the scene and then fixate on regions
that seem of high salient values. Drivers’ choice of attended regions and
their levels of attention to them (e.g., measured by the fixation duration)
vary over time. Their attention level increases when they perceive accident
risks visually. Inspired by humans’ dynamic visual attention in perceiving
traffic risks, this paper developed a computer vision-based deep neural
network for the early anticipation of traffic accidents. The network has
embedded spatial-temporal attention modules that reinforce its ability to
anticipate traffic accidents. The technical contributions of this paper are
threefold:

  * •

A new framework named Dynamic Spatial-Temporal Attention (DSTA) network that
learns to dynamically attend to the salient temporal segments and spatial
regions of the driving scene video for the early accident anticipation. The
network has outperformed the state-of-the-art;

  * •

A new method named score-level late-fusion that can be implemented on any set
of complementary trained networks. The fusion method achieves higher
prediction accuracy than the constituent networks;

  * •

A detailed analysis of the applicability and limitations of representative
traffic accident datasets.

The remainder of this paper summarizes the literature in Section II,
delineates the proposed DSTA-network in Section III, and discusses the
implementation detail and the experimental assessment of the network in
Section IV. At the end, the paper summarizes findings and important future
work.

##  II The Literature

Accident anticipation, in general, falls into the category of problems that
predict the probability of a future event. This problem is studied thoroughly
to anticipate human actions. The computer vision-based approach to human
action anticipation commonly uses appearance features as cues for prediction,
including those at the object, activity, and context levels, respectively [12,
13, 14, 15, 16]. Other cues that may further strengthen the prediction ability
include the temporal relationship of sub-activities or activity-related
entities, and the spatial relationship between humans, other entities, and the
environment [5]. Furthermore, the design of the loss function for training
prediction neural networks is a mechanism for encouraging early predictions
[17]. While these studies have built a strong methodological foundation for
the probabilistic prediction of future events, they analyze video data
captured by static surveillance cameras, not applicable to mobile cameras
mounted on vehicles.

Various efforts are made to anticipate traffic accidents from dashcam videos.
Yao et al. [18] developed an unsupervised approach that utilizes the ego-
vehicle motion information to monitor and predict future locations of traffic
agents. Takimoto et al. [19] incorporated physical location data with video
data to predict the occurrence of traffic accidents. Closely related to [20,
17], Suzuki et al. [10] proposed an adaptive loss function for promoting the
early anticipation of accidents. The loss function assigns penalty
coefficients according to the achieved mean time-to-accident during training.

Recently, attention mechanisms are receiving growing interest. Like humans, a
neural network with attention units can learn to selectively concentrate on
most relevant features. Chan et al. [8] introduced the dynamic soft-attention
to the traffic accident anticipation, which fuses the weighted sum of object-
level features with the frame-level features of each video frame. The weights
represent the attention levels on different objects. Inspired by [8], Zeng et
al. [9] proposed a soft-attention RNN that models the nonlinear interaction
between traffic agents and locates risky regions where the agents may involve
in a future accident. Fatima et al. [11] introduced a feature aggregation
block that calculates the weights for aggregating object-level features to
capture the inter-object interactions. The above-discussed studies mainly
focus on learning attentions to spatially distributed agents that may be
related to accidents. The temporal importance of appearance features are
ignored either at the frame-level, the object-level, or both. From a different
application domain, Cui et al. [21] integrated both a spatial attention module
and a temporal attention module with a GRU [22] to estimate the state-of-
health of batteries. The model in [21] processes the feature vector measured
at any time using a one-dimensional (1D) convolution layer, assigns “spatial”
weights to filter kernels, and learns a set of temporal weights that do not
update over time. This approach is not applicable to the recurring task of
accident anticipation from 2D video frames.

Very recently, Bao et al. [23] used a graph convolutional recurrent neural
network (GCRNN) to capture the spatial-temporal relations among candidate
objects. By measuring the spatial distance between objects in each frame, a
graph is created to capture their spatial relationships. However, spatial
distances between objects in a frame do not capture their true spatial
relationships in the real world. The study further used a folded vector of all
the hidden representations as temporal attention. Although it is beneficial to
model training to a certain extent, temporal attention does not update
dynamically to reflect the latest temporal information received.

##  III Methodology

The proposed network for accident anticipation integrates a Gated Recurrent
Unit (GRU) with a Dynamic Spatial Attention (DSA) module, a Dynamic Temporal
Attention (DTA) module, and Temporal Self-Attention Aggregation (TSAA) module,
as Fig.1 illustrates. First, the network reads a dashcam video. Each frame of
the video flows into an object detector to get multiple objects detected.
Then, a feature extractor extracts both the frame-level features and object-
level features. The weighted aggregation of the object-level features is
concatenated with the frame-level features, becoming the overall feature input
to the GRU. The GRU reads the input feature and the weighted aggregation of
multiple hidden representations in the past to generate the hidden
representation of the current frame. This hidden representation is used to
predict the probability of seeing an accident in future frames. The DTA module
learns the attention weights for aggregating hidden representations in the
past, and the DSA module learns the attention weights for aggregating object-
level features. Additionally, the appended auxiliary network TSAA learns the
weights to aggregate all the hidden representations of each training video to
predict the class of the video only in the training phase. Details of the
proposed network are delineated below.

Figure 1: Overview of the DSTA-framework

###  III-A Dynamic Spatial-Temporal Attention Framework

The Dynamic Spatial-Temporal Attention (DSTA) network uses a GRU, a simple yet
well-performed type of recurrent neural network [22], to recurrently predict
the probability of seeing an accident in future frames. The spatial-temporal
relations of objects and the context information provide important cues for
the accident anticipation. The GRU integrates a DSA module and a DTA module to
learn the impact of the spatial-temporal relations to accident risks.

####  III-A1 Feature Extraction and Aggregation

An object detector detects spatially distributed objects from each video frame
and keeps the top $N$ objects of the highest detection score for
consideration. The VGG-16 [24] feature extractor extracts both frame-level and
object-level features from each frame through multiple convolution and pooling
operations. Then, two fully connected layers flatten the extracted feature
maps to become a $D$ dimensional feature vector. This study adds an additional
fully connected layer to further reduce the dimension of the feature vector to
$d$ ($<D$). Therefore, the extracted object-level features are
$\boldsymbol{o}_{t}\in\mathbb{R}^{d\times N}$, and the frame-level features
are $\boldsymbol{f}_{t}\in\mathbb{R}^{d}$. After passing through the DSA
module to be introduced in the next section, $\boldsymbol{o}_{t}$ is turned
into a weighted aggregation, $\boldsymbol{o}^{\prime}_{t}\in\mathbb{R}^{d}$.
Then, $\boldsymbol{o}^{\prime}_{t}$ is concatenated with the frame-level
feature $\boldsymbol{f}_{t}$,

$\boldsymbol{X}_{t}=[\boldsymbol{o}^{\prime}_{t};\boldsymbol{f}_{t}],$ (1)

to form the overall feature vector of each frame,
$\boldsymbol{X}_{t}\in\mathbb{R}^{2d}$.

####  III-A2 Dynamic Spatial Attention (DSA)

Attentions to the spatially distributed objects in a frame are unequal. This
fact is modeled by the spatial attention weights,
$\boldsymbol{\alpha}_{t}\in\mathbb{R}^{N}$, calculated using the weighted
aggregation of hidden representations from the last step,
$\boldsymbol{h}^{\prime}_{t-1}$, to be introduced in the next section and the
object-level features extracted from the current frame, $\boldsymbol{o}_{t}$:

$\boldsymbol{\alpha}_{t}=\gamma(\boldsymbol{W}_{sa}^{\text{T}}\tanh(\boldsymbol{W}_{g}\boldsymbol{h}^{\prime}_{t-1}+\boldsymbol{W}_{\theta}\boldsymbol{o}_{t}+\boldsymbol{B}_{\theta})),$
(2)

where $\boldsymbol{W}_{sa}\in\mathbb{R}^{d}$,
$\boldsymbol{W}_{g}\in\mathbb{R}^{d\times d}$,
$\boldsymbol{W}_{\theta}\in\mathbb{R}^{d\times d}$, and
$\boldsymbol{B}_{\theta}\in\mathbb{R}^{d}$ are parameters of the DSA module,
which are learned during training. $\tanh$ is an activation function that
regulates the values flowing through the network, and $\gamma$ in this section
is the softmax operator that normalizes the attention scores as spatial
attention weights. Given the weights, the object-level features are turned
into a weighted aggregation, $\boldsymbol{o}^{\prime}_{t}\in\mathbb{R}^{d}$:

$\boldsymbol{o}^{\prime}_{t}=\boldsymbol{o}_{t}\boldsymbol{\alpha}_{t}.$ (3)

####  III-A3 Dynamic Temporal Attention (DTA)

Frames in a video are not equally important for the accident anticipation.
Some frames may contain more discriminative information for accident
prediction, whereas others mainly provide contextual information. Motivated by
this fact, the study designs the DTA module to provide the temporal attention
weights for aggregating the hidden representations of the most recent $M$
frames. Denote $\boldsymbol{H}_{t-1}\in\mathbb{R}^{d\times M}$ as the hidden
representations of the past $M$ frames indexed by $t-M$ to $t-1$:

$\boldsymbol{H}_{t-1}=[\boldsymbol{h}_{t-1},\dots,\boldsymbol{h}_{t-M}].$ (4)

The sliding window size $M$ is chosen experimentally. A very long temporal
window may distract the network from the accident anticipation task and a very
short window would not provide sufficient temporal context for the task. The
temporal attention weights, $\boldsymbol{\beta}_{t-1}\in\mathbb{R}^{d\times
M}$, are computed as

$\boldsymbol{\beta}_{t-1}=\gamma(\boldsymbol{W}_{ta}\tanh(\boldsymbol{H}_{t-1})),$
(5)

where $\boldsymbol{W}_{ta}\in\mathbb{R}^{d\times d}$ are learnable parameters.
Then, $\boldsymbol{\beta}_{t-1}$ is used to turn $\boldsymbol{H}_{t-1}$ into a
weighted aggregation, $\boldsymbol{h}^{\prime}_{t-1}\in\mathbb{R}^{d}$:

$\boldsymbol{h}^{\prime}_{t-1}=\langle\boldsymbol{\beta}_{t-1},\boldsymbol{H}_{t-1}\rangle_{r},$
(6)

where $\langle\,,\,\rangle_{r}$ represents the row-wise inner product of two
equally-sized matrix.

####  III-A4 Spatial-Temporal Relational Learning with GRU

At any frame $t$, the GRU takes the feature vector $\boldsymbol{X}_{t}$ and
the hidden representation $\boldsymbol{h}^{\prime}_{t-1}$ as the inputs to
obtain the hidden representation of the current frame, $\boldsymbol{h}_{t}$.
GRU has two gates, a reset gate and an update gate, which retain the most
relevant information, $\boldsymbol{g}_{t}^{(r)}$ and
$\boldsymbol{g}_{t}^{(u)}$, respectively from the video sequence by filtering
out irrelevant information. The data flow through the GRU are expressed
mathematically in equations (7-10):

${\color[rgb]{0,0,1}\boldsymbol{g}_{t}^{(r)}=\sigma(\boldsymbol{W}_{g}^{(r)}\boldsymbol{X}_{t}+\boldsymbol{B}_{g}^{(r)}\boldsymbol{h}^{\prime}_{t-1}),}$
(7)
${\color[rgb]{0,0,1}\boldsymbol{r}_{t}=\tanh(\boldsymbol{W}_{r}\boldsymbol{X}_{t}+\boldsymbol{B}_{r}(\boldsymbol{g}_{t}^{(r)}\circ\boldsymbol{h}^{\prime}_{t-1})),}$
(8)
${\color[rgb]{0,0,1}\boldsymbol{g}_{t}^{(u)}=\sigma(\boldsymbol{W}_{g}^{(u)}\boldsymbol{X}_{t}+\boldsymbol{B}_{g}^{(u)}\boldsymbol{h}^{\prime}_{t-1}),}$
(9)
${\color[rgb]{0,0,1}\boldsymbol{h}_{t}=(1-\boldsymbol{g}_{t}^{(u)})\circ\boldsymbol{r}_{t}+\boldsymbol{g}_{t}^{(u)}\circ\boldsymbol{h}^{\prime}_{t-1},}$
(10)

wherein $\sigma$ represents the sigmoid activation function, $\circ$ is the
element-wise product operator, $\boldsymbol{W}$’s and $\boldsymbol{B}$’s
($\in\mathbb{R}^{d\times d}$) are learnable parameters.

After passing two fully connected layers, $\phi$, the hidden representation
$\boldsymbol{h}_{t}$ is turned into the scores of positive and negative
classes. The scores are further normalized by the softmax operator to become
the accident anticipation probability, $a_{t}$:

${\color[rgb]{0,0,1}a_{t}=\gamma[\phi(\phi(\boldsymbol{h}_{t};\boldsymbol{W}_{0},\boldsymbol{B}_{0});\boldsymbol{W}_{1},\boldsymbol{B}_{1})]},$
(11)

where $\boldsymbol{W}$’s and $\boldsymbol{B}$’s ($\in\mathbb{R}^{d\times d}$)
are learnable parameters of the fully connected layers.

####  III-A5 Temporal Self-Attention Aggregation (TSAA)

To help train the hidden layers of GRU better, the auxiliary module TSAA is
included in the training stage only. The TSAA module provides learnable
weights $\boldsymbol{W}_{saa}\in\mathbb{R}^{T}$ to perform a temporal self-
attention aggregation [25] of all the $T$ hidden representations of each
training video to predict the video class. As illustrated in Fig. 1, all the
$T$ hidden representations of the training video indexed by $v$ are stored as
a matrix $\boldsymbol{H}_{v}$:

${\color[rgb]{0,0,1}\boldsymbol{H}_{v}=[\boldsymbol{h}_{1},\dots,\boldsymbol{h}_{T}]_{v},}$
(12)

Then, the self-attention operation is applied to obtain the weighted
aggregation of all hidden representations,
$\boldsymbol{Z}_{v}\in\mathbb{R}^{d}$:

${\color[rgb]{0,0,1}\boldsymbol{Z}_{v}=\boldsymbol{H}_{v}\gamma(\boldsymbol{H}^{\text{T}}_{v}\boldsymbol{H}_{v})\boldsymbol{W}_{saa},}$
(13)

where $\boldsymbol{W}_{saa}\in\mathbb{R}^{\text{T}}$ are learnable parameters
of TSAA. This aggregated video-level representation flows into two additional
fully connected layers and one softmax operator to predict the video-level
score, $a_{v}$:

${\color[rgb]{0,0,1}a_{v}=\gamma[\phi(\phi(\boldsymbol{Z}_{v};\boldsymbol{W}_{v_{0}},\boldsymbol{B}_{v_{0}});\boldsymbol{W}_{v_{1}},\boldsymbol{B}_{v_{1}})],}$
(14)

where $\boldsymbol{W}_{v}$’s and $\boldsymbol{B}_{v}$’s
($\in\mathbb{R}^{d\times d}$) are learnable parameters of the fully connected
layers.

###  III-B Training Procedure

The goal of the training process is to fit GRU, DSA, DTA, and TSAA to the
training data by determining their learnable parameters described in Section
III-A. A video dataset of size $V$, indexed by $v$, is used to train the DSTA-
network. Each video contains $T$ frames in total, captured at the rate of $f$
per second. The training dataset has two classes: positive and negative. If a
video contains an accident starting from frame $\tau$ ($<T$), it belongs to
the positive class and the video-level label is $l_{v}=1$. Otherwise, it
belongs to the negative class and the label $l_{v}=0$ meaning that the video
contains no accident. Prediction results of the DSTA-network are compared
against the ground truth of training data to determine the loss that guides
the learning process.

$a_{t,v}$, calculated in Eq. (11), denotes the probability that video $v$
contains an accident, anticipated by the DSTA-network at frame $t$ of the
video. The frame-level loss function on the training dataset is calculated as:

$\displaystyle\mathcal{L_{F}}=$
$\displaystyle\sum_{v=1}^{V}\left[-l_{v}\sum_{t=1}^{T}e^{-\max\left(\frac{\tau-t}{f},0\right)}\log(a_{t,v})\right.$
(15) $\displaystyle\left.-(1-l_{v})\sum_{t=1}^{T}\log(1-a_{t,v})\right].$

The first term within the square bracket in Eq. (15) is the loss of a positive
video, and the second term is the loss of a negative video. Negative videos
use a regular binary cross entropy loss function. But the frame-level loss
coefficient for positive videos increases exponentially as approaching the
accident (e.g., $t<\tau$). After that, it reaches the same loss coefficient
for negative videos. The use of the exponentially increasing loss coefficient
for positive videos encourages the early anticipation of accidents.

The auxiliary TSAA module is trained to predict the probability that a video
contains an accident using the video-level loss function:

$\mathcal{L_{V}}=\sum_{v=1}^{V}\left[-l_{v}\log(a_{v})-(1-l_{v})\log(1-a_{v})\right],$
(16)

where $a_{v}$, calculated in Eq. (14), is the predicted probability that video
$v$ is positive. The video-level loss function is a regular binary cross
entropy loss function. The first term within the square bracket in Eq. (14) is
the loss of a positive video and the second term is the loss of a negative
video.

The objective to achieve in training the DSTA-network is to minimize the
following loss function:

$\mathcal{L}=\mathcal{L_{F}}+w_{a}\mathcal{L_{V}}.$ (17)

Here, $w_{a}$ is a hyper-parameter that helps adjust the relative importance
of the auxiliary loss $\mathcal{L_{V}}$ relative to the primary loss
$\mathcal{L_{F}}$ to achieve an appropriate balance between them. This study
chooses 15 as the value for $w_{a}$ experimentally. Finally, the training
progresses by backpropagating the loss $\mathcal{L}$ to update the learnable
parameters of DSA, DTA, GRU, and TSAA.

###  III-C Score-Level Late Fusion

Any model for the early anticipation of traffic accidents, including the
proposed DSTA-network, faces an unavoidable trade-off between the earliness of
prediction and the correctness. Different models have their respective trade-
off points. A deep ensemble of two or more complimentary models may further
improve performance. Motivated by this hypothesis, the study proposes a method
that fuses the prediction scores of two models that can complement each other
to derive a more reliable prediction. This study found that models of the same
architecture, but different parameters may complement each other, and so those
of different architectures.

Algorithm 1 below summarizes the proposed score-level late fusion method.
Consider a video sequence flowing into two models. On each frame $t$, the two
models give their own independent frame-level prediction scores, $a_{t}^{(1)}$
and $a_{t}^{(2)}$, respectively. If both models give scores greater than or
equal to their respective classification thresholds, $\bar{a}^{(1)}$ and
$\bar{a}^{(2)}$, the driving scene is anticipated to involve in a future
accident confidently. Therefore, the fusion method takes the maximum score
from the two models as the new anticipation probability. Similarly, if both
the models return scores lower than their respective threshold values, the
driving scene is anticipated to be safe confidently. Accordingly, the fusion
method takes the minimum score from the two model as the new anticipation
probability. If one model returns a score greater than or equal to its
threshold while the other returns a score lower than its threshold, The two
models give conflict classification. In such a case, the fusion method takes
the mean of their scores as the new anticipation probability.

Algorithm 1 The Score-level Late Fusion Method

Notation:

$T$: number of frames in a video sequence, indexed by $t$

$a_{t}^{(1)}$ and $a_{t}^{(2)}$: prediction scores by models 1 and 2,
respectively, at frame $t$

$\bar{a}^{(1)}$ and $\bar{a}^{(2)}$: classification thresholds of models 1 and
2, respectively

Input: Frames of a video sequence

Output: Accident prediction scores, $\\{a_{t}|t=1,\dots,T\\}$

for $t=1\>to\>T$ do

if $a_{t}^{(1)}\geq\bar{a}^{(1)}$ and $a_{t}^{(2)}\geq\bar{a}^{(2)}$ then

$a_{t}=\max(a_{t}^{(1)},a_{t}^{(2)})$

else if $a_{t}^{(1)}<\bar{a}^{(1)}$ and $a_{t}^{(2)}<\bar{a}^{(2)}$ then

$a_{t}=\min(a_{t}^{(1)},a_{t}^{(2)})$

else

$a_{t}=\text{mean}(a_{t}^{(1)},a_{t}^{(2)})$

end if

end for

The threshold values of the two constituent models in Algorithm 1 are design
parameters that impact the effectiveness of the fusion method. Different
models have different prediction behavior. For example, one model may focus
more on the earliness of prediction, while the other may focus more on the
correctness. Therefore, using separate threshold values for the constituent
models allow for taking advantage of their respective strengths. The values of
$\bar{a}^{(1)}$ and $\bar{a}^{(2)}$ are determined in a numerical optimization
approach. That is, given a pair of threshold values, the corresponding
performance of the fusion method on the testing dataset is obtained. The
decision is to select a pair of threshold values which leads to the best
performance.

##  IV Implementation and Experimental results

The study illustrates the implementation and evaluation of the proposed DSTA-
network for the early accident anticipation on two publicly available
datasets.

###  IV-A Datasets

####  IV-A1 Dashcam Accident Dataset (DAD) [8]

It contains a diverse set of videos captured across different cities in
Taiwan. The frame rate of the video is 20 frames per second (fps). Each video
lasts 5 seconds and thus consists of 100 ($T$) frames. This dataset has 1,130
negative videos without any accident and 620 positive videos that each
contains an accident in the last 0.5 seconds. The training dataset includes
455 positive and 829 negative videos, whereas the testing dataset comprises
165 positive and 301 negative videos.

####  IV-A2 Car Crash Dataset (CCD) [23]

It is a dashcam dataset with diverse environmental attributes. It contains
4,500 videos. 80% are training data and 20% are testing data. For both
training and testing data, the ratio of positive videos to negative videos is
1:2. Each video lasts 5 seconds with 50 frames in total (i.e., fps is 10). For
a positive video, the accident starting time is randomly placed in the last 2
seconds of the video.

###  IV-B The Detail of Implementation

The study implemented the proposed approach using PyTorch [26]. Training and
testing were performed using an Nvidia Tesla V100 GPU with 32GB of memory. In
use of the DAD dataset, this study adopted the features of 19 ($N$) candidate
objects and the frame-level appearance features provided by [8]. The candidate
object classes are human, bicycle, motorbike, car, and bus which were detected
using Faster R-CNN [27]. For the CCD dataset, the study also directly uses the
features provided by [23] for a fair result comparison, where candidate
objects were detected using Cascade R-CNN [28]. The dimension of VGG-16
features in both datasets is 4,096 ($D$). These features were passed through
fully-connected embedding layers to reduce the dimension to 512 ($d$). The
dimension of hidden representations returned by GRU is 512 too. Features and
hidden representations of this dimension can represent input video frames well
and they are learned within a reasonable amount of training time. Parameters
of the DSTA-network were initialized randomly, by taking values from a normal
distribution with 0 mean and 0.01 standard deviation. The temporal sliding
window for the DTA module is 0.5 seconds ($M=0.5f$), which is verified
experimentally to be suitable. Similar to Chan et al. [8], a learning rate of
0.0001 and a batch size 10 were used to train the network. ReduceLROnPlateau
was used as the learning rate scheduler. Adam optimizer was used to optimize
the network for 60 epochs that are sufficiently long for this study.

###  IV-C Evaluation Metrics

The evaluation of a model for the early accident anticipation needs to
consider two aspects: the earliness and the correctness of prediction.

####  IV-C1 Correctness

The ability of a model to correctly anticipate accidents can be measured by
its performance in classifying a set of testing videos. This study adopts the
following three classic metrics [29].

  * •

Recall (R): the ratio of correctly predicted positives over the total number
of positive videos

  * •

Precision (P): the ratio of correctly predicted positive videos over the total
number of positive predictions

  * •

Average Precision (AP): The recall value and the precision value are changing
when the classification threshold changes. The average precision is the area
below the precision-recall curve:

$\text{AP}=\int\text{P}_{\text{R}}d\text{R},$ (18)

where $\text{P}_{\text{R}}$ is the precision corresponding to the recall R. AP
is independent of the choice of classification threshold.  A high recall value
certainly is critical for accident anticipation due to the severe consequence
of false negatives. But a very high recall could be unrealistic, especially
when it is at the cost of very low precision. In accident anticipation studies
[8, 23], 80% represents a reasonably good recall value for evaluating the
corresponding precision. This study adopts the precision corresponding to 80%
recall, denoted by P${}_{80\text{R}}$, as another metric for the performance
evaluation.

####  IV-C2 Earliness

The sooner a model can anticipate an accident, the more capable the accident
prevention is. This study measures the earliness of accident anticipation
using the following metrics[8, 23, 9, 10]:

  * •

Time-to-Accident (TTA) : The first time when the frame-level anticipation
probability $a_{t}$ goes across the classification threshold $\bar{a}$ is the
time to classify a video as a positive. TTA is the period between this time
and the starting time of accident, $\tau$:

$\text{TTA}=\max\\{\tau-t|a_{t}\geq\bar{a},1\leq t\leq\tau\\}$ (19)

  * •

mean Time-to-Accident (mTTA): TTA is changing if the classification threshold
$\bar{a}$ changes. The study thus calculates the mean value of TTA:

$\text{mTTA}=E[\text{TTA}].$ (20)

mTTA is independent of the classification threshold. AP and mTTA are commonly
used as a pair of metrics for the model assessment. TTA${}_{80\text{R}}$ is
the TTA at 80% recall. TTA${}_{80\text{R}}$ reflects the TTA at a relatively
high requirement on recall.

###  IV-D Evaluation of the DSTA-Network

####  IV-D1 Experimental Evaluation

A learning behavior is observed from training the DSTA-network, which attempts
to seek models that have larger AP and longer mTTA over multiple epochs. A
trade-off between the earliness and the correctness of accident anticipation
presents because the exponentially increasing loss coefficient for positive
videos in Eq. (15) encourages predicting accidents earlier. The network learns
a set of models that are positioned at different places on the mTTA-AP
diagram. Models on the Pareto-optimal frontier [30] are efficient models in
that neither their mTTA nor AP can be further improved without sacrificing the
other. Fig. 2 is the mTTA-AP diagram that displays a group of models the
proposed DSTA-network learnt on the DAD dataset. It keeps models with at least
1 second of mTTA and at least 50% of AP because a very short mTTA or a very
low AP is not applicable in real-world applications. As seen from the figure,
the efficient model with the longest mTTA (3.75 seconds) achieves 53.7% AP.
Increasing the AP of an efficient model requires to shorten the mTTA. The
rightmost point in this mTTA-AP diagram is the efficient model with the best
AP, which achieves 1.5 seconds of mTTA at 72.3% AP. A similar relationship
between mTTA and AP is also observed when training the DSTA-network on the CCD
dataset.

Figure 2: Relationship between mTTA and AP on the DAD dataset

Selection of an efficient model from the Pareto-optimal frontier for the
implementation should consider the specific need of users. Requirements on
accident anticipation systems can vary, depending on the weather conditions,
time of the day, and areas where vehicles are running [5]. Some may demand
super early prediction where false alarms are handled well. Others may require
a high accuracy where earliness is not essential.

####  IV-D2 Comparison to the State-of-the-Art Models

The DSTA-network is compared to the state-of-the-art models targeting longer
mTTA [8, 9, 10, 23]. TABLE I summarizes the results of the comparative study.
The performance of [8, 9, 10, 23] are cited from [9] and [23]. On the DAD
dataset, DSA published in 2016 [8] achieved 48.1% AP and 1.34 seconds mTTA.
Within four years, other studies have progressively increased the AP to 53.7%
and extended the mTTA to 3.66 seconds [9, 10, 23]. The proposed DSTA-network
further leverages the AP by another 2.4% and mTTA by 0.13 seconds, achieving
56.1% AP and 3.66 secconds mTTA. This confirms that DSTA can improve both AP
and mTTA on such a challenging dataset. On the CCD dataset, the state-of-the-
art performance seems has saturated already. But DSTA still outperforms it
with a small margin, achieving 99.6% AP with 4.87 seconds mTTA. On both DAD
and CCD datasets, the proposed DSTA-network has outperformed the existing
state-of-the-art performance.

TABLE I: Comparison of models seeking longer mTTA on DAD and CCD Dataset | Year |  Model |  AP(%) | mTTA(s)
---|---|---|---|---
DAD | 2016 |  DSA [8] |  48.1 | 1.34
| 2017 |  L-RAI [9] |  51.4 | 3.01
| 2018 |  adaLEA [10] |  52.3 | 3.43
| 2020 |  GCRNN [23] |  53.7 | 3.53
| 2021 |  DSTA (Ours) |  56.1 | 3.66
CCD | 2016 |  DSA [8] |  99.6 | 4.52
| 2020 |  GCRNN [23] |  99.5 | 4.74
| 2021 |  DSTA (Ours) |  99.6 | 4.87

TABLE II further compares the best AP model that the DSTA-network attains to
that of GCRNN [23]. Compared to GCRNN, the best AP model of DSTA extends the
mTTA by 13% and TTA${}_{80\text{R}}$ by 16%. Meanwhile, it increases AP
slightly, to 72.34 %. Not to mention that extending the TTA even for a
fraction of second can create better opportunities for accident prevention.
Results in TABLES I and II verify the competitiveness of the DSTA-network in
meeting various requirements for prediction correctness and earliness. The
above-cited performances are associated with individual models. The proposed
fusion method, to be discussed in Section IV-E, will achieve further
improvement.

TABLE II: Comparison of Best AP models on DAD Model |  AP(%) |  mTTA(s) |  TTA${}_{80\text{R}}$(s)
---|---|---|---
GCRNN [23] |  72.22 |  1.33 |  1.56
DSTA (Ours) |  72.34 |  1.50 |  1.81

####  IV-D3 Ablation Study

An ablation study comprising 10 experiments is performed to assess the
architecture of the proposed DSTA-network. A decrease in the model’s
performance due to the removal or replacement of a key component measures the
component’s contribution. TABLE III summarizes the best AP and corresponding
mTTA attained in each of the experiments.

TABLE III: Ablation study on the DAD dataset

Experiment | RNN | DSA | DTA | TSAA | AP (%) | mTTA (s)
---|---|---|---|---|---|---
1 | GRU | ✓ | ✓ | ✓ | 72.34 | 1.50
2 | GRU | ✓ | ✓($M$=1) | ✓ | 70.74 | 1.36
3 | GCRNN | ✓ | ✓ | ✓ | 70.47 | 1.22
4 | LSTM | ✓ | ✓ | ✓ | 69.08 | 1.48
5 | GRU | ✓ | ✓ |  | 69.89 | 1.70
6 | GRU | ✓ |  | ✓ | 69.72 | 1.43
7 | GRU |  | ✓ | ✓ | 68.15 | 1.33
8 | GRU |  |  | ✓ | 66.04 | 1.39
9 | GRU |  | ✓ |  | 65.99 | 1.42
10 | GRU | ✓ |  |  | 65.03 | 1.55

Experiment 1 is the DSTA-network with all the components in place. The
attained AP is higher than that of any other experiment. It shall be mentioned
that the DSTA-network can provide another model (see Fig. 2), whose AP is
70.5% and mTTA is 1.74 seconds, longer than that of any other experiment in
TABLE III.

In experiment 2, the sliding window of the temporal attention model, $M$, is
reduced to one frame (i.e., the last frame). This change reduces the AP for
1.6%, which signifies the effectiveness of using several recent frames’ hidden
representations to learn the temporal relation.

In experiment 3, a GCRNN replaces the GRU, resulting a drop of the AP for
1.87% and a decrease of mTTA for 0.28 seconds. This is mostly because the edge
weights of the GCRNN are less effective than the dynamic spatial attention
weights that the DSTA-network learns. Similarly, the replacement of the GRU by
an LSTM in experiment 4 lowers the AP for 3.26% and reduces the mTTA slightly,
for 0.02 seconds, despite of increasing the number of parameters.
Additionally, the training time of experiment 3 is about five times longer
than the training times of experiments 1 and 4.

Experiment 5 drops the TSAA module, which decreases the AP for 2.45% but
increases the mTTA for 0.20 seconds. Experiment 6 drops the DTA module,
resulting a decrease in the AP for 2.62% and the mTTA for 0.07 seconds.
Experiment 10 drops both modules, which decreases the AP for 7.31% and but
slightly increases the mTTA for 0.05 seconds. These experiments verify the
importance of the temporal attention modules in improving AP.

Compared to experiment 1, experiment 7 drops the DSA module. This change
reduces the AP for 4.19% and the mTTA for 0.17 seconds. Similar observations
are seen from the comparison between experiments 9 and 5 and the comparison
between experiments 8 and 6. These experiments verify that the DSA module in
the DSTA-network is effective in increasing both the AP and the mTTA.

The ablation study verifies the merits of the DSTA-network design. Each of the
attention modules positively contributes to the improvement of AP. Their net
effect on mTTA is also positive. This study also confirms that, given all
these attention modules, GRU is a better choice than both LSTM and GCRNN.

####  IV-D4 Qualitative Results

The DSTA-network can focus on the most semantically significant spatial and
temporal regions in a video sequence. Fig. 3 illustrates the dynamics of the
spatial and temporal attentions using a few sample frames from a positive
video wherein two vehicles (cars) are approaching to each other in an
intersection. The color bar on the right indicates the magnitude of the
attention weights.

The first row is the original sample frames. In the second row, the top two
attended objects in each frame are indicated by colored rectangular boxes. The
object receiving the highest spatial attention weight and the magnitude of
this weight may vary from one frame to another. In this example, the top two
attended objects in frames #50, #54, and #56 are traffic agents that will soon
involve in an accident. In frames #48 and #52, only the object with the top
attention weight is a related traffic agent. The observation indicates that
the network will get more contextual information as time goes by and it will
become more accurate in assigning the spatial attention.

According to Eq.(5), the temporal attention assigned to the hidden
representation of frame $t-j$, $\boldsymbol{\beta}_{t-j}$, for $j=1,\dots,M$,
is a vector. This study calculates the mean value of this vector and uses it
to visualize the temporal attention on this frame assigned at $t$. The third
row of Fig. 3 shows the frames overlaid with their averaged temporal attention
weights assigned at frame #58. Frame #52 receives the highest attention
whereas frame #50 receives the lowest attention. The fourth row further shows
the frames overlaid with their averaged temporal attention weights obtained at
frame #60. The change of attention weights from row three to row four
illustrates the dynamic nature of the temporal attention.

Figure 3: Illustrating the dynamics of the spatial and temporal attentions of
the DSTA-network

Fig. 4 illustrates three examples of accident anticipation by the proposed
DSTA-network. a) and b) are successful examples on analyzing a positive video
and a negative video, respectively. c) is a false-negative example. For the
illustration purpose, a threshold value of 0.5 is used to trigger the
prediction of a future accident. In each example, sample frames of the video
are shown on the top and the time series of the accident anticipation
probability (i.e., the red colored curve) is displayed at the bottom. In a),
the probability of anticipating an accident has reached the threshold at frame
#54, yielding 1.75 seconds TTA. From the sample frames, it is clear that the
network successfully identifies the risk of accident when a few motorcycles
are coming to the direct lane of the car involved in the accident later on. In
b), the network does not predict an accident because the probability is always
below 50%. In c), the accident happens between a yellow car and a motorcycle,
which are relatively far from the ego vehicle. The prediction score never
exceeds the selected threshold value 0.5. The long-distance of the accident-
involved agents from the ego vehicle might cause difficulty in extracting
salient spatial-temporal relational dynamics of the two vehicles from their
features. Therefore, the network missed predicting the accident ahead of time.

Figure 4: Examples of accident anticipation on the DAD dataset: a) a true-
positive sample, b) a true-negative sample, and c) a false-negative sample.
For better visualization, white bounding boxes with a shade are added to the
top two attended objects. The red curve indicates the prediction probability
at each frame.

###  IV-E Performance of the Fusion Method

The study evaluates the effectiveness of the proposed score-level late-fusion
method on the DAD dataset. The top portion of TABLE IV lists four candidate
models for fusion, which are efficient and may complement one and another. The
first two candidate models are learnt by the DSTA-network, which only differ
in model parameters. DSTA(I) is the model yielding the best AP (see Fig. 2)
and DSTA(II) is the one achieving the best P${}_{80\text{R}}$. The third
candidate model is the best AP model of GCRNN [23], and its performance is
attained by reproducing the model using the publicly available
code111https://github.com/Cogito2012/UString. The last candidate model is a
modification of GCRNN, which is trained using the adaptive loss function of
AdaLEA [10]. Here, AdaLEA provides another degree of flexibility to explore a
higher AP. Compared to GCRNN, GCRNN+AdaLEA attains higher AP and
P${}_{80\text{R}}$, but shorter mTTA and TTA${}_{80\text{R}}$. The last two
candidate models have their respective network architecture, different than
the DSTA-network.

TABLE IV: Assessment of the fusion method | Model-1 | Model-2 | $\bar{a}^{(1)}$ | $\bar{a}^{(2)}$ | AP (%) | P${}_{80\text{R}}$ (%) | mTTA (s) | TTA${}_{80\text{R}}$ (s)
---|---|---|---|---|---|---|---|---
Individual | DSTA (I) | - | - | - | 72.3 | 46.6 | 1.50 | 1.81
DSTA (II) | - | - | - | 68.6 | 53.1 | 1.50 | 1.69
GCRNN | - | - | - | 68.3 | 50.8 | 1.31 | 1.58
GCRNN+AdaLEA | - | - | - | 69.4 | 50.6 | 1.02 | 1.18
Fusion | DSTA (I) | GCRNN+AdaLEA | 0.72 | 0.55 | 74.1 | 53.1 | 1.08 | 1.18
DSTA (I) | GCRNN | 0.40 | 0.20 | 73.8 | 48.9 | 1.47 | 1.60
DSTA (I) | DSTA (II) | 0.42 | 0.50 | 72.6 | 56.4 | 1.50 | 1.70

The bottom portion of Table IV summarizes three selected fusion results.
Fusing the predicted scores of DSTA(I) and GCRNN+AdaLEA achieves a remarkably
high AP, 74.1%, with the corresponding mTTA 1.08 seconds. This result is
achieved by selecting the prediction threshold 0.72 for DSTA(I), and 0.55 for
GCRNN+AdaLEA. Compared to the first fusion result, the fusion of DSTA(I) with
GCRNN slightly lowers the AP for 0.3% but extends the mTTA for 0.39 seconds.
Fusing DSTA(I) and DSTA(II) effectively leverages the $P_{80\text{R}}$ to
56.4%, equivalent to a margin of 9.8% compared to DSTA(I) and 3.3% compared to
DSTA(II). It is noted that the fusion of DSTA(I) and DSTA(II) provides a
practical solution to real-world applications. This fusion result dominates
all of the four individual models on the mTTA-AP diagram. Meantime, it
provides probably the most attractive combination of precision and TTA, at 80%
recall. This fusion result further highlights the contribution of the proposed
DSTA-network. The study found that the improvement from fusing more than two
efficient models is marginal.

###  IV-F Impact of the Datasets

TABLE I shows that performances of the accident anticipation models on the DAD
dataset are significantly lower than on the CCD dataset. The literature has
not revealed the underlying reasons for the observed dramatic difference. In
attempt to reveal some of the unknowns, a thorough experimental study is
conducted to evaluate the applicability and limitations of representative
datasets for the early accident anticipation.

CCD is over 2.5 times larger than DAD. To determine if the performance gap is
caused by the data size difference, a subset of CCD, which is in the same size
as DAD and named as CCD-small, is created by randomly selecting videos from
the original CCD dataset. Despite of its reduced size, the DSTA-network
trained on CCD-small still achieves 98.51% AP and 4.83 seconds mTTA. This
verifies that the larger size of CCD is not the main reason for the superior
performance of early accident anticipation on CCD.

DAD and CCD datasets are dramatically different. Accidents in the DAD dataset
always start from the last 0.5 seconds of positive videos, only 10% of the
total video length. Most importantly, 90% of the positive videos in DAD
contains extremely complex urban traffic accidents, where ego-vehicles are not
involved in accidents, and traffic agents involved in, or affected by
accidents, may appear in the video for a very short time in relative to the
full length of the training videos. In the CCD dataset, 53.4% of the accidents
are ego-vehicle involved. Changes in motion, size, and other appearance
features in accident videos are relatively large. The starting time of
accidents in CCD is placed randomly in the last 2 seconds, and the mean
starting time is in the last 1.28 seconds. These differences are important
reasons that accident anticipation models have largely different performance
on the two datasets.

TABLE V: The Impact of Dataset Differences Training | Testing | AP(%) | mTTA (s)
---|---|---|---
DAD | CCD | 35.99 | 4.98
CCD | DAD | 35.74 | 4.99
CCD-small$\cup$DAD | CCD | 98.89 | 4.48
CCD-small$\cup$DAD | DAD | 69.67 | 1.40

To determine if the above-discussed data differences would impact the
generalization capability of the proposed DSTA-network, four experiments
listed in TABLE V are conducted. In these experiments, the frequency of DAD
videos is reduced to 10 fps to be compatible with CCD. The first two
experiments in this table are cross-testing. Both of the tests achieve a low
AP value, near 36%. The dramatically reduced AP implies a challenge for the
early accident anticipation arising from differences in traffic scenes and
accident types. The network is further trained on the dataset that mixes the
CCD-small and the DAD training dataset. This model performs better than in the
cross-testing experiments, achieving 98.89% AP and 4.48 secconds mTTA when
tested on the CCD dataset, and 69.67% AP and 1.40 sec mTTA on DAD. But the
performances are below those in TABLE I.

This experimental study verifies that the data foundation for the early
accident anticipation is crucial. A comprehensive dataset is desired, which
has balanced ego-vehicle involved and uninvolved accidents and diverse scene
configurations from different regions and countries.

##  V Conclusion

This paper presented a novel end-to-end DSTA-network for the early
anticipation of traffic accidents from widely deployed dashcams. Through
designing a new model fusion method and analyzing existing datasets, this
paper identifies opportunities for further advancing the early accident
anticipation.

Computer vision has been playing vital roles in traffic safety enhancement and
autonomous driving. It can augment or substitute for human vision in the
transportation system. Just like human vision that is one, but not the only,
sensing and learning mechanism, computer-vision based early accident
anticipation should be further integrated with other safety-enhancement
technologies and methods such as LiDAR, internet of things, and transportation
safety surrogate measurements (SSM) to deliver a multimodal, multifunctional
system for the early accident anticipation. The convergence of these
technologies with the proposed method is an important direction that will
bring exciting opportunities for safety enhancement. This study has envisaged
multiple lines of future work. The proposed DSTA-network has a common
limitation as many other deep learning models, which is the unexplainability
of the rationale behind the network’s decision-making process. A follow-up
study is to reveal the risk perception mechanism of the network to make it
explainable and trustworthy to humans. In transportation, various surrogate
safety measurements are effectively used to identify accident risks. An
exciting future direction is to develop sensor-fusion based, SSM-informed deep
learning networks for safety enhancement. This study also evidences the need
for a strong data foundation for the accident anticipation. Finally, accident
anticipation is also a desired capability for mobile robots such as drones and
remotely operated underwater vehicles. The application of the DSTA-network to
those robots faces additional challenges. This paper lays a foundation for
exploring the above-discussed exciting opportunities.

## Acknowledgment

Qin, Karim, and Li receive support from National Science Foundation (NSF)
through the grant ECCS-#2026357. Yin and Karim receive support from NSF
through ECCS-#2025929.

## References

  * [1] A. Eskandarian, C. Wu, and C. Sun, “Research advances and challenges of autonomous and connected ground vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , 2019.
  * [2] K. Muhammad, A. Ullah, J. Lloret, J. Del Ser, and V. H. C. de Albuquerque, “Deep learning for safe autonomous driving: Current challenges and future directions,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020\.
  * [3] California DMV, “Autonomous vehicle collision reports,” 2021, available at:https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/, accessed September, 2021.
  * [4] WHO, “Global status report on road safety 2018: Summary,” World Health Organization, Tech. Rep., 2018.
  * [5] Y. Li, M. M. Karim, R. Qin, Z. Sun, Z. Wang, and Z. Yin, “Crash report data analysis for creating scenario-wise, spatio-temporal attention guidance to support computer vision-based perception of fatal crash risks,” _Accident Analysis & Prevention_, vol. 151, p. 105962, 2021.
  * [6] NHTSA, “Critical reasons for crashes investigated in the national motor vehicle crash causation survey,” Washington, DC: US Department of Transportation, Tech. Rep., 2015.
  * [7] J. Janai, F. Güney, A. Behl, A. Geiger _et al._ , “Computer vision for autonomous vehicles: Problems, datasets and state of the art,” _Foundations and Trends® in Computer Graphics and Vision_ , vol. 12, no. 1–3, pp. 1–308, 2020.
  * [8] F.-H. Chan, Y.-T. Chen, Y. Xiang, and M. Sun, “Anticipating accidents in dashcam videos,” in _Asian Conference on Computer Vision_ , 2016, pp. 136–153.
  * [9] K.-H. Zeng, S.-H. Chou, F.-H. Chan, J. Carlos Niebles, and M. Sun, “Agent-centric risk assessment: Accident anticipation and risky region localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2222–2230.
  * [10] T. Suzuki, H. Kataoka, Y. Aoki, and Y. Satoh, “Anticipating traffic accidents with adaptive loss and large-scale incident db,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3521–3529.
  * [11] M. Fatima, M. U. K. Khan, and C.-M. Kyung, “Global feature aggregation for accident anticipation,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 2809–2816.
  * [12] H. S. Koppula and A. Saxena, “Anticipating human activities using object affordances for reactive robotic response,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 1, pp. 14–29, 2015.
  * [13] C. Vondrick, H. Pirsiavash, and A. Torralba, “Anticipating visual representations from unlabeled video,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 98–106.
  * [14] R. Q. Mínguez, I. P. Alonso, D. Fernández-Llorca, and M. Á. Sotelo, “Pedestrian path, pose, and intention prediction through gaussian process dynamical models and pedestrian activity recognition,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 20, no. 5, pp. 1803–1814, 2018.
  * [15] M. Xu, M. Gao, Y.-T. Chen, L. S. Davis, and D. J. Crandall, “Temporal recurrent networks for online action detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5532–5541.
  * [16] J. Liang, L. Jiang, J. C. Niebles, A. G. Hauptmann, and L. Fei-Fei, “Peeking into the future: Predicting future person activities and locations in videos,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5725–5734.
  * [17] M. Sadegh Aliakbarian, F. Sadat Saleh, M. Salzmann, B. Fernando, L. Petersson, and L. Andersson, “Encouraging LSTMs to anticipate actions very early,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 280–289.
  * [18] Y. Yao, M. Xu, Y. Wang, D. J. Crandall, and E. M. Atkins, “Unsupervised traffic accident detection in first-person videos,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 273–280.
  * [19] Y. Takimoto, Y. Tanaka, T. Kurashima, S. Yamamoto, M. Okawa, and H. Toda, “Predicting traffic accidents with event recorder data,” in _Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Prediction of Human Mobility_ , 2019, pp. 11–14.
  * [20] S. Ma, L. Sigal, and S. Sclaroff, “Learning activity progression in LSTMs for activity detection and early detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1942–1950.
  * [21] S. Cui and I. Joe, “A dynamic spatial-temporal attention-based gru model with healthy features for state-of-health estimation of lithium-ion batteries,” _IEEE Access_ , vol. 9, pp. 27 374–27 388, 2021.
  * [22] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” in _NIPS 2014 Workshop on Deep Learning, December 2014_ , 2014.
  * [23] W. Bao, Q. Yu, and Y. Kong, “Uncertainty-based traffic accident anticipation with spatio-temporal relational learning,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 2682–2690.
  * [24] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5998–6008.
  * [26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 8026–8037, 2019.
  * [27] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016\.
  * [28] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6154–6162.
  * [29] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [30] Y. Censor, “Pareto optimality in multiobjective problems,” _Applied Mathematics and Optimization_ , vol. 4, no. 1, pp. 41–59, 1977.


14 | $0.016132$ | $0.033179$ | $0.017001$ | $0.033501$ | $0.011320$ | $0.031089$
15 | $0.009571$ | $0.010922$ | $0.009835$ | $0.010946$ | $0.007536$ | $0.010668$
16 | $0.003143$ | $0.001099$ | $0.003165$ | $0.001099$ | $0.002882$ | $0.001097$
Table 31: Four loop Landau gauge Banks-Zaks fixed points and $\omega$ in the
MOMq, MOMc and MOMg schemes.

$N_{\\!f}$ | $a_{[3,1]}$ | $\alpha_{[3,1]}$ | $\rho_{m{[3,1]}}$ | $a_{[2,2]}$ | $\alpha_{[2,2]}$ | $\rho_{m{[2,2]}}$ | $a_{[1,3]}$ | $\alpha_{[1,3]}$ | $\rho_{m{[1,3]}}$
---|---|---|---|---|---|---|---|---|---
$9$ | $0.081552{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $-0.355655$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.081552{\mbox{\footnotesize{zp}}}$ | $-0.009802$ | $-0.355655$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.081552{\mbox{\footnotesize{zp}}}$ | $-4.117330$ | $-0.355655$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$10$ | $0.060179{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.211071$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.060179{\mbox{\footnotesize{zp}}}$ | $-0.708479$ | $0.211071$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.060179{\mbox{\footnotesize{zp}}}$ | $-3.428390$ | $0.211071$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$11$ | $0.044777{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.293997$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.044777{\mbox{\footnotesize{zp}}}$ | $-1.299001$ | $0.293997$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.044777{\mbox{\footnotesize{zp}}}$ | $-3.069015$ | $0.293997$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$12$ | $0.032429{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.255133$ | $0.050422$ | $0.000000$ | $0.215820$ | $0.048898$ | $0.000000$ | $0.229795$
| $0.032429{\mbox{\footnotesize{zp}}}$ | $-1.777608$ | $0.255133$ | $0.050422$ | $-2.076662$ | $0.215820$ | $0.048898$ | $-2.111467$ | $0.229795$
$13$ | $-$ | $-$ | $-$ | $0.029939$ | $0.000000$ | $0.230199$ | $0.029869$ | $0.000000$ | $0.229907$
| $-$ | $-$ | $-$ | $0.029939$ | $-2.355729$ | $0.230199$ | $0.029869$ | $-2.370730$ | $0.229907$
$14$ | $-$ | $-$ | $-$ | $0.018432$ | $0.000000$ | $0.153358$ | $0.018430$ | $0.000000$ | $0.153346$
| $-$ | $-$ | $-$ | $0.018432$ | $-2.600534$ | $0.153358$ | $0.018430$ | $-2.603503$ | $0.153346$
$15$ | $0.010131$ | $0.000000$ | $0.084781$ | $0.010075$ | $0.000000$ | $0.084311$ | $0.010075$ | $0.000000$ | $0.084311$
| $0.022171$ | $0.000000$ | $0.173566$ | $0.010075$ | $-2.797725$ | $0.084311$ | $0.010075$ | $-2.797936$ | $0.084311$
| $0.010131$ | $-2.799955$ | $0.084781$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.022171$ | $-3.269499$ | $0.173566$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$16$ | $0.003171$ | $0.000000$ | $0.025904$ | $0.003171$ | $0.000000$ | $0.025902$ | $0.003171$ | $0.000000$ | $0.025902$
| $0.024469$ | $0.000000$ | $0.179317$ | $0.003171$ | $-2.945965$ | $0.025902$ | $0.003171$ | $-2.945965$ | $0.025902$
| $0.003171$ | $-2.945970$ | $0.025904$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.024469$ | $-3.831623$ | $0.179317$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$

Table 32: Values of $a$ and $\alpha$ in the $\overline{\mbox{MS}}$ scheme from
the Padé analysis.

$N_{\\!f}$ | $a_{[3,1]}$ | $\alpha_{[3,1]}$ | $\rho_{m{[3,1]}}$ | $a_{[2,2]}$ | $\alpha_{[2,2]}$ | $\rho_{m{[2,2]}}$ | $a_{[1,3]}$ | $\alpha_{[1,3]}$ | $\rho_{m{[1,3]}}$
---|---|---|---|---|---|---|---|---|---
$9$ | $0.081552{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $-2.193550$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$10$ | $0.060179{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $-0.635482$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$11$ | $0.044777{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.056006$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$12$ | $0.032429{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.209582$ | $0.050422$ | $0.000000$ | $-0.296439$ | $0.048898$ | $0.000000$ | $-0.209082$
| $0.032429{\mbox{\footnotesize{zp}}}$ | $2.511522$ | $0.445578$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.032429{\mbox{\footnotesize{zp}}}$ | $-1.557388$ | $0.204125$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.032429{\mbox{\footnotesize{zp}}}$ | $-6.482723$ | $0.617028$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$13$ | $-$ | $-$ | $-$ | $0.029939$ | $0.000000$ | $0.196188$ | $0.029869$ | $0.000000$ | $0.196332$
| $-$ | $-$ | $-$ | $0.029939$ | $2.893770$ | $0.456669$ | $0.029869$ | $2.890160$ | $0.453356$
| $-$ | $-$ | $-$ | $0.029939$ | $-4.293260$ | $0.210744$ | $0.029869$ | $-6.304360$ | $0.413156$
| $-$ | $-$ | $-$ | $0.029939$ | $-6.348796$ | $0.428731$ | $-$ | $-$ | $-$
$14$ | $-$ | $-$ | $-$ | $0.018432$ | $0.000000$ | $0.150990$ | $0.018430$ | $0.000000$ | $0.150979$
| $-$ | $-$ | $-$ | $0.018432$ | $3.414938$ | $0.235117$ | $0.018430$ | $3.350179$ | $0.229641$
| $-$ | $-$ | $-$ | $0.018432$ | $-4.055378$ | $0.155694$ | $0.018430$ | $-4.086352$ | $0.155884$
| $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $0.018430$ | $-6.934702$ | $0.241291$
$15$ | $0.010131$ | $0.000000$ | $0.084699$ | $0.010075$ | $0.000000$ | $0.084238$ | $0.010075$ | $0.000000$ | $0.084238$
| $0.022171$ | $0.000000$ | $0.165259$ | $0.010075$ | $4.664518$ | $0.118824$ | $0.010075$ | $4.533597$ | $0.115735$
| $0.010131$ | $4.575310$ | $0.117700$ | $0.010075$ | $-3.561433$ | $0.084894$ | $0.010075$ | $-3.562379$ | $0.084896$
| $0.010131$ | $-3.555677$ | $0.085353$ | $-$ | $-$ | $-$ | $0.010075$ | $-8.200740$ | $0.122989$
| $0.022171$ | $3.134973$ | $0.266982$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.022171$ | $-3.762350$ | $0.170434$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.022171$ | $-6.703760$ | $0.282523$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.022171$ | $-7.981369$ | $0.656479$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$16$ | $0.003171$ | $0.000000$ | $0.025903$ | $0.003171$ | $0.000000$ | $0.025902$ | $0.003171$ | $0.000000$ | $0.025902$
| $0.024469$ | $0.000000$ | $0.171220$ | $0.003171$ | $-3.027379$ | $0.025902$ | $0.003171$ | $9.180136$ | $0.036049$
| $0.003171$ | $9.158681$ | $0.035969$ | $0.003171$ | $-12.733830$ | $0.037015$ | $0.003171$ | $-3.027380$ | $0.025902$
| $0.003171$ | $-3.027359$ | $0.025903$ | $-$ | $-$ | $-$ | $0.003171$ | $-12.812101$ | $0.037342$
| $0.003171$ | $-12.734028$ | $0.037018$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.003171$ | $-27.524404$ | $1.952290$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.024469$ | $2.987578$ | $0.273801$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.024469$ | $-3.565082$ | $0.177193$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
| $0.024469$ | $-6.591011$ | $0.298554$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$

Table 33: Values of $a$ and $\alpha$ in the RI′ scheme from the Padé analysis.

$N_{\\!f}$ | $a_{[3,1]}$ | $\alpha_{[3,1]}$ | $\rho_{m{[3,1]}}$ | $a_{[2,2]}$ | $\alpha_{[2,2]}$ | $\rho_{m{[2,2]}}$ | $a_{[1,3]}$ | $\alpha_{[1,3]}$ | $\rho_{m{[1,3]}}$
---|---|---|---|---|---|---|---|---|---
$9$ | $0.068777{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $1.555108$ | $0.034658$ | $0.000000$ | $0.459046$ | $0.015650$ | $0.000000$ | $0.156758$
| $0.037028$ | $-3.586442$ | $0.411340$ | $0.021048$ | $-3.613442$ | $0.207376$ | $0.016895$ | $-0.385698$ | $0.172742$
$10$ | $0.019275{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.199491$ | $0.035297$ | $0.000000$ | $0.438099$ | $0.022986$ | $0.000000$ | $0.248950$
| $0.021361{\mbox{\footnotesize{zp}}}$ | $-1.236002$ | $0.227161$ | $0.027313$ | $-3.724616$ | $0.273199$ | $0.023299$ | $-1.634361$ | $0.251958$
| $0.035930$ | $-3.625784$ | $0.376564$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$11$ | $0.026800{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.292618$ | $0.033149$ | $0.000000$ | $0.379081$ | $0.027364$ | $0.000000$ | $0.300178$
| $0.115245$ | $1.079306$ | $-5.582485$ | $-$ | $-$ | $-$ | $0.028914$ | $-2.681089$ | $0.307396$
| $0.033687$ | $-3.664896$ | $0.337578$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$12$ | $0.028658$ | $0.000000$ | $0.305342$ | $0.028698$ | $0.000000$ | $0.305832$ | $0.026874{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.283733$
| $0.017894$ | $-2.195156$ | $0.173967$ | $0.030517{\mbox{\footnotesize{zp}}}$ | $-3.159403$ | $0.308537$ | $0.028747$ | $-3.067509$ | $0.290452$
| $0.031308{\mbox{\footnotesize{zp}}}$ | $-3.208981$ | $0.316187$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$13$ | $0.021998$ | $-1.158753$ | $0.221442$ | $0.022688{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.227889$ | $0.022319{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.223707$
| $0.102810$ | $-2.332708$ | $-3.089860$ | $0.023447$ | $-3.139124$ | $0.226299$ | $0.023378$ | $-3.137254$ | $0.225576$
| $0.023456$ | $-3.135595$ | $0.226429$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
$14$ | $0.015768{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.149002$ | $0.016059$ | $0.000000$ | $0.152079$ | $0.016013{\mbox{\footnotesize{zp}}}$ | $0.000000$ | $0.151600$
| $0.016559$ | $-3.122146$ | $0.152248$ | $0.015603$ | $4.098613$ | $0.125880$ | $0.016520$ | $-3.121172$ | $0.151856$
| $0.021578$ | $-3.473532$ | $0.200960$ | $0.016521$ | $-3.121165$ | $0.151871$ | $-$ | $-$ | $-$
| $-$ | $-$ | $-$ | $0.021567$ | $-3.476577$ | $0.200823$ | $-$ | $-$ | $-$
$15$ | $0.009664$ | $-3.078780$ | $0.084260$ | $0.009475$ | $0.000000$ | $0.084205$ | $0.009473$ | $0.000000$ | $0.084181$
| $-$ | $-$ | $-$ | $0.009658$ | $-3.078493$ | $0.084202$ | $0.009657$ | $-3.078420$ | $0.084194$
| $-$ | $-$ | $-$ | $0.016853$ | $-3.622266$ | $0.151513$ | $-$ | $-$ | $-$
$16$ | $0.003122$ | $0.000000$ | $0.025904$ | $0.003122$ | $0.000000$ | $0.025902$ | $0.003122$ | $0.000000$ | $0.025902$
| $0.015066$ | $0.000000$ | $0.139013$ | $0.003143$ | $-3.027379$ | $0.025902$ | $0.003143$ | $-3.027379$ | $0.025902$
| $0.003143$ | $-3.027381$ | $0.025902$ | $0.014135$ | $-3.897042$ | $0.123605$ | $-$ | $-$ | $-$
| $0.012358$ | $-3.769284$ | $0.107543$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$

Table 34: Values of $a$ and $\alpha$ in the mMOM scheme from the Padé
analysis.