# Where and When: Space-Time Attention for Audio-Visual Explanations

Yanbei Chen1, Thomas Hummel1, A. Sophia Koepke1, Zeynep Akata1,2,3
1University of Tübingen 2MPI for Informatics 3MPI for Intelligent Systems
{yanbei.chen, thomas.hummel, a-sophia.koepke<EMAIL_ADDRESS>
###### Abstract

Explaining the decision of a multi-modal decision maker requires to determine
the evidence from both modalities. Recent advances in XAI provide explanations
for models trained on still images. However, when it comes to modeling
multiple sensory modalities in a dynamic world, it remains underexplored how
to demystify the mysterious dynamics of a complex multi-modal model. In this
work, we take a crucial step forward and explore learnable explanations for
audio-visual recognition. Specifically, we propose a novel space-time
attention network that uncovers the synergistic dynamics of audio and visual
data over both space and time. Our model is capable of predicting the audio-
visual video events, while justifying its decision by localizing where the
relevant visual cues appear, and when the predicted sounds occur in videos. We
benchmark our model on three audio-visual video event datasets, comparing
extensively to multiple recent multi-modal representation learners and
intrinsic explanation models. Experimental results demonstrate the clear
superior performance of our model over the existing methods on audio-visual
video event recognition. Moreover, we conduct an in-depth study to analyze the
explainability of our model based on robustness analysis via perturbation
tests and pointing games using human annotations.

##  1 Introduction

Figure 1:  For a given video, our space-time attention network (STAN) predicts
the audio-visual event and provides audio-visual explanations, which pinpoint
where the discriminative visual cues appear in the video and when the
predicted sounds occur.

A real-world event is often perceived and interpreted by processing
information from various sensory modalities, such as audio and vision [12].
For instance, the video event of playing acoustic guitar can be recognized
because of the presence of a musician playing the guitar in the scene (visual
cues) and the sound of the guitar (audio cues). In fact, perceiving the audio
and visual cues simultaneously makes it easier to distinguish acoustically or
visually similar events such as playing acoustic guitar and playing mandolin.
The synergy of audio and visual modalities has also been shown to be
beneficial for learning more powerful multi-modal representations for video
recognition. Recently, a line of works shows that integrating the audio and
visual data by multi-modal learning can greatly boost the model performance,
e.g. for recognizing human action [43], speech [2], or sound events [13] in
videos.

Although existing works have discovered the strength of multi-modal networks
for video recognition, understanding how different modalities are composed and
utilized for model predictions remains an unresolved challenge. Inspired by
the recent advances in explainable AI (XAI) [8, 20, 48, 27, 28], our goal in
this work is to uncover the underlying rationale for the audio-visual model
predictions. Instead of simply fusing the two modalities, we introduce an
intrinsically explainable audio-visual model that can predict the audio-visual
video events, while explaining how each data modality contributes to the model
predictions.

To achieve the aforementioned intrinsic explainability in audio-visual
learning, we need to tackle several challenges: (i) the audio and visual
information could vary drastically over space and time; (ii) obtaining desired
audio-visual recognition and explanation both require learning good
representations. We tackle these challenges via a unified audio-visual model
built with an explainable space-time attention mechanism (Figure 1). In
contrast to most existing explainable models that operate on the static images
and texts [36, 39, 33, 50], our model learns to provide explanations for the
audio and visual data over space and time. In other words, unlike visual
recognition on still images, it is crucial to simultaneously reason on the
spatial and temporal dynamics across different modalities for audio-visual
video recognition. As a pioneer in exploring explanations for audio-visual
learning, our model is designed to predict the audio-visual video events and
offer explanations that localize where the related visual cues appear in space
and when the predicted sounds occur along time.

Our contributions are as follows. (1) We propose a novel explainable space-
time attention network (STAN) which uncovers the underlying spatial and
temporal dynamics to justify how the audio and visual modalities are utilized
for recognizing an audio-visual event. (2) We establish a comprehensive
benchmark on multiple audio-visual event datasets, comparing our model to
multiple recent multi-modal models and intrinsic explanation models.
Experimental results show that our model achieves superior performance on
audio-visual event recognition. (3) We provide an insightful study to analyze
the explainablity of our model. We show that our model serves as a good proxy
for model explanations and offers human-interpretable explanations.

##  2 Related Work

Visual Explanations. A group of existing works provide visual explanations
(e.g., a saliency or attention map) that uncover the underlying focus of deep
neural networks in a visual recognition task [37, 49, 29, 54, 36, 36, 38, 39,
8, 51, 14, 28, 27]. Existing visual explanation models can be grouped into
three families: CNN visualization [49, 29], gradient-based post-hoc
explanations [36, 38, 8, 39], and response-based intrinsic explanations [54,
14]. While the former two families obtain visualizations without making any
architectural change, the latter one incorporates visual explanations as an
intrinsic component in the model architecture. The gradient-based visual
explanation models (such as Grad-CAM [36] and Integrated Gradients [39])
utilize the back-propagated gradients of a pre-trained network to derive an
activation or saliency map for identifying the image regions or pixels that
most contribute to the model’s decisions. For response-based visual
explanation models (e.g., CAM [54], ABN [14] and one-class explanation models
[27, 28]), an activation or attention map is designed as an intrinsic property
of the model to provide transparent explanations, which often yield an
interpretable heatmap on the image to highlight the decisive regions. In a
similar spirit as CAM and ABN, our proposed model is also an intrinsic
explanation model with a learnable class activation mechanism to localize the
salient regions. However, we stretch this mechanism to model the multi-modal
audio-visual data in videos, and propose a novel explainable space-time
attention mechanism that jointly learns class activation maps over space and
class activation values along time.

Multi-Modal Explanations. Another recent line of works offers multi-modal
explanations for deep neural networks via generating human-interpretable
justifications [19, 33, 25, 20, 50, 24, 48], such as text explanations that
explain a model’s decisions by generating human-readable words or natural
language sentences [25], or other cues like grounding bounding boxes that
point to the important visual regions [24]. In general, these explanation
models are trained in a multi-task learning framework by jointly optimizing a
primary task objective and an auxiliary explanation objective. Among these
works, several of them explore to learn from multi-modal visual and textual
data and generate text explanations to justify the answers for visual question
answering [33] or visual commonsense reasoning [50]. Similarly, we also
explore multi-modal data, but in the task of audio-visual event recognition
that especially requires the synergistic space-time understanding of two data
modalities. Although text explanations or bounding boxes are human-friendly,
they often require manual annotations to provide supervision on explanations
during training. To avoid using expensive annotations, we propose a space-time
attention mechanism to explain the spatial and temporal dynamics for
recognizing a video event, without using any additional supervision on
explanations.

Audio-Visual Learning. Integrating the audio and visual modalities for multi-
modal representation learning can benefit a wide variety of tasks, such as lip
reading [3], speech recognition [2], speech separation [11], emotion
recognition [31, 5], sound source separation [17, 4], sound localization [41,
46, 15], action recognition [43, 47, 9], sound recognition [13], audio-driven
image synthesis [44, 23, 42], and visual-driven audio synthesis [55, 16]. A
line of works has also explored various ways to localize the visual regions
related to sounds [6, 53, 52, 4], or to localize the sounds [41, 46]. In this
work, we consider to jointly localize the audio and visual aspects over space
and time. However, rather than simply fusing the two modalities for learning
an audio-visual recognition model [43, 13], we introduce explainability into
audio-visual learning. While existing works on localizing sound sources [46,
15] or sound source separation [17, 4] focus mainly on the audio aspect and do
not consider explanability, our work considers both recognition and
explanations on audio and visual modalities simultaneously. Moreover, we
evaluate our model under the explanation evaluation criteria to study its
underlying connections to model explanations and human explanations.

##  3 Space-Time Attention Network (STAN)

Figure 2:  Given audio and visual data from a video, STAN first composes the
audio and visual features from the audio and video encoders (Section 3.1), and
learns space-time attention (Section 3.2), constrained by a learning objective
for three classifiers (Section 3.3). Once trained, STAN can recognize the
audio-visual events and offer explanations on the audio-visual data. While the
space attention tells where the salient visual cues appear in space, the time
attention tells when the sounds occur along time. CAV/CAM: class activation
values/maps.

Given a video consisting of audio and image frames, our task is to recognize
the audio-visual video events, while being capable of explaining where (i.e.,
space) and when (i.e., time) are decisive for recognition. For instance, to
predict the audio-visual event of playing acoustic guitar, an audio-visual
recognition model is expected to “see” where the musicians play the string
instruments in the video, and “hear” when the sounds of string music occur in
the audio track. To this end, we propose a unified audio-visual recognition
model with an explainable space-time attention mechanism. As Figure 2 shows,
the attention mechanism is decomposed and learned separately along the space
and time dimensions, conditioning on the audio-visual features. By design, the
space-time attention can pinpoint the salient visual and audio cues, and
selectively activate the space-time features to learn explainable
representations for video event recognition using heterogeneous audio and
visual data.

###  3.1 Representing Audio and Visual Modalities

We refer to the video dataset as
$\\{\textbf{X}_{a[i]},\textbf{X}_{v[i]},y_{[i]}\\}_{i=1}^{N}$, where
$\textbf{X}_{a}$ denotes the audio, $\textbf{X}_{v}$ is the video,
$y\in\mathbb{R}^{{K}}$ is the corresponding video event label and $K$ is the
number of classes. To learn from the fine-grained information over space and
time, $\textbf{X}_{a},\textbf{X}_{v}$ are decomposed into a sequence of $T$
segments, where $T$ is the total time length. Each audio or video segment is
of a fixed time length (e.g. 1 second). Next, we detail how each modality is
represented and how they are composed for audio-visual representation
learning.

Audio Encoder. An audio track $\textbf{X}_{a}$ is represented by a sequence of
audio features, i.e., $\bm{a}=\\{a_{1},\dots,a_{t},\dots,a_{T}\\}$. Each audio
segment is a $D_{a}$-dimensional embedding $a_{t}$ extracted from the log-mel
spectrogram by an audio encoder.

Video Encoder. To encode the space-time visual content, the video
$\textbf{X}_{v}$ is represented by a sequence of visual features, i.e.,
$\bm{v}=\\{v_{1},...,v_{t},...,v_{T}\\}$. For each visual segment, a ResNet
[18] is used to extract a $(H\times W\times D_{v})$-dimensional feature map
$v_{t}$, where $H\times W$ denotes its spatial dimensions. As each visual
segment could contain multiple image frames, temporal average pooling is
applied to aggregate all the frame-wise image features and derive one feature
map per segment. Thus, each video $\textbf{X}_{v}$ is encoded by a space-time
visual feature tensor $\bm{v}\in\mathbb{R}^{T\times H\times W\times D_{v}}$.

Audio-Visual Composition. As audio and visual modalities encode heterogeneous
and complementary information, we propose to compose the two modalities. To
achieve this aim, an intuitive strategy would be to add, average or
concatenate the audio and visual features, which however, may not be directly
applicable due to a mismatch in feature dimensions across modalities. Hence,
we apply linear transformations to project the audio and visual feature
tensors to the size of $T\times D$ and $T\times H\times W\times D$, followed
by tiling the audio features spatially to kept the audio and visual features
in an identical size of $T\times H\times W\times D$. Formally, the linear
transformations can be written as:

$\displaystyle\hat{a}_{t}$
$\displaystyle=\mathcal{F}_{\text{tile}}(\mathcal{F}_{\text{MLP}}^{st}([a_{t},\bar{a}];W_{a}^{st})),$
(1) $\displaystyle\hat{v}_{t}$
$\displaystyle=\mathcal{F}_{\text{conv}}^{st}(v_{t};W_{v}^{st}),$

where $\mathcal{F}_{\text{MLP}}^{st}(\cdot;W_{a}^{st})$ is an MLP;
$\mathcal{F}_{\text{tile}}(\cdot)$ is a spatial tiling operator;
$\mathcal{F}_{\text{conv}}^{st}(\cdot;W_{v}^{st})$ is a convolutional layer.
To take the global audio information into account, each audio segment is
represented by concatenating its feature $a_{t}$ and the temporal average
pooling feature of all audio segment features, i.e.,
$\bar{a}=\frac{1}{T}\sum_{t=1}^{T}a_{t}$. The features of the audio and video
sequences can be referred as $\bm{\hat{a}}$ and $\bm{\hat{v}}$, where
$\bm{\hat{a}}=\\{\hat{a}_{1},...\hat{a}_{t},...,\hat{a}_{T}\\}^{T\times
H\times W\times D}$ and
$\bm{\hat{v}}=\\{\hat{v}_{1},...\hat{v}_{t},...,\hat{v}_{T}\\}^{T\times
H\times W\times D}$. To compose the audio and visual features $\bm{\hat{a}}$
and $\bm{\hat{v}}$, simple addition or concatenation can be applied to derive
a compositional space-time feature tensor
$\bm{x}^{\bm{st}}\in\mathbb{R}^{T\times H\times W\times D^{\prime}}$ which
encodes the audio-visual data. We concatenate $\bm{\hat{a}}$ and
$\bm{\hat{v}}$ to get the space-time feature.

###  3.2 Learning to Explain Where and When

To demystify how the audio and visual modalities are composed for recognizing
a video event, our space-time attention mechanism is learned to explain where
the visual cues appear in the video and when the sounds occur in the audio.
More precisely, the space and time attentions are learned separately and then
integrated to obtain attention-weighted space-time features. Since the audio
and visual modalities are both important for recognition, the space and time
attentions are learned upon the space features and time features respectively,
as elaborated in the following.

Space Attention. To explain the where dynamics in space, we first derive the
space features, followed by learning the space attention. Specifically, the
space representations are obtained using Eq. (1), which gives an audio-visual
space-time feature tensor
$\bm{x}^{\bm{s}}=\\{x_{1}^{\bm{s}},...,x_{t}^{\bm{s}},...x_{T}^{\bm{s}}\\}$.
To learn the space attention, spatial class activation maps (CAM) [54] are
learned for each individual space feature tensor
${x}_{t}^{\bm{s}}\in\mathbb{R}^{H\times W\times D}$, followed by a learnable
space gating function:

$\displaystyle{A}_{t}^{\bm{s}}=\mathcal{F}_{\text{space-
gate}}(\mathcal{F}_{\text{CAM}}({x}_{t}^{\bm{s}};W_{s});W_{sa}),$ (2)
$\displaystyle\text{with}\
\mathcal{F}_{\text{CAM}}({x}_{t}^{\bm{s}};W_{s})=M_{t}=W_{s}*{x}_{t}^{\bm{s}},$

where $*$ denotes 2D convolution; $\mathcal{F}_{\text{CAM}}(\cdot;W_{s})$
learns the class activation maps for a space feature tensor $x_{t}^{\bm{s}}$
at time step $t$: $M_{t}\in\mathbb{R}^{H\times W\times K}$.
$\mathcal{F}_{\text{space-gate}}(\cdot;W_{sa})$ is a space gating function
with a convolutional layer and a sigmoid function, which maps the class
activation maps to a space attention map
${A}_{t}^{\bm{s}}\in\\{0,1\\}^{H\times W}$. As ${A}_{t}^{\bm{s}}$ is learned
upon the spatial CAM, it summarizes how the model arrives at its decision in
space. As Figure 2 shows, a space attention map ${A}_{t}^{\bm{s}}$ essentially
pinpoints the most discriminative visual regions. For a video, the overall
space attention is written as
$\bm{A}^{\bm{s}}=\\{{A}_{1}^{\bm{s}},...,{A}_{t}^{\bm{s}},...,{A}_{T}^{\bm{s}}\\}^{T\times
H\times W}$, which includes $T$ space attention maps.

Time Attention. Although the space attention can explain the visual dynamics,
it does not tell when the sounds occur. The latter aspect is however essential
to explain how the audio contributes to the recognition of an audio-visual
event. To achieve this, we propose to first derive the time features, followed
by learning the time attention. In the time dimension, the spatial information
is no longer important; thus, we rewrite Eq. (1) to learn the time
representations by linear transformations on the audio and visual features:

$\displaystyle\hat{a}_{t}$
$\displaystyle=\mathcal{F}_{\text{MLP}}^{t}([a_{t},\bar{a}];W_{a}^{t}),$ (3)
$\displaystyle\hat{v}_{t}$
$\displaystyle=\mathcal{F}_{\text{MLP}}^{t}(\bar{v}_{t};W_{v}^{t}),$

where
$\mathcal{F}_{\text{MLP}}^{t}(\cdot;W_{a}^{t}),\mathcal{F}_{\text{MLP}}^{t}(\cdot;W_{v}^{t})$
are MLP layers. To discard the spatial information,
$\bar{v}_{t}\in\mathbb{R}^{T\times D_{v}}$ is derived by spatial average
pooling on the original visual feature map ${v}_{t}$. Given Eq. (3), the audio
and visual time features are kept in an identical size, where
$\bm{\hat{a}}^{\bm{t}}=\\{\hat{a}_{1},...\hat{a}_{t},...,\hat{a}_{T}\\}^{T\times
D}$ and
$\bm{\hat{v}}^{\bm{t}}=\\{\hat{v}_{1},...\hat{v}_{t},...,\hat{v}_{T}\\}^{T\times
D}$. Thus, $\bm{\hat{a}}^{\bm{t}}$ and $\bm{\hat{v}}^{\bm{t}}$ can be composed
by addition to obtain a compositional time feature tensor:
$x^{\bm{t}}=\\{x_{1}^{\bm{t}},...,x_{t}^{\bm{t}},...,x_{T}^{\bm{t}}\\}^{T\times
D}$.

To learn an explainable time attention in a similar fashion as the space
attention (Eq. (2)), we propose to first learn the temporal class activation
values (CAV) per time step feature $x_{\bm{t}_{t}}$, followed by a learnable
time gating function:

$\displaystyle{A}_{t}^{\bm{t}}=\mathcal{F}_{\text{time-
gate}}(\mathcal{F}_{\text{CAV}}(\bm{x}_{t}^{\bm{t}};W_{t});W_{ta}),$ (4)
$\displaystyle\text{with}\
\mathcal{F}_{\text{CAV}}(\bm{x}_{t}^{\bm{t}};W_{t})=V_{t}=W_{t}\bm{x}_{t}^{\bm{t}},$

where $\mathcal{F}_{\text{CAV}}(\cdot;W_{t})$ is an MLP that learns the class
activation values for the time feature $\bm{x}_{t}^{\bm{t}}$ at time $t$;
$V_{t}\in\mathbb{R}^{K}$ denotes the class activation values.
$\mathcal{F}_{\text{time-gate}}(\cdot;W_{ta})$ is a time gating function with
an MLP and a sigmoid function, which maps the class activation values to an
attention scalar value per time step, i.e., ${A}_{t}^{\bm{t}}{\in}\\{0,1\\}$.
For a video, the time attention is written as
$\bm{A}^{\bm{t}}=\\{{A}_{1}^{\bm{t}},...,{A}_{t}^{\bm{t}},...,{A}_{T}^{\bm{t}}\\}^{T}$.
$\bm{A}^{\bm{t}}$ has higher values when relevant sounds occur (see Figure 2).

Remark. It is worth noting that the temporal CAV differs from the spatial CAM.
While CAM learns an activation map per class per space tensor, CAV learns an
activation value per class per time step. However, both CAM and CAV learn the
per-class relevance scores at a specific spatial or temporal location, which
are also constrained by classification losses (detailed in Section 3.3). Thus,
CAM and CAV can both offer class activation scores to learn the attention maps
that indicate the feature importance over space and time.

Space-Time Attention. Given the space and time attention tensors
${A}^{\bm{s}}$, ${A}^{\bm{t}}$, we integrate the two attention tensors by
outer product, resulting in a space-time attention tensor:

$\displaystyle\bm{A}^{\bm{st}}$
$\displaystyle=\bm{A}^{\bm{s}}\otimes\bm{A}^{\bm{t}},$ (5)

where $\bm{A}^{\bm{st}}\in\\{0,1\\}^{T\times H\times W}$ is a holistic
attention tensor that tells how the space-time dynamics contribute to audio-
visual representation learning. For each video, ${A}^{\bm{st}}$ operates on
its audio-visual space-time feature tensor $\bm{x}^{\bm{st}}$ (Eq. (1)) as
follows.

$\displaystyle\bm{\hat{x}}^{\bm{st}}$
$\displaystyle=\bm{x}^{\bm{st}}\odot\bm{A}^{\bm{st}},$ (6)

where $\odot$ is the element-wise product of two tensors. The new space-time
feature tensor $\bm{\hat{x}}^{\bm{st}}$ (reweighted by attention tensor
${A}^{\bm{st}}$) is further passed towards the final classifier for predicting
the video events, as detailed in the next section.

###  3.3 Learning Objective

Given the attention-weighted audio-visual feature (Eq. (6)), the STAN model is
trained to predict the video events. Each audio-visual feature tensor
$\bm{\hat{x}}^{\bm{st}}\in\mathbb{R}^{T\times H\times W\times D^{\prime}}$ is
first passed through spatial average pooling and temporal average pooling to
obtain a $D^{\prime}$-dimensional feature embedding, followed by a fully-
connected layer with sigmoid activation to obtain the final probabilistic
predictions $p$ for the video. As each video may be tagged with more than one
event label, we cast this recognition task as a multi-label multi-class
classification problem. Hence, STAN can be optimized with a multi-label binary
cross-entropy loss:

$\displaystyle\mathcal{L}_{\text{BCE}}$
$\displaystyle=\sum_{j=1}^{K}y(j)\text{log}p(j)+(1{-}y(j))\text{log}(1{-}p(j)),$
(7)

where $y$ is the groundtruth audio-visual event label of the video. In theory,
the CAM and CAV (from Eq. (2) and Eq. (4)) both learn the class activations.
Therefore, we constrain the learning of CAM and CAV by mapping their average
pooling features to the class predictions by adding the classifier upon CAM
and CAV, using the same loss in Eq. (7). In particular, for CAM, we perform
spatial average pooling and temporal average pooling of all the class
activation maps in the full video; while for CAV, we perform temporal average
pooling of all the class activation values. The overall learning objective for
STAN can be written as:

$\displaystyle\mathcal{L}_{\text{STAN}}$
$\displaystyle=\mathcal{L}_{\text{BCE}}+\mathcal{L}_{\text{BCE}}^{\text{CAM}}+\mathcal{L}_{\text{BCE}}^{\text{CAV}}.$
(8)

As there are three classifiers, STAN is trained with
$\mathcal{L}_{\text{STAN}}$ as a multi-task model. To simplify model
optimization, we use pre-trained audio and video encoders and train only the
space-time attention modules and classifiers, while keeping the encoders
frozen during training. At test time, the final classifier (constrained by
$\mathcal{L}_{\text{BCE}}$) is used for model inference.

##  4 Experiments

We first detail the experimental setup (Section 4.1), and evaluate on audio-
visual recognition (Section 4.2). Finally, we analyze the explainability of
our model (Section 4.3).

###  4.1 Experimental Setup

Datasets. We evaluate on three audio-visual event datasets. First, AVE [41] is
an audio-visual event dataset, including 4,143 videos of 28 video event
categories (e.g., motorcycle and violin). Most videos are around 10 seconds,
and each video is tagged with one or more video event label. Second, LLP [40]
is a recent audio-visual video event dataset, which includes 11,849 video
clips of 25 video categories (e.g. car and cat). Each video is 10 seconds and
tagged with one or more than one video event label. Finally, we construct a
new Audio-Visual Recognition (referred as AVR) dataset by unifying the AVE and
LLP datasets into one dataset to obtain a large class space. AVR contains
15,992 videos of 43 video categories. We summarize the class spaces and
statistics in the supplementary. For all the datasets, we use only the video
event labels for training.

We adopt the above datasets as they contain the human annotated temporal
bounding boxes that indicate when the predicted sounds occur. For instance, a
temporal bounding box “$[0,4]$” means the sounds occur from seconds 0 to 4,
which can be converted to a binary mask to evaluate whether the temporal
attention values are in line with human annotations. We detail this evaluation
in Section 4.3.

Evaluation Metrics. Following the common practice for evaluating multi-label
multi-class classification [45], we adopt the following three evaluation
metrics. Top-1 accuracy measures the fraction of instances whose most
confident label is relevant. Mean average precision (mAP) is the average
fraction of relevant labels ranked higher than other labels. F-score is the
F-measure averaged over the instances, which is commonly adopted in sound
event recognition [30]. As computing the F-score requires binary predictions,
we convert the model predictive scores (${\in}[0,1]$) to binary values by a
threshold of 0.5. For all the metrics, higher percentages (%) indicate better
model performance.

Implementation Details. We use a VGGish network [21] pre-trained on the
YouTube dataset [1] to extract embeddings for each audio segment (1 second).
We use a ResNet-152 [18] pre-trained on ImageNet [10] to extract a visual
feature map that encodes each video segment. To ensure fair comparison, we use
the Adam optimizer [26] with the same learning rate schedule and random seed
for all the methods that we compare to. More implementation details are given
in the supplementary. Code will be publicly available.

###  4.2 Evaluating Audio-Visual Recognition

In this section, we conduct an ablation study on STAN, and compare to the
state of the art of two types of models: (1) intrinsic explanation models and
(2) audio-visual learners. All experiments on audio-visual recognition are
conducted on three datasets: AVE, LLP and AVR, using three metrics: top-1, mAP
(mean average precision) and F-score.

Data |  Method | AVE | LLP | AVR
---|---|---|---|---
top-1 | mAP | F-score | top-1 | mAP | F-score | top-1 | mAP | F-score
Audio |  audio baseline | 67.7 | 66.8 | 40.1 | 79.1 | 65.5 | 60.3 | 72.3 | 58.0 | 51.7
STAN (audio) | 74.4 | 77.0 | 68.2 | 81.8 | 72.3 | 69.1 | 76.5 | 67.1 | 64.1
Vision |  visual baseline | 77.4 | 83.7 | 71.4 | 61.1 | 47.4 | 46.1 | 64.1 | 58.5 | 47.2
STAN (visual) | 79.4 | 84.0 | 72.5 | 70.8 | 59.4 | 54.0 | 71.4 | 64.7 | 54.1
Audio+Vision |  STAN (audio-visual) | 91.8 | 93.6 | 85.3 | 87.2 | 82.6 | 75.5 | 86.2 | 83.1 | 74.5
Table 1: Ablation study of STAN for audio-visual video event recognition on the AVE, LLP and AVR datasets, using the evaluation metrics: top-1, mAP (mean average precision) and F-score.  Model |  Method | AVE | LLP | AVR
---|---|---|---|---
top-1 | mAP | F-score | top-1 | mAP | F-score | top-1 | mAP | F-score
Explanation Models |  CAM (visual) [54] | 78.6 | 82.1 | 71.1 | 61.0 | 45.8 | 46.1 | 62.9 | 56.9 | 47.7
ABN (visual) [14] | 79.6 | 83.8 | 72.4 | 63.2 | 47.4 | 45.1 | 63.8 | 58.6 | 47.8
AV-CAM | 85.6 | 88.3 | 79.6 | 84.9 | 76.2 | 70.8 | 83.4 | 78.4 | 71.1
AV-ABN | 87.8 | 91.0 | 81.6 | 85.6 | 78.6 | 71.9 | 84.6 | 80.4 | 70.9
Audio-Visual Models |  Relationship [35] | 85.3 | 88.4 | 78.6 | 83.8 | 76.1 | 70.4 | 83.2 | 77.8 | 68.9
Average Ensemble [32] | 86.8 | 90.0 | 62.8 | 82.0 | 76.8 | 65.9 | 79.6 | 76.4 | 59.5
FiLM [34] | 87.8 | 90.8 | 81.6 | 85.0 | 79.6 | 71.3 | 83.4 | 81.2 | 70.2
Attention Fusion [13] | 89.3 | 93.2 | 81.8 | 86.3 | 80.4 | 72.2 | 84.8 | 81.8 | 72.6
Ours |  STAN | 91.8 | 93.6 | 85.3 | 87.2 | 82.6 | 75.5 | 86.2 | 83.1 | 74.5
Table 2: Evaluating state of the art intrinsic explanation models, audio-
visual learners and STAN for audio-visual video event recognition on the AVE,
LLP and AVR datasets, using the evaluation metrics: top-1, mAP (mean average
precision) and F-score.

Baselines. We ablate our STAN model to evaluate the effects of audio and
visual data on our model. Our audio baseline is a unimodal baseline that
trains an MLP and a classifier upon the audio features. Our visual baseline is
also a unimodal baseline that trains a convolutional layer and a classifier
upon the visual features. Furthermore, our STAN (audio) is an ablation of our
model that uses only audio features and learns only the time attention; while
our STAN (visual) is also an ablation of our model that uses only visual
features and learns space-time attention. Finally, STAN (audio-visual) is our
full model that integrates audio and visual data to learn space-time
attention.

Ablation Study. Table 1 shows our ablative evaluation on audio-visual video
event recognition. As can be seen, compared to the audio baseline and visual
baseline that learn from one single modality, STAN (audio) and STAN (visual)
greatly boost the model performance by learning the explainable attention on
the audio or visual data. For instance, when comparing the audio baseline and
STAN (audio), the F-score is increased from 40.1/60.3/51.7 to 68.2/69.1/64.1
on the AVE/LLP/AVR datasets. Similarly, when comparing the visual baseline and
STAN (visual), the F-score is improved from 71.4/46.1/47.2 to 72.4/54.0/54.1
on AVE/LLP/AVR. Compared to the unimodal models, our STAN (audio-visual)
further enhances the model performance. The F-score of STAN (audio-visual) is
raised upon STAN (visual) from 72.4/54.0/54.1 to 85.3/75.5/74.5 on
AVE/LLP/AVR. The similar improving trends can also be observed in other
evaluation metrics including top-1 and mAP. Overall, these results suggest the
joint benefits of composing the audio and visual modalities and learning the
explainable space-time attention. In the following, we refer to our full model
STAN (audio-visual) as STAN.

Compared Methods. We compare STAN to two groups of models: (1) intrinsic
explainable models that incorporate explanation as an intrinsic model
component; (2) multi-modal representation learners that integrate audio and
visual data. We describe these two groups of models as below.

  * •

CAM [54] (class activation maps): A visual explanation model that learns to
localize the discriminative regions.

  * •

ABN [14] (attention branch network): An attention-based visual explanation
model that learns visual attention maps to pinpoint class-agnoistic
discriminative regions.

  * •

AV-CAM: We extend the vanilla CAM to learn upon the concatenation of visual
feature maps and audio features.

  * •

AV-ABN: Same as AV-CAM, we extend the vanilla ABN to learn from audio and
visual data.

Remark. CAM is used as a model component in the above methods. However, our
model especially learns class activation values (CAV) along time, which allows
to uncover both the spatial and temporal dynamics in videos.

  * •

Relationship [35]: A multi-modal module. It takes in the visual features and
audio features, followed by concatenation and an MLP to fuse the audio-visual
features.

  * •

Average Ensemble [32]: An ensemble of the unimodal audio and visual baselines
by averaging their predictions.

  * •

FiLM [34]: A Feature-wise Linear Modulation module, where the audio features
are used to modulate the visual features through a learnable affine
transformation.

  * •

Attention Fusion [13]: A recent state-of-the-art audio-visual feature learner
that fuses the audio and visual features by an attention-based weighted
average.

Remark. Relationship and FiLM are two representative multi-modal models
proposed to learn from images and texts. We adapt these two methods by
replacing the text features with audio features, and use temporal average
pooling to obtain one audio-visual feature per video. As Average Ensemble and
Attention Fusion are both designed for audio-visual learning, we apply them to
our context directly.

Comparative Results. Table 2 shows the comparative evaluation on two groups of
models and our STAN on audio-visual video event recognition. Among the group
of intrinsic explanation models including CAM, ABN, AV-CAM and AV-ABN, we can
observe the benefits of learning from audio-visual data jointly, e.g., the
performance of CAM and ABN are improved substantially in their multi-modal
variants. However, compared to the best model AV-ABN in this group, our STAN
still outperforms AV-ABN significantly, increasing the F-score from
81.6/71.9/70.9 to 85.3/75.5/74.5 on AVE/LLP/AVR respectively. The performance
advantages of STAN over AV-CAM and AV-ABN indicate the synergistic merits of
learning the space attention based on class activation maps and the time
attention based on class activation values. By design, the space-time
attention also selectively activates the space-time audio-visual features (Eq.
(6)) to facilitate better audio-visual learning.

In the group of audio-visual learners, we see that STAN still outperforms the
best model Attention Fusion, improving the F-score from 81.8/72.2/72.6 to
85.3/75.5/74.5 on AVE/LLP/AVR respectively. Although Attention Fusion also
learns attention to selectively fuse the two modalities, STAN learns a more
advanced space-time attention that can select the essential audio and visual
information both over space and along time, thus offering superior model
performance for audio-visual video event recognition.

###  4.3 Evaluating Audio-Visual Explanations

To study how our explainable attention relates to model explanations and human
explanations, we design two experiments to analyze its explainablity: (1)
robustness analysis via perturbation tests, and (2) pointing games on
localizing sounds. The former aims to uncover whether the attention value
provides the importance of each feature towards model prediction; while the
latter focuses on examining whether the attention is in line with human
annotations.

Figure 3:  Perturbation tests on (a) relevant features of high attention
values; (b) irrelevant features of low attention values. Note: Increasing the
perturbation noise level on features of high/low attention values should lead
to higher/lower TVD. Dataset: AVR.

Perturbation Tests. Here, we examine if our learned attention-based
explanations uncover the importance of each feature towards the model’s
decision. We conduct perturbation tests following the evaluation criteria in
robustness analysis for evaluating model explanations [22]. Specifically, our
perturbation tests are driven by two evaluation criteria. First, perturbing
the relevant features (i.e., features with high attention scores or importance
values) should lead to an appreciable change in the model predictions. Second,
perturbing the irrelevant features (i.e., features with low attention scores
or importance values) should not lead to an appreciable change in model
predictions. Based on these criteria, we add perturbation noise to the
features, following a Gaussian distribution
$\eta\sim\mathcal{N}(0,\sigma^{2})$, where $\sigma^{2}$ is increased from 0 to
1 to vary the level of noise. As the attention scores lie in $[0,1]$, we count
the features with higher attention scores (${\geqslant}0.5$) as relevant, and
the features with lower attention scores (${<}0.5$) as irrelevant. To quantify
the changes in the model predictions, we measure the total variation distance
(TVD) between two probability distributions P and Q derived before and after
adding perturbation noise, where
$\text{TVD}(P,Q)=\frac{1}{2}\sum_{i=1}^{K}|P_{i}-Q_{i}|$.

Figure 3 presents the results of perturbation tests on the AVR dataset for two
types of features: (a) relevant features that have higher attention values;
and (b) irrelevant features with lower attention values. We compare our STAN
to two strong intrinsic explainable models: ABN and AV-ABN (described in
Section 4.2), which also learn attention maps to locate the essential
features. For each level of perturbation noise, we randomly sample the 100
different noises and compute the mean TVD of 100 tests. We give an algorithm
overview of our perturbation tests in the supplementary.

On the one hand, as Figure 3 (a) shows, when increasing the perturbation noise
level in relevant features, the TVD is increased among all the methods.
However, the changes are much larger in STAN as compared to those of ABN and
AV-ABN. This means that the features of high attention values predicted by
STAN correspond to the regions essential towards model predictions. On the
other hand, as Figure 3 (b) shows, when increasing the perturbation noise
level in irrelevant features, the predictive changes in STAN are almost
imperceptible compared to the much bigger changes in ABN and AV-ABN. This
indicates the features of low attention values in STAN are indeed not
important for making model predictions. In summary, these results indicate
that our model offers explainable attention that picks up the features of
importance for model decision, and serves as a good proxy of model
explanations.

Figure 4:  Visualizing attention maps with STAN. (a) Space attention on image frames from different video classes. (b) Space attention maps over different image frames in the videos and the time attention on the co-occurring audios. Dataset: AVR.  Method | AVE | LLP | AVR
---|---|---|---
STAN (audio)+soft attention | 0.47 | 0.42 | 0.44
STAN (audio-visual)+soft attention | 0.46 | 0.40 | 0.37
STAN (audio)+binary attention | 0.33 | 0.23 | 0.25
STAN (audio-visual)+binary attention | 0.29 | 0.24 | 0.20
Table 3: Evaluation of pointing games for localizing sounds on AVE, LLP and
AVR. Metric: MAE (the lower the better).

Pointing Games on Localizing Sounds. To further evaluate how our attention-
based explanations relate to human explanations, we conduct pointing games
using available human annotations on audio. This evaluation shares a similar
spirit as the pointing games for localizing visual objects in images [51].
However, we design this test in the audio domain to evaluate the temporal
localization ability of sounds. We exploit the human-annotated temporal
bounding boxes that indicate when the sounds occur, as described in Section
4.1. Hence, they serve as a proxy of human explanations.

We compare the time attention to the binary masks obtained by converting a
human annotation to a grounding mask: $[0,...,1,1,0]$, where $1$ refers to the
time when the sounds occur. We compute the mean absolute error (MAE) between
the time attention and the binary mask, which measures the difference between
our explanation and the human explanation. We evaluate STAN in two modes, i.e.
STAN (audio) and STAN (audio-visual). As described in Section 4.2, these are
the only two explanation models that offer the time attention on audio. We
also evaluate soft attention obtained from the model and binary attention
derived by binarizing the soft attention with a threshold of 0.5.

As Table 3 shows, STAN (audio-visual) overall offers the lower MAE compared to
STAN (audio) on two types of attentions across three different datasets.
Notably, when using soft attention, STAN (audio-visual) obtains a lower MAE of
0.46/0.40/0.37 compared to 0.47/0.42/0.44 by STAN (audio) on AVE/LLP/AVR. This
indicates that STAN (audio-visual) has a better temporal localization ability
of sounds. In other words, the time attention learned by STAN (audio-visual)
is closer to the groundtruth human annotations.

Visualizing Attention Maps. We visualize the attention maps of STAN in Figure
4. As Figure 4 (a) shows, the space attention maps of different video classes
cover the most discriminative visual regions. For instance, in the video Flute
and Violin, the flute and violin are highlighted by STAN. This means that the
space attention can pick up the class-relevant visual cues. In Figure 4 (b),
we can observe that the space attention on video and the time attention on
audio work in different but synergistic ways. For instance, in the video Goat,
the space attention operates on the image frames to locate the moving goats in
the video; while the time attention operates on the audio to pick up the
relevant audio cues along time. Overall, these results confirm that our
formulation of space and time attention works synergistically to discover the
informative audio and visual cues over space and time, which also resembles
how humans interpret video events with both audio and visual content.

##  5 Conclusion

We presented a novel intrinsic explanation model for audio-visual recognition,
which sheds light on a new aspect for explainable modeling on multi-modal data
collected over space and time. Our proposed space-attention network (STAN)
first composes the audio and visual features, and learns attention upon class
activation maps over space and class activation values along time. The learned
space and time attention maps can be considered as explanations on the visual
and audio modalities for video event recognition. Our comprehensive
experiments demonstrate that STAN is a strong audio-visual representation
learners and offers impressive model performance on audio-visual event
recognition. Our analysis on explainablity also shows that STAN provides
meaningful explanations that are closely in line with model explanations and
human explanations.

Acknowledgements This work has been partially funded by the ERC (853489 -
DEXIM) and by the DFG (2064/1 – Project number 390727645).

## References

  * [1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  arXiv preprint arXiv:1609.08675, 2016.
  * [2] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman.  Deep audio-visual speech recognition.  IEEE TPAMI, 2018.
  * [3] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.  Asr is all you need: Cross-modal distillation for lip reading.  In ICASSP, 2020.
  * [4] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman.  Self-supervised learning of audio-visual objects from video.  In ECCV, 2020.
  * [5] Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman.  Emotion recognition in speech using cross-modal transfer in the wild.  In ACM MM, 2018.
  * [6] Relja Arandjelovic and Andrew Zisserman.  Objects that sound.  In ECCV, 2018.
  * [7] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [8] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In WACV, 2018.
  * [9] Yanbei Chen, Yongqin Xian, A. Sophia Koepke, Ying Shan, and Zeynep Akata.  Distilling audio-visual knowledge by compositional contrastive learning.  In CVPR, 2021.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [11] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation.  TOG, 2018.
  * [12] Marc O Ernst and Heinrich H Bülthoff.  Merging the senses into a robust percept.  Trends in cognitive sciences, 2004.
  * [13] Haytham M Fayek and Anurag Kumar.  Large scale audiovisual learning of sounds with weakly labeled data.  In IJCAI, 2020.
  * [14] Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi.  Attention branch network: Learning of attention mechanism for visual explanation.  In CVPR, 2019.
  * [15] Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba.  Self-supervised moving vehicle tracking with stereo sound.  In ICCV, 2019.
  * [16] Ruohan Gao and Kristen Grauman.  2.5 d visual sound.  In CVPR, 2019.
  * [17] Ruohan Gao and Kristen Grauman.  Co-separating sounds of visual objects.  In ICCV, 2019.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [19] Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell.  Generating visual explanations.  In ECCV, 2016.
  * [20] Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata.  Grounding visual explanations.  In ECCV, 2018.
  * [21] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.  Cnn architectures for large-scale audio classification.  In ICASSP, 2017.
  * [22] Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsieh.  Evaluations and methods for explanation through robustness analysis.  arXiv preprint arXiv:2006.00442, 2020.
  * [23] Amir Jamaludin, Joon Son Chung, and Andrew Zisserman.  You said that?: Synthesising talking faces from audio.  IJCV, 2019.
  * [24] Atsushi Kanehira, Kentaro Takemoto, Sho Inayoshi, and Tatsuya Harada.  Multimodal explanations by predicting counterfactuality in videos.  In CVPR, 2019.
  * [25] Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata.  Textual explanations for self-driving vehicles.  In ECCV, 2018.
  * [26] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [27] Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard J Radke, and Octavia Camps.  Towards visually explaining variational autoencoders.  In CVPR, 2020.
  * [28] Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller.  Explainable deep one-class classification.  In ICLR, 2021.
  * [29] Aravindh Mahendran and Andrea Vedaldi.  Understanding deep image representations by inverting them.  In CVPR, 2015.
  * [30] Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen.  Metrics for polyphonic sound event detection.  Applied Sciences, 2016.
  * [31] Fatemeh Noroozi, Marina Marjanovic, Angelina Njegus, Sergio Escalera, and Gholamreza Anbarjafari.  Audio-visual emotion recognition in video clips.  IEEE Transactions on Affective Computing, 2017.
  * [32] Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc QK Duong, Patrick Pérez, and Gaël Richard.  Weakly supervised representation learning for unsynchronized audio-visual events.  In CVPRW, 2018.
  * [33] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach.  Multimodal explanations: Justifying decisions and pointing to the evidence.  In CVPR, 2018.
  * [34] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In AAAI, 2018.
  * [35] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap.  A simple neural network module for relational reasoning.  In NeurIPS, 2017.
  * [36] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, 2017.
  * [37] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [38] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [39] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In ICML, 2017.
  * [40] Yapeng Tian, Dingzeyu Li, and Chenliang Xu.  Unified multisensory perception: weakly-supervised audio-visual video parsing.  In ECCV, 2020.
  * [41] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu.  Audio-visual event localization in unconstrained videos.  In ECCV, 2018.
  * [42] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy.  Mead: A large-scale audio-visual dataset for emotional talking-face generation.  In ECCV, 2020.
  * [43] Weiyao Wang, Du Tran, and Matt Feiszli.  What makes training multi-modal classification networks hard?  In CVPR, 2020.
  * [44] Olivia Wiles, A. Sophia Koepke, and Andrew Zisserman.  X2face: A network for controlling face generation using images, audio, and pose codes.  In ECCV, 2018.
  * [45] Xi-Zhu Wu and Zhi-Hua Zhou.  A unified view of multi-label performance measures.  In ICML, 2017.
  * [46] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang.  Dual attention matching for audio-visual event localization.  In CVPR, 2019.
  * [47] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer.  Audiovisual slowfast networks for video recognition.  arXiv preprint arXiv:2001.08740, 2020.
  * [48] Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos.  Explainable object-induced action decision for autonomous vehicles.  In CVPR, 2020.
  * [49] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In ECCV, 2014.
  * [50] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In CVPR, 2019.
  * [51] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff.  Top-down neural attention by excitation backprop.  IJCV, 2018.
  * [52] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba.  The sound of motions.  In ICCV, 2019.
  * [53] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.  The sound of pixels.  In ECCV, 2018.
  * [54] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In CVPR, 2016.
  * [55] Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang.  Vision-infused deep audio inpainting.  In ICCV, 2019.

## Supplementary

We provide additional details about the datasets in Section A, more
implementation details of our proposed model STAN in Section B, and the
algorithm descriptions for the perturbation tests in Section C. Further, we
present additional qualitative results in Section D.

##  A Datasets

Tables A, B and C summarize the class label space of the three audio-visual
datasets: AVE, LLP and AVR, which include 28, 25 and 43 different classes
respectively. In Table D, we summarize the statistics of the training,
validation and test set.

2 Hyper Class | Class Names
---|---
Animal | Bark; Cat; Goat; Horse; Rodents, rats, mice
Instrument | Accordion; Acoustic guitar; Banjo; Flute;
Mandolin; Shofar; Ukulele; Violin, fiddle
Home | Clock; Frying (food); Toilet flush
Human | Baby cry, infant cry; Female speech, woman speaking;
Male speech, man speaking
Vehicle | Bus; Fixed-wing aircraft, airplane; Helicopter;
Motorcycle; Race car, auto racing; Train horn; Truck
Others | Chainsaw; Church bell
2 |
Table A: The audio-visual class space of the AVE dataset. 2 Hyper Class | Class Names
---|---
Animal | Cat; Chicken rooster; Dog
Instrument | Accordion; Acoustic guitar; Banjo; Cello; Violin, fiddle
Home | Blender; Frying (food); Lawn mower;
Telephone bell ringing; Vacuum cleaner
Human | Baby cry, infant cry; Baby laughter;
Cheering; Clapping; Singing; Speech
Vehicle | Car; Helicopter; Motorcycle
Others | Basketball bounce; Chainsaw; Fire alarm
2 |
Table B: The audio-visual class space of the LLP dataset. 2 Hyper Class | Class Names
---|---
Animal | Bark (Dog); Cat; Chicken rooster; Goat; Horse;
Rodents, rats, mice
Instrument | Accordion; Acoustic guitar; Banjo; Cello;
Flute; Mandolin; Shofar; Ukulele; Violin, fiddle
Home | Blender; Clock; Frying (food); Lawn mower;
Telephone bell ringing; Toilet flush; Vacuum cleaner
Human | Baby cry, infant cry; Baby laughter; Cheering; Clapping;
Female speech, woman speaking;
Male speech, man speaking; Singing; Speech
Vehicle | Car; Bus; Fixed-wing aircraft, airplane; Helicopter;
Motorcycle; Race car, auto racing; Train horn; Truck
Others | Basketball bounce; Chainsaw; Church bell; Fire alarm
2 |
Table C: The audio-visual class space of the AVR dataset. 2 Dataset | # Total | # Training | # Val | # Test
---|---|---|---|---
AVE | 4,143 | 3,339 | 402 | 402
LLP | 11,850 | 10,000 | 650 | 1,200
AVR | 15,993 | 13,339 | 1,602 | 1,052
2 |  |  |  |
Table D: Dataset statistics for the AVE, LLP, and AVR datasets.

##  B Implementation Details

As aforementioned in Section 4.1 in the main paper, the audio and visual
features are extracted from a VGGish network [21] pre-trained on YouTube-8M
[1] and a ResNet-152 pre-trained on ImageNet [10]. For each 10 second video
(i.e., $T=10$), audio features (with size $T\times D_{a}$) and visual feature
maps (with size $T\times H\times W\times D_{c}$) are extracted to encode the
audio and video segment per second. For pre-trained networks, we empirically
find that a pre-trained image encoder works better than a video encoder pre-
trained on a video dataset such as Kinetics [7]. This is likely caused by the
fact that most audio-visual video events do not contain a lot of motion
information for actions, while an ImageNet pre-trained network offers more
generic visual features than the visual features extracted from a pre-trained
video network. The audio and visual features, extracted from pre-trained
networks, further serve as the inputs to STAN. To understand the learned
attention maps, we show the space attention maps on image frames and the time
attention maps on raw audio data. Since the convolutions in the image encoder
reduce the spatial sizes of the visual features, we resize the space attention
maps to match the original image size. Similarly, the raw audio time attention
maps are of small resolution. Hence, we upsample the attention maps to a
larger size for better visual quality. PyTorch code will be released publicly.

##  C Algorithm for Perturbation Tests

We give an algorithm overview for a general perturbation test (as mentioned in
Section 4.3 in the main paper) in Algorithm A, which either adds input noise
($\eta\sim\mathcal{N}(0,\sigma^{2})$) to the relevant features (i.e. features
with higher attention values: $\mathbbm{1}_{[{A}\geq 0.5]}$) or the irrelevant
features (i.e. features with lower attention values:
$\mathbbm{1}_{[{A}<0.5]}$). The robustness under perturbation is measured as
the total variation distance (TVD) between the predictive distributions before
and after perturbation:
$\text{TVD}(\hat{y},\hat{y}^{p})=\frac{1}{2}\sum_{i=1}^{K}|\hat{y}-\hat{y}^{p}|$.
We also detail the perturbation test for STAN in Algorithm B, which adds
perturbation noise to the audio and visual features similar to Algorithm A.
Note that for each input, we average the TVD values over 100 random
perturbations.

1:$\bm{x}\leftarrow\mathcal{F}_{\textrm{feat}}(\textbf{X})$,
${A}\leftarrow\mathcal{F}_{\textrm{attn}}(\bm{x})$,
$\hat{y}\leftarrow\mathcal{F}_{\textrm{classify}}(\bm{x},{A})$

2:if perturb relevant features then

3: $\epsilon=\mathbbm{1}_{[{A}\geq 0.5]}$ $\triangleright$ mask on input of
high attention

4:else

5: $\epsilon=\mathbbm{1}_{[{A}<0.5]}$ $\triangleright$ mask on input of low
attention

6:end if

7:for $p\leftarrow 1\textrm{ to }100$ do

8: $\eta\sim\mathcal{N}(0,\sigma^{2})$ $\triangleright$ sample random input
noise

9: $\bm{x}^{p}\leftarrow\bm{x}+\eta*\epsilon$ $\triangleright$ add noise to
input features

10: $\hat{y}^{p}\leftarrow\mathcal{F}_{\textrm{classify}}(\bm{x}^{p},{A})$
$\triangleright$ forward perturbed input

11: $\Delta\hat{y}^{p}\leftarrow\text{TVD}(\hat{y}^{p},\hat{y})$
$\triangleright$ compute TVD

12:end for

13:$\Delta\hat{y}^{mean}\leftarrow\textrm{Mean}_{p}(\Delta\hat{y}^{p})$
$\triangleright$ average TVD

Algorithm A General perturbation test on input features.

1:$\bm{a}\leftarrow\mathcal{F}_{\textrm{featAudio}}(\textbf{X}_{a})$,
$\bm{v}\leftarrow\mathcal{F}_{\textrm{featVideo}}(\textbf{X}_{v})$

2:${A}^{\bm{s}}\leftarrow\mathcal{F}_{\text{space-branch}}(\bm{a},\bm{v})$,
${A}^{\bm{t}}\leftarrow\mathcal{F}_{\text{time-branch}}(\bm{a},\bm{v})$

3:${A}^{\bm{st}}\leftarrow{A}^{\bm{s}}\otimes{A}^{\bm{t}}$,
$\bm{x}^{\bm{st}}\leftarrow\mathcal{F}_{\text{space-time}}(\bm{a},\bm{v})$

4:$\hat{y}\leftarrow\mathcal{F}_{\textrm{classify}}(\bm{x}^{\bm{st}},{A}^{\bm{st}})$

5:if perturb relevant features then

6: $\epsilon=\mathbbm{1}_{[{A}\geq 0.5]}$ $\triangleright$ mask on input of
high attention

7:else

8: $\epsilon=\mathbbm{1}_{[{A}<0.5]}$ $\triangleright$ mask on input of low
attention

9:end if

10:for $p\leftarrow 1\textrm{ to }100$ do

11: $\eta\sim\mathcal{N}(0,\sigma^{2})$ $\triangleright$ sample random input
noise

12: $\bm{a}^{p}\leftarrow\bm{a}+\eta*\epsilon$,
$\bm{v}^{p}\leftarrow\bm{v}+\eta*\epsilon$ $\triangleright$ add noise

13: $\bm{x}^{\bm{st},p}\leftarrow\mathcal{F}_{\text{space-
time}}(\bm{a}^{p},\bm{v}^{p})$ $\triangleright$ forward propagation

14:
$\hat{y}^{p}\leftarrow\mathcal{F}_{\textrm{classify}}(\bm{x}^{\bm{st},p},{A}^{\bm{st}})$
$\triangleright$ forward perturbed input

15: $\Delta\hat{y}^{p}\leftarrow\text{TVD}(\hat{y}^{p},\hat{y})$
$\triangleright$ compute TVD

16:end for

17:$\Delta\hat{y}^{mean}\leftarrow\textrm{Mean}_{p}(\Delta\hat{y}^{p})$
$\triangleright$ average TVD

Algorithm B STAN perturbation test on input features.

##  D Additional Results

Figure A:  Perturbation tests on the AVE and LLP datasets for (a) relevant
features with high attention values, and (b) irrelevant features with low
attention values. Note: Increasing the perturbation noise level on features
with high / low attention values should lead to higher / lower TVD.  Figure B:
Visualizing attention maps with STAN, which shows space attention maps over
different image frames in the videos and the time attention maps on the co-
occurring audio waveform files. Dataset: AVR.

Additional Results on Perturbation Tests. Figure A shows the perturbation
tests on the AVE and LLP datasets. Similar to Figure 3 in the main paper that
shows perturbation tests on the AVR dataset, we can observe the same trends in
Figure A. As Figure A (1)-(a) and (2)-(a) show, when increasing the
perturbation noise level in relevant features, the TVD is increased among all
the methods. The changes in predictions (as quantified by TVD) are much larger
for STAN than the changes made by ABN and AV-ABN. As Figure A (1)-(b) and
(2)-(b) show, when increasing the perturbation noise level in irrelevant
features, the predictive changes in STAN are much smaller compared to the much
bigger changes made by ABN and AV-ABN. These observations again confirm that
the features with high attention values predicted by STAN correspond to the
regions essential for model predictions; while the features with low attention
values in STAN are less essential for making model predictions. In other
words, our model offers explainable attention which picks up on the features
of importance for making the model decision.

Additional Results on Attention Visualization. Similar to Figure 4 in the main
paper, we show more visual examples of the space and time attention maps
obtained from our STAN in Figure B. As can be seen, the space and time
attention maps work in different ways to pick up the discriminative visual and
audio cues over space and time. As shown in the main paper, our STAN not only
offers meaningful attention-based explanations, but also provides better model
performance in audio-visual video event recognition.


# A Decentralized Resource Management System Proposal For Disasters: NGO-RMSD
(STK-AKYS)

1st Arzu Özkan  Department of English Language & Literature
Muğla Sıtkı Koçman University
Muğla, Turkey
<EMAIL_ADDRESS>2nd Umutcan Korkmaz  Department of Computer Engineering
Muğla Sıtkı Koçman University
Muğla, Turkey
<EMAIL_ADDRESS>3rd Cemal Dak  Department of Computer Engineering
Muğla Sıtkı Koçman University
Muğla, Turkey
<EMAIL_ADDRESS>4th Enis Karaarslan  Department of Computer Engineering
Muğla Sıtkı Koçman University
Muğla, Turkey
<EMAIL_ADDRESS>
###### Abstract

Disaster and emergency management are under the responsibility of many
organizations and there are serious coordination problems in post-disaster
crisis management. This paper proposes a decentralized non-governmental
organization resource management system for disasters (NGO-RMSD). This system
is based on blockchain technology and it will enable the non-governmental
organizations (NGO) and public institutions to manage and coordinate the
resources in a trusted environment in the case of disasters. A proof of
concept implementation is developed by using the Quorum blockchain framework
which is more energy-efficient than crypto currency-based blockchain
solutions. Smart contracts are developed for the autonomous working of the
system. These smart contacts are used for the verification of the needs of the
one who is in need, delivering resources to the right people, and identifying
the urgent needs. The system aims to reach more disaster victims in a more
timely manner. NGO-RMSD is designed according to the needs of the NGOs in the
field. The application is shared with the free software license and further
development with the community is aimed.

###### Index Terms:

Disaster, Non-Governmental Organizations, Crisis management, Organization,
Source management, Coordination, Blockchain

##  I Introduction

The disaster management processes of the local governments are not sufficient
even in developed countries such as Japan and America [1]. Disaster and
emergency management are under the responsibility of many organizations, there
are critical coordination problems in post-disaster crisis management.

Non-governmental organizations and volunteers have significant roles and
contributions in the case of disasters. Today, NGOs intensely work on the
following about the disasters:

  * •

Support the preparation of communities for the disasters

  * •

Work on the reduction of pre-disaster risks and post-disaster losses,

  * •

Work on short-term intervention, and long-term improvement efforts.

Without the help of voluntary organizations, it does not seem possible for
local governments and the state to meet the needs of disaster victims. There
is a need for Joint disaster management that is possible by assigning the
roles of voluntary organizations at all stages of the disasters and
establishing coordination between parties [2].

The duty of NGOs is to support local governments and the public, as needed
especially in times of crisis within the framework of their chosen mission
[3]. As an example; Ahbap NGO took an active role in the post-disaster process
after the 6.8 magnitude earthquake in Elazig Sivrice/TURKEY, which happened in
January 2020. One of our team members, who was an active Ahbap volunteer in
this aid team, observed the following:

  * •

Non-governmental organizations working in the disaster area act independently
of each other

  * •

There is a deficiency and problem to reach the real needy people

  * •

The resources can not be used efficiently.

When NGOs are analyzed, it is seen that there are a variety of NGOs which have
different missions and visions. Each has various resources and strengths. The
most significant deficiency that we have observed in the post-disaster
management is; the coordination problem between NGOs and the government. This
results in the efficient usage of the capabilities. This situation causes
deficiencies in resource management and slows down the post-disaster recovery
efforts.

In the next section, the fundamentals will be given. Then the related works
will be given in Section 3. In Section 4, details of the system proposal are
given. The proof of concept implementation of NGO-RMSD system is given in
Section 5. Conclusion and future works will be given in the last section.

##  II Fundamentals

###  II-A Blockchain

Blockchain is a distributed ledger technology that ensures trust between the
users without an intermediary. This is accomplished by keeping track of
records linked to each other[4]. The records are called transactions and
transactions are bundled into blocks at the nodes (the computers that are in
charge of the blockchain system). These records are stored in immutable record
storage called the ledger. Since the blockchain system is distributed, there
is a need for a way that the nodes can agree upon the transactions, blocks,
etc. Consensus protocols are used in blockchain networks to provide that
functionality to the distributed system. The consensus protocol in the
blockchain is chosen according to the type of the blockchain.

Different types of blockchain networks exist. Their types mainly depend on who
can join as a node and participate in the network and also the visibility of
the blockchain records. Public/private types are according to the visibility
of the records. In public blockchain networks; all the transactions in the
network are visible by anyone on the internet. Private networks only allow
authorized users to reach the records. Permissioned/Permissionless type
determines who can join and participate in the network. Anyone can join
permissionless networks. In Permissioned networks, nodes need to be allowed by
an authority of the network to join in and participate. Bitcoin [4] is an
example of a Public/Permissionless network while Quorum [5] and R3 Corda [6]
are examples of Private/Permissioned networks.

###  II-B Resource Management Need

According to The Republic of Turkey Ministry of the Interior Disaster and
Emergency Management Presidency Earthquake Office Management’s general
earthquake statistics report by years in Turkey; Earthquake rates in Turkey
are increasing every year. In the last century, 192 damaging earthquakes
occurred in Turkey, one hundred thousand citizens died, and more than 650
thousand houses were demolished or damaged heavily [7].

After our researches and interviews, İt’s concluded that there is a lack of
coordination and resources management in post-crisis management in Turkey.
These problems are in the following areas:

  * •

Coordination between NGOs

  * •

Management of human resources

  * •

Resource management

    * –

Logistics

    * –

Storage

    * –

Distribution

  * •

Post-disaster necessity tracking

    * –

Verifying the necessity

    * –

Support tracking and management

    * –

Tracking the provided support

    * –

Preventing abuse of resources and supports

  * •

Protection of trust in NGOs

###  II-C The need for coordination Between NGO’s

NGOs, which have voluntary activities in disaster situations, have access to
different resources such as aid and human resources. It is very important to
provide coordination between NGOs to manage these resources correctly. The
collaboration of NGOs will enhance the post-disaster recovery.

Autonomous systems can be used to manage these resources effectively and
promptly. In this context, utilizing the technology’s facilities can have the
potential to simplify resource management and ensure coordination.

##  III RELATED WORKS

There are few studies on disaster management. The use of decision support
systems for this purpose is discussed in [8]. Usage of digital twins is given
in a recent study [9]. Interestingly, there are comprehensive open-source
solutions such as SAHANA EDEN (https://sahanafoundation.org/) that have
several functions. Interestingly, Sahane Eden or similar solutions are not
that widely used. Our interviews with experts in the field and NGOs revealed
some possible reasons for that. It is mainly because these solutions are not
built with people in the field. Also, there is always a need for transparency
of the transactions and a trust issue in the used systems.

Blockchain technology can be used to overcome these needs. Blockchain
technology can remove intermediaries and ensure trust. Blockchain technology
was mostly used for cryptocurrency implementations and is famous for high
energy usage. However, with the advance in enterprise blockchain frameworks,
it is possible to obtain a trusted environment without high energy usage.
Different fields of using blockchain are possible such as supply chain,
sharing medical information, AI marketplace, DAO machines economy, etc. Also,
blockchain can be used for social impact [10].

There are a few decentralized system proposals in the literature regarding
this topic. These are mostly for collecting monetary aid [11] or
disaster/refugee aid scenarios [12]. An interesting study [13] is about
integrating IoT and blockchain in the supply chains of various units to
increase the efficiency and effectiveness of humanitarian aid. These papers
mostly do not contain technical details and also do not involve NGOs. A very
recent study [14] gives a theoretical model which questions the applicability
of a trusted supply chain for humanitarian aid. We are not aware of a resource
management system for disasters that focuses on NGO collaboration. We focused
on generating a prototype to make a social impact.

##  IV SYSTEM PROPOSAL

The proposed NGO-RMSD is shown in Figure 1, nodes to be found in the
blockchain network will be added by non-governmental organizations within the
system by using blockchain technology. All transactions performed on the
system for each node will be saved encrypted in a tamper-proof way. Actors
within the distributed application to be created with smart contracts to be
deployed on the blockchain network are as follows;

  * •

Non-Governmental Organizations: Voluntary platforms that are approved by the
ministry, supporting local governments and ministries in case of disasters.

  * •

People In Need: Materially and morally damaged victims due to disasters.

  * •

Supporter: People, institutions, or organizations that are wishing to provide
in-kind and cash support to disaster victims.

Figure 1: Basic design of the NGO-RMSD System

###  IV-A Architecture

The overall system consists of two parts that are given in Figure 2. A Quorum
network for the nodes of the NGO’s and the web service. Those two parts will
communicate via Web3JS library from node to JS Back-End. Front-End includes
HTML and CSS code to present users from varying platforms with a web page
interface. Front-End and Back-End components of the web service will
communicate via the ReactJS framework.

To initialize the system, our research group will install the initial
(startup) nodes of the network. There is no defined minimum node number, the
more the better. The first node account (of the first node) added to the
system will have the role “admin”. After the initialization of the system, the
account with the admin role will distribute the admin roles to the regarding
accounts (pre-defined accounts on the pre-defined nodes) according to the
protocol of the community.

At least one node for each NGO is required in the blockchain network to
represent the institution. As the system will run in a docker image, there is
no need to reserve the node machine only for this purpose. This machine may
also be used for general computing purposes by the NGO.

The volunteers or staff of the NGOs are going to create their accounts via
those nodes. These accounts are needed to check the support offers or requests
and take action.

RAFT is chosen as the consensus protocol of the proposed system. This protocol
uses the “leader-follower” role principle where the leader node is selected
internally in the network. The leader and follower nodes can be thought of as
minter and verifier nodes in Ethereum. The leader node mines a block to the
network while follower nodes verify it. Every node has the chance to be a
leader, they have to mine a block for this. Every node has an enode ID that
has the raftport parameter as a query string. This parameter will be used to
define the communication port that will be used for communicating with the
other nodes. These definitions are grouped in the static-nodes.json file.
Every node should have the same static-nodes.json file in its filesystem.

The addition of another node or another NGO to the system is as follows:

  1. 1.

The candidate NGO node installs the required Docker image.

  2. 2.

A node can not be initialized as the leader, there is an election for it in
RAFT protocol. So every node is added as a new follower node.

  3. 3.

The required files with the candidate node’s information (static-nodes.json,
node keys, etc. ) should be set. The details will be given in the project
Github page.

  4. 4.

The static-nodes.json files of all nodes in the network need to be updated.

  5. 5.

One node in the network has to run eth.addPeer() command from its geth RPC
console with the enode ID of the candidate node. Then, a new node is added and
running functionally in the network.

We are currently working on automation for this process and the automatization
process will be given on the project Github page.

Figure 2: NGO-RMSD System Architecture

###  IV-B Smart Contracts

Although the blockchain concept is said to be decentralized, not all solutions
have to be like that. Private/Permissioned networks like Quorum are not fully
decentralized. The system gives privileges to some authorities to add nodes or
users.

The proposed system aims to form trust and transparency. NGOs have the ADMIN
role which has the privileges that can set the user’s roles as CHECKER or
CREATOR. Also, NGOs with the ADMIN privileges add the new NGOs to the system.
Users are assigned roles in the smart contract structure. These roles are as
follows:

  1. 1.

Admin: Admin is the system administrator.

  2. 2.

Checker: The role of the checker belongs to NGOs and is used to check whether
the proposed support or request is correct.

  3. 3.

Creator: The role of the creator is open to everyone and it is required to
make a request or support offered.

Smart contracts are written for the following functions:

  1. 1.

Role Assignment and Listing Functions: These are called by the system
administrator and they aim to assign the role to the user with the setUser
function and to display the role of the user with the getUserAuth function.

  2. 2.

Authority-Role Control Modifier Functions: These functions are called by the
organizer to compare the hash of the user’s role with the hash of the role
assigned to the user.

  3. 3.

Requirement Creation Function: This function enables users in the role of
“Creator” to create a new need with the data that they enter. Need; is created
with features such as type, and amount. The created need is put in the need
list and the label ”waiting for confirmation” is added.

  4. 4.

Support Creation Function: This function enables users in the role of
”Creator” to create new support with the data that they enter. Support; is
created with features such as type, amount, and shipping type. The created
support is added to the support list and the label ”waiting for approval” is
added.

  5. 5.

Approve Functions: These functions enable users with the role of ”Checker” to
confirm the needs and supports labeled ”waiting for approval”. While the
approveNeed function confirms the pending needs, the approveSupport function
confirms the pending supports. Once the need is approved, the label changes to
”approved” and not ”waiting for approval”. Once the support has been approved,
it is added to the approved support list.

  6. 6.

Request-Support Listing Functions: These functions are used to display all
needs and supports. While the showSupport function listed the known support,
the showSupports function shows all the supports. The showAllApprovedSupports
function lists approved supports. The showNeedOffers function lists all the
needs records while the showNeed function lists the known need records.

The last 2 functions (showNeedStatus and showSupportStatus) could be run by
users with the role of ”Checker”, and these are in charge to list the known
requests and supports confirmation status.

###  IV-C System Application Flow

Application flow of the system as shown in Figure 3, the user chooses their
application type. That application is sent to the Quorum node and stored as an
unapproved application. A transaction is sent to the network and the ledgers
of nodes get updated. If there are any unapproved applications in the network,
an NGO Staff checks the application by communicating with the user that
applied. If the application is valid, in that case, the requirement is real or
support is accurate; the information of the applicant user is written into the
local database, and the application gets approved. After the application
validation, the application is stored as an approved application. Then,
another transaction is sent to the network, and the ledgers of nodes get
updated again. If there are no unapproved applications, the application flow
process is finished.

Figure 3: System Activity Diagram

##  V IMPLEMENTATION

The default system works initially as three nodes with two NGOs. To
participate in the network and the system, each NGO must add at least one
node. The scenario with five nodes was executed. The proof of concept
implementation of NGO-RMSD is implemented on the Quorum framework. The Quorum
framework, which provides corporate solutions, is used as blockchain
technology. It offers a private/permissioned blockchain structure with low
energy consumption [5]. Docker container technology is used to integrate this
system into every platform. The system does not need special hardware and does
not need high power. Ordinary virtual machines (single CPU, 4 GB ram, 256 GB
disk) are used for the nodes.

Web3JS is used for the communication between the Quorum network and Web
interface backend. Solidity language is used to write smart contracts. RAFT
protocol is used as the consensus protocol as it can support more nodes than
Istanbul Byzantine Fault Tolerance (IBFT). NodeJS is used in the backend
because it provides the asynchronous functionality that this application
needs.

The privacy of personal data is prioritized in the design. Turkish Law on the
Protection of Personal Data (KVKK) [15] and the EU General Data Protection
Rules (GDPR) [16] requirements are taken into consideration. Since it is not
possible to delete the data on the blockchain, the storage of personal data in
the blockchain constitutes a situation that is not suitable for personal data
protection laws. In our design, personal data will never be stored on the
blockchain. Personal data will be stored in a private database instead, which
could be deleted at any time by NGOs. In a privacy breach case, it will be the
NGO’s responsibility if it didn’t take the required actions to apply to the
necessary legal processes. However, Non-interactive zero-knowledge proof
(NIZK) based autonomous codes can be developed to ensure privacy [17], which
is out of the scope of this study.

In this project, open-source and free software licensed software is used, and
dependency on any company or institution is minimized. This project is also
published on the Github page.

The prototype is created appropriately with the scenario specified.
Requirement Creation, Requirement Confirmation, Support Creation, Support
Confirmation, Support, and Requirement Listing processes are directed to smart
contracts running on nodes in the Quorum network through the web interface.

###  V-A Smart Contracts

The smart contract is open to improvement and updating and debugging process
continues. The recent version of the contract structure and functions are
given on the public Github page https://github.com/MSKU-BcRG/akys.

###  V-B GUI Design

Prototype web interfaces of the system are designed. Disaster victims can make
requests or give support through forms that are shown in Figure 4. All
operations in the system are stored on the blockchain as transactions. All the
users can track the status of the transactions through the web interface that
is given in Figure 5.

Figure 4: Request and Support Forms Figure 5: Interface to track approved
requests

##  VI ACKNOWLEDGEMENT

This study won the second prize in the TUBİTAK 2242 University Students
Research Project Competition in the field of “Information and Communication
Technologies” in 2020.

##  VII RESULTS AND CONCLUSION

Decentralized resource management and NGO coordination system are proposed for
disasters in this study. The proposed NGO-RMSD system will enable NGOs and
public institutions to act in coordination in the case of disasters. A trusted
system is designed where all transactions are transparent. Proof of concept
implementation of the proposed system NGO-RMSD had promising results. Smart
contracts which allow the autonomous working of the system is written and
tested on the live platform.

Coordination of NGOs will be provided with this system which will lead to
improvements in workforce and resource management. The proposed system will
enable NGOs to reach more people in need. Also, it will ensure urgent needs
could be determined to resolve as soon as possible. Users’ personal data’s
unauthorized sharing will be prevented and made sure that the data-sharing is
legal.

The smart contracts and project details of the system are shared on the
project’s Github page with a free software license. We are in contact with
related NGOs and further development with the community will continue. The
live tests of the proposed system is being implemented in the DS4H blockchain
research network. Dynamically establishing new nodes and the automation for
the authentication will be given on the project’s Github page. Details of the
project can be reached at http://wiki.netseclab.mu.edu.tr/index.php?title=STK-
AKYS.

In future works, it is aimed to ensure that the system is easily integrated
into new laws and regulations. The personal data will be protected
autonomously with new smart contracts. It is also aimed to create a more
efficient system by integrating the data collected in the system with the data
collected from the territory. The collected data will be processed with
artificial intelligence. The requirement and support requests will be matched
by using natural language processing and machine learning. A dataset for
machine learning processes will be formed with the collected data on the live
system. Thus, the data to be entered into the system could be cleared by using
Natural Language Processing (NLP) and a classification algorithm could be
applied with keywords. It will be possible to apply the need and support
matching infrastructure through this classification algorithm. In all these
processes, preliminary studies will be done to receive extra information from
the Twitter environment with sustained queries to be made with these keywords.

## References

  * [1] Uluğ, A. (2009). Nasıl bir afet yönetimi [What kind of disaster management], TMMOB İzmir Kent Sempozyumu, İzmir, 1-18.
  * [2] Mustafa, K. A. Y. A. Afet Yönetiminde Sivil Toplum Kuruluşları Ve Gönüllülük İşlevi. [Civil Society Organizations And Volunteering Function In Disaster Management].
  * [3] Birleşmiş Milletler Gönüllüleri. (2011). Türkiye’de gönüllülük [Volunteering in Turkey] (çev. Bordo Tercüme Bürosu ve E. Erdem).
  * [4] Nakamoto, Satoshi. Bitcoin: A peer-to-peer electronic cash system. Manubot, 2019.
  * [5] Baliga, A., Subhod, I., Kamat, P., & Chatterjee, S. (2018). Performance evaluation of the quorum blockchain platform. arXiv preprint arXiv:1809.03421.
  * [6] Mohanty, Debajani. R3 Corda for Architects and Developers: With Case Studies in Finance, Insurance, Healthcare, Travel, Telecom, and Agriculture. Apress, pp. 49-50, 2019
  * [7] Genel istatistikler, [General statistics], AFAD, https://deprem.afad.gov.tr/genelistatistikler
  * [8] Yaqoob, Lubna, N. Ahmed Khan, and Fazli Subhan. ”An overview of existing decision support systems for disasters management.” Sci Int (Lahore) 26 (2014): 1765-76.
  * [9] Fan, Chao, et al. ”Disaster City Digital Twin: A vision for integrating artificial and human intelligence for disaster management.” International Journal of Information Management (2019): 102049.
  * [10] Al-Saqaf, Walid, and Nicolas Seidler. ”Blockchain technology for social impact: opportunities and challenges ahead.” Journal of Cyber Policy 2.3 (2017): 338-354.
  * [11] Nor, Rizal Mohd, et al. ”Blockchain sadaqa mechanism for disaster aid crowd funding.” Proceedings of the 6th International Conference on Computing and Informatics: Embracing Eco-Friendly Computing, Kuala Lumpur. 2017.
  * [12] McIsaac, Joseph, et al. ”Blockchain Technology for Disaster and Refugee Relief Operations.” Prehospital and Disaster Medicine 34.s1 (2019): s106-s106.
  * [13] Aranda, Daniel Arias, Luis Miguel Molina Fernández, and Vladimir Stantchev. ”Integration of Internet of Things (IoT) and Blockchain to increase humanitarian aid supply chains performance.” 2019 5th International Conference on Transportation Information and Safety (ICTIS). IEEE, 2019.
  * [14] Dubey, Rameshwar, et al. ”Blockchain technology for enhancing swift-trust, collaboration and resilience within a humanitarian supply chain setting.” International Journal of Production Research (2020): 1-18.
  * [15] [dated 24.3.2016 numbered 6698 Personal Data Protection Law] https://www.mevzuat.gov.tr/MevzuatMetin/1.5.6698.pdf, Last access date : 2 April 2020
  * [16] Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free
  * [17] Feng, Q., He, D., Zeadally, S., Khan, M. K., & Kumar, N., 2019. A survey on privacy protection in blockchain system. Journal of Network and Computer Applications, 126, 45-58.