# Let’s Ask Again: Refine Network for Automatic Question Generation

Preksha Nema†‡ Akash Kumar Mohankumar†11footnotemark: 1 Mitesh M. Khapra†‡
Balaji Vasan Srinivasan∙ Balaraman Ravindran†‡
†IIT Madras, India ∙Adobe Research
‡ Robert Bosch Center for Data Science and Artificial Intelligence, IIT Madras
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>* The first two authors have
contributed equally to this work.

###### Abstract

In this work, we focus on the task of Automatic Question Generation (AQG)
where given a passage and an answer the task is to generate the corresponding
question. It is desired that the generated question should be (i)
grammatically correct (ii) answerable from the passage and (iii) specific to
the given answer. An analysis of existing AQG models shows that they produce
questions which do not adhere to one or more of the above-mentioned qualities.
In particular, the generated questions look like an incomplete draft of the
desired question with a clear scope for refinement. To alleviate this
shortcoming, we propose a method which tries to mimic the human process of
generating questions by first creating an initial draft and then refining it.
More specifically, we propose Refine Network (RefNet) which contains two
decoders. The second decoder uses a dual attention network which pays
attention to both (i) the original passage and (ii) the question (initial
draft) generated by the first decoder. In effect, it refines the question
generated by the first decoder, thereby making it more correct and complete.
We evaluate RefNet on three datasets, viz., SQuAD, HOTPOT-QA, and DROP, and
show that it outperforms existing state-of-the-art methods by 7-16% on all of
these datasets. Lastly, we show that we can improve the quality of the second
decoder on specific metrics, such as, fluency and answerability by explicitly
rewarding revisions that improve on the corresponding metric during training.
The code has been made publicly available
111https://github.com/PrekshaNema25/RefNet-QG.

##  1 Introduction

Passage 1: Liberated by Napoleon’s army in 1806, Warsaw was made the capital
of the newly created Duchy of Warsaw.
---
Generated Questions
Baseline What was the capital of the newly duchy of Warsaw? RefNet Who
liberated Warsaw in 1806? Reward-RefNet Whose army liberated Warsaw in 1806?
Passage 2: To fix carbon dioxide into sugar molecules in the process of
photosynthesis, chloroplasts use an enzyme called rubisco
Generated Questions
Baseline What does chloroplasts use? RefNet What does chloroplasts use to fix
carbon dioxide into sugar molecules? Reward-RefNet What do chloroplasts use to
fix carbon dioxide into sugar molecules?

Table 1: Samples of generated questions from Baseline, RefNet and Reward-
RefNet model on the SQuAD dataset. Answers are shown in blue

.

Over the past few years, there has been a growing interest in Automatic
Question Generation (AQG) from text - the task of generating a question from a
passage and optionally an answer. AQG is used in curating Question Answering
datasets, enhancing user experience in conversational AI systems Shum et al.
(2018) and for creating educational materials Heilman and Smith (2010). For
the above applications, it is essential that the questions are (i)
grammatically correct (ii) answerable from the passage and (iii) specific to
the answer. Existing approaches focus on encoding the passage, the answer and
the relationship between them using complex functions and then generate the
question in one single pass. However, by carefully analysing the generated
questions, we observe that these approaches tend to miss one or more of the
important aspects of the question. For instance, in Table 1, the question
generated by the single-pass baseline model for the first passage is
grammatically correct but is not specific to the answer. In the second
example, the generated question is both syntactically incorrect and
incomplete.

The above examples indicate that there is clear scope of improving the general
quality of the questions. Additionally, the quality can be specifically
improved in terms of aspects like: fluency (Example 2) and answerability
(Example 1). One way to approach this is by re-visiting the passage and answer
with the aim to refine the initial draft by generating a better question in
the second pass and then improving it with respect to a certain aspect. We can
draw a comparison between this process and how humans tend to write a rough
initial draft first and then refine it over multiple passes, where the later
revisions focus on improving the draft aiming at certain aspects like fluency
or completeness. With this motivation, we propose Refine Network (RefNet),
which examines the initially generated question and performs a second pass to
generate a revised question. Furthermore, we propose Reward-RefNet which uses
explicit reward signals to achieve refinement focused on specific properties
of the question such as fluency and answerability.

Our RefNet is a seq2seq based model that comprises of two decoders:
Preliminary and Refinement Decoder. The Refinement Decoder takes the initial
draft of the question generated by the Preliminary decoder as an input along
with passage and answer, and generates the refined question by attending onto
both the passage and the initial draft using a Dual Attention Network. The
proposed dual attention aids RefNet to generate the final question by
revisiting the appropriate parts of the input passage and initial draft. From
Table 1, we can infer that our RefNet model is able to generate better
questions in the second pass by fixing the errors in the initial draft. Our
Reward-RefNet model uses REINFORCE with a baseline algorithm to explicitly
reward the Refinement Decoder for generating a better question as compared to
the Preliminary Decoder based on certain desired parameters like fluency and
answerability. This leads to more answerable (see Reward-RefNet example for
passage $1$ in Table 1) and fluent (see Reward-RefNet example for passage $2$
in Table 1) questions as compared to vanilla RefNet model.

Our experiments show that the proposed RefNet model outperforms existing
state-of-the-art models on the SQuAD dataset by $12.3$% and $3.7$% (on BLEU)
given the relevant sentence and passage respectively. We also achieve state-
of-the-art results on HOTPOT-QA and DROP datasets with an improvement of
$7.57$% and $15.25$% respectively over the single-decoder baseline (on BLEU).
Our human evaluations further validate these results. We further analyze and
explain the impact of including the Refinement Decoder by examining the
interaction between both the decoders. Interestingly, we observe that the
inclusion of the Refinement Decoder boosts the quality of the questions
generated by the initial decoder also. Lastly, our human evaluation of the
questions generated by Reward-RefNet corroborate empirical results, i.e., it
improves the question w.r.t. to fluency and answerability as compared to
RefNet questions.

Figure 1: Our RefNet model with Preliminary and Refinement Decoder.

##  2 Refine Networks (RefNet) Model

In this section, we discuss various components of our proposed model as shown
in Figure 1. For a given passage $\mathbf{P}=\\{w^{p}_{1},\dots,w^{p}_{m}\\}$
of length $m$ and answer $\mathbf{A}=\\{w^{a}_{1},\dots,w^{a}_{n}\\}$ of
length $n$, we first obtain answer-aware latent representation,
$\mathbf{U}=\\{\tilde{\mathbf{h}}^{p}_{1},\dots,\tilde{\mathbf{h}}^{p}_{m}\\}$,
for every word of the passage and an answer representation $\mathbf{h}^{a}$
(as described in Section 2.1). We then generate an initial draft
$\mathbf{\tilde{Q}}=$ $\\{\tilde{q}_{1},\dots,\tilde{q}_{T}\\}$ by computing
$\tilde{q}_{t}$ as

$\displaystyle\tilde{q}_{t}$
$\displaystyle=\arg\max_{\tilde{q}}\prod_{t=1}^{l}\tilde{p}(\tilde{\textrm{q}}_{t}|\tilde{\textrm{q}}_{t-1},\dots,\tilde{\textrm{q}}_{1},\mathbf{U},\mathbf{h}^{a})$

Here $\tilde{p}(.)$ is a probability distribution modeled using the
Preliminary Decoder. We then refine the initial draft $\mathbf{\tilde{Q}}$
using the Refinement Decoder to obtain the refined draft
$\mathbf{Q}=\\{{q}_{1},\dots q_{T}\\}$:

$\displaystyle q_{t}$
$\displaystyle=\arg\max_{q}\prod_{t=1}^{l}p(\textrm{q}_{t}|\textrm{q}_{t-1},\dots,\textrm{q}_{1},\mathbf{\tilde{Q}},\mathbf{U},\mathbf{h}^{a})$

We then use explicit rewards to enforce refinement on a desired metric, such
as, fluency or answerability through our Reward-RefNet model. In the following
sub-sections, we describe the passage encoder, preliminary and refinement
decoders and our reward mechanism.

###  2.1 Passage and Answer Encoder

We use a 3 layered encoder consisting of: (i) Embedding, (ii) Contextual and
(iii) Passage-Answer Fusion layers as described below. To capture interaction
between passage and answer, we ensure that the passage and answer
representations are fused together at every layer.

Embedding Layer:

In this layer, we compute a $d$-dimensional embedding for every word in the
passage and the answer. This embedding is obtained by concatenating the word’s
Glove embedding Pennington et al. (2014) with its character based embedding as
discussed in Seo et al. (2016). Additionally, for passage words, we also
compute a positional embedding based on the relative position of the word
w.r.t. the answer span as described in Zhao et al. (2018). For every passage
word, this positional embedding is also concatenated to the word and character
based embeddings. We discuss the impact of character embeddings and answer
tagging in Appendix A. In the subsequent sections, we will refer to embedding
of the $i$-th passage word $w^{p}_{i}$ as $\mathbf{e(w^{p}_{i})}$ and the
$j$-th answer word $w^{a}_{j}$ as $\mathbf{e(w^{a}_{j})}$.

Contextual Layer:

In this layer, we compute a contextualized representation for every word in
the passage by passing the word embeddings (as computed above) through a
bidirectional-LSTM Hochreiter and Schmidhuber (1997):

$\displaystyle\overrightarrow{\mathbf{h}^{p}_{t}}$
$\displaystyle=\text{LSTM}(\mathbf{e(w^{p}_{t})},\overrightarrow{\mathbf{h}^{p}}_{t-1})\;\;\forall
t\in[1,m]$

where $\overrightarrow{\mathbf{h}^{p}_{t}}$ is the hidden state of the forward
LSTM at time $t$. We then concatenate the forward and backward hidden states
as
$\mathbf{h}^{p}_{t}=[\overrightarrow{\mathbf{h}^{p}_{t}};\overleftarrow{\mathbf{h}^{p}_{t}}]$.

The answer could correspond to a span in the passage. Let $j+1$ and $j+n$ be
the start and end indices of the answer span in the passage respectively. We
can thus refer to $\\{\mathbf{h}^{p}_{j+1},\dots,\mathbf{h}^{p}_{j+n}\\}$ as
the representation of the answer words in the context of the passage. We then
obtain contextualized representations for the $n$ answer words by passing them
through LSTM as follows:

$\displaystyle\overrightarrow{\mathbf{h}^{a}_{t}}$
$\displaystyle=\text{LSTM}([\mathbf{e(w^{a}_{t})},\mathbf{h}^{p}_{j+t}],\overrightarrow{\mathbf{h}^{a}}_{t-1})\;\;\forall
t\in[1,n]$

The final state
$\mathbf{h}^{a}=[\overrightarrow{\mathbf{h}^{a}_{n}};\overleftarrow{\mathbf{h}^{a}_{n}}]$
of this Bi-LSTM is used as the answer representation in the subsequent stages.
When the answer is not present in the passage, only $\mathbf{e(w^{a}_{t})}$ is
passed to the LSTM.

Passage-Answer Fusion Layer:

In this layer, we refine the representations of the passage words based on the
answer representation as follows:

$\displaystyle\tilde{\mathbf{h}}^{p}_{i}=\tanh{(\mathbf{W}_{u}\left[\,\mathbf{h}^{p}_{i};\,\mathbf{h}^{a};\,\mathbf{h}^{p}_{i}\odot\mathbf{h}^{a}\right])}\;\;\forall
i\in[1,m]$

Here $\mathbf{W}_{u}\in\mathbb{R}^{l\times 3l}$. $l$ is the hidden size of
LSTM. This is similar to how Seo et al. (2016) capture interactions between
passage and question for QA. We use
$\mathbf{U}=\\{\tilde{\mathbf{h}}^{p}_{1},\dots,\tilde{\mathbf{h}}^{p}_{m}\\}$
as the fused passage-answer representation which is then used by our
decoder(s) to generate the question $\mathbf{Q}$.

###  2.2 Preliminary and Refinement Decoders

As discussed earlier, RefNet has two decoders, viz., Preliminary Decoder and
Refinement Decoder, as described below:

Preliminary Decoder:

This decoder generates an initial draft of the question, one word at a time,
using an LSTM as follows:

$\displaystyle\tilde{\mathbf{h}}^{d}_{t}$
$\displaystyle=\text{LSTM}([\mathbf{e_{w}(\tilde{q}_{t-1})};\tilde{\mathbf{c}}_{t-1};\mathbf{h}^{a}],\tilde{\mathbf{h}}^{d}_{t-1})$
$\displaystyle\tilde{\mathbf{c}}_{t}$
$\displaystyle=\sum_{i=1}^{m}\alpha^{i}_{t}\tilde{\mathbf{h}}_{i}^{p}$ (1)

Here $\tilde{\mathbf{h}}^{d}_{t}$ is the hidden state at time $t$,
$\mathbf{h}^{a}$ is the answer representation as computed above,
$\tilde{\mathbf{c}}_{t-1}$ is an attention weighted sum of the contextualized
passage word representations, $\alpha^{i}_{t}$ are parameterized and
normalized attention weights Bahdanau et al. (2014). Let’s call this attention
network as $\mathbf{A}_{1}$. $\mathbf{e_{w}(\tilde{q_{t}})}$ is the embedding
of the word $\tilde{q_{t}}$. We obtain $\tilde{q_{t}}$ as:

$\displaystyle\tilde{q}_{t}=\text{arg}\max_{\tilde{q}}\;(\text{softmax}(\mathbf{W}_{o}[\mathbf{W}_{c}[\tilde{\mathbf{h}}^{d}_{t};\tilde{\mathbf{c}}_{t}])),$
(2)

where $\mathbf{W}_{c}$ is a $\mathbb{R}^{l\times 2l}$ matrix and
$\mathbf{W}_{o}$ is the output matrix which projects the final representation
to $\mathbb{R}^{V}$ where $V$ is the vocabulary size.

Refinement Decoder: Once the preliminary decoder generates the entire
question, the refinement decoder uses it to generate an updated version of the
question using a Dual Attention Network. It first computes an attention
weighted sum of the embeddings of the words generated by the first decoder as:

$\displaystyle\mathbf{g}_{t}=\sum_{i=1}^{T}\beta^{t}_{i}\mathbf{e_{w}(\tilde{q_{i}})}$

where $\beta^{t}_{i}$ are parameterized and normalized attention weights
computed by attention network $\mathbf{A}_{3}$. Since the initial draft could
be erroneous or incomplete, we obtain additional information from the passage
instead of only relying on the output of the first decoder. We do so by
computing a context vector $\mathbf{c}_{t}$ as

$\displaystyle\mathbf{c}_{t}=\sum_{i=1}^{m}\gamma^{t}_{i}\tilde{\mathbf{h}}_{i}^{p}$

where $\gamma^{t}_{i}$ are parameterized and normalized attention weights
computed by attention network $\mathbf{A}_{3}$. The hidden state of the
refinement decoder at time $t$ is computed as follows:

$\displaystyle\mathbf{h}^{d}_{t}=\text{LSTM}([\mathbf{e(q_{t-1})};\mathbf{c}_{t-1};\mathbf{g}_{t-1};\mathbf{h^{a}}],\mathbf{h}^{d}_{t-1})$

Finally, ${q}_{t}$ is predicted using

$\displaystyle{q}_{t}=\text{arg}\max_{q}(\text{softmax}({\mathbf{W}}_{o}[{\mathbf{W}^{\prime}}_{c}[{\mathbf{h}}^{d}_{t};{\mathbf{c}}_{t-1};\mathbf{g}_{t-1}]))$

where $\mathbf{W}^{\prime}_{c}$ is a weight matrix and $\mathbf{W}_{o}$ is the
output matrix which is shared with the Preliminary decoder (Equation 2). Note
that RefNet generates two variants of the question : initial draft
$\tilde{\mathbf{Q}}$ and final draft $\mathbf{{Q}}$. We compare these two
versions of the generated questions in Section 4.

###  2.3 Reward-RefNet

Next, we address the following question: Can the refinement decoder be
explicitly rewarded for generating a question which is better than that
generated by the preliminary decoder on certain desired parameters? For
example, Nema and Khapra (2018) define fluency and answerability as desired
qualities in the generated question. They evaluate fluency using BLEU score
and answerability using a score which captures whether the question contains
the required {named entities, important words, function words, question types}
(and is thus answerable). We use these fluency and answerability scores
proposed by Nema and Khapra (2018) as reward signals. We first compute the
reward $r(\mathbf{\tilde{Q}})$ and $r(\mathbf{Q})$ for the question generated
by the preliminary and refinement decoder respectively. We then use “REINFORCE
with a baseline” algorithm Williams (1992) to reward Refinement Decoder using
the Preliminary Decoder’s reward $r(\mathbf{\tilde{Q}})$ as the baseline. More
specifically, given the Preliminary Decoder’s generated word sequence
$\mathbf{\tilde{Q}}=\\{\tilde{q}_{1},\tilde{q}_{2},\dots,\tilde{q}_{T}\\}$ and
the Refinement Decoder’s generated word sequence
$\mathbf{Q}=\\{q_{1},q_{2},\dots,q_{T}\\}$ obtained from the distribution
$p(\textrm{q}_{t}|\textrm{q}_{t-1},\dots,\textrm{q}_{1},\mathbf{\tilde{Q}},\mathbf{U},\mathbf{h}^{a})$,
the training loss is defined as follows

$\displaystyle L(\mathbf{Q})=$
$\displaystyle(r(\mathbf{Q})-r(\mathbf{\tilde{Q}}))\cdot$
$\displaystyle\sum_{t=1}^{T}\textrm{log}\;p({q}_{t}|\
{q}_{t-1},\dots,{q}_{1},\mathbf{\tilde{Q}},\mathbf{U},\mathbf{h}^{a})$

where $r(\mathbf{Q})$ and $r(\mathbf{\tilde{Q}})$ are the rewards obtained by
comparing with the reference question $\mathbf{Q^{*}}$. As mentioned, this
reward $r(.)$ can be the fluency score or answerability score as defined by
Nema and Khapra (2018).

###  2.4 Copy Module

Along with the above-mentioned three modules, we adopt the pointer-network and
coverage mechanism from See et al. (2017). We use it to (i) handle Out-of-
Vocabulary words and (ii) avoid repeating phrases in the generated questions.

Dataset | Model | n-gram |
---|---|---|---
BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE-L | METEOR | QBLEU4
SQuAD (Sentence Level) | Sun et al. (2018) | $43.02$ | $28.14$ | $20.51$ | $15.64$ | - | - | -
Zhao et al. (2018) | $44.51$ | $29.07$ | $21.06$ | $15.82$ | $44.24$ | $19.67$ | -
Kim et al. (2019) | - | - | - | $16.17$ | - | - | -
EAD | $44.74$ | $29.79$ | $22.00$ | $16.84$ | $44.78$ | $20.60$ | $24.7$
RefNet | $\mathbf{47.27}$ | $\mathbf{31.88}$ | $\mathbf{23.65}$ | $\mathbf{18.16}$ | $\mathbf{47.14}$ | $\mathbf{23.40}$ | $\mathbf{27.4}$
SQuAD (Passage Level) | Zhao et al. (2018) | $45.07$ | $29.58$ | $21.60$ | $16.38$ | $44.48$ | $20.25$ | -
EAD | $44.61$ | $29.37$ | $21.50$ | $16.36$ | $43.95$ | $20.11$ | $24.2$
RefNet | $\mathbf{46.41}$ | $\mathbf{30.66}$ | $\mathbf{22.42}$ | $\mathbf{16.99}$ | $\mathbf{45.03}$ | $\mathbf{21.10}$ | $\mathbf{26.6}$
HOTPOT |  Zhao et al. (2018)* | $45.29$ | $32.06$ | $24.43$ | $19.29$ | $40.40$ | $19.29$ | $25.7$
EAD | $\mathbf{46.00}$ | $32.47$ | $24.82$ | $19.68$ | $41.52$ | $23.27$ | $26.2$
RefNet | $45.45$ | $\mathbf{33.13}$ | $\mathbf{26.05}$ | $\mathbf{21.17}$ | $\mathbf{43.12}$ | $\mathbf{25.81}$ | $\mathbf{28.7}$
DROP Dataset |  Zhao et al. (2018)* | $39.56$ | $29.19$ | $22.53$ | $18.07$ | $45.01$ | $19.68$ | $31.4$
EAD | $39.21$ | $29.10$ | $22.65$ | $18.42$ | $45.07$ | $19.56$ | $31.8$
RefNet | $\mathbf{42.81}$ | $\mathbf{32.63}$ | $\mathbf{25.78}$ | $\mathbf{21.23}$ | $\mathbf{47.49}$ | $\mathbf{22.25}$ | $\mathbf{33.6}$
|  |  |  |  |  |  |  |

Table 2: Comparsion of RefNet model with existing approaches and EAD model.
Here * denotes our implementation of the corresponding work.

##  3 Experimental Details

In this section, we discuss (i) the datasets for which we tested our proposed
model, (ii) implementation details and (iii) evaluation metrics used to
compare our model with the baseline and existing works.

###  3.1 Datasets

SQuAD Rajpurkar et al. (2016): It contains $100$K (question, answer) pairs
obtained from $536$ Wikipedia articles, where the answers are a span in the
passage. For SQuAD, AQG has been tried from both sentences and passages. In
the former case, only the sentence which contains the answer span is used as
input, whereas in the latter case the entire passage is used. We use the same
train-validation-test splits as used in Zhao et al. (2018).

Hotpot QA Yang et al. (2018) : Hotpot-QA is a multi-document and multi-hop QA
dataset. Along with the triplet (P, A, Q), the authors also provide supporting
facts that potentially lead to the answer. The answers here are either yes/no
or answer span in P. We concatenate these supporting facts to form the
passage. We use $10$% of the training data for validation and use the original
dev set as test set.

DROP Dua et al. (2019): The DROP dataset is a reading comprehension benchmark
which requires discrete reasoning over passage. It contains $96$K questions
which require discrete operations such as addition, counting, or sorting to
obtain the answer. We use $10$% of the original training data for validation
and use the original dev set as test set.

###  3.2 Implementation Details

We use $300$ dimensional pre-trained Glove word embeddings, which are fixed
during training. For character-level embeddings, we initially use a $20$
dimensional embedding for the characters which is then projected to $100$
dimensions. For answer-tagging, we use embedding size of $3$. The hidden size
for all the LSTMs is fixed to $512$. We use 2-layer, 1-layer and 2-layer
stacked BiLSTM for the passage encoder, answer encoder and the decoders (both)
respectively. We take the top $30,000$ frequent words as the vocabulary. We
use Adam optimizer with a learning rate of $0.0004$ and train our models for
$10$ epochs using cross entropy loss. For the Reward-RefNet model, we fine-
tune the pre-trained model with the loss function mentioned in Section 2.3 for
$3$ epochs. The best model is chosen based on the BLEU Papineni et al. (2002)
score on the validation split. For all the results we use beam search decoding
with a beam size of $5$.

###  3.3 Evaluation

We evaluate our models, based on $n$-gram similarity metrics BLEU Papineni et
al. (2002), ROUGE-L Lin (2004), and METEOR Lavie and Denkowski (2009) using
the package released in Sharma et al. (2017)222https://github.com/Maluuba/nlg-
eval. We also quantify the answerability of our models using
QBLEU-4333https://github.com/PrekshaNema25/Answerability-MetricNema and Khapra
(2018).

##  4 Results and Discussions

In this section, we present the results and analysis of our proposed model
RefNet. Throughout this section, we refer to our models as follows:

Encode-Attend-Decode (EAD) model

is our single decoder model containing the encoder and the Preliminary Decoder
described earlier. Note that the performance of this model is comparable to
our implementation of the model proposed in Zhao et al. (2018).

Refine Network (RefNet) model

includes the encoder, the Preliminary Decoder and the Refinement Decoder.
We will (i) compare RefNet’s performance with EAD and existing models across
all the mentioned datasets (ii) report human evaluations to compare RefNet and
EAD (iii) analyze Refinement and Preliminary Decoders iv) present the
performance of Reward RefNet with two different reward signal (fluency and
answerability).

###  4.1 RefNet’s performance across datasets

In Table 2, we compare the performance of RefNet with existing single decoder
architectures across different datasets. On BLEU-4 metric, RefNet beats the
existing state-of-the-art model by $12.30$%, $9.74$%, $17.48$%, and $3.71$%
respectively on SQuAD (sentence), HOTPOT-QA, DROP and SQuAD (passage) dataset.
Also it outperforms EAD by $7.83$%, $7.57$%, $15.25$% and $3.85$% respectively
on SQuAD (sentence), HOTPOT-QA, DROP and SQuAD (passage). In general, RefNet
is consistently better than existing models across all $n$-gram scores (BLEU,
ROUGE-L and METEOR). Along with $n$-gram scores, we also observe improvements
on Q-BLEU4 as well, which as described earlier, gives a measure of both
answerability and fluency.

###  4.2 Human Evaluations

We conducted human evaluations to analyze the quality of the questions
produced by EAD and RefNet. We randomly sampled $500$ questions generated from
the SQuAD (sentence level) dataset and asked the annotators to compare the
quality of the generated questions. The annotators were shown a pair of
questions, one generated by EAD and one by RefNet from the same sentence, and
were asked to decide which one was better in terms of Fluency, Completeness,
and Answerability. They were allowed to skip the question pairs where they
could not make a clear choice. Three annotators rated each question and the
final label was calculated based on majority voting. We observed that the
RefNet model outperforms the EAD model across all three metrics. Over $68.6$%,
$66.7$% and $64.2$% of the generated questions from RefNet were respectively
more fluent, complete and answerable when compared to the EAD model. However,
there are some cases where EAD does better than RefNet. For example, in Table
3, we show that while trying to generate a more elaborate question, RefNet
introduces an additional phrase “in the united” which is not required. Due to
such instances, annotators preferred the EAD model in around $30$% of the
instances.

Passage: Before the freeze ended in 1952, there were only 108 existing
television stations in the United States; a few major cities (such as Boston)
had only two television stations, …
---
Questions
EAD: how many television stations existed in boston ?
RefNet: how many television stations did boston have in the united ?

Table 3: An example where EAD model was better than RefNet. The ground truth
answers are shown in blue.

Model | Decoder | BLEU-4 | QBLEU-4
---|---|---|---
without $\mathbf{A_{3}}$ | RefNet | $17.16$ | $25.80$
Initial Draft | $17.59$ | $26.00$
with $\mathbf{A_{3}}$ | RefNet | $18.37$ | $27.40$
Initial Draft | $17.89$ | $26.00$

Table 4: Comparison between Preliminary Decoder and Refinement Decoder in
RefNet Model for SQuAD Sentence Level QG.

Sample: |  |  Sentence: For instance , the language { xx — x is any binary string } can be solved in
---
linear time on a multi-tape Turing machine , but necessarily requires
quadratic time in the model of single-tape Turing machines .
Reference Question: A multi-tape Turing machine requires what type of time for
a solution ?
with $\mathbf{A}_{3}$ |  |  Refinement Decoder: in what time can the language be solved on a multi-tape turing machine ?
---
Preliminary Decoder: in what time can the language be solved ?
without $\mathbf{A}_{3}$ |  |  Refinement Decoder: in what time can the language { xx — x x x x is any binary string ?
---
Preliminary Decoder: in what time can the language — x x x x is solved ?

Table 5: Generated samples by Preliminary Decoder and Refinement Decoder in
RefNet model.

###  4.3 Analysis of Refinement Decoder and Preliminary Decoder

The two decoders impact each other through two paths: (i) indirect path, where
they share the encoder and the output projection to the vocabulary $V$, (ii)
direct path, via the dual attention network, where the initial draft of the
question is attended by the Refinement Decoder. When RefNet has only indirect
path, we can infer from row 1 of Table 4 that the performance of Preliminary
Decoder improves when compared to the EAD model ($16.84$ v/s $17.59$ BLEU).
This suggests that generating two variants of the question improves the
performance of the first decoder pass as well. This is perhaps due to the
additional feedback that the shared encoder and output layer get from the
Refinement Decoder. When we add the direct path (attention network) between
the two decoders, the performance of the Refinement Decoder improves as
compared to the Preliminary Decoder as shown in rows 3 and 4 of the Table 4
Comparison on Answerability: We also evaluate both the initial and refined
draft using QBLEU4. As discussed earlier, Q-Metric measures Answerability
using four components, viz., Named Entities, Important Words, Function Words,
and Question Type. We observe that the increase in Q-Metric for refined
questions is because the RefNet model can correct/add the relevant Named
Entities in the question. In particular, we observe that the Named Entity
component score in Q-Metric increases from $32.42$ for the first draft to
$37.81$ for the refined draft.

Figure 2: Generated Question Length Distribution for Preliminary Decoder
(First Decoder) and Refinement Decoder (Second Decoder).

Qualitative Analysis: Figure 2 shows that the RefNet model indeed generates
more elaborate questions when compared to the Preliminary Decoder. As shown in
Table 5, the quality of the refined question is better than the initial draft
of the questions. Here RefNet adds the phrase “multi-tape Turing Machine,”
(row 2) which removes any ambiguity in the question.

###  4.4 Analysis of Reward-RefNet

In this section, we analyze the impact of employing different reward signals
in Reward-RefNet. As discussed earlier in section 2.3, we use fluency and
answerability scores as reward signals. As shown in Table 6, when BLEU-4
(fluency) is used as a reward signal, there is improvement in BLEU-4 scores of
Reward-RefNet as compared to RefNet model. We validated these results through
human evaluations across $200$ samples. Annotators prefer the Reward-RefNet
model in $67$% of the cases for fluency. Similarly when we use Answerability
score as a reward signal, answerability improves for the model and annotators
prefer the Reward-RefNet in $70$% of the cases for answerability. The
performance of Reward-RefNet on fluency and answerability is similar for other
datasets (see Appendix B).

Model |  | BLEU
---
Reward Signal
| Answerability
---
Reward Signal
BLEU4 |  | %prefer-
---
ence
Ans. |  | %prefer
---
ence
RefNet | $18.37$ |  $32.9$% | $36.9$ |  $30$%
Reward-RefNet | $18.52$ |  $67.1$% | $37.5$ |  $70$%

Table 6: Impact of Reward-RefNet on fluency and answerability. %preference
denotes the percentage of times annotators prefer the generated output from
the model for fluency in case of BLEU Reward signal and answerability in case
of Answerability Reward signal.

Passage: Cost engineers and estimators apply expertise to relate the work and
materials involved to a proper valuation
---
Questions
Generated: Who apply expertise to relate the work and materials involved to a
proper valuation ?
True: Who applies expertise to relate the work and materials involved to a
proper valuation ?

Table 7: An example of question with significant overlap with the passage. The
answer is shown in blue.

Case Study: Originality of the Questions
We observe that current state-of-the-art models perform very well in terms of
BLEU/QBLEU scores when the actual question has significant overlap with the
passage. For example, consider a passage from the SQuAD dataset in Table 7,
where except the question word who, the model sequentially copies everything
from the passage and achieves a QBLEU score of $92.4$. However, the model
performs poorly in situations where the true question is novel and does not
contain a large sequence of words from the passage itself. In order to
quantify this, we first sort the true questions based on its BLEU-2 overlap
with the passage in ascending order. We then select the first $N$ true
questions and compute the QBLEU score with the generated questions. The
results are shown in red in Figure 3. Towards the left, where there are true
questions with low overlap with the passage, the performance is poor, but it
gradually improves as the overlap increases.

The task of generating questions with high originality (where the model
phrases the question in its own words) is a challenging aspect of AQG since it
requires complete understating of the semantics and syntax of the language. In
order to improve questions generated on originality, we explicitly reward our
model for having low $n$-gram score with the passage as compared to the
initial draft. As a result we observe that with Reward-RefNet(Originality),
there is an improvement in the performance where the overlap with the passage
was less (as shown in blue in Figure 3).As shown in Table 8, although both
questions are answerable given the passage, the question generated from
Reward-RefNet(Originality) is better.

Figure 3: Originality Analysis: Plot of Q-BLEU score vs $N$ \- the number
points selected.

Passage: McLetchie was elected on the Lothian regional list and the
Conservatives suffered a net loss of five seats , with leader Annabel Goldie
claiming that their support had held firm, nevertheless, she too announced she
would step down as leader of the party.
---
Questions
True: Who announced she would step down as leader of the Conservatives ?
RefNet: who claiming that their support had held firm ?
Reward-RefNet: who was the leader of the conservatives?

Table 8: An example where Reward-RefNet(Originality) is better than RefNet.

##  5 Related Work

Early works on Question Generation were essentially rule based systems Heilman
and Smith (2010); Mostow and Chen (2009); Lindberg et al. (2013); Labutov et
al. (2015). Current models for AQG are based on the encode-attend-decode
paradigm and they either generate questions from the passage alone Du and
Cardie (2017); Du et al. (2017); Yao et al. (2018) or from the passage and a
given answer (in which case the generated question must result in the given
answer). Over the past couple of years, several variants of the encode-attend-
decode model have been proposed. For example, Zhou et al. (2018) proposed a
sequential copying mechanism to explicitly select a sub-span from the passage.
Similarly, Zhao et al. (2018) mainly focuses on efficiently incorporating
paragraph level content by using Gated Self Attention and Maxout pointer
networks. Some works Yuan et al. (2017) even use Question Answering as a
metric to evaluate the generated questions. There has also been some work on
generating questions from images Jain et al. (2017); Li et al. (2017) and from
knowledge bases Serban et al. (2016); Reddy et al. (2017). The idea of multi
pass decoding which is central to our work has been used by Xia et al. (2017)
for machine translation and text summarization albeit with a different
objective. Some works have also augmented seq2seq models Rennie et al. (2017);
Paulus et al. (2018); Song et al. (2017) with external reward signals using
REINFORCE with baseline algorithm Williams (1992). The typical rewards used in
these works are BLEU and ROUGE scores. Our REINFORCE loss is different from
the previous ones as it uses the first decoder’s reward as the baseline
instead of reward of the greedy policy.

##  6 Conclusion and Future Work

In this work, we proposed Refine Networks (RefNet) for Question Generation to
focus on refining and improving the initial version of the generated question.
Our proposed RefNet model consisting of a Preliminary Decoder and a Refinement
Decoder with Dual Attention Network outperforms the existing state-of-the-art
models on the SQuAD, HOTPOT-QA and DROP datasets. Along with automated
evaluations, we also conducted human evaluations to validate our findings. We
further showed that using Reward-RefNet improves the initial draft on specific
aspects like fluency, answerability and originality. As a future work, we
would like to extend RefNet to have the ability to decide whether a refinement
is needed on the generated initial draft.

## Acknowledgements

We thank Amazon Web Services for providing free GPU compute and Google for
supporting Preksha Nema’s contribution in this work through Google Ph.D.
Fellowship programme. We would like to acknowledge Department of Computer
Science and Engineering, IIT Madras and Robert Bosch Center for Data Sciences
and Artificial Intelligence, IIT Madras (RBC-DSAI) for providing us sufficient
resources. We would also like to thank Patanjali SLPSK, Sahana Ramnath, Rahul
Ramesh, Anirban Laha, Nikita Moghe and the anonymous reviewers for their
valuable and constructive suggestions.

## References

  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _CoRR_ , abs/1409.0473.
  * Du and Cardie (2017) Xinya Du and Claire Cardie. 2017.  Identifying where to focus in reading comprehension for neural question generation.  In _EMNLP_ , pages 2067–2073. Association for Computational Linguistics.
  * Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017.  Learning to ask: Neural question generation for reading comprehension.  In _ACL (1)_ , pages 1342–1352. Association for Computational Linguistics.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proc. of NAACL_.
  * Heilman and Smith (2010) Michael Heilman and Noah A. Smith. 2010.  Good question! statistical ranking for question generation.  In _HLT-NAACL_ , pages 609–617. The Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780.
  * Jain et al. (2017) Unnat Jain, Ziyu Zhang, and Alexander G. Schwing. 2017.  Creativity: Generating diverse questions using variational autoencoders.  In _CVPR_ , pages 5415–5424. IEEE Computer Society.
  * Kim et al. (2019) Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019.  Improving neural question generation using answer separation.  _CoRR_ , abs/1809.02393.
  * Labutov et al. (2015) Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015.  Deep questions without deep understanding.  In _ACL (1)_ , pages 889–898. The Association for Computer Linguistics.
  * Lavie and Denkowski (2009) Alon Lavie and Michael J. Denkowski. 2009.  The meteor metric for automatic evaluation of machine translation.  _Machine Translation_ , 23(2-3):105–115.
  * Li et al. (2017) Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, and Xiaogang Wang. 2017\.  Visual question generation as dual task of visual question answering.  _CoRR_ , abs/1709.07192.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Proc. ACL workshop on Text Summarization Branches Out_ , page 10.
  * Lindberg et al. (2013) David Lindberg, Fred Popowich, John C. Nesbit, and Philip H. Winne. 2013.  Generating natural language questions to support learning on-line.  In _ENLG_ , pages 105–114. The Association for Computer Linguistics.
  * Mostow and Chen (2009) Jack Mostow and Wei Chen. 2009.  Generating instruction automatically for the reading strategy of self-questioning.  In _AIED_ , volume 200 of _Frontiers in Artificial Intelligence and Applications_ , pages 465–472. IOS Press.
  * Nema and Khapra (2018) Preksha Nema and Mitesh M. Khapra. 2018.  Towards a better metric for evaluating question generation systems.  _CoRR_ , abs/1808.10192.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_ , pages 311–318. ACL.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  _CoRR_ , abs/1705.04304.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100, 000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2383–2392.
  * Reddy et al. (2017) Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Josh. 2017.  Generating natural language question-answer pairs from a knowledge graph using a RNN based question generation model.  In _EACL (1)_ , pages 376–385. Association for Computational Linguistics.
  * Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1179–1195.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _ACL_.
  * Seo et al. (2016) Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016.  Bidirectional attention flow for machine comprehension.  _CoRR_ , abs/1611.01603.
  * Serban et al. (2016) Iulian Vlad Serban, Alberto García-Durán, Çaglar Gülçehre, Sungjin Ahn, Sarath Chandar, Aaron C. Courville, and Yoshua Bengio. 2016.  Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus.  In _ACL (1)_. The Association for Computer Linguistics.
  * Sharma et al. (2017) Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017.  Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation.  _CoRR_ , abs/1706.09799.
  * Shum et al. (2018) Heung-yeung Shum, Xiao-dong He, and Di Li. 2018.  From eliza to xiaoice: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_, 19(1):10–26.
  * Song et al. (2017) Linfeng Song, Zhiguo Wang, and Wael Hamza. 2017.  A unified query-based generative model for question generation and question answering.  _CoRR_ , abs/1709.01058.
  * Sun et al. (2018) Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018.  Answer-focused and position-aware neural question generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3930–3939, Brussels, Belgium. Association for Computational Linguistics.
  * Williams (1992) Ronald J. Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Mach. Learn._ , 8(3-4):229–256.
  * Xia et al. (2017) Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017.  Deliberation networks: Sequence generation beyond one-pass decoding.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 1784–1794. Curran Associates, Inc.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Yao et al. (2018) Kaichun Yao, Libo Zhang, Tiejian Luo, Lili Tao, and Yanjun Wu. 2018.  Teaching machines to ask questions.  In _IJCAI_.
  * Yuan et al. (2017) Xingdi Yuan, Tong Wang, Çaglar Gülçehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017\.  Machine comprehension by text-to-text neural question generation.  In _Rep4NLP@ACL_ , pages 15–25. Association for Computational Linguistics.
  * Zhao et al. (2018) Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018.  Paragraph-level neural question generation with maxout pointer and gated self-attention networks.  In _EMNLP_.
  * Zhou et al. (2018) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2018.  Sequential copying networks.  In _AAAI_.

##  Appendix A Impact of Various Embeddings

We perform an ablation study to identify the impact of various word embeddings
used in RefNet. When character embedding is not used in RefNet, the
performance on SQuAD sentence-level drops from $18.16$ to $17.97$ BLEU-4
score. Meanwhile, when positional embeddings are dropped the performance
decreases to $17.87$ BLEU-4 score.

##  Appendix B Reward-RefNet on Various Datasets

Table 9 shows the comparison between RefNet and Reward-RefNet on BLEU-4 score
and answerability score when the respective scores are used as rewards in
Reward-RefNet. We can infer from Table 9 that there is improvement in fluency
and answerability across all the datasets.

Datasets | Model |  | BLEU-4
---
Reward Signal
| Answerability
---
Reward Signal
SQuAD (Passage Level) | RefNet | 16.99 | 26.6
Reward-RefNet | 17.11 | 27.3
HOTPOT-QA | RefNet | 21.17 | 28.7
Reward-RefNet | 21.32 | 29.2
DROP | RefNet | 21.23 | 33.6
Reward-RefNet | 21.60 | 34.3

Table 9: Impact of Reward-RefNet on various datasets when fluency and
answerability are used as reward signals.

##  Appendix C Visualization of Attention Weights

We plot the aggregated attention given to the passage and initial draft of the
generation question across the various time-steps of the decoder in Figure 4.
Although, both the questions are specific to the answer, $\mathbf{A_{2}}$ pays
some attention to the context surrounding the answer, which leads to a
complete question. Also, note that in $\mathbf{A_{3}}$, while attending on to
initial draft “oncogenic” word is not paid attention to and thus the final
draft revises over the initial draft by correcting it to generate a better
question.

(a) $\mathbf{A_{1}}$ attention plot

(b) $\mathbf{A_{2}}$ attention plot

(c) $\mathbf{A_{3}}$ attention plot

Figure 4: Attention plots for a) $\mathbf{A_{1}}$, b) $\mathbf{A_{2}}$, c)
$\mathbf{A_{3}}$ respectively
Initial Generated Question: “What is the name of the oncogenic virus?”
Refined Generated Question: “What is the name of the organism that causes
cervical
cancer?”
Passage: “The antigens expressed by tumors have several sources ; some are
derived from oncogenic viruses like human papillomavirus , which causes
cervical cancer , while others are the organism’s own proteins that occur at
low levels in normal cells but reach high levels in tumor cells.”


#  Deep multi-stations weather forecasting: explainable recurrent
convolutional neural networks

Ismail Alaoui Abdellaoui  Siamak Mehrkanoon  Department of Data Science and
Knowledge Engineering, Maastricht University, The Netherlands

###### Abstract

Deep learning applied to weather forecasting has started gaining popularity
because of the progress achieved by data-driven models. The present paper
compares two different deep learning architectures to perform weather
prediction on daily data gathered from 18 cities across Europe and spanned
over a period of 15 years. We propose the Deep Attention Unistream Multistream
(DAUM) networks that investigate different types of input representations
(i.e. tensorial unistream vs. multistream ) as well as the incorporation of
the attention mechanism. In particular, we show that adding a self-attention
block within the models increases the overall forecasting performance.
Furthermore, visualization techniques such as occlusion analysis and score
maximization are used to give an additional insight on the most important
features and cities for predicting a particular target feature of target
cities.

###### keywords:

Weather data , deep learning , convolutional neural network , attention
mechanism , explainability

††journal: arXiv

##  1 Introduction

Weather forecasting is of major importance as it affects the daily activities
of fundamental fields such as agriculture, transportation and international
commerce among others. The ability to forecast the precipitation rates, the
risk of flood or the likelihood of a hurricane can potentially lead to saving
of lives and to the well-being of humans. Moreover, the change of the climate
on earth has led to increasing research and world-wide efforts to halt the
environmental and ecological consequences [26].

Traditional approaches of weather forecasting rely on priors like the
thermodynamic properties of the atmosphere [12, 25, 5], statistical
distribution of the data [9], or ensemble learning that incorporates multiple
models with different initial conditions [10]. This family of models belongs
to the “Numerical Weather Prediction” (NWP) methodologies [15] and usually
rely on the processing power of supercomputers, thus being resource heavy [2].
In addition of the high computational cost, it has been shown that a priori
information about the data that constitutes the initial state is the source of
errors in weather prediction [29].

While traditional NWP methods aim at extracting useful dynamics from a model
or to transfer information between models, the purpose of recent data-driven
approaches is to simulate an entire system to predict its future state [27].
Machine learning data-driven based models have already been successfully
applied in various domains such as healthcare, dynamical systems, biomedical
signal analysis, neuroscience among others [20, 22, 21, 18, 23, 1, 4, 33, 17].
The recent advances of machine learning models has increased the capability to
automatically learn the underlying nonlinear complex patterns of weather
dynamics [19, 30, 31]. In particular, the combination of convolutional neural
networks (CNNs) and long short-term memory (LSTM) networks proved to be a
successful deep learning approach for climate modeling and weather forecasting
[6, 8].

This paper presents three contributions. The first one is an investigation of
the unistream and multistream approaches as input representation for the
neural networks. The second contribution aims at enriching the proposed
networks with a self-attention mechanism. Finally, the third contribution
addresses the explainability of the networks by using modern visualization
techniques to determine the features and cities that contribute the most to
the output predictions of a particular city or group of cities. It is of utter
importance to gain interpretability from the data driven models given that
weather prediction is the basis of many human real-life decisions. This paper
is organized as follows. A brief review of the existing machine learning
methodologies for weather forecasting is given in Section 2. A formal
definition of the Conv-LSTM layer and the visualization techniques used are
presented in Section 3. Our proposed models are introduced in Section 4.
Furthermore, the dataset used is introduced in Section 5. The experimental
results are reported in section 6. Finally, a discussion followed by the
conclusion are drawn in sections 7 and 8, respectively.

##  2 Related Work

Multiple approaches have been recently proposed to tackle weather forecasting
using deep machine learning models. The author in [19] introduced the
convolutional neural networks to learn the underlying spatio-temporal patterns
of weather data. This work used hourly past data from cities in the
Netherlands, Belgium and Denmark to predict the temperature and wind speed of
multiple cities. It has been shown that the convolutional operations benefit
from a tensorial representation in order to improve the prediction capability.

In another work, a feed forward neural network has been used to investigate
the volume of data needed as well as its recency to yield accurate weather
predictions [3]. In terms of data volume, it has been shown that more data
consistently leads to better predictions. The impact of the data recency
remains unknown as there was no significant impact on the predictions when
tuning the recency of the data.

In [35], the authors used deep learning to predict weather phenomena related
to the heating of the air (i.e. heavy rains and thunderstorms among others),
more commonly known as severe convective weather (SCW). A deep CNN has been
utilized and proved to yield superior results compared to traditional machine
learning models such as support vector machines or random forests. Another
model which used stacked ConvLSTM layers, DeepRain, was compared to linear
regression models and reduced the RMSE by a large margin. This model used past
radar data with a 6-min time resolution over a period of two years [13].

Similarly to the previous work, the authors in [28] used a multi-input network
with past radar data to perform precipitation forecasting. However, a
different approach was used since the regression problem was transformed into
a multi-class classification of possible precipitation ranges. The model also
made use of axial self-attention [11] as spatial aggregator. This model was
able to outperform the system used in the National Oceanic and Atmospheric
Administration (NOAA). In [7], the different challenges of using deep learning
for weather forecasting are addressed. In particular, it has been stated that
while neural networks can be useful for short-term predictions, the need for
domain knowledge is essential when tackling forecasting of longer term ranges.

##  3 Preliminaries

###  3.1 Self-Attention

The self-attention mechanism was first introduced by Vaswani et. al [32] to
capture dependencies within sequence of words. It relies on the dot product
operation to assess the similarity of each word with respect to all the other
words of a sequence. The query $Q$, key $K$, and value $V$ matrices are
computed through the sequence of inputs $I\in\mathbb{R}^{S\times E}$ where $S$
is the sequence length and $E$ is the embedding dimension of each input
feature:

$\quad Q=IW_{q},\quad K=IW_{k},\quad and\quad V=IW_{v},$ (1)

where $W_{q}$, $W_{k}$, and $W_{v}$ are learnable weights through a linear
function. The attention matrix, also called the head output, is then computed
through the softmax of a scaled dot product as follows:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V,$ (2)

where $d_{k}$ is the dimension of the key vector $K\in\mathbb{R}^{1\times
d_{k}}$.

###  3.2 Conv-LSTM

In this section we give an overview of the ConvLSTM layer that is used in the
proposed models.

It is based on the LSTM cell and was introduced in [34] to address the issue
of capturing the spatial structure of the data. In this model, the input gate
$i_{t}$, the forget gate $f_{t}$, the output gate $o_{t}$, the hidden state
$h_{t-1}$, the candidate cell state $\hat{C}_{t}$, the current cell state
$C_{t}$ and the input $x_{t}$ are all 3D tensors. The first dimension of each
tensor is the sequence length while the two last dimensions represent the rows
and columns. This model has first been used on weather data for precipitation
nowcasting, outperforming other models based on the LSTM only.

###  3.3 Activation maximization

Activation maximization is a visualization technique that looks for patterns
that maximize a particular activation function inside a neural network [16].
Following the taxonomy of interpretability methods presented in [24],
activation maximization is a post hoc method since it aims at understanding
the model after the training. This methodology focuses on finding a new input
that maximizes the activation of a neuron:

$I^{*}=\operatorname*{argmax}_{I}h_{l,z}(I),$ (3)

where $I$ is the input data of the network, $h$ is the activation function
used in the neuron $z$ of the layer index $l$. For our case, we want to find
the input data that contributes the most to minimizing the error between the
model prediction and ground truth data. Since in our study weather element
forecasting is reduced to a regression problem, here we define $h$, a custom
objective function, as the inverse of the mean squared error (MSE):

$h=\frac{1}{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}},$ (4)

where $y_{i}$ and $\hat{y}_{i}$ are the true measured data and the model
prediction of a particular weather feature for the ith target city,
respectively. Here, $n$ denotes the number of target cities. The pseudocode of
maximizing the $h$ score and getting the score map $I^{*}$ is provided in
Algorithm 1.

Input: The number of iterations $s$

Pretrained model $m_{p}$

Sample input $I$

Input ranges $I_{min}$ and $I_{max}$

Learning rate $\eta$

Output: New input $I^{*}$

for  _the number of iterations $s$_ do

Perform a forward pass of $m_{p}$ on $I$ to get a prediction $\hat{y}$.

Use eq. (4) to obtain the score $h$.

Apply $L_{2}$ normalization on the obtained score.

Compute the gradient $dI$ of the normalized score with respect to input $I$.

Update the input using: $I\leftarrow I+\eta\;dI$.

end for

Obtain $I^{*}$ by clipping $I$ based on $I_{min}$ and $I_{max}$.

Algorithm 1 Score Maximization

###  3.4 Occlusion Analysis

The occlusion analysis is a simplistic, yet effective way to determine which
features contribute the most to a minimal error between the actual and
prediction data. In this paper, we are concerned with two types of occlusion
analysis: a spatial and a temporal occlusion. While the spatial occlusion
analysis focuses on the important cities and weather features, the temporal
analysis aims at determining the most important lags. The spatial occlusion
analysis can be used to either focus on the cities only, or on the weather
features only, or on a group of cities and features. However, all of these
approaches rely on the same principle, which is to compute the percentage
change between a reference MSE, obtained from the prediction of an unmasked
data sample and its corresponding ground truth target data, and a new MSE,
computed from the prediction of a masked data sample and the same ground truth
label. We perform this percentage change each time the mask is slided to a new
location of the input. Focusing on either the cities or the weather features
on one hand, or a group of cities and features on the other hand, will
determine the shape of the mask (i.e. a vector or a matrix, respectively). We
present in Algorithm 2 the specific pseudocode of the occlusion analysis, when
using a square matrix of size $p$ as a mask over the input dataset
$\mathcal{X}$, and for a particular target city.

Input: Input dataset $\mathcal{X}=\\{x_{i}\\}_{i=1}^{k}$

Target dataset $\mathcal{O}=\\{o_{i}\\}_{i=1}^{k}$

Pretrained model $m_{p}$

Target city index $c$

Mask size $p$

Number of horizontal slidings $s_{h}$

Number of vertical slidings $s_{v}$

Output: Occlusion map $M_{o}$

for  _the number of data samples_ do

Perform a prediction of the sample $x_{i}$ using $m_{p}$.

Compute MSEi between the real target data $o_{i}$ and the model prediction for
the $c$th city.

for  _the number of horizontal slidings $s_{h}$_ do

for  _the number of vertical slidings $s_{v}$_ do

Mask the sample $x_{i}$ by the patch to get the masked sample $\tilde{x}_{i}$.

Make a new prediction using masked sample $\tilde{x}_{i}$.

Compute the $\widetilde{MSE}$i between $o_{i}$ and the recent model prediction
for the $c$th city.

Calculate the percentage change $\Delta$ between MSEi and $\widetilde{MSE}$i.

Store $\Delta$ in a tensor $\mathcal{M}$ for this particular patch location.

Displace the mask vertically by the distance $p$ over the sample $x_{i}$.

end for

Displace the mask horizontally by the distance $p$ over the sample $x_{i}$.

end for

end for

For each mask location, compute an average of the stored $\Delta$ over all
data samples from $\mathcal{M}$ to get the occlusion map $M_{o}$.

Algorithm 2 Occlusion analysis

##  4 Proposed Models

The two proposed models aim at studying the impact of the input representation
for the task of weather forecasting. To this end, these models use two types
of input representation: a unistream tensorial representation and a
multistream representation.

###  4.1 Unistream model

The input of the model is a tensor $\mathcal{T}\in\mathbb{R}^{L\times F\times
C}$ where $L$ is the number of lags used, $F$ is the number of weather
features, and $C$ the number of cities. This tensor is the input of a ConvLSTM
layer. As previously seen in section 3.2, the ConvLSTM layer essentially
processes this tensor input while taking the number of lags $L$ as the
sequence length. This layer is then followed by batch normalization and a
flatten operation since the output of the ConvLSTM is tensorial. We then use
two fully connected layers with a $ReLu$ activation function before the output
layer.

###  4.2 Multistream model

This architecture uses a multistream approach. Each input stream uses a tensor
$\mathcal{U}\in\mathbb{R}^{V\times F\times C}$ where $V$ is the number of lags
used in each tensor. Since the two models, i.e. Unistream and Multistream, use
the same number of lags, then $V$ evenly divides the total number of lags $L$
in each sample. Two ConvLSTM layers are used in each stream to capture the
spatial and temporal features. The output of each stream is then concatenated
on the axis of the channels. Similarly to the previous model, we use batch
normalization and layer flattening. A dense layer with a $ReLu$ activation
function is used before the final output layer. Some hyperparameters like the
kernel size or the number of fully connected nodes have been adapted to have a
comparable number of paramaters with respect to the previous model.

###  4.3 Attention enriched models

The two models presented above have also been augmented with a self-attention
mechanism. More specifically, one layer encoder block introduced in [32] has
been incorporated.

In the tensorial input model, it has been added right after the ConvLSTM
layer. For the multistream approach, the attention block is inserted after the
merging. For both approaches, a reshaping is necessary before the attention
encoder block since it is designed for matrices. It should also be noted that
we only use one head attention.

Fig. 1 shows the schema of the proposed models. The output of these models is
a vector $o$ of length $n$, i.e. the number of target cities. Each value in
this vector represents the same target feature for all the target cities. The
hyperparameters of these models were selected so that they all have a
comparable number of learnable parameters.

Figure 1: Schemas of (a) the Unistream and (b) Multistream models.

##  5 Data Description

The dataset used has been collected from Weather Underground and includes 18
cities across Europe and 18 weather features, for a period of 15 years from
May 2005 to April 2020. The data is made publicly available
111https://github.com/IsmailAlaouiAbdellaoui/weather-forecasting-explanable-
recurrent-convolutional-NN. Its time resolution is daily and weather features
include for instance the temperature, wind speed, condition and sea level
pressure among others. Table 1 presents the list of all features used. At each
time step $t$, a data sample is represented by a matrix
$M_{t}\in\mathbb{R}^{F\times C}$, where $F$ is the number of features and $C$
is the number of cities. Therefore the whole dataset is a tensor
$\mathcal{D}\in\mathbb{R}^{L\times F\times C}$, where $L$ is the total number
of days used. Fig. 2 shows a map of the different cities that are contained in
the dataset.

Table 1: Features used in the dataset.

Feature name |  Remarks
---|---
Highest temperature (°F) |  -
Lowest temperature (°F) |  -
Average temperature (°F) |  -
Dew point (°F) |  -
Highest dew point (°F) |  -
Lowest dew point (°F) |  -
Average dew point (°F) |  -
Maximum wind speed (mph) |  -
Visiblity (mi) |  Discrete value expressed in miles to measure the distance at which an object can be clearly distinguished
Sea level pressure (Hg) |  Measured in inch of mercury
Observed temperature (°F) |  Temperature in Fahrenheit observed at 10 am
Observed dew point (°F) |  Dew point in Fahrenheit observed at 10 am
Humidity (%) |  -
Wind direction |  Discrete values indicating 16 possible directions of the wind
Wind speed (mph) |  -
Wind gust (mph) |  -
Pressure (in) |  -
Condition |  21 possible discrete values that describe the overall weather state (cloudy, rainy, fog …)
|

Figure 2: Map showing the 18 cities used in the dataset.

##  6 Experimental Results

###  6.1 Data Preprocessing

The weather data is first scaled by means of equation 5. In this way, for each
feature and city, we take the values corresponding to every date and scale
them down between 0 and 1.

$x_{scaled}=\frac{x-min(c_{ij})}{max(c_{ij})-min(c_{ij})},i\in[1,F],j\in[1,C],$
(5)

where $c_{ij}\in\mathbb{R}^{L}$ refers to a column vector and $i$ and $j$ are
the ith feature of the jth city.

###  6.2 Experimental setup

For all the experiments, the following six cities have been used as target
cities: Paris, Luxembourg, London, Brussels, Frankfurt and Rotterdam. We also
selected two target features: the wind speed, in miles per hour, and the
average temperature of the day, in degree Fahrenheit. It should be noted that
for each training instance, the output vector corresponds to only one type of
feature, for all the target cities. Moreover, we performed experiments for 2,
4 and 6 days ahead, using 10 lags. All the experiments used $90\%$ of the data
for training and validation, while the remaining $10\%$ was used for testing.
Adam method [14] is used to optimize the mean square error (MSE) with a
learning rate of $1e^{-4}$ and a batch size of $16$ for all of the proposed
models.

###  6.3 Results

The obtained mean squared error (MSE) of the proposed two models for wind
speed as well as average temperature prediction for six target cities over 2,
4, and 6 days ahead are tabulated in Table 2 and Table 3 respectively. The
results of the two models with incorporated attention mechanism is also
tabulated in Table 2 and Table 3. For every city and days ahead, the MSE of
the best model is underlined. It should be noted that the reported MSEs are
calculated after descaling the models prediction.

Table 2: The MSE comparison of the four models, for 2, 4, and 6 days ahead
wind speed prediction.

Days ahead | City | Unistream | Att-Unistream | Multistream | Att-Multistream
---|---|---|---|---|---
2 | Luxembourg | 28.85 | 22.96 | 26.94 | 23.05
| Rotterdam | 48.04 | 38.22 | 44.85 | 38.38
| Frankfurt | 41.11 | 22.70 | 38.38 | 32.84
| Brussels | 36.78 | 29.26 | 34.34 | 29.38
| London | 30.75 | 24.46 | 28.71 | 24.56
| Paris | 25.25 | 20.09 | 23.58 | 20.16
4 | Luxembourg | 32.27 | 25.82 | 29.85 | 25.35
| Rotterdam | 53.73 | 42.98 | 49.69 | 42.20
| Frankfurt | 45.97 | 36.78 | 42.52 | 36.11
| Brussels | 41.14 | 32.91 | 38.05 | 32.31
| London | 34.39 | 27.51 | 31.80 | 27.01
| Paris | 28.24 | 22.59 | 26.12 | 22.18
6 | Luxembourg | 38.22 | 26.34 | 30.63 | 25.36
| Rotterdam | 63.64 | 43.87 | 51.00 | 42.23
| Frankfurt | 54.45 | 37.53 | 43.64 | 36.14
| Brussels | 48.72 | 33.58 | 39.05 | 32.33
| London | 40.73 | 28.07 | 32.64 | 27.03
| Paris | 33.45 | 23.06 | 26.81 | 22.20
|  |  |  |  |

From Tables 2 and 3, one can observe that often the models with attention are
the most successful ones. Indeed, for both the prediction of the temperature
and the wind speed, the models with attention always have an edge over their
corresponding models without attention.

If we consider the models without the addition of attention, the one that uses
a multistream approach is the most dominant one for both weather features,
consistently outperforming the unistream model. Concerning the models with
attention, if we compare them over all the results, there is no clear winner.
However, if we perform the same analysis within each weather feature, a
different pattern emerges. Indeed, the unistream model is the best one for
predicting the temperature. It should also be noted that the unistream model
is better than the other one for short time horizons (e.g. 2 days ahead). The
city for which wind speed is easier to predict is Paris, while the average
temperature of Brussels is the one yielding the minimum MSE error among all
the target cities. Fig. 3 shows the result of real data versus its prediction
for 2, 4 and 6 days ahead using the Att-Multistream model and for the cities
of Paris and Brussels.

Table 3: The MSE comparison of the four models, for 2, 4, and 6 days ahead
average temperature prediction.

Days ahead | City | Unistream | Att-Unistream | Multistream | Att-Multistream
---|---|---|---|---|---
2 | Luxembourg | 58.40 | 40.83 | 46.35 | 47.46
| Rotterdam | 52.89 | 37.14 | 41.98 | 43.23
| Frankfurt | 53.73 | 37.68 | 42.65 | 43.89
| Brussels | 42.87 | 30.15 | 34.02 | 35.18
| London | 44.80 | 31.48 | 35.56 | 36.69
| Paris | 53.15 | 37.27 | 42.19 | 43.41
4 | Luxembourg | 67.71 | 53.88 | 59.41 | 42.16
| Rotterdam | 61.32 | 48.91 | 53.81 | 38.19
| Frankfurt | 62.29 | 49.67 | 54.66 | 38.80
| Brussels | 49.70 | 39.70 | 43.61 | 30.97
| London | 51.94 | 41.46 | 45.58 | 32.36
| Paris | 61.62 | 49.13 | 54.07 | 38.29
6 | Luxembourg | 75.87 | 54.91 | 65.96 | 55.76
| Rotterdam | 68.72 | 49.84 | 59.74 | 50.55
| Frankfurt | 69.80 | 50.60 | 60.68 | 51.35
| Brussels | 55.69 | 40.43 | 48.41 | 40.99
| London | 58.20 | 42.23 | 50.60 | 42.84
| Paris | 69.05 | 50.06 | 60.03 | 50.80
|  |  |  |  |

(a)  (b)

Figure 3: Actual vs. prediction of the average temperature (a) and wind speed
(b) using the two proposed models.

Concerning the training time, we observed that it keeps growing as we add the
attention mechanism and use the multistream approach. Interestingly enough,
the multistream architecture that incorporates attention takes more time to
train despite less trainable parameters. We should also note that the
attention mechanism has more impact on the training time of the unistream
architecture.

##  7 Discussion

An effective way to understand which input features and cities affect the
outputs is using the techniques explained in sections 3.3 and 3.4. In this
section, we present the results of spatial and temporal occlusion analysis and
score maximization techniques for the two proposed models. In addition, for
this analysis the models have been trained to predict six days ahead.

####  7.0.1 Occlusion analysis

In order to determine which features contribute the most to a minimal error
between the actual data and the prediction for a particular city, we first
determine the MSE between the prediction of a sample and the actual data,
which is used as a reference MSE. We then use a mask vector
$m_{f}\in\mathbb{R}^{1\times F}$ which is used in a sliding fashion across all
the feature rows of the input data. We make an inference everytime $m_{f}$
masks a row, compute the corresponding MSE and finally obtain the percentage
change between this MSE and the reference MSE. We repeat the same process
along all row features to obtain all the percentage change for that particular
data sample. The masked row feature that leads to the biggest MSE increase
corresponds to the most important feature. Moreover, we repeat the same
computations using multiple data samples and we average the percentage changes
for each feature row. The same process applies to the rest of the other cities
in order to obtain their corresponding important features. On the other hand,
to determine the most important cities, we use the same algorithm, but with a
mask vector $m_{c}\in\mathbb{R}^{C\times 1}$ that is slided across all the
column cities of the input data.

Fig. 4 shows the most important features and cities of the Att-Unistream model
after performing the occlusion analysis. This model was trained on the 6
target cities described above with the average temperature as target feature.
Fig. 4 (a) shows the most important features for each target city while Fig. 4
(b) presents the most relevant cities for each target city. Fig. 4 (a)
highlights the importance of the dew point on the temperature since the plot
shows an agreement around this weather feature. This feature makes sense since
the dew point is the temperature to which air must be cooled to transform into
water vapor. Concerning the cities, Brussels plays a major role into the
predictions. This finding is reasonable since we can observe from Fig. 2 that
Brussels is the centroid of the city cluster that makes up the target cities.
Indeed, these models are multioutput, and affecting the city of Brussels would
affect the prediction of all the target cities.

(a)  (b)

Figure 4: Occlusion analysis visualization of the Att-Unistream model showing
the most relevant weather features (a) and cities (b) for each target city.
The model was trained to predict the 6 days ahead average temperature. Figure
5: Occlusion analysis visualization of the Att-Multistream model showing the
most relevant weather features for each target city. The model was trained to
predict the 6 days ahead wind speed.

Fig. 5 presents the same information as in Fig. 4, but for the Att-Multistream
model, with the wind speed as target feature. Interestingly enough, the
condition and to a lesser extent the wind gust and pressure play some role in
these predictions.

Apart from determining the important features or cities only, occlusion
analysis can also be useful to determine whether a group of cities or features
is important as well. We applied the same process described above, however
instead of using vector masks, we used square patch matrices that are slided
along both the rows and columns directions, without overlapping. Fig. 6 shows
the visualization of this occlusion analysis, where the reference MSE
corresponds to the error between the prediction and the actual data of the
cities of Paris. The top and bottom rows show the visualization of this
analysis for the Att-Unistream and Att-Multistream models, respectively. These
models were trained for a 6 days ahead prediction of the temperature and wind
speed. The first, second, and third columns use patch sizes of 1$\times$1,
2$\times$2, and 3$\times$3, respectively. Brighter colors correspond to
features and cities that are more important. A first observation is that
occlusions seem decisive about the important features and cities since only a
specific mask region is brighter than the other regions in each occlusion map.
The important features to predict the temperature shown in subfigures (a),
(b), and (c) are the temperature and the pressure. These findings bring
complementary information to the outcomes shown by subfigure (a) of Fig. 4.
Subfigures (d), (e), and (f) also reveal complementary information when
compared to the analysis shown in Fig. 5. Among the weather features that are
important to predict the wind speed, we can find the maximum wind speed and
the wind speed. Moreover, Brussels as well as cities near the sea like
Barcelona or Amsterdam are critical cities for wind speed prediction.

(a)  (b)  (c)

(d)  (e)  (f)

Figure 6: Results of the spatial occlusion analysis using different mask
sizes. (a,d): mask size 1$\times$1\. (b,e): mask size 2$\times$2\. (c,f): mask
size of 3$\times$3\. For both Att-Unistream and Att-Multistream models, the
target city is Paris. The Att-Unistream and Att-Multistream models were
trained to predict the temperature and wind speed, respectively.

(a)

(b)

Figure 7: Temporal occlusion analysis visualization of the Att-Unistream (a)
and Att-Multistream (b) models. These models were trained to perform the 6
days ahead prediction of temperature and wind speed, respectively.

(a)  (b)  (c)

Figure 8: Score maximization maps of the Att-Unistream for the 1st (a), 5th
(b), and 10th (c) lags. This model was trained to predict the wind speed.

Fig. 7 displays the visualization of the temporal occlusion analysis where
brighter values mean more relevant lags. As previously mentioned, this
approach aims at finding which lag contributes the most to a minimal error
between the actual and prediction data. The leftmost region corresponds to the
oldest lags while the rightmost one are the most recent lags. For the Att-
Unistream model, Fig. 7 (a), there is an emphasis on the recent lags, with the
exception of the cities of London and Rotterdam. However, for the Att-
Multistream model in Fig. 7 (b), the older lags yield consistently highest
importance, with the exception of Rotterdam. Interestingly enough, the 1st and
2nd lags seem the most important ones for the cities of Paris, Luxembourg,
Brussels, and Frankfurt.

####  7.0.2 Score maximization

Here we show the visualization of the score maximization introduced in section
3.3. Fig. 8 are the score maximization maps that show the relevant weather
features for all the cities of the Att-Unistream model, trained to predict six
days ahead daily average temperature. Higher pixel values in the score maps
refer to a more important city-feature pair. There is no discernible pattern
except the consistent feature across the three lags. This weather feature is
the visibility. Moreover, the pressure feature seems important as well, but to
a lesser extent.

##  8 Conclusion

In this paper, two deep neural networks architectures have been proposed and
investigated to perform weather elements forecasting. Moreover, a self-
attention mechanism proved to be beneficial to these models since it
consistently improved the results. From the analysis of the experimental
results, it has been shown that a multistream input representation is globally
more suitable for this task. In addition, interpretability techniques such as
occlusion analysis and score maximization have been used to extract the most
relevant input features (i.e. weather features and cities). These methods
revealed that in general, Brussels is important for the prediction of
temperature since it is located in the center of the target cities, while
cities near the sea like Barcelona or Amsterdam are more suitable for wind
speed prediction. It has also been shown that dew point is an important
feature for the prediction of the temperature while the maximum wind speed and
the condition heavily influence the wind speed prediction. From a temporal
perspective, each model favors specific lags, with the unistream model having
more emphasis on the recent lags and the multistream model favouring the older
lags. The data and code used can be found at
github.com/IsmailAlaouiAbdellaoui/weather-forecasting-explanable-recurrent-
convolutional-nn.

## Acknowledgment

Simulations were performed with computing resources granted by RWTH Aachen
University and Cloud TPUs from Google’s TensorFlow Research Cloud (TFRC).

## References

  * Alaoui Abdellaoui et al. [2020] Alaoui Abdellaoui, I., García Fernández, J., Şahinli, C., Mehrkanoon, S., 2020\.  Deep brain state classification of meg data.  arXiv preprint arXiv:2007.00897 .
  * Bauer et al. [2015] Bauer, P., Thorpe, A., Brunet, G., 2015.  The quiet revolution of numerical weather prediction.  Nature 525, 47–55.
  * Booz et al. [2019] Booz, J., Yu, W., Xu, G., Griffith, D., Golmie, N., 2019\.  A deep learning-based weather forecast system for data volume and recency analysis, in: 2019 International Conference on Computing, Networking and Communications (ICNC), IEEE. pp. 697–701.
  * Breiman [2001] Breiman, L., 2001.  Random forests.  Machine learning 45, 5–32.
  * Campbell and Diebold [2005] Campbell, S.D., Diebold, F.X., 2005\.  Weather forecasting for weather derivatives.  Journal of the American Statistical Association 100, 6–16.
  * Chen et al. [2019] Chen, R., Wang, X., Zhang, W., Zhu, X., Li, A., Yang, C., 2019.  A hybrid cnn-lstm model for typhoon formation forecasting.  Geoinformatica 23, 375–396.
  * Dueben and Bauer [2018] Dueben, P.D., Bauer, P., 2018\.  Challenges and design choices for global weather and climate models based on machine learning.  Geoscientific Model Development 11, 3999–4009.
  * Fu et al. [2019] Fu, Q., Niu, D., Zang, Z., Huang, J., Diao, L., 2019\.  Multi-stations’ weather prediction based on hybrid model using 1d cnn and bi-lstm, in: 2019 Chinese Control Conference (CCC), IEEE. pp. 3771–3775.
  * Glahn [1985] Glahn, H.R., 1985.  Statistical weather forecasting.
  * Gneiting and Raftery [2005] Gneiting, T., Raftery, A.E., 2005\.  Weather forecasting with ensemble methods.  Science 310, 248–249.
  * Ho et al. [2019] Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T., 2019\.  Axial attention in multidimensional transformers.  arXiv preprint arXiv:1912.12180 .
  * Holtslag et al. [1990] Holtslag, A., De Bruijn, E., Pan, H., 1990.  A high resolution air mass transformation model for short-range weather forecasting.  Monthly Weather Review 118, 1561–1575.
  * Kim et al. [2017] Kim, S., Hong, S., Joh, M., Song, S.k., 2017.  Deeprain: Convlstm network for precipitation prediction using multichannel radar data.  arXiv preprint arXiv:1711.02316 .
  * Kingma and Ba [2014] Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Lorenc [1986] Lorenc, A.C., 1986.  Analysis methods for numerical weather prediction.  Quarterly Journal of the Royal Meteorological Society 112, 1177–1194.
  * Mahendran and Vedaldi [2016] Mahendran, A., Vedaldi, A., 2016\.  Visualizing deep convolutional neural networks using natural pre-images.  International Journal of Computer Vision 120, 233–255.
  * Mehrkanoon [2019a] Mehrkanoon, S., 2019a.  Cross-domain neural-kernel networks.  Pattern Recognition Letters 125, 474–480.
  * Mehrkanoon [2019b] Mehrkanoon, S., 2019b.  Deep neural-kernel blocks.  Neural Networks 116, 46–55.
  * Mehrkanoon [2019c] Mehrkanoon, S., 2019c.  Deep shared representation learning for weather elements forecasting.  Knowledge-Based Systems 179, 120–128.
  * Mehrkanoon et al. [2012] Mehrkanoon, S., Falck, T., Suykens, J.A., 2012.  Approximate solutions to ordinary differential equations using least squares support vector machines.  IEEE transactions on neural networks and learning systems 23, 1356–1367.
  * Mehrkanoon et al. [2014] Mehrkanoon, S., Mehrkanoon, S., Suykens, J.A., 2014.  Parameter estimation of delay differential equations: an integration-free ls-svm approach.  Communications in Nonlinear Science and Numerical Simulation 19, 830–841.
  * Mehrkanoon and Suykens [2015] Mehrkanoon, S., Suykens, J.A., 2015\.  Learning solutions to partial differential equations using ls-svm.  Neurocomputing 159, 105–116.
  * Mehrkanoon and Suykens [2018] Mehrkanoon, S., Suykens, J.A., 2018\.  Deep hybrid neural-kernel networks using random fourier features.  Neurocomputing 298, 46–54.
  * Molnar [2020] Molnar, C., 2020.  Interpretable Machine Learning.  Lulu. com.
  * Niziol et al. [1995] Niziol, T.A., Snyder, W.R., Waldstreicher, J.S., 1995.  Winter weather forecasting throughout the eastern united states. part iv: Lake effect snow.  Weather and Forecasting 10, 61–77.
  * O’Neill et al. [2017] O’Neill, B.C., Oppenheimer, M., Warren, R., Hallegatte, S., Kopp, R.E., Pörtner, H.O., Scholes, R., Birkmann, J., Foden, W., Licker, R., et al., 2017\.  Ipcc reasons for concern regarding climate change risks.  Nature Climate Change 7, 28–37.
  * Scher [2018] Scher, S., 2018.  Toward data-driven weather and climate forecasting: Approximating a simple general circulation model with deep learning.  Geophysical Research Letters 45, 12–616.
  * Sønderby et al. [2020] Sønderby, C.K., Espeholt, L., Heek, J., Dehghani, M., Oliver, A., Salimans, T., Agrawal, S., Hickey, J., Kalchbrenner, N., 2020.  Metnet: A neural weather model for precipitation forecasting.  arXiv preprint arXiv:2003.12140 .
  * Tolstykh and Frolov [2005] Tolstykh, M., Frolov, A., 2005\.  Some current problems in numerical weather prediction.  Izvestiya Atmospheric and Oceanic Physics 41, 285–295.
  * Trebing and Mehrkanoon [2020a] Trebing, K., Mehrkanoon, S., 2020a.  Smaat-unet: Precipitation nowcasting using a small attention-unet architecture.  arXiv preprint arXiv:2007.04417 .
  * Trebing and Mehrkanoon [2020b] Trebing, K., Mehrkanoon, S., 2020b.  Wind speed prediction using multidimensional convolutional neural networks.  arXiv preprint arXiv:2007.12567 .
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017\.  Attention is all you need, in: Advances in neural information processing systems, pp. 5998–6008.
  * Webb [2018] Webb, S., 2018.  Deep learning for biology.  Nature 554.
  * Xingjian et al. [2015] Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.c., 2015\.  Convolutional lstm network: A machine learning approach for precipitation nowcasting, in: Advances in neural information processing systems, pp. 802–810.
  * Zhou et al. [2019] Zhou, K., Zheng, Y., Li, B., Dong, W., Zhang, X., 2019\.  Forecasting different types of convective weather: A deep learning approach.  Journal of Meteorological Research 33, 797–809.