11institutetext: LIT AI Lab & Institute for Machine Learning
Johannes Kepler University Linz
A-4040 Linz, Austria
22institutetext: Institute of Hydrology and Watermanagement
University of Natural Resources and Life Sciences, Vienna
A-1190 Vienna, Austria

# NeuralHydrology - Interpreting LSTMs in Hydrology

Frederik Kratzert 1st corresponding author<EMAIL_ADDRESS>Mathew
Herrnegger 2nd corresponding author<EMAIL_ADDRESS>Daniel
Klotz 11 Sepp Hochreiter 11 Günter Klambauer 11

###### Abstract

Despite the huge success of Long Short-Term Memory networks, their
applications in environmental sciences are scarce. We argue that one reason is
the difficulty to interpret the internals of trained networks. In this study,
we look at the application of LSTMs for rainfall-runoff forecasting, one of
the central tasks in the field of hydrology, in which the river discharge has
to be predicted from meteorological observations. LSTMs are particularly well-
suited for this problem since memory cells can represent dynamic reservoirs
and storages, which are essential components in state-space modelling
approaches of the hydrological system. On basis of two different catchments,
one with snow influence and one without, we demonstrate how the trained model
can be analyzed and interpreted. In the process, we show that the network
internally learns to represent patterns that are consistent with our
qualitative understanding of the hydrological system.

###### Keywords:

Neural Networks LSTM Interpretability Hydrology Rainfall-Runoff modelling

##  1 Introduction

Describing the relationship between rainfall and runoff is one of the central
tasks in the field of hydrology [21]. This involves the prediction of the
river discharge from meteorological observations from a river basin. The basin
or catchment of a river is defined by the area of which all (surface) runoff
drains to a common outlet [40]. Predicting the discharge of the river is
necessary for e.g. flood forecasting, the design of flood protection measures,
or the efficient management of hydropower plants.

Within the basin of a river, various hydrological processes take place that
influence and lead to the river discharge, including, for example,
evapotranspiration, where water is lost to the atmosphere, snow accumulation
and snow melt, water movement in the soil or groundwater recharge and
discharge (see Fig. 1).

Figure 1: Simplified visualization of processes and fluxes that influence the
river discharge, such as precipitation, snow melt, surface runoff or
subsurface flows.

The hydrological processes have highly non-linear interactions and depend, to
a large degree, on the states of the system, which represent the memory of,
e.g. a river basin. Consequently, hydrological models are formulated in a
state-space approach where the states at a specific time depend on the input
$\bm{I}_{t}$, the system states at the previous time step $\bm{S}_{t-1}$, and
a set of parameters $\Theta_{i}$ [14]:

$\bm{S}_{t}=f(\bm{I}_{t},\bm{S}_{t-1};\Theta_{i})$ (1)

The discharge at a given time step $t$ is driven by the system states and in
consequences by the meteorological events of the preceding time steps. More
generally, any output $\bm{O}_{t}$ of a hydrological system (e.g. the runoff)
can be described as:

$\bm{O}_{t}=g(\bm{I}_{t},\bm{S}_{t};\Theta_{j}),$ (2)

where $g(\cdot)$ is a mapping function that connects the states of the system
and the inputs to the system output, and $\Theta_{j}$ is the corresponding
subset of model parameters.

For making proficient predictions these non-linearities make it inevitable (at
least in classical process-based hydrological models) to explicitly implement
the hydrological processes [15, 25, 32, 38]. However, defining the
mathematical representations of the processes, including the model structures
and determining their effective parameters so that the resulting system
exhibits good performance and generalizable properties (e.g. in the form of
seamless parameter fields) still remains a challenge in the field of hydrology
[11, 22, 35].

A significant problem and limiting factor in this context is the frequently
missing information regarding the physical properties of the system [5, 10].
These tend to be highly heterogeneous in space (e.g. soil characteristics) and
can additionally change over time, e.g. vegetation cover. Our knowledge of the
properties on, or near the surface has increased significantly in the last
decades. This is mainly due to advances in high-resolution air- as well as
spaceborne remote sensing [7, 13, 29]. However, hydrology, to a significant
part, takes place underground, for which detailed information is rarely
available. In essence, the process-based models try to describe a system
determined by spatially and temporally distributed system states and physical
parameters, which are most of the time unknown.

In contrast, data-driven methods, such as Neural Networks, are solely trained
to predict the discharge, given meteorological observations, and do not
necessitate an explicit definition of the underlying processes. But these
models have not the best reputation among many hydrologists because of the
prevailing opinion that models “must work well for the right reasons” [20].
However, due to their predictive power, first studies of using Neural Networks
for predicting the river discharge date back to the early 90s [9, 12].

Recently, Kratzert et al. [23] used Long Short-Term Memory networks (LSTMs)
[17] for daily rainfall-runoff modelling and could show that LSTMs achieve
competitive results, compared to the well established Sacramento Soil Moisture
Accounting Model [8] coupled with the Snow-17 snow model [2]. LSTM is an
especially well-suited network architecture for Hydrological applications,
since the evolution of states can be modelled explicitly through time and
mapped to a given output. The approach is very similar to rainfall-runoff
models defined by Equations 1-2 (in the case of the LSTM the system states are
the memory cell states and the parameters are the learnable network weights,
[23]).

The aim of this chapter is to show different possibilities that enable the
interpretation of the LSTM and its internals in the context of rainfall-runoff
simulations. Concretely, we explore and investigate the following questions:
How many days of the past influence the output of the network at a given day
of the year? Do some of the memory cells correlate with hydrological states?
If yes, which input variables influence these cells and how? Answering these
questions is important to a) gain confidence in data-driven models, e.g. in
case of the necessity for extrapolation, b) have tools to understand possible
mistakes and difficulties in the learning process and c) potentially learn
from findings for future applications.

###  1.1 Related work

In the field of water resources and hydrology, a lot of effort has been made
on interpreting neural networks and analyzing the importance of input
variables (see [6] for an overview). However, so far only feed-forward neural
networks have been applied in these studies. Only recently, Kratzert et al.
[23] have demonstrated the potential use of LSTMs for the task of rainfall-
runoff modelling. In their work they have also shown that memory cells with
interpretable functions exist, which were found by visual inspection.

Outside of hydrology, LSTMs found a wide range of applications, which
attracted researches to study on analyzing and interpreting the network
internals. For example Hochreiter et al. [16] found new protein motifs through
analyzing LSTM memory cells. Karpathy et al. [18] inspected memory cells in
character level language modelling and identify some interpretable memory
cells, e.g. cells that track the line-length or cells that check if the
current text is inside brackets of quotation marks. Li et al. [24] inspected
trained LSTMs in the application of sentence- and phrase-based sentiment
classification and showed through saliency analysis, which parts of the inputs
are influencing the network prediction most. Arras et al. [3] used Layer-wise
Relevance Propagation to calculate the impact of single words on sentiment
analysis from text sequences. Also for sentiment analysis, Murdoch et al. [28]
present a decomposition strategy for the hidden and cell state of the LSTM to
extract the contributions of single words on the overall prediction. Poerner
et al. [33] summarize various interpretability efforts in the natural language
processing domain and present an extension of the LIME framework, introduced
originally by Reibiero et al. [34]. Strobelt et al. [36] developed LSTMVis, a
general purpose visual analysis tool for inspecting hidden state values in
recurrent neural networks.

Inspired by these studies, we investigate the internals of LSTMs in the domain
of environmental science and compare our findings to hydrological domain
knowledge.

##  2 Methods

###  2.1 Model architecture

In this study, we will use a network consisting of a single LSTM layer with 10
hidden units and a dense layer, that connects the output of the LSTM at the
last time step to a single output neuron with linear activation. To predict
the discharge of a single time step (day) we provide the last 365 time steps
of meteorological observations as inputs. Compared to Eq. 1 we can formulate
the LSTM as:

$\\{\bm{c}_{t},\bm{h}_{t}\\}=f_{\mathrm{LSTM}}(\bm{i}_{t},\bm{c}_{t-1},\bm{h}_{t-1};\Theta_{k}),$
(3)

where $f_{\mathrm{LSTM}}(\cdot)$ symbolizes the LSTM cell that is a function
of the meteorological input $\bm{i}_{t}$ at time $t$, and the previous cell
state $\bm{c}_{t-1}$ as well as the previous hidden state $\bm{h}_{t-1}$,
parametrized by the network weights $\Theta_{k}$. The output of the system,
formally described in Eq. 2, would in this specific case be given by:

$y=f_{\mathrm{Dense}}(\bm{h}_{365};\Theta_{l}),$ (4)

where $y$ is the output of a dense layer $f_{\mathrm{Dense}}(\cdot)$
parametrized by the weights $\Theta_{l}$, which predicts the river discharge
from the hidden state at the end of the input sequence $\bm{h}_{365}$.

The difference between the LSTM and conventional rainfall-runoff models is
that the former has the ability to infer the needed structure/parametrization
from data without preconceived assumptions about the nature of the processes.
This makes them extremely attractive for hydrological applications.

The network is trained for 50 epochs to minimize the mean squared error using
RMSprop [39] with an initial learning rate of 1e-2. The final model is
selected based on the score of an independent validation set.

###  2.2 Data

In this work, we concentrate on two different basins from the publicly
available CAMELS data set [1, 31]. Basin A, which is influenced by snow, and
basin B, which is not influenced by snow. Some key attributes of both basins
can be found in Table 1.

Table 1: Basin overview. Basin | IDiiiUSGS stream gauge ID | Snow | Area (km2) | NSE | NSE
---|---|---|---|---|---
fractioniiiiiiFraction of precipitation falling with temperatures below 0∘C | validation | test
A | 13340600iiiiiiiiiClearwater river, CA | 56 % | 3357 | 0.79 | 0.76
B | 11481200ivivivLittle river, CA | 0 % | 105 | 0.72 | 0.72

For meteorological forcings, the data set contains basin averaged daily
records of precipitation (mm/d), solar radiation (W/m2), minimum and maximum
temperature (∘C) and vapor pressure (Pa). The streamflow is reported as daily
average (m3/s) and is normalized by the basin area (m2) to (mm/d).
Approximately 33 years of data is available, of which we use the first 15 for
training the LSTMs. Of the remaining years the first 25 % is used as
validation data by which we select the final model. The remaining data points
(approx. 13 years) are used for the final evaluation and for all experiments
in this study. The meteorological input features, as well as the target
variable, the discharge are normalized by the mean and standard deviation of
the training period.

One LSTM is trained for each basin separately and the trained model is
evaluated using the Nash-Sutcliffe-Efficiency [30], an established measure
used to evaluate hydrological time series given by the following equation:

$\text{NSE}=1-\frac{\sum_{t=1}^{T}(Q_{m}^{t}-Q_{o}^{t})^{2}}{\sum_{t=1}^{T}(Q_{o}^{t}-\bar{Q}_{o})^{2}},$
(5)

where $T$ is the total number of time steps, $Q_{m}^{t}$ is the simulated
discharge at time $t$ $(1\leq t\leq T)$, $Q_{o}^{t}$ is the observed discharge
at time $t$ and $\bar{Q}_{o}$ is the mean observed discharge. The range of the
NSE is (-inf, 1], where a value of 1 means a perfect simulation, a NSE of 0
means the simulation is as good as the mean of the observation and everything
below zero means the simulation is worse compared to using the observed mean
as a prediction.

In the test period the LSTM achieves a NSE of above 0.7 (see Table 1), which
can be considered a reasonably good result [27].

Figure 2: Example of predicted (dashed line) and observed discharge (solid
line) of two years of the test period in the snow influenced basin.
Corresponding daily precipitation sums are plotted upside down, where snow
(temperature below 0∘C) is plotted darker.

Figure 2 shows observed and simulated discharge of two years of the test
period in the snow influenced basin A, as well as the input variable
precipitation. We can see that the discharge has its peak in the spring/early
summer und that the model in the year 2012 underestimates the discharge, while
in the second year it fits the observed discharge pretty well. The time lag
between precipitation and discharge can be explained by snow accumulation in
the winter months and subsequent melt of this snow layer in spring.

###  2.3 Integrated gradients

Different methods have been presented recently to analyze the attribution of
input variables on the network output (or any in-between neuron) (e.g. [3, 4,
19, 26, 37])

In this study we focus on Integrated gradients by Sundarajan et al. [37].
Here, the attribution of each input to e.g. the output neuron is calculated by
looking at the change of this neuron when the input shifts from a baseline
input to the target input of interest. Formally, let $\bm{x}$ be the input of
interest, (in our case a sequence of 365 time steps with 5 meteorological
observations each), ${\bm{x}}^{\prime}$ the baseline input and $F(\cdot)$ the
neuron of interest. Then the integrated gradients, for the i-th input variable
$x_{i}$, can be appoximated by:

$\text{IntegratedGrads}_{i}^{\mathrm{approx}}(\bm{x}):=\frac{\mathbf{x}_{i}-\mathbf{x^{\prime}}_{i}}{m}\sum_{k=1}^{m}\frac{\partial
F\left(\mathbf{\tilde{x}}\right)}{\partial\mathbf{\tilde{x}}_{i}}\bigg{|}_{\mathbf{\tilde{x}}=\mathbf{x^{\prime}}+\frac{k}{m}(\mathbf{x}-\mathbf{x^{\prime}})},$
(6)

where $m$ is the number of steps used to approximate the integral (here
$m=1000$). As baseline ${\bm{x}}^{\prime}$, we used an input sequence of
zeros.

###  2.4 Experiments

####  2.4.1 Question 1: How many days of past influence the network output?

The discharge of a river in a seasonal influenced region varies strongly
throughout the year. For e.g. snow influenced basins the discharge usually
peaks in the spring or early summer, when not only precipitation and
groundwater but also snow melt contributes to the discharge generation.
Therefore, at least from a hydrological point of view, the precipitation of
the entire winter might be influential for the correct prediction of the
discharge. In contrast, in drier periods (e.g. here at the end of summer) the
discharge likely depends on far fewer time steps of the meteorological past.
Since we provide a constant number of time steps (365 days) of meteorological
data as input, it is interesting to see how many of the previous days are
really used by the LSTM for the final prediction.

To answer this question, we calculate the integrated gradients for one sample
of the test period w.r.t. the input variables and sum the integrated gradients
across the features for each time step. We then calculate the difference from
time step to time step and determine the first timestep $t$ $(1\leq t\leq T)$,
at which the difference surpasses a threshold of 2e-3, with $T$ being the
total length of the sequence. We have chosen the threshold value empirically
so that noise in the integrated gradient signal is ignored. For each sample
the number of Time Steps Of Influence (TSOI) on the final prediction can then
be calculated by:

$\text{TSOI}=T-n$ (7)

This is repeated for each day of each year in the test period.

####  2.4.2 Question 2: Do memory cells correlate with hydrological states?

The discharge of a river basin is frequently approximated by decomposing its
(hypothetical) components into a set of interacting reservoirs or storages
(see Fig. 1). Take snow as an example, which is precipitation that falls if
temperatures are below 0∘C. It can be represented in a storage $\bm{S}$ (see
Eq. 1), which generally accumulates during the winter period and depletes from
spring to summer when the temperatures rise above the melting point. Similarly
other components of the system can be modelled as reservoirs of lower or
higher complexity. The soil layer, for example, can also be represented by a
bucket, which is filled - up to a certain point - by incoming water (e.g.
rainfall) and depleted by evapotranspiration, horizontal subsurface flow and
water movement into deeper layers, e.g. the groundwater body.

Theoretically, memory cells of LSTMs could learn to mimic these storage
processes. This is a crucial property, at least from a hydrological point of
view, to be able to correctly predict the river discharge. Therefore the aim
of this experiment is to see if certain memory cells $\bm{c}_{t}$ (Eq. 3)
correlate to these hydrological states $\bm{S}_{t}$ (Eq. 1).

Because the CAMELS data does not include observations for these states, we
take the system states of the included SAC-SMA + Snow-17 model as a proxy.
This is far from optimal, but since this is a well established and studied
hydrological model, we can assume that at least the trend and tendencies of
these simulations are correct. Furthermore, we only want to test in this
experiment if memory cells correlate with these system states and not if they
quantitatively match these states exactly. Of the calibrated SAC-SMA + Snow-17
we use the following states as a reference in this experiment:

  * •

SWE (snow water equivalent): This is the amount of water stored in the snow
layer. This would be available if the entire snow in the system would melt.

  * •

UZS (upper zone storage): This state is calculated as the sum of the UZTWC
(upper zone tension water storage content) and the UZFWC (upper zone free
water storage) of the SAC-SMA + Snow-17 model. This storage represents upper
layer soil moisture and controls the fast response of the soil, e.g. direct
surface runoff and interflow.

  * •

LZS (lower zone storage): This state is calculated by the sum of LZTWC (lower
zone tension water content), LZFSC (lower zone free supplemental water storage
content) and LZFPC (lower zone free primary water storage content). This
storage represents the groundwater storage and is relevant for the
baseflowvvvFollowing [40] the baseflow is defined as: “Discharge which enters
a stream channel mainly from groundwater, but also from lakes and glaciers,
during long periods when no precipitation or snowmelt occurs.”.

For each sample in the test period we calculate the correlation of the cell
states with the corresponding time series of these four states.

####  2.4.3 Question 3: Which inputs influence a specific memory cell?

Suppose that we find memory cells that correlate with time series of
hydrological system states, then a natural question would be if the inputs
influencing these memory cells agree with our understanding of the
hydrological system. For example, a storage that represents the snow layer in
a catchment, should be influenced by precipitation and solar radiation in a
contrarious way. Solid precipitation or snow would increase the amount of snow
available in the system during winter. At the same time, solar radiation,
providing energy for sublimation, would effectively reduce the amount of snow
stored in the snow layer. Therefore, in this experiment we look in more detail
at specific cells that emerged from the previous experiment and analyse the
influencing variables on this cell. We do this by calculating the integrated
gradients from a single memory cell at the last time step of the input
sequence w.r.t. the input variables.

##  3 Results and Discussion

###  3.1 Timesteps influencing the network output

Figure 3 shows how many time steps with meteorological inputs from the past
have an influence on the LSTM output at the time step of prediction (TSOI).
The TSOI does thereby not differentiate between single inputs. It is rather
the integrated signal of all inputs. Instead of using specific dates, we here
show the temporal dimension in the unit day of year (DOY). Because all years
of the test period are integrated in the plot, we show the 25 %, 50 % and 75 %
percent quantiles. For the sake of interpretation and the seasonal context,
Fig. 3 also includes the temporal dynamics of the median precipitation,
temperature and discharge.

Figure 3: Time steps of influence (TSOI) on the network output over the unit
day of year (DOY) for the snow influenced basin A (left column) and the basin
B without snow influence (right column). Corresponding median precipitation,
discharge and minimum temperature are shown for reference. For the snow-
influenced basin A, we can see for example that the TSOI increases during the
winter period and is largest during the snow melting period ($\sim$DOY
100-160), which matches our understanding of the hydrological processes.

The left column of Fig. 3 shows the results for snow influenced basin A. Here,
3 different periods can be distinguished in the TSOI time series:

(1) Between DOY 200 and 300 the TSOI shows very low values of less than 10-20
days. This period corresponds to the summer season characterised by high
temperatures and low flows, with fairly little precipitation. In this period
the time span of influence of the inputs on the output is short. From a
hydrological perspective this makes sense, since the discharge in this period
is driven by short-term rainfall events, which lead to a very limited response
in the discharge. The short time span of influence can be explained by higher
evapotranspiration rates in this season and lower precipitation amounts.
Higher evapotranspiration rates lead to the loss of precipitation to the
atmosphere, which is then missing in the discharge. The behaviour of the
hydrograph is fairly easy to predict and does not necessitate much information
about the past.

(2) In the winter period, starting with DOY 300, the TSOI increases over time
reaching a plateau of 140-150 days around DOY 100. In this period the daily
minimum temperature is below 0∘C, leading to solid precipitation (snow) and
therefore water being stored as snow in the catchment without leading to
runoff. This is underlined with the low discharge values, despite high
precipitation input. Thus the LSTM has to understand that this input does not
lead to an immediate output and therefore the TSOI has to increase. The
plateau is reached, as soon as the minimum temperature values are higher than
the freezing point. From a hydrological perspective, it is interesting to
observe that the TSOI at the end of the winter season has a value which
corresponds to the beginning of the winter season ($\sim$DOY 300), when the
snow accumulation begins. It should be noted that the transition between the
winter period and the following spring season is not sharp, at least when
taking the hydrograph as a reference. It is visible that, although the TSOI is
still increasing, the discharge is also increasing. From a hydrological point
of view this can be explained by a mixed signal in discharge - appart from
melting snow (daily maximum temperature is larger than 0∘C), we still have
negative minimum temperatures, which would lead to snow fall.

(3) In spring, during the melting season, the TSOI stays constant (DOY
100-160) followed by a sharp decrease until the summer low flows. During the
main melting period, the TSOI of approximately 140-150 days highlights that
the LSTM uses the entire information of the winter period to predict the
discharge. The precipitation in this period now falls as rain, directly
leading to runoff, without increasing the TSOI. At the same time, all the
inputs from the winter still influence the river discharge explaining the
stable plateau. The sharp decrease of the TSOI around DOY 160 represents a
transition period where the snow of the winter continuously loses its
influence until all snow has melted.

Although it has the same precipitation seasonality, Basin B (Fig. 3, right
column) has different characteristics compared to basin A, since it is not
influenced by snow. Here, only 2 different periods can be distinguished, where
the transitions periods are however more pronounced:

(1) Between DOY 180 and 280, the warm and dry summer season, the catchment is
characterised by very low flows. In this period the TSOI is also constantly
low, with values of around 10-15 days. The discharge can be predicted with a
very short input time series, since rainfall as input is missing and the
hydrology does not depend on any inputs.

(2) Following summer, the wet period between DOY 280 and 180 is characterised
by a steady increase in the TSOI. The temporal influence of rainfall on runoff
becomes longer, the longer the season lasts. The general level of runoff is
now significantly higher compared to the summer. It does not solely depend on
single rainfall events, but is driven by the integrated signal of inputs from
the past, explaining the increasing TSOI. The TSOI reaches a maximum median
value of around 120 days. This peak is followed by a decrease towards the end
of the wet period, which is however not as rapid as in basin A. This distinct
transition period in TSOI between wet and dry ($\sim$DOY 100-180) season
corresponds very well with the observed falling limb in the hydrograph. As the
runoff declines, the influence of past meteorological inputs also declines.
Compared to basin A, a higher variability in TSOI is evident, which can be
explained with a higher variability in precipitation inputs in the single
years. In basin A, a high interannual variability in the rainfall is also
observable. However, the lower temperatures below freezing level lead to the
precipitation falling as snow, and therefore act as a filter. This leads to a
lower variability in discharge and in consequence in TSOI.

Overall, the TSOI results of the two contrasting basins match well with our
hydrological understanding of the anterior days influencing the runoff signal
at a specific day. It is interesting to see that the LSTM shows the capability
to learn these differing, basin specific properties of long-term dependencies.

###  3.2 Correlation of memory cells with hydrological states

Figure 4 shows the average correlation of every memory cell with the
hydrological states considered in this experiment. The correlation is averaged
over all samples in the test period and only correlations with $\rho>0.5$ are
shown. We can see, that in both basins some of the memory cells have a
particularly high correlation with the provided hydrological states. For both
basins several cells exhibit a high correlation with both, the upper (UZS) and
lower (LZS), soil states. Although the majority of the cells show a positive
correlation, negative correlations are also visible, which are however of a
lower absolute magnitude.

Figure 4: Average correlations between memory cells and hydrological states
for basin A (upper plot) and basin B (lower plot). Only correlations with
$\rho>0.5$ are shown. Ellipse are scaled by the absolute correlation and
rotated by their sign (left inclined for negative correlation and right
inclined for positive correlation)

The correlation between the LSTM cells and the baseflow influencing state LZS
are significantly higher for the drier basin B. However, the baseflow index, a
measure to define the importance of the contribution of the baseflow to total
runoff, is lower for this basin. In basins A and B the ratios of mean daily
baseflow to mean daily discharge are about 66 % and 44 %, respectively.
Currently, we cannot explain this discrepancy. In the snow influenced basin,
the trained LSTM also has some cells with high correlation with the snow-
water-equivalent (SWE). The occurrence of multiple cells with high correlation
to different system states can be seen as an indicator that the LSTM is not
yet defined in a parsimonious way. Therefore, hydrologist can use this
information to restrict the neural network even further.

In general, the correlation analysis is difficult to interpret in detail.
Frequently, high correlations however exist, indicating a strong relationship
between LSTM memory cells and system states from a well-established
hydrological model.

###  3.3 Inspection of memory cells

In the first experiment, we used the integrated gradient method to calculate
the attribution of the input variables on the output of the model. In this
experiment, we apply it to analyse interactions between arbitrary neurons
within the neural network (e.g. here, a memory cell at the last time step).
The previous experiment proved that memory cells with a high correlation to
some of the hydrological system states exist. The aim of this experiment is
therefore to analyse the influences and functionality of a given cell. Here,
we can explore (i) which (meteorological) inputs are important and (ii) at
what time in the past this influence was high. We chose a “snow-cell” from the
previous experiment to demonstrate this task, since the accumulation and
depletion of snow is a particularly illustrative example. To be more precise,
we chose to depict a single sample of the test period from the cell with the
highest correlation to the SWE, which is memory cell 4 from basin A (see Fig.
4). Figure 5 shows the integrated gradients of the meteorological inputs in
the top row, the evolution of the memory cell value in the second row and the
corresponding change in minimum and maximum temperature in the third row.

Figure 5: Integrated gradient analysis of the snow-cell (cell 4) of the LSTM
trained for basin A. Upper plot shows the integrated gradient signal on each
input variable at each time step. The plot in the center shows the memory cell
state over time for reference and the bottom plot the minimum and maximum
daily temperature.

One can see that the snow-cell is mainly influenced by the meteorological
inputs of precipitation, minimum temperature, maximum temperature and solar
radiation. Precipitation shows the largest magnitude of positive influence.
Solar radiation in contrast has a negative sign of influence, possibly
reproducing the sublimation from the snow layer and leading to a reduction of
the snow state. All influencing factors only play a role at temperatures
around or below freezing level. This matches the expected behaviour of a snow
storage from a hydrological point of view: Lower temperatures and concurrent
precipitation are associated with snow accumulation. Consequently, this leads
to an increase in magnitude of the memory cell value (especially for the first
temperature below the freezing point). This can be observed e.g. in October
1999, where the temperature values decrease, the influences of the
meteorological parameters appear and the snow-cell begins to accumulate. In
contrast, as the temperature rises, the value of the cell decreases again,
especially when the daily minimum temperature also rises above 0∘C.

This suggests that the LSTM realistically represents short- as well as long-
term dynamics in the snow cell storage and their connection to the
meteorological inputs.

##  4 Conclusion

LSTM networks are a versatile tool for time series predictions, with many
potential applications in hydrology and environmental sciences in general.
However, currently they do not enjoy a wide-spread application. We argue that
one reason is the difficulty to interpret the LSTMs. The methods presented in
this book provide solutions regarding interpretability and allow a deeper
analysis of these models. In this chapter, we demonstrate this for the task of
rainfall-runoff modelling (where the aim is to predict the river discharge
from meteorological observations). In particular, we were able to show that
the processes learned by the LSTM matches our comprehension of a real-world
environmental system.

For this study, we focused on a qualitative analysis of the correspondence
between the hydrological system and the learned behaviour of the LSTM. In a
first experiment, we looked at the number of time steps of influence on the
network output (TSOI) and how this number varies throughout the year. We saw
that the TSOI pattern matches our hydrological understanding of the yearly
pattern. In the next experiment, we looked at the correlation of the memory
cells of the network with some selected states of the hydrological system
(such as snow or soil moisture). We found some cells that exhibited relatively
high correlation with the chosen states, which strengthens the hypothesis that
the LSTM obtained some general understanding of the runoff-generation
processes. In the last experiment, we inspected a single memory cell that
exhibited a high correlation with the snow state. We analyzed the influencing
inputs over time through the integrated gradient method and could see, that
the behavioral patterns manifested in the cell, closely resemble the ones
suggested by hydrological theory. We view this as a further underpinning of
the observation that the internally modelled processes of the network follow
some sort of physically viable pattern. We hypothesize that this relation can
be seen as a legitimization of the LSTM usage within environmental sciences
applications, and thus believe that the presented methods will pave the way
for its future in environmental-modelling. The correspondence of the memory
cells and the physical states can be especially useful in novel situations,
which often arise in this context. Environmental scientists and practitioners
can exploit it (together with the proposed techniques) to “peek into the LSTM”
and argue about potential behaviours. Our demonstration was certainly not
exhaustive and should rather be seen as indicative application study.

The most important message is that the combination of domain-knowledge (in
this case hydrology) and the insights provided by the proposed interpretation
techniques, provide the fundamentals for designing environmental forecasting
systems with neural networks. Consequently, we expect that the combination of
powerful data driven models (such as LSTMs) with the possibility of
interpretation by experts will lead to new insights in the field of
application.

## References

  * [1] Addor, N., Newman, A.J., Mizukami, N., Clark, M.P.: Catchment attributes for large-sample studies. Boulder, CO: UCAR/NCAR (2017)
  * [2] Anderson, E.A.: National Weather Service River Forecast System - Snow Accumulation and Ablation Model. Tech. Rep. November, US Department of Commerce, Silver Spring (1973)
  * [3] Arras, L., Montavon, G., Müller, K.R., Samek, W.: Explaining Recurrent Neural Network Predictions in Sentiment Analysis. arXiv preprint arXiv:1706.07206 (2017)
  * [4] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), 1–46 (2015)
  * [5] Beven, K.: How far can we go in distributed hydrological modelling ? Hydrological and Earth System Sciences 5(1), 1–12 (2001)
  * [6] Bowden, G.J., Dandy, G.C., Maier, H.R.: Input determination for neural network models in water resources applications. Part 1 - Background and methodology. Journal of Hydrology 301(1-4), 75–92 (2005)
  * [7] Brenner, C., Thiem, C.E., Wizemann, H.D., Bernhardt, M., Schulz, K.: Estimating spatially distributed turbulent heat fluxes from high-resolution thermal imagery acquired with a UAV system. International Journal of Remote Sensing 38(8-10), 3003–3026 (2017)
  * [8] Burnash, R.J.C., Ferral, R.L., McGuire, R.A.: A generalised streamflow simulation system–conceptual modelling for digital computers. Tech. rep., US Department of Commerce National Weather Service and State of California Department of Water Resources (1973)
  * [9] Daniell, T.M.: Neural networks—applications in hydrology and water resources engineering. In: Proceedings of the International Hydrology and Water Resources Symposium. vol. 3, pp. 797–802. Institution of Engineers, Perth, Australia (1991)
  * [10] Freeze, R.A., Harlan, R.L.: Blueprint for a physically-based, digitally-simulated hydrologic response model. Journal of Hydrology 9(3), 237–258 (1969)
  * [11] Gupta, H.V., Sorooshian, S., Yapo, P.O.: Status of Automatic Calibration for Hydrologic Models: Comparison with Multilevel Expert Calibration. Journal of Hydrologic Engineering 4(2), 135–143 (1999)
  * [12] Half, A.H., Half, H.M., Azmoodeh, M.: Predicting runoff from rainfall using neural networks. In: ASCE, New York, 760-765. pp. 760–765. New York, USA (1993)
  * [13] Hengl, T., De Jesus, J.M., Heuvelink, G.B., Gonzalez, M.R., Kilibarda, M., Blagotić, A., Shangguan, W., Wright, M.N., Geng, X., Bauer-Marschallinger, B., Guevara, M.A., Vargas, R., MacMillan, R.A., Batjes, N.H., Leenaars, J.G., Ribeiro, E., Wheeler, I., Mantel, S., Kempen, B.: SoilGrids250m: Global gridded soil information based on machine learning, vol. 12 (2017)
  * [14] Herrnegger, M., Nachtnebel, H.P., Schulz, K.: From runoff to rainfall: Inverse rainfall-runoff modelling in a high temporal resolution. Hydrology and Earth System Sciences 19(11), 4619–4639 (2015)
  * [15] Herrnegger, M., Nachtnebel, H.P., Haiden, T.: Evapotranspiration in high alpine catchments – an important part of the water balance! Hydrology Research 43(4), 460 (2012)
  * [16] Hochreiter, S., Heusel, M., Obermayer, K.: Fast model-based protein homology detection without alignment. Bioinformatics 23(14), 1728–1736 (2007)
  * [17] Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9(8), 1735–1780 (1997)
  * [18] Karpathy, A., Johnson, J., Fei-Fei, L.: Visualizing and Understanding Recurrent Networks. arXiv preprint arXiv:1506.02078 (2015)
  * [19] Kindermans, P.J., Schütt, K.T., Alber, M., Müller, K.R., Erhan, D., Kim, B., Dähne, S.: Learning how to explain neural networks: PatternNet and PatternAttribution pp. 1–12 (2017)
  * [20] Klemeš, V.: Dilettantism in hydrology: Transition or destiny? Water Resources Research 22(9 S), 177S–188S (1986)
  * [21] Klemes, V.: Stochastic models of rainfall-runoff relationship (1982)
  * [22] Klotz, D., Herrnegger, M., Schulz, K.: Symbolic Regression for the Estimation of Transfer Functions of Hydrological Models. Water Resources Research 53(11), 9402–9423 (2017)
  * [23] Kratzert, F., Klotz, D., Brenner, C., Schulz, K., Herrnegger, M.: Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks. Hydrology and Earth System Sciences 22(11), 6005–6022 (2018)
  * [24] Li, J., Chen, X., Hovy, E., Jurafsky, D.: Visualizing and Understanding Neural Models in NLP. arXiv preprint arXiv:1506.01066 (2015)
  * [25] Lindström, G., Pers, C., Rosberg, J., Strömqvist, J., Arheimer, B.: Development and testing of the HYPE (Hydrological Predictions for the Environment) water quality model for different spatial scales. Hydrology Research 41(3-4), 295 (2010)
  * [26] Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition 65(May 2016), 211–222 (2017)
  * [27] Moriasi, D.N., Gitau, M.W., Pai, N., Daggupati, P.: Hydrologic and Water Quality Models: Performance Measures and Evaluation Criteria. Transactions of the ASABE 58(6), 1763–1785 (2015)
  * [28] Murdoch, W.J., Liu, P.J., Yu, B.: Beyond word importance: Contextual decomposition to extract interactions from LSTMs. In: International Conference on Learning Representations (2018)
  * [29] Myneni, R.B., Hoffman, S., Knyazikhin, Y., Privette, J.L., Glassy, J., Tian, Y., Wang, Y., Song, X., Zhang, Y., Smith, G.R., Lotsch, A., Friedl, M., Morisette, J.T., Votava, P., Nemani, R.R., Running, S.W.: Global products of vegetation leaf area and fraction absorbed PAR from year one of MODIS data. Remote Sensing of Environment 83(1-2), 214–231 (2002)
  * [30] Nash, J.E., Sutcliffe, J.V.: River flow forecasting through conceptual models part I - A discussion of principles. Journal of Hydrology 10(3), 282–290 (1970)
  * [31] Newman, A., Sampson, K., Clark, M., Bock, A., Viger, R., Blodgett, D.: A large-sample watershed-scale hydrometeorological dataset for the contiguous USA. Boulder, CO: UCAR/NCAR. (2014)
  * [32] Perrin, C., Michel, C., Andréassian, V.: Improvement of a parsimonious model for streamflow simulation. Journal of Hydrology 279(1-4), 275–289 (2003)
  * [33] Poerner, N., Schütze, H., Roth, B.: Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). vol. 1, pp. 340–350 (2018)
  * [34] Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144. ACM (2016)
  * [35] Samaniego, L., Kumar, R., Thober, S., Rakovec, O., Zink, M., Wanders, N., Eisner, S., Müller Schmied, H., Sutanudjaja, E., Warrach-Sagi, K., Attinger, S.: Toward seamless hydrologic predictions across spatial scales. Hydrology and Earth System Sciences 21(9), 4323–4346 (2017)
  * [36] Strobelt, H., Gehrmann, S., Pfister, H., Rush, A.M.: LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks. IEEE Transactions on Visualization and Computer Graphics 24(1), 667–676 (2018)
  * [37] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. pp. 3319–3328. JMLR. org (2017)
  * [38] Thielen, J., Bartholmes, J., Ramos, M.H., de Roo, A.: The European Flood Alert System – Part 1: Concept and development. Hydrology and Earth System Sciences Discussions 5(1), 257–287 (2008)
  * [39] Tieleman, T., Hinton, G.: Lecture 6.5 - RMSProp, COURSERA: Neural Networks for Machine Learning. Tech. rep. (2012)
  * [40] WMO, W.M.O., UNESCO, U.N.E., Organization, S.: International Glossary of Hydrology. No. 12, Geneva, Switzerland (1998)