11institutetext: Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Improving Context-aware Neural Machine Translation with Target-side Context

Hayahide Yamagishi 11 Mamoru Komachi 11

###### Abstract

In recent years, several studies on neural machine translation (NMT) have
attempted to use document-level context by using a multi-encoder and two
attention mechanisms to read the current and previous sentences to incorporate
the context of the previous sentences. These studies concluded that the
target-side context is less useful than the source-side context. However, we
considered that the reason why the target-side context is less useful lies in
the architecture used to model these contexts.

Therefore, in this study, we investigate how the target-side context can
improve context-aware neural machine translation. We propose a weight sharing
method wherein NMT saves decoder states and calculates an attention vector
using the saved states when translating a current sentence. Our experiments
show that the target-side context is also useful if we plug it into NMT as the
decoder state when translating a previous sentence.

###### Keywords:

n

eural machine translation; document; context; weight sharing.

##  1 Introduction

---

Figure 1: Separated source model.

Figure 2: Separated target model. Figure 3: Shared source model.

Figure 4: Shared target model. Figure 5: Proposed methods: dashed line
represents the weight sharing with the encoders or decoders.

Neural machine translation (NMT; Sutskever et al. [1], Bahdanau et al. [2],
Vaswani et al. [3]) has become popular in recent years because it can handle
larger contexts compared to conventional machine translation systems. However,
most of the NMTs do not employ document-level contexts due to lack of an
efficient mechanism, similar to other machine translation systems.

Recently, a few studies have attempted to expand the notion of a sentence-
level context in NMT to that of a document-level context111Hereinafter,
“document-level context” is simply referred to as a “context”.. It is reported
that the information of one or more previous sentences improves the scores of
automatic and human evaluations.

Context-aware NMT systems typically have two encoders: one is for a current
sentence and the other is for a previous sentence. For instance, Bawden et al.
[4] showed that encoding a previous target sentence does not improve the
performance in an English–French task even though encoding a previous source
sentence works well. Other studies that utilized a multi-encoder (Jean et al.
[5], Voita et al. [6], Zhang et al. [7]) did not use a previous target
sentence. Thus, there are a few works on handling the target-side context.
Moreover, these previous works mainly used language pairs that belonged to the
same language family. In distant language pairs, the information of discourse
structures in the target-side document might be useful because distant
languages might have different discourse structures.

Therefore, this study investigates how the target-side context can be used in
context-aware NMT. We hypothesize that the source-side contexts should be
incorporated into an encoder and the target-side contexts should be
incorporated into a decoder. To validate this hypothesis, we propose a weight
sharing method, in which NMT saves the decoder states and calculates an
attention vector using the saved states when translating a current sentence.
We find that target-side contexts are also useful if they are inserted into
the NMT as the decoder states. This method can obtain competitive or even
better results compared to a baseline model using source-side features.

The main findings of this study are as follows:

  * •

The target-side context is as important as the source-side context.

  * •

The effectiveness of source-side context depends on language pairs.

  * •

Weight sharing between current and context states is effective for context-
aware NMT.

##  2 Model Architecture

Figure 5 presents our methods. We build context-aware NMT based on the multi-
encoder model proposed by Bawden et al. [4]. A parallel document $D$
consisting of $L$ sentence pairs, is denoted by
$D=(X^{1},Y^{1}),...,(X^{i},Y^{i}),...,(X^{L},Y^{L})$, where $X$ and $Y$ are
source and target sentences, respectively. Each sentence, $X^{i}$ or $Y^{i}$,
is denoted as $X^{i}=x_{1}^{i},...,x_{m}^{i},...,x_{M^{i}}^{i}$ or
$Y^{i}=y_{1}^{i},...,y_{n}^{i},...,y_{N^{i}}^{i}$, where $x_{m}^{i}$ or
$y_{n}^{i}$ are the tokens, and $M^{i}$ or $N^{i}$ are the sentence lengths.
The objective is to maximize the following probabilities:

$\displaystyle
p(Y^{i}|X^{i},Z^{i-1})=\prod_{n=1}^{N^{i}}p(y_{n}^{i}|y_{<n}^{i},X^{i},Z^{i-1})$
(1)

where $Z^{i-1}$ represents a previous sentence, $X^{i-1}$ or $Y^{i-1}$,
depending on the experimental settings. Each $p$ is calculated as follows:

$\displaystyle p(y_{n}^{i}|y_{<n}^{i},X^{i},Z^{i-1})$ $\displaystyle=$
$\displaystyle\mathrm{softmax}(W_{\mathrm{o}}\tilde{\bm{h}}_{n}^{i})$ (2)
$\displaystyle\tilde{\bm{h}}_{n}^{i}$ $\displaystyle=$ $\displaystyle
W_{\mathrm{h}}[\bm{h}_{n}^{i};\bm{c}_{n}^{i};\bm{c}_{n}^{i-1}]$ (3)
$\displaystyle\bm{c}_{n}^{i}$ $\displaystyle=$
$\displaystyle\sum_{m=1}^{M^{i}}\alpha_{n,m}^{i}\bm{s}_{m}^{i}$ (4)
$\displaystyle\alpha_{n,m}^{i}$ $\displaystyle=$
$\displaystyle\mathrm{softmax}(\bm{s}_{m}^{i}\cdot\bm{h}_{n}^{i})$ (5)

where $\bm{s}_{m}^{i}$, $\bm{h}_{n}^{i}$, and $\bm{c}_{n}^{i}$ represents
encoder states, decoder states, and attention, respectively.
$W_{\mathrm{o}}\in\mathbb{R}^{V\times H}$ and
$W_{\mathrm{h}}\in\mathbb{R}^{H\times 3H}$ represents weights. We calculate
the encoder state $\bm{s}_{m}^{i}$ and the decoder state $\bm{h}_{n}^{i}$ as
follows:

$\displaystyle\bm{s}_{m}^{i}$ $\displaystyle=$
$\displaystyle\mathrm{LSTM}_{enc}(W_{\mathrm{x}}x_{m}^{i},\bm{s}_{m-1}^{i})$
(6) $\displaystyle\bm{h}_{n}^{i}$ $\displaystyle=$
$\displaystyle\mathrm{LSTM}_{dec}(W_{\mathrm{y}}y_{n}^{i},\bm{h}_{n-1}^{i})$
(7)

where $W_{\mathrm{x}}\in\mathbb{R}^{E\times V}$ and
$W_{\mathrm{y}}\in\mathbb{R}^{E\times V}$ represents word embeddings of
source- and target sides, respectively. We use the dot product of encoder
states and hidden states as an attention score $\alpha_{n,m}^{i}$, proposed by
Luong et al. [8].

The multi-encoder model has an additional attention, $\bm{c}_{n}^{i-1}$, which
is for using the information of a previous sentence.

$\displaystyle\bm{c}_{n}^{i-1}$ $\displaystyle=$
$\displaystyle\sum_{t=1}^{|Z^{i-1}|}\beta_{n,t}^{i-1}\bm{z}_{t}^{i-1}$ (8)
$\displaystyle\beta_{n,t}^{i-1}$ $\displaystyle=$
$\displaystyle\mathrm{softmax}(\bm{z}_{t}^{i-1}\cdot\bm{h}_{n}^{i})$ (9)

We experiment using two methods, separated model and shared model. The
separated model represents the conventional multi-encoder model, and the
shared model is our proposed method. The difference between the two methods is
the calculation of $\bm{z}_{t}^{i-1}$.

---
Figure 6: Shared mix model.

###  2.1 Separated model

Context-aware NMT saves and encodes a source or target sentence in a context
encoder when translating a current sentence. Previous works on multi-encoder
models have an additional encoder, referred to as a context encoder. Each
context encoder $\bm{u}^{i-1}$ or $\bm{v}^{i-1}$ reads a previous source-side
or target-side sentence as context, respectively.

$\displaystyle\bm{u}_{t}^{i-1}$ $\displaystyle=$
$\displaystyle\mathrm{LSTM}_{\mathrm{src\\_enc}}(W_{\mathrm{x}}x_{t}^{i-1},\bm{u}_{t-1}^{i-1})$
(10) $\displaystyle\bm{v}_{t}^{i-1}$ $\displaystyle=$
$\displaystyle\mathrm{LSTM}_{\mathrm{trg\\_enc}}(W_{\mathrm{y}}y_{t}^{i-1},\bm{v}_{t-1}^{i-1})$
(11)

We refer to this architecture as a separated model in this paper. In the
separated model, the weights of a context encoder are different from those of
a current encoder which encodes a current source sentence. If
$\bm{u}_{t}^{i-1}$ is used as $\bm{z}_{t}^{i-1}$, we call this model separated
source model; otherwise, we call this model separated target model.

###  2.2 Shared model

A shared model saves the hidden states of an encoder or decoder and then
calculates $\bm{c}_{n}^{i-1}$ using these states when translating a current
sentence. The strength of this model is that the target-side context can be
incorporated into a decoder instead of an encoder. Moreover, the shared model
does not require much additional parameters and extra computational times
because this model simply loads the saved hidden states. Thus, we can see
these models as examples of weight sharing between a current encoder or
decoder and a context encoder. The shared source model uses $\bm{s}_{t}^{i-1}$
as $\bm{z}_{t}^{i-1}$, and the shared target model uses $\bm{h}_{t}^{i-1}$ as
$\bm{z}_{t}^{i-1}$.

###  2.3 Shared mix model

We propose a shared mix model, which incorporates the source- and target-side
contexts. Figure 6 presents the shared mix model. The attention vector of the
shared mix model $\bm{c}^{i-1}$ is calculated as
$\bm{c}^{i-1}=\bm{c}_{source}^{i-1}+\bm{c}_{target}^{i-1}$, where
$\bm{c}_{source}^{i-1}$ and $\bm{c}_{target}^{i-1}$ are the context attentions
calculated by the equation (8). The reason for calculating the sum of two
attention is to arrange the same number of parameters as the other shared
models. Other architectures are the same as the other shared models.

##  3 Experiments

###  3.1 Data

Corpus | Train | Dev | Test
---|---|---|---
TED De–En | 203,998 | 888 | 1,305
TED Zh–En | 226,196 | 879 | 1,297
TED Ja–En | 194,170 | 871 | 1,285
Recipe Ja–En | 108,990 | 3,303 | 2,804
Table 1: Number of sentences in each dataset. Experiment | Baseline | Separated | Shared
---|---|---|---
Source | Target | Source | Target | Mix
TED De–En | 26.55 | $26.29\pm.37$ | $26.52\pm.12$ | ${}^{*}27.20\pm.11$ | ${}^{*}\textbf{27.34}\pm.11$ | $27.18\pm.21$
TED En–De | 21.26 | $21.04\pm.64$ | $20.77\pm.10$ | $21.63\pm.27$ | $\textbf{21.83}\pm.30$ | $21.50\pm.29$
TED Zh–En | 12.54 | $12.52\pm.33$ | $12.63\pm.24$ | ${}^{*}13.36\pm.41$ | ${}^{*}\textbf{13.52}\pm.10$ | ${}^{*}13.23\pm.09$
TED En–Zh | 8.97 | $8.94\pm.11$ | $8.71\pm.06$ | $9.45\pm.22$ | ${}^{*}\textbf{9.58}\pm.13$ | $9.42\pm.19$
TED Ja–En | 5.84 | ${}^{*}6.64\pm.26$ | ${}^{*}6.37\pm.12$ | ${}^{*}6.95\pm.07$ | ${}^{*}\textbf{6.96}\pm.18$ | ${}^{*}6.81\pm.16$
TED En–Ja | 8.40 | $8.58\pm.12$ | $8.26\pm.00$ | $8.51\pm.31$ | $8.59\pm.08$ | $\textbf{8.66}\pm.14$
Recipe Ja–En | 25.34 | ${}^{*}26.51\pm.09$ | ${}^{*}26.69\pm.15$ | ${}^{*}26.90\pm.17$ | ${}^{*}\textbf{26.92}\pm.10$ | ${}^{*}26.78\pm.11$
Recipe En–Ja | 20.81 | ${}^{*}21.87\pm.12$ | ${}^{*}21.45\pm.14$ | ${}^{*}\textbf{22.02}\pm.20$ | ${}^{*}21.97\pm.09$ | ${}^{*}21.81\pm.15$
Table 2: BLEU scores of our context-aware NMT in each language pair. Each
score is the average of three runs. “$*$” represents the statistically
significant results against the baseline at $p<0.05$ in all the runs.

We mainly use the IWSLT2017 German–English, Chinese–English, and
Japanese–English datasets from TED [9] for experiments. We consider each talk
of TED as a document, which includes sentences that cannot be translated using
only sentence-level information. Japanese and Chinese sentences are segmented
by the MeCab222http://taku910.github.io/mecab/ (dictionary: IPADic 2.7.0) and
jieba333https://github.com/fxsjy/jieba, respectively. English and German
sentences are segmented by tokenizer.perl included in
Moses444http://www.statmt.org/moses/. The documents that include sentences
consisting of more than 100 words are eliminated from the training corpus. We
evaluate our methods on the 2014 test set. The statistics of preprocessed
corpora are shown in Table 1. Byte pair encoding [10] is used separately for
source and target languages for subword segmentation. The number of merge
operations is 32,000.

Moreover, we use the Recipe
Corpus555http://lotus.kuee.kyoto-u.ac.jp/WAT/recipe-corpus/, which consists of
Japanese–English user-posted recipes, to investigate the influences in the
different domains. The procedures of data preprocessing are the same as those
for the TED corpus, except for the number of merge operations (8,000).

###  3.2 Settings

The baseline system of this experiment is our implementation of RNN-based NMT.
The encoder is two-layer bi-LSTM, and the decoder is two-layer uni-LSTM. The
dimensions of hidden states and embeddings are set to be 512. We use dropout
with $p=0.2$. The optimizer is AdaGrad with initial learning rate $=0.01$.
Each batch consists of up to 128 documents. These settings are the same in the
baseline and all context-aware models. Dot global attention is used for
calculating context attention $\bm{c}^{i-1}$. We set $\bm{c}^{0}=\bf{0}$
because the first sentences in documents do not have any previous contexts.

The context-aware models are pretrained with the baseline system. Each model
is trained for 30 epochs; then, the best model is selected with a development
set. The results are evaluated using BLEU [11]. We calculate the statistical
significance between the baseline and our methods by the bootstrap resampling
toolkit in Travatar [12]. Experiments are performed three times with different
random seeds.

###  3.3 Results

Table 2 shows the results. The shared target model improves the performances
in all language pairs. In the experiments on several language pairs, the
separated target model used in Bawden et al. [4] also improves performances
compared to the baseline. However, improvement is less compared to the shared
target model. Therefore, these results show that the target-side context
should be introduced from a decoder.

##  4 Discussion

Experiment |  Sentences
---|---
Input |  わかめ は よく 洗っ て 塩 を 落とし 、 10 分 ほど 水 に 浸け て おい て から ざく 切り に する 。 長 ねぎ は 小口切り に する 。
熱し た 鍋 に ごま油 を ひき 、 わかめ と 長 ねぎ を 入れ て 30 秒 ほど 軽く 炒める 。
Reference |  wash the wakame well to remove the salt , put into a bowl of water for 10 minutes and drain . cut into large pieces . slice the Japanese leek .
heat a pan and pour the sesame oil . stir fry the wakame and leek for 30
seconds .
Baseline |  wash the wakame seaweed well and remove the salt . soak in water for 10 minutes , then roughly chop . cut the Japanese leek into small pieces .
heat sesame oil in a heated pot , add the wakame and leek , and lightly sauté
for about 30 seconds .
Separated Source |  wash the wakame well , remove the salt , soak in water for about 10 minutes , then roughly chop . cut the Japanese leek into small pieces .
heat sesame oil in a heated pot and add the wakame and leek . stir-fry for
about 30 seconds .
Shared Source |  wash the wakame well, remove the salt , soak in water for about 10 minutes , then roughly chop . cut the Japanese leek into small pieces .
heat sesame oil in a heated pot and add the wakame and leek . stir-fry for
about 30 seconds .
Separated Target |  wash the wakame well , soak in water for about 10 minutes . cut into small pieces. cut the Japanese leek into small pieces .
heat the sesame oil in a frying pan , add the wakame and leek , and stir-fry
for about 30 seconds .
Shared Target |  wash the wakame well , remove the salt , soak in water for about 10 minutes , then roughly chop . chop the Japanese leek into small pieces .
heat sesame oil in a heated pan , add the wakame and Japanese leek , and
lightly stir-fry for about 30 seconds .
Shared Mix |  wash the wakame well , remove the salt , soak in water for about 10 minutes , then roughly chop . chop the Japanese leek into small pieces .
heat sesame oil in a heated pan , add the wakame and Japanese leek , and stir-
fry for about 30 seconds .
Table 3: The output examples in Recipe Japanese–English experiments. Each
upper sentence represents a previous sentence, and each lower sentence
represents a current sentence. Each sequence may comprise several sentences
because each sentence in Recipe corpus corresponds to “one step” of cooking.

---

Figure 7: TED German–English

Figure 8: TED English–German Figure 9: TED Chinese–English

Figure 10: TED English–Chinese Figure 11: TED Japanese–English

Figure 12: TED English–Japanese Figure 13: Recipe Japanese–English

Figure 14: Recipe English–Japanese Figure 15: The graph of BLEU scores using
each development set. BLEU score is calculated at the end of each epoch.

###  4.1 Weight sharing

We expected that there would be no differences between the results of the
shared source and separated source models because both models can introduce
source-side context into the encoder. However, the results obtained for the
language pairs used in this study show that the shared source model also
improves the BLEU scores with fewer parameters. Dabre et al. [13] found that
translation performances could be boosted even if the weights of stacked
layers were shared. Our shared models can be seen as an instance of weight
sharing for stacking sentence-level RNNs in chronological order. Shared models
can also be seen as an instance of multitask learning that shares the same
weights for encoder–decoders of neighboring sentences such as skip-thought
[14]. Thus, it is possible that weight sharing leads to a more efficient model
space by regularization, rather than by learning discourse structures.

###  4.2 Language dependency

The tendencies of the scores vary depending on language pairs. The result of
the TED English–German task shows that the source-side context decreases the
performance. Müller et al. [15] obtained similar results in other datasets
using the concatenation method proposed in Tiedemann et al. [16]. However, in
the Japanese–English and English–Japanese tasks, the importance of the source-
side context is equivalent to that of the target-side one. The reason is that
Japanese requires contexts more than English because Japanese is a pro-drop
language, which allows for the omission of agents and object arguments when
they are pragmatically or syntactically inferable. Comparing the result of the
TED and Recipe corpora, the difference of corpus domains does not affect such
tendencies. In the Chinese–English task, where they have more similar word
order, the importance of target-side context is equivalent or even better
compared to that of the source-side one. Therefore, these results imply that
the necessity of the source-side context depends on language pairs, while the
target-side context is generally important.

The shared mix model obtains competitive results compared to the shared source
model, in most of the language pairs. Therefore, either of contexts helps the
improvement without both side information if we choose the source- or target-
side context depending on the language pairs.

###  4.3 Output examples

We analyze the output examples in terms of the phrase coherence. We select the
Recipe Japanese–English task because Japanese is a pro-drop language that
needs context due to many omissions but it is difficult to draw any definitive
conclusions on the TED Japanese–English task as the BLEU score is too low to
analyze. Table 3 shows the examples. When the model translates the previous
sentence, this model does not use the context information because this is the
first sentence of a document. The examples written in each lower row are the
result using the information of the upper sentence as a context.

Looking at the result of the baseline, “長 ねぎ” (naga negi, Japanese leek) is
translated into “Japanese leek” in the previous sentence, even though this is
translated into “leek” in the current sentence. This phenomenon can be
commonly seen in the results of separated models. If we independently evaluate
these sentences, these sentences will be rated with high fluency and adequacy.
BLEU scores are also high because reference sentences also follow this
translation. However, these sentences have low coherence because the same noun
phrase in the Japanese sentence is translated into different phrases.

On the contrary, “長 ねぎ” is translated into “Japanese leek” in both sentences
in the experiments of shared target model and shared mix model, which use
target-side context. Our models improve phrase coherence using weight sharing.

###  4.4 Convergence of training

Figure 15 plots the BLEU scores on development sets. Shared models and
separated source model seem to be stable. However, the separated target model
is unstable and does not lead to an improvement. This is due to the exposure
bias problem [17] in the context encoder as well as the decoder. At the test
phase, the separated target model has to read the low-quality sentence with
well-trained encoder if the learning speed of encoding is faster than that of
decoding. Thus, the separated target model should fill the gap between the
learning speed of the context encoder and decoder.

##  5 Related Works

Wang et al. [18], Maruf et al. [19], and Tu et al. [20] incorporated the
information of previous sentences by using a hierarchical encoder, a memory
network and cache mechanism respectively. Although they used several sentences
as contexts, the former two works found that the information of distant
sentences in a document does not improve translation quality. Our
investigation is focused on a previous sentence.

Tiedemann et al. [16] used the concatenation of a previous sentence and a
current sentence as an input or output sentence to incorporate source-side and
target-side contexts in conventional NMT. Müller et al. [15] evaluated the
performance of existing context-aware NMT in the English–German task in terms
of pronoun translation. They concluded that generating concatenated sentence
is more effective than inputting concatenated sentence. Our results of the
shared target model support their results.

Voita et al. [6] and Zhang et al. [7] proposed Transformer-based context-aware
NMT. The former suggested that self-attention solves anaphora resolution. The
latter performed fine-tuning with small document-level data to adapt a single-
sentence NMT trained with large data to context-aware NMT. However, they did
not investigate the influence of the target-side contexts.

##  6 Conclusion

We reported how context-aware neural machine translation effectively employs
target-side contexts. We proposed a weight sharing to model the target-side
context in a decoder. This method achieves high performances in several
language pairs, even though it does not require much additional parameters. In
the future, we will analyze whether this model can handle longer contexts.

## References

  * [1] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Advances in Neural Information Processing Systems 27_ , 2014, pp. 3104–3112.
  * [2] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _Proceedings of the International Conference on Learning Representations_ , 2015.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30_ , 2017, pp. 5998–6008.
  * [4] R. Bawden, R. Sennrich, A. Birch, and B. Haddow, “Evaluating discourse phenomena in neural machine translation,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018, pp. 1304–1313.
  * [5] S. Jean, S. Lauly, O. Firat, and K. Cho, “Does neural machine translation benefit from larger context?” _CoRR_ , vol. abs/1704.05135, 2017.
  * [6] E. Voita, P. Serdyukov, R. Sennrich, and I. Titov, “Context-aware neural machine translation learns anaphora resolution,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 1264–1274.
  * [7] J. Zhang, H. Luan, M. Sun, F. Zhai, J. Xu, M. Zhang, and Y. Liu, “Improving the Transformer translation model with document-level context,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 533–542.
  * [8] T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015, pp. 1412–1421.
  * [9] M. Cettolo, C. Girardi, and M. Federico, “WIT3: Web inventory of transcribed and translated talks,” in _Proceedings of the 16 th Conference of the European Association for Machine Translation_, May 2012, pp. 261–268.
  * [10] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 1715–1725.
  * [11] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [12] G. Neubig, “Travatar: A forest-to-string machine translation engine based on tree transducers,” in _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2013, pp. 91–96.
  * [13] R. Dabre and A. Fujita, “Recurrent stacking of layers for compact neural machine translation models,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, pp. 6292–6299, 2019.
  * [14] R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler, “Skip-thought vectors,” in _Advances in Neural Information Processing Systems 28_ , 2015, pp. 3294–3302.
  * [15] M. Müller, A. Rios, E. Voita, and R. Sennrich, “A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation,” in _Proceedings of the Third Conference on Machine Translation: Research Papers_ , 2018, pp. 61–72.
  * [16] J. Tiedemann and Y. Scherrer, “Neural machine translation with extended context,” in _Proceedings of the Third Workshop on Discourse in Machine Translation_ , 2017, pp. 82–92.
  * [17] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba, “Sequence level training with recurrent neural networks,” in _Proceedings of the International Conference on Learning Representations_ , 2016.
  * [18] L. Wang, Z. Tu, A. Way, and Q. Liu, “Exploiting cross-sentence context for neural machine translation,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017, pp. 2826–2831.
  * [19] S. Maruf and G. Haffari, “Document context neural machine translation with memory networks,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 1275–1284.
  * [20] Z. Tu, Y. Liu, S. Shi, and T. Zhang, “Learning to remember translation history with a continuous cache,” _Transactions of the Association for Computational Linguistics_ , vol. 6, pp. 407–420, 2018.


# GRAPH NEURAL NETWORK SURROGATE FOR SEISMIC RELIABILITY ANALYSIS OF HIGHWAY
BRIDGE SYSTEM

Tong Liu
Department of Civil and Environmental Engineering
University of Illinois at Urbana-Champaign
<EMAIL_ADDRESS>
&Hadi Meidani
Department of Civil and Environmental Engineering
University of Illinois at Urbana-Champaign
<EMAIL_ADDRESS>

###### Abstract

Rapid reliability assessment of transportation networks can enhance
preparedness, risk mitigation and response management procedures related to
these systems. Network reliability approaches commonly consider network-level
responses, and due to computational cost do not consider the more detailed
node-level responses. In this paper, we propose a rapid seismic reliability
assessment approach for bridge networks based on graph neural networks, where
node-level connectivities, between points of interest and other nodes, are
quantified under probabilistic bridge conditions and earthquake events. Via
numerical experiments on transportation systems in California, we demonstrate
the accuracy, computational efficiency and robustness of the proposed approach
compared to the Monte Carlo approach.

##  1 Introduction

Extreme hazards continue to influence the infrastructure system, with losses
exceeding hundreds of billion of dollars. For instance, the overall cost to
repair or replace the infrastructure system during hurricane Katrina in 2005
[1] is over $1 billion. The cost for postdisaster reconstruction and
ecological rehabilitation after Wenchuan Earthquake in 2008 is over $100
billion [2]. The transportation system is one of the 16 critical
infrastructure sectors according to Cyber Infrastructure Security Agency
(CISA) [3]. The highway system in the US, for instance, encompasses over
164,000 miles in the National Highway System (NHS). The society relies on the
highway system also during emergencies for access to critical facilities,
including hospitals, airports, fire stations, etc. Highway bridges are
critical, and yet vulnerable, components of the transportation infrastructure.
The United States has a national inventory of almost 600,000 highway bridges,
many of which are deteriorated. Maintaining a reliable highway bridge system
can be properly achieved via a framework that quantitatively assesses highway
reliability in an accurate and computationally tractable way.

In the aftermath of natural hazards, the objectives of a reliability
assessment framework are to identify the risk potentials with respect to human
life and the infrastructure system; to adjust the emergency plan beforehand to
reduce the potential losses and damages; to allocate the funding effectively
for repair and recovery planning. In the area of the reliability analysis of
transportation systems, recent methods can be grouped into two categories:
qualitative and quantitative approaches. A review of qualitative approaches
can be found in [4]. As an example of a qualitative approach, [5] builds a
framework considering the human factor in the transportation reliability
analysis and then identifies the need for both transportation systems and
driving assistance systems. Recently, intelligent transportation systems
(ITSs) have leveraged advanced technologies to enable effective, efficient,
and safe transport [6, 7]. [8] presents the principles and selected
requirements accompanying the implementation of the ITSs and the reliability
and operation analysis of this type of system. [9] applies the vehicle-to-
infrastructure (V2I) in the ITSs to reduce the end-to-end delay of the vehicle
and improve the system reliability. [10] integrates sensor technology with the
transportation infrastructure to improve road safety and traffic reliability.
Among quantitative approaches, the works are either empirical surveys or
numerical simulations. [11] quantifies resilience measures of road
transportation and generalizes the characteristics of the resilience response
with the GPS information during severe winter storms in Wisconsin in 2008. Due
to the difficulty and unavailability of data collection, the simulation-based
approach is more frequently used. [12] assesses post-earthquake network damage
through predictive simulation of damage states of all highway bridges. Various
metrics are proposed and applied to the reliability of the transportation
system, including $k$-terminal connectivity [13, 14, 15, 16], centrality [17,
18, 19], travel time [20, 21, 22], and resilience index [23, 24].

However, the simulation-based reliability assessments of large infrastructure
networks are computationally intractable or expensive. To address this issue,
[25] reduced computational complexity by using the principal component
analysis (PCA) before a Monte Carlo (MC)-based uncertainty quantification.
Also, surrogate techniques such as polynomial chaos expansions [26], kriging
[27], and support vector machines (SVM) [28] have been proposed for network
reliability analysis. Using advances in machine learning, [16] provided a deep
learning framework to accelerate seismic reliability analysis of a
transportation network where the $k$-terminal connectivity measure was used.
As another example, [29] proposed a neural network surrogate for system-level
seismic risk assessment of bridge transportation networks using total system
travel time as the evaluation metric.

Transportation networks are expected to satisfy two requirements after an
extreme event [30]: (1) maintain connectivity and desired capacity between
critical origin-destination (OD) pairs immediately and (2) maintain acceptable
mobility conditions for people and goods in the presence of disruptions and
congestion. A rapid connectivity analysis for transportation systems is
necessary to evaluate how these two requirements are satisfied. For roadway
transportation systems, [16, 29] provide a graph-level evaluation including
$k$-terminal connectivity probability and total traffic time, which is a
network-level scalar measure for the whole graph. In this paper, we seek to
compute a detailed node-level connectivity measure in an efficient way using a
neural network-based analysis framework. Specifically, the model evaluates the
connectivity probability for all origin-destination pairs after earthquakes
using a graph neural network (GNN) model.

The major contributions of this work are as follows: (1) to the best of
authors’ knowledge, this is the first work that evaluates the connectivity for
all origin-destination pairs using the graph neural network; (2) the proposed
end-to-end structure of the model achieves high efficiency by avoiding
extensive sampling otherwise used in Monte Carlo simulations (MCS) which
reduces the computational time; and (3) the proposed model has the ability of
inductive learning, i.e., it can predict node connectivities in unseen graphs.
We will numerically show that the proposed framework can effectively
accelerate the connectivity reliability analysis of highway bridge networks
with a case study in the Bay Area in California.

The remainder of this article is structured as follows. A general simulation-
based framework for connectivity analysis of transportation networks is
described in Section 2. Then, Section 3 presents the node-level bridge
connectivity analysis for earthquake events using a graph neural network.
Furthermore, a case study of the highway bridge system in California is
presented to demonstrate the accuracy and efficiency of the proposed framework
in Section 4. Finally, the discussion and conclusion of the proposed framework
are presented in Section 5.

##  2 Transportation System Reliability Analysis

A transportation network can be represented by a graph $G=(V,E)$ where $V$ and
$E$ denote the sets of nodes and edges between these nodes, respectively. In a
highway bridge system, a node is a highway intersection, and an edge is a
highway segment between two intersections. Each link is denoted by $(u,v)\in
E$ with $u$ and $v$ being the indices of its two end nodes. In this work, we
assume the bridge is the only component that can fail in the network due to an
earthquake, and an edge $(u,v)$ is removed from the graph only because of a
bridge failure.

Given a source node $s$ and a target node $t$, $s$ and $t$ are connected when
there exists at least one active path
$l_{s,t}=\\{({s,v_{1}}),({v_{1},v_{2}}),({v_{1},v_{2}}),\dots,({v_{n},t})\\}$
between $s$ and $t$. Let us denote the the set of all possible paths as the
set $L_{s,t}=\\{l^{i}_{s,t}\\}_{i=1}^{n_{s,t}}$ where $n_{s,t}$ is the total
number of active path between $s$ and $t$. We seek to compute the node-to-node
connectivity (as a probability) using the survival probability of the bridges
in the network.

To do so, we first model the survival state of the path $i$ between $s$ and
$t$ as a binary Bernoulli variable given by

$x_{s,t,i}=\left\\{\begin{array}[]{cl}1,&\quad\mathrm{with\ probability\
}p_{s,t,i},\\\ 0,&\quad\mathrm{with\ probability\
}1-p_{s,t,i},\end{array}\right.$ (1)

where $\\{1,0\\}$ denotes the survived and failed states, respectively, with a
survival probability of $p_{s,t,i}$. A path with at least one failed bridge
will be removed. If path $i$ between $s$ and $t$ consist of $m_{s,t,i}$
bridges, the survival probability of the path can be represented with

$p_{s,t,i}=\prod_{j=1}^{m_{s,t,i}}p_{s,t,i,j},$ (2)

where $p_{s,t,i,j}$ is the survival probability of the $j$th bridge on path
$i$ between $s$ and $t$. The detail of calculating bridge survival probability
is shown in Section 3.2.1.

Given bridge failure probabilities, it is however computationally intractable
to directly compute the node-to-node connectivity probability from the bridge
survival probability. As a result, a Monte Carlo approach is adopted, where
realizations of the network are obtained by randomly removing paths based on
path survival probabilities $p_{s,t,i}$ calculated using Equation 2.
Specifically, for the $k$th MC realization of the network with the removed
paths, we check all the possible paths between a given pair $s$ and $t$ using
the breadth-first search (BFS) algorithm with time complexity of
$\mathcal{O}(|\mathcal{V}|+|\mathcal{E}|)$. Then, the binary node-to-node
connectivity $p_{s,t}^{k}$ is set equal to 1, if there is at least one path
between $s$ and $t$, and it is set to zero, otherwise. With $N$ MC samples,
the none-to-node connectivity probability can be approximated as

$P_{s,t}=\frac{1}{N}\sum_{k=1}^{N}p^{k}_{s,t}.$ (3)

Enough MC samples are used until the quantity of interest converges, which is
set to be when the standard deviation of connectivity probability becomes less
than 0.01.

##  3 Graph Neural Network for Node-Level Transportation System Reliability
Analysis

###  3.1 Graph Neural Network

In most of the first applications of neural networks, e.g. in image
classification, the input data is structured and typically of Euclidean
structure. In graph applications, on the other hand, input is non-Euclidean
and unstructured, containing different types of topologies. As such, normal
neural network frameworks cannot handle graph data since the structure of the
input data is not fixed. This paper leverages the graph neural network (GNN)
[31] and [32] to tackle the non-Euclidean graph-structured inputs. GNN is an
organic combination of connectionism and symbolism, which not only enables the
deep learning model to be applicable to the non-Euclidean structure of the
graph but also gives the deep learning model a certain causal reasoning
ability [33].

Using the previously introduced notation for the graph, $G=(V,E)$, features
used in the GNN consist of node features $X_{n}\subset\mathbb{R}^{|V|\times
F_{n}}$ and edge features $X_{e}\subset\mathbb{R}^{|E|\times F_{e}}$, where
$|V|$ and $|E|$ denote the number of nodes and edges, $|F_{v}|$ and $|F_{e}|$
denote the dimension of features for each node and edge. The process of
combining features from a node and other nodes is called message passing.
There are various approaches to exploiting node features. For instance, graph
convolutional network (GCN)[34] accomplishes message passing by using the
adjacency matrix $A$, or in an improved way by using a normalized adjacency
matrix. GCN has shown great potential in many areas, for instance text
classification [35], image captioning [36], and traffic prediction [37, 38].

However, the adjacency matrix contains the intra-node information, and it can
only be used for transductive learning tasks, where the model is not capable
of training on partial graphs and testing on unseen parts of the graph or
different graphs. To overcome this obstacle, GraphSAGE [39] is proposed to
enable inductive learning tasks without exploiting the adjacency matrix. This
model learns to generate low-dimensional node embeddings by means of sampling
and aggregating node neighbor features through the aggregation function. The
widely used aggregation functions are the average operation and maximum
operation [40]. GraphSAGE uses learned neural networks to aggregate
information and compute the embedding based on the central vertex and its
neighbors. The learned aggregation consists of two stages: feature aggregation
and feature update. The learned aggregation step is modeled as a single layer
in the graph neural network.

In the feature aggregation stage, at step $k$ (or in the $k^{\mathrm{th}}$
layer) for each node $v$, we aggregate the features of its neighbors denoted
by $x_{u}^{k}\in\mathbb{R}^{1\times d_{n}^{k}},\forall u\in\mathcal{N}(v)$
according to:

$x^{k+1}_{\mathcal{N}(v)}=f\left(\\{x_{u}^{k},\forall
u\in\mathcal{N}(v)\\}\right),\ \forall v\in V,$ (4)

where $f$ is the aggregation function, here chosen to be the mean aggregator
function, and $d_{n}^{k}$ is the node embedding dimension at layer $k$. Then
in the update stage at step $k$ for each node $v$, the updated node features
$x_{v}^{k+1}$ will be computed using the previous node features
$x_{v}^{k}\in\mathbb{R}^{1\times d_{n}^{k}}$ and the aggregated features
$x^{k+1}_{\mathcal{N}(v)}\in\mathbb{R}^{1\times d_{n}^{k+1}}$ according to:

$x^{k+1}_{v}=\sigma\left(g\left(\phi\left(x^{k}_{v},x^{k+1}_{\mathcal{N}(v)}\right);W\right)\right),\
\forall v\in V,$ (5)

where $\phi$ is the concatenating function, $\sigma$ is a nonlinear activation
function, $g(\cdot;W)$ represents a fully connected layer with parameter $W$.

One weakness of GraphSAGE in the original paper [39] is that it only considers
the node features and neglects the edge features. In this work, we extend the
learning aggregation to include both node and edge features. At step $k$, edge
features $x_{e}^{k}\in\mathbb{R}^{1\times d_{e}^{k}}$ of the edge connecting
each node $v,\forall u\in\mathcal{E}(v)$, are aggregated through

$x^{k+1}_{\mathcal{E}(v)}=f\left(\\{x_{e}^{k},\forall
e\in\mathcal{E}(v)\\}\right),\ \forall v\in V.$ (6)

where $d_{e}^{k}$ is the edge embedding dimension at layer $k$. The
aggregation step for node embedding is the same as Equation 4. Then in the
update step, both node features and edge features are passed through a neural
network:

$x^{k+1}_{v}=\sigma\left(g\left(\phi\left(x^{k}_{v},x^{k+1}_{\mathcal{N}(v)},x^{k+1}_{\mathcal{E}(v)}\right);W\right)\right),\
\forall v\in V.$ (7)

By repeatedly using Equations 4, 6 and 7, the node and edge features of
multiple-hop neighbors of the central node are aggregated into the features of
that central node. Figure 1 illustrates the process of message-passing when
both node and edge features are passed. The left figure represents the
original graph at step 0 where the node features $x_{n}^{0}$ and edge feature
$x_{e}^{0}$ are initialized. Then in the aggregation step, which is shown in
the middle figure, the node feature and edge feature are passed following the
arrow direction. Taking the red node as an example, in step 1, the node
features and edge features with blue color are passed into the red node and
then updated, which are 1-hop neighbors. Then in step 2, the node features and
edge features with green color are passed into the red node and then updated,
which are 2-hop neighbors. After the k-step update, the node feature
$x_{n}^{k}$ includes the information from all k-hop neighbors, which is shown
in the right figure. Then the node embedding can be further used for node
regression and other downstream tasks.

Figure 1: Message-passing with both node and edge features.

###  3.2 Graph Neural Network for Bridge Connectivity Analysis

The pipeline of graph neural network surrogate for bridge connectivity
analysis is shown in Figure 2. The pipeline consists of two major components:
a bridge seismic analysis module and a graph neural network module.

Figure 2: Pipeline of connectivity analysis of highway bridge system.

####  3.2.1 Bridge Seismic Analysis Module

This module consists of generating a bridge graph and computing bridge failure
probability. To generate the graph, we first need to define a region of
interest for reliability analysis. The edges in the graph are highways,
freeways, and expressways within the region of interest. The intersections of
these roadways are considered as nodes in the graph. To calculate the bridge
failure probability, the first step is to extract bridge information in the
considered highway system, which could be obtained from the National Bridge
Inventory (NBI)[41]. The useful information for bridge failure probability
calculation includes built year, structural material, structural type, number
of spans, maximum span length, bridge length, and skew angle. It should be
noted that in general, an edge may consist of zero, one, or more than one
bridge.

The second step for bridge failure probability calculation is to determine the
ground motion at each bridge site. In this paper, the ground motion prediction
equation (GMPE) is adopted, which predicts the characteristics of ground
motion, including peak ground acceleration (PGA), spectral acceleration (SA),
and its associated uncertainty [42, 43]. In the past decades, hundreds of
GMPEs have been proposed for predicting PGA and SA. In this work, Graizer-
Kalkan GMPE (GK15)[44] is adopted. The updated ground-motion prediction model
for PGA has six independent predictor parameters: moment magnitude $M$, the
closest distance to fault rupture plane in kilometers $R$, average shear-wave
velocity in the upper 30m $V_{\mathrm{S30}}$, style of faulting $F$, regional
quality factor $Q_{0}$, and basin depth under the site $B_{\mathrm{depth}}$.
In GK15, the peak ground acceleration is calculated as a multiplication of a
series of functions. In the natural logarithmic scale, it is given by:

$\ln(\mathrm{PGA})=\ln(G_{1})+\ln(G_{2})+\ln(G_{3})+\ln(G_{4})+\ln(G_{5})+\sigma_{\ln(\mathrm{PGA})},$
(8)

where $G_{1}$ is a scaling function for magnitude and style of faulting,
$G_{2}$ models the ground-motion distance attenuation, $G_{3}$ adjusts the
distance attenuation rate considering regional anelastic attenuation, $G_{4}$
models the site amplification owing to shallow site conditions, $G_{5}$ is a
basin scaling function, and $\sigma_{\ln(\mathrm{PGA})}$ represents
variability in the ground motion.

In order to conduct quick assessments, rather than using the finite element
analysis (FEA), the failure probability of a bridge is computed using the
fragility function following the guideline of HAZUS-HM [45], developed by the
Federal Emergency Management Agency (FEMA). HAZUS-HM categorizes the bridges
into 28 primary types based on materials, structure type, build year, number
of spans, etc. To estimate the probability of failure of bridges, the
geographical location of the bridge, SA at 0.3 s and 1.0 s at the bridge
location, and soil condition are also required. The form for 5% damped
spectral acceleration at spectral period T is:

$S_{a,T}=\mathrm{PGA}\times\mu\left(T,M,R,V_{\mathrm{S30}},B_{\mathrm{depth}}\right),$
(9)

where $\mu$ is the spectral shape function. In HAZUS-HM, the failure
probabilities of bridges are computed using the fragility curve. Five damage
states are defined: none, slight, moderate, extensive, and complete damage
states. For bridges, extensive damage is defined by shear failure, major
settlement approach, and the vertical offset of the abutment. In this work, we
assume the bridges will stop functioning when bridge damage is beyond the
extensive damage state. The fragility curve for the bridge component is
modeled as log-normal distribution functions characterized by median and
dispersion. Furthermore, the failure probability of the $j$th edge on path $i$
between $s$ and $t$, $p_{s,t,i,j}\in[0,1]$ is computed based on the assumption
that the edge fails when at least one bridge fails.

####  3.2.2 Bridge Connectivity Analysis Module

To build the training data and testing data for graph neural networks, we need
first to create node feature and edge feature using the failure probability
computed in 3.2.1. The edge feature is the failure probability of each edge,
which has already been computed. It should be noted that in conventional
connectivity analysis there is no feature directly assigned to the nodes, i.e.
intersection of the roadways. In this work, however, we consider node features
in the graph neural network models. Four local and global graph
characteristics are considered as features of node $v$: the degree of the node
$\mathrm{deg}(v)$; the largest failure probability of edges connected to the
node $\max(\\{p(e),\forall e\in\mathcal{E}(v)\\})$; the smallest failure
probability of edges connected to the node $\min(\\{p(e),\forall
e\in\mathcal{E}(v)\\})$ and number of hops on the shortest path to the target
node $t$ h(v, t). All the node features in the graph are denoted by
$X_{n}\in\mathbb{R}^{|N|\times F_{n}}$ with the number of node feature
$F_{n}=4$.

Figure 3: Architecture of graph neural network model for node-level
connectivity analysis.

The proposed node features characterize each node from several perspectives:
the degree represents the possible paths connecting the node to its neighbors;
the range of failure probabilities of edges connected to the node represents
the distribution of the failure probability; the number of hops represents a
global feature characterizing the distance with respect to the target node. It
should be noted that in addition to the range of edge failure probabilities
(i.e. the largest and smallest failure probabilities of connected edges), one
can also consider multiple quantiles to more precisely capture the
distribution of failure probabilities for a potentially more accurate
analysis.

Another difficulty in the graph neural network is the over-smoothing issue. In
particular, because of the aggregation and update operations in the message
passing, the interacting nodes have similar representations and the feature
embeddings of different nodes in the deep graph neural network would tend to
be similar. To alleviate this issue, differentiable group normalization [46]
and dropEdge[47] were proposed to add normalization into graph neural networks
in different forms. In this work, we add a dropout layer after the message-
passing layer since dropout can randomly remove neurons in the forward
propagation. It is equivalent to randomly removing edge features and node
features from the original graph in the training process, which can be
considered as a normalization strategy.

The architecture of the proposed graph neural network model for bridge
connectivity analysis is shown in Figure 3. A single block of message passing
consists of a message passing layer, a dropout layer, and an activation layer.
The skipping layer connection is used in our proposed architecture to mitigate
vanishing or exploding gradient problems. A regression block with multiple
feed-forward fully connected layers is concatenated after the last message
passing block. The output of the model is the node-level connectivity
probability $P_{c}\in\mathbb{R}^{|V|\times 1}$. The L1 loss is used for
quantifying the difference between the MCS and prediction.

##  4 Experiments and Results

###  4.1 Experiment Setup

The proposed GNN-based reliability analysis is applied to the highway bridge
system of the California Bay Area. The transportation system contains
highways, freeways, and expressways connecting major airports and hospitals.
The considered study area includes the major cities of Santa Clara, Mountain
View, and San Jose, with a large population base. Therefore, it is important
to maintain the connectivity of the highway bridge system after extreme
events. In this experiment, three different regions of study are considered
(levels 1 to 3). The map illustrations are shown in Figure 4 obtained from
Google Map [48]. The characteristics of these three regions, specifically
their number of nodes, $N_{n}$, number of edges, $N_{b}$, and number of
bridges, $N_{b}$, are shown in Table 1.

The 1989 Loma Prieta earthquake is chosen as the seismic event. The earthquake
is scaled to different magnitudes, $M$, in the range between 6.5 and 8.0. To
generate the training data, the earthquake magnitude $M_{i}$ is sampled from

$M_{i}=M_{u}-\lambda_{i},$ (10)

Where $M_{u}$ is the upper bound of the possible earthquake magnitude and set
to be 8.0 in this study, and $\lambda_{i}$ is a random sample following
truncated exponential distribution with the shape parameter of 1.5. Figure 2
shows the pipeline of data generation. In the Monte Carlo simulation (MCS)
approach, the number of samples $N_{s}$ is set to 10,000. For each node in the
graph, $N_{k}$ earthquake realizations are generated, and in total,
$N_{k}\times N_{n}$ network realizations are generated.

The graph neural network used in this work consists of five graph message
passing layers and three fully connected layers for regression purposes. The
dimension of the hidden message passing layer and fully connected layer is 64.
The rectified linear unit (ReLU) is chosen as the activation function. The
dropout rate is set to 0.1, and the Adam optimizer is used to minimize the L1
loss, which is the mean absolute error, with a learning rate of 0.001. The
model is trained using mini-batch training with 200 epochs and a batch size of
64.

Figure 4: Map illustration of the studied regions in different levels [48]. Table 1: Statistic of different region levels Graph Level | # Node $N_{n}$ | # Edge $N_{e}$ | # Bridge $N_{b}$
---|---|---|---
Level 1 | 39 | 64 | 245
Level 2 | 84 | 133 | 448
Level 3 | 103 | 159 | 628

For the train-test split, we first held out 20% of the nodes for testing. To
ensure that the removed nodes are more or less evenly scattered on the
network, one can partition the graph and select a number of nodes from each
partition. However, uniform graph partitioning or a balanced graph partition
problem can be shown to be NP-complete. As the problem size increases, the
time to find the optimal solution increases exponentially [49]. As a result,
the serial graph partitioning algorithm [50] is applied for approximately
partitioning of the graph. Then we determine the percentage of training nodes
to determine the number of nodes used for training and the training node is
selected from each partitioned graph.

###  4.2 Prediction Results

We test the performance of the proposed approach from multiple perspectives.
First, we consider a regression problem where we calculate the probability of
connectivity between a target node and the rest of the nodes in the region.
The target node is located in downtown San Jose, which is a critical location
for the large population base. We do this for three regions of interest. The
prediction results for the Level 1 region are shown in Figure 5. The mean
absolute error (MAE) between Monte Carlo simulation (MCS) and GNN predictions
was calculated. The maximum MAE in the graph is 0.032 and the average MAE is
0.013. This demonstrates the accuracy of the proposed GNN surrogate model.
Figure 6 shows the prediction results and the corresponding error for the
Level 3 graph. For this case, the maximum MAE in the graph is 0.055 and the
mean prediction MAE is 0.019.

(a) Predicted connectivity probability from GNN

(b) Absolute error between GNN and MCS predictions

Figure 5: Prediction of connectivity probabilities between the target node
(red circle) and the other nodes in the Level 1 region. The nodes are color-
coded based on the predicted connectivity probability (top) and the absolute
error between predicted connectivity probabilities from GNN and MCS (bottom).

(a) Predicted connectivity probability from GNN

(b) Absolute error between MCS and GNN predictions

Figure 6: Prediction of connectivity probabilities between the target node
(red circle) and the other nodes in the Level 3 region. The nodes are color-
coded based on the predicted connectivity probability (top) and the absolute
error between predicted connectivity probabilities from GNN and MCS (bottom).

Furthermore, from the perspective of decision-making, the actual node
connectivity probability may not be needed. Instead, stakeholders may choose
to consider a set of node connectivity classes. In this case, we evaluate how
accurate the predicted classes will be compared to the MCS approach. It should
be noted that a different classification model is not trained in this case. We
only predict the connectivity classes by assigning the predicted connectivity
probability of each node into a class. In this example, three classes for node
connectivity probabilities, namely normal connection, minor disconnection and
major disconnection, for the connectivity probabilities falling in the ranges
[0.75, 1.0], [0.5, 0.75] and [0, 0.5], respectively. The node connectivity
conditions are directly classified based on the connectivity probability from
the regression result. Figure 7 shows the classification result using the same
example in Figure 5. F1 score is chosen as the classification evaluation
metric F1 score, which is 1.0 in this example.

(a) Predicted connectivity classes from MCS

(b) Predicted connectivity class from GNN

Figure 7: Identical predictions of connectivity classes obtained from GNN and
MCS models for the Level 1 region. The red circle shows the target node. Three
classes represent normal connection, minor disconnection, and major
disconnection.

In order to assess the robustness of the proposed approach, we run several
experiments in which the number of training nodes within the graph varies.
Specifically, a subset of the nodes is considered as the target nodes in
training. In all the experiments, 20% of the nodes in the graph are selected
as the "test" target nodes, which are made sure to be different from the
"training" target nodes. For each case, a specific ratio of the graph nodes is
selected as the "training" target nodes. These ratios are chosen between 5%
and 80%. The evaluation metrics include mean square error, mean absolute
error, and F1 score. The performances with different levels are demonstrated
in Figure 8, which shows that when the ratio of training target nodes is from
20% to 40%, the F1 score can reach 0.85 and MAE is less than 0.08.
Furthermore, when the training target node ratio is 60%, the F1 score can
reach above 0.9, and MAE is less than 0.01.

To further evaluate the robustness of the proposed approach, Figure 9 compares
the MCS and GNN predictions of node connectivity for all earthquake
realizations. The percentage in each figure indicates the ratio of false
positive (FP) and false negative (FN) samples among all samples with the
cutoff threshold 0.75, which is less than 5%. It can also be seen that the
Pearson Correlation coefficients between MCS and GNN prediction for all region
levels are higher than 0.93. Table 2 summarizes the performance metrics and
the computational times for different regions. The F1 scores are calculated
for two cases, where the predicted connectivity probability is turned into one
of two classes, [0-0.5] or [0.5-1] , or into one of the three aforementioned
classes. The computational time for the proposed GNN model considers both
training time and inference time, and as can be seen is 20$\times$ shorter
than that of MCS.

(a) Level 1

(b) Level 2

(c) Level 3

Figure 8: Relationship between different metrics and the percentage of
training nodes shown for the three regions.

(a) Level 1

(b) Level 2

(c) Level 3

Figure 9: Relationship between predicted node connectivity and MCS predictions
shown for the three regions. The percentage in the figure represents the
proportion of false positive (FP) and false negative (FN) results. Table 2:
Comparison between performance metrics and computational times between GNN and
MCS, averaged over 100 experiments for each region. The cutoff thresholds for
two-class and three-class classification are {0.5} and {0.5, 0.75},
respectively.

Graph Level | MAE | MSE | F1 score (2 class) | F1 score (3 class) | Time (GNN) | Time (MC)
---|---|---|---|---|---|---
Level 1 | 0.012 | 0.001 | 0.947 | 0.935 | 4.7 min | 118.2 min
Level 2 | 0.025 | 0.008 | 0.952 | 0.906 | 13.6 min | 490.3 min
Level 3 | 0.013 | 0.002 | 0.972 | 0.942 | 29.6 min | 698.2 min

###  4.3 Model Performance Under Special Cases

Neural networks are vulnerable to adversarial examples and [51, 52], where a
small perturbation to the original data will change the neural network
prediction drastically. To illustrate the robustness of our approach in these
cases, the proposed GNN model is evaluated on the slightly modified testing
data. As the context, we consider the cases where bridge failure probabilities
are decreased or increased because of repairs or deterioration through the
life cycle. To test the robustness of the proposed model among all region
levels, we generate perturbed features by adding to each edge failure
probability a zero-mean Gaussian noise with a standard deviation set equal to
20% of the original edge failure probability. The resulting failure
probability is truncated at 0 and 1. Following this perturbation to the edge
features, the node features are also updated accordingly. Then the GNN
prediction of node connectivities is performed for all three regions as
described in Section 4.2 and the comparison between the prediction and MCS in
this perturbed case is also shown in Figures 10 and 11. It can be seen in
Figure 11, that even though there is more discrepancy between the GNN and MCS
compared to the original case in Figure 9, overall the performance is
acceptable for an unseen case where all edges have different features. The F1
score of prediction with perturbated data is 0.958, 0.949, and 0.961 for three
region levels respectively. The correlation coefficient of prediction with
perturbated data is 0.942, 0.923, and 0.969 for three region levels
respectively.

(a) Level 1

(b) Level 2

(c) Level 3

Figure 10: Relationship between different performance metrics and different
ratios of training target nodes, shown for the three regions with perturbed
edge features.

(a) Level 1

(b) Level 2

(c) Level 3

Figure 11: Comparison of predicted node connectivity between MCS and GNN in
the three regions with perturbed edge features.

Finally, we assess the capability of the proposed approach in generalization
or _inductive_ inference and prediction. Specifically, we evaluate whether the
model can be used for predicting out-of-distribution (OOD) data, that is,
producing accurate predictions for test cases that are distributed differently
from the way training data was distributed. As opposed to transfer learning
where the model is retrained on the new data, in generalization or inductive
learning, we do not modify the previously trained model and only evaluate it
on the new data. In practice, this can be used to assess the network
connectivity when new bridges or roads are either added in a network expansion
or removed due to a disaster.

In these cases, to assess whether the trained model has generalization
capacity without additional fine-tuning, we consider the base training of the
model to be done at the Level 2 region level. We then test the performance of
the trained model on the Level 3 region. In general, this is a challenging
task because many of the nodes in the larger Level 3 region are unseen and
were not included in the previous training. However, as can be seen in Figure
12, the proposed GNN model has an acceptable performance. It can be seen that
the F1 score in this case is 0.92 even though only 60% of the nodes in the
smaller region (Level 2) were used as training nodes, for a prediction at a
large region (Level 3). This good performance in handling inductive learning
tasks is achieved because the mechanism of the message passing used in the GNN
architecture enables a modular understanding of how the graph features are
aggregated and collectively influence the task at hand.

Figure 12: Assessment of the performance of inductive learning from the
smaller Level 2 region to Level 3 region, in terms of the classification
results (top) and regression results (bottom).

##  5 Conclusion and Discussion

Rapid evaluation of large-scale infrastructure system reliability is critical
to enhance the preparation, risk mitigation and response management under
probabilistic natural hazard events. In this paper, we propose a rapid seismic
reliability assessment approach for roadway transportation systems with
seismic damage on bridges using the graph neural network. Compared with the
common reliability assessment approaches, which consider the response at the
system or graph level, we focus on a more detailed notion of reliability where
the seismic impact is quantified at the node-level for transportation systems.
The proposed GNN surrogate bypasses extensive sampling that is required in
Monte Carlo-based approaches and offers high efficiency while preserving the
accuracy. Additionally, the message passing component of the proposed GNN
model creates a modular model structure which can offer a good generalization
capacity, i.e., can effectively predict system reliability in unseen graphs.
The numerical experiments in this work, carried out on a transportation system
in California, demonstrate the effectiveness, robustness, and efficiency of
the proposed approach in enabling rapid assessment of large-scale network
reliability with high accuracy. As extensions to this work, measures other
than node-to-node connectivity can be considered. Examples include travel
distance, travel time, or traffic flow. Another area for future research is
the investigation of an efficient optimization framework based on the proposed
GNN approach for selecting the best maintenance planning strategy.

## References

  * [1] Jamie Padgett, Reginald DesRoches, Bryant Nielson, Mark Yashinsky, Oh-Sung Kwon, Nick Burdette, and Ed Tavera.  Bridge damage and repair costs from hurricane katrina.  Journal of Bridge Engineering, 13(1):6–14, 2008.
  * [2] Caiyu Sun and Jiuping Xu.  Estimation of time for wenchuan earthquake reconstruction in china.  Journal of Construction Engineering and Management, 137(3):179–187, 2011.
  * [3] Critical infrastructure sectors, 2022.
  * [4] Chengpeng Wan, Zaili Yang, Di Zhang, Xinping Yan, and Shiqi Fan.  Resilience in transportation systems: a systematic review and future directions.  Transport Reviews, 38(4):479–498, Jul 2018.
  * [5] Subeer Rangra, Mohamed Sallak, Walter Schön, and Frédéric Vanderhaegen.  On the study of human reliability in transportation systems of systems.  In 2015 10th System of Systems Engineering Conference (SoSE), pages 208–213. IEEE, 2015.
  * [6] Li Zhu, Fei Richard Yu, Yige Wang, Bin Ning, and Tao Tang.  Big data analytics in intelligent transportation systems: A survey.  IEEE Transactions on Intelligent Transportation Systems, 20(1):383–398, 2018.
  * [7] Agachai Sumalee and Hung Wai Ho.  Smarter and more connected: Future intelligent transportation system.  Iatss Research, 42(2):67–71, 2018.
  * [8] Miroslaw Siergiejczyk, Karolina Krzykowska, Adam Rosinski, and Luigi Alfredo Grieco.  Reliability and viewpoints of selected its system.  In 2017 25th International Conference on Systems Engineering (ICSEng), pages 141–146. IEEE, 2017.
  * [9] Hong Zhang and Xinxin Lu.  Vehicle communication network in intelligent transportation system based on internet of things.  Computer Communications, 160:799–806, 2020.
  * [10] Juan Guerrero-Ibáñez, Sherali Zeadally, and Juan Contreras-Castillo.  Sensor technologies for intelligent transportation systems.  Sensors, 18(4):1212, 2018.
  * [11] Teresa M Adams, Kaushik R Bekkem, and Edwin J Toledo-Durán.  Freight resilience measures.  Journal of Transportation Engineering, 138(11):1403–1409, 2012\.
  * [12] S Banerjee and M Shinozuka.  Uncertainties in seismic risk assessment of highway transportation systems.  In TCLEE 2009: Lifeline Earthquake Engineering in a Multihazard Environment, pages 1–9. 2009.
  * [13] Benny Sudakov and Van H Vu.  Local resilience of graphs.  Random Structures & Algorithms, 33(4):409–433, 2008.
  * [14] Sybil Derrible and Christopher Kennedy.  The complexity and robustness of metro networks.  Physica A: Statistical Mechanics and its Applications, 389(17):3678–3691, 2010.
  * [15] Xiaodong Zhang, Elise Miller-Hooks, and Kevin Denny.  Assessing the role of network topology in transportation network resilience.  Journal of transport geography, 46:35–45, 2015.
  * [16] Mohammad Amin Nabian and Hadi Meidani.  Deep learning for accelerated seismic reliability analysis of transportation networks.  Computer-Aided Civil and Infrastructure Engineering, 33(6):443–458, 2018.
  * [17] Yew-Yih Cheng, Roy Ka-Wei Lee, Ee-Peng Lim, and Feida Zhu.  Measuring centralities for transportation networks beyond structures.  In Applications of social media and social network analysis, pages 23–39. Springer, 2015.
  * [18] Andrew Cook, Henk AP Blom, Fabrizio Lillo, Rosario Nunzio Mantegna, Salvatore Micciche, Damián Rivas, Rafael Vázquez, and Massimiliano Zanin.  Applying complexity science to air traffic management.  Journal of Air Transport Management, 42:149–158, 2015.
  * [19] Ralihalizara Julliard, PJ Yun, and Ranaivoson Rado.  Robustness of transportation networks: The case of madagascar’s road network.  International Journal of Advancements in Research & Technology, 4(5):1–4, 2015.
  * [20] Mayada Omer, Ali Mostashari, and Roshanak Nilchiani.  Measuring the resiliency of the manhattan points of entry in the face of severe disruption.  American Journal of Engineering and Applied Sciences, 4(1):153–161, 2011.
  * [21] Reigna Jewel Ritz Meman Macababbad and Jose Regin Fajardo Regidor.  A study on travel time and delay survey and traffic data analysis and visualization methodology.  In Proceedings of the Eastern Asia Society for Transportation Studies Vol. 8 (The 9th International Conference of Eastern Asia Society for Transportation Studies, 2011), pages 318–318. Eastern Asia Society for Transportation Studies, 2011.
  * [22] Pavithra Parthasarathi, David Levinson, and Hartwig Hochmair.  Network structure and travel time perception.  PloS one, 8(10):e77718, 2013.
  * [23] Aman Karamlou and Paolo Bocchini.  From component damage to system-level probabilistic restoration functions for a damaged bridge.  Journal of Infrastructure Systems, 23(3):04016042, 2017.
  * [24] A Karamlou, P Bocchini, and V Christou.  Metrics and algorithm for optimal retrofit strategy of resilient transportation networks.  Maintenance, monitoring, safety, risk and resilience of bridges and bridge networks, pages 1121–1128, 2016.
  * [25] Mohammad Amin Nabian and Hadi Meidani.  Uncertainty quantification and pca-based model reduction for parallel monte carlo analysis of infrastructure system reliability.  In 96th Annual Meeting of the Transportation Research Board, Washington DC, 2017.
  * [26] Nestor V Queipo, Raphael T Haftka, Wei Shyy, Tushar Goel, Rajkumar Vaidyanathan, and P Kevin Tucker.  Surrogate-based analysis and optimization.  Progress in aerospace sciences, 41(1):1–28, 2005.
  * [27] Vincent Dubourg, Bruno Sudret, and Jean-Marc Bourinet.  Reliability-based design optimization using kriging surrogates and subset simulation.  Structural and Multidisciplinary Optimization, 44(5):673–690, 2011\.
  * [28] Raphael E Stern, Junho Song, and Daniel B Work.  Accelerated monte carlo system reliability analysis through machine-learning-based surrogate models of network connectivity.  Reliability Engineering & System Safety, 164:1–9, 2017.
  * [29] Sungsik Yoon, Jeongseob Kim, Minsun Kim, Hye-Young Tak, and Young-Joo Lee.  Accelerated system-level seismic risk assessment of bridge transportation networks through artificial neural network-based surrogate model.  Applied Sciences, 10(18):6476, 2020.
  * [30] Yafeng Yin and Hitoshi Ieda.  Assessing performance reliability of road networks under nonrecurrent congestion.  Transportation Research Record, 1771(1):148–155, 2001.
  * [31] Alessandro Sperduti and Antonina Starita.  Supervised neural networks for the classification of structures.  IEEE Transactions on Neural Networks, 8(3):714–735, 1997.
  * [32] Marco Gori, Gabriele Monfardini, and Franco Scarselli.  A new model for learning in graph domains.  In Proceedings. 2005 IEEE international joint conference on neural networks, volume 2, pages 729–734, 2005.
  * [33] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph Neural Networks: A Review of Methods and Applications.  arXiv, Dec 2018.
  * [34] Thomas N. Kipf and Max Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  arXiv, Sep 2016.
  * [35] Liang Yao, Chengsheng Mao, and Yuan Luo.  Graph convolutional networks for text classification.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370–7377, 2019.
  * [36] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Exploring visual relationship for image captioning.  In Proceedings of the European conference on computer vision (ECCV), pages 684–699, 2018.
  * [37] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting.  arXiv, Jul 2017.
  * [38] Tanwi Mallick, Prasanna Balaprakash, Eric Rask, and Jane Macfarlane.  Graph-partitioning-based diffusion convolutional recurrent neural network for large-scale traffic forecasting.  Transp. Res. Rec., 2674(9):473–488, Jul 2020.
  * [39] William L Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025–1035, 2017.
  * [40] Simon Geisler, Daniel Zügner, and Stephan Günnemann.  Reliable graph neural networks via robust aggregation.  Advances in Neural Information Processing Systems, 33:13272–13284, 2020.
  * [41] National bridge inventory, 2018.
  * [42] Jonathan P Stewart, John Douglas, Mohammad Javanbarg, Yousef Bozorgnia, Norman A Abrahamson, David M Boore, Kenneth W Campbell, Elise Delavaud, Mustafa Erdik, and Peter J Stafford.  Selection of ground motion prediction equations for the global earthquake model.  Earthquake Spectra, 31(1):19–45, 2015.
  * [43] Julian J Bommer, John Douglas, Frank Scherbaum, Fabrice Cotton, Hilmar Bungum, and Donat Fah.  On the selection of ground-motion prediction equations for seismic hazard analysis.  Seismological Research Letters, 81(5):783–793, 2010.
  * [44] Vladimir Graizer and Erol Kalkan.  Summary of the GK15 Ground-Motion Prediction Equation for Horizontal PGA and 5% Damped PSA from Shallow Crustal Continental Earthquakes Summary of the GK15 GMPE for Horizontal PGA and 5% Damped PSA.  Bull. Seismol. Soc. Am., 106(2):687–707, Apr 2016.
  * [45] FEMA.  HAZUS-MH MR5 Technical Manual - Earthquake Model.  U.S. Department of Homeland Security, 2010.
  * [46] Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu.  Towards deeper graph neural networks with differentiable group normalization.  arXiv preprint arXiv:2006.06972, 2020.
  * [47] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang.  Dropedge: Towards deep graph convolutional networks on node classification.  arXiv preprint arXiv:1907.10903, 2019.
  * [48] Google.  Google map, 2022.
  * [49] Tomas Feder, Pavol Hell, Sulamita Klein, and Rajeev Motwani.  Complexity of graph partition problems.  In Proceedings of the thirty-first annual ACM symposium on Theory of computing, pages 464–472, 1999.
  * [50] George Karypis and Vipin Kumar.  MeTis: Unstructured Graph Partitioning and Sparse Matrix Ordering System, Version 4.0, 2022.
  * [51] Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.  Computer Science Review, 37:100270, 2020.
  * [52] Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K Jain.  Adversarial attacks and defenses in images, graphs and text: A review.  International Journal of Automation and Computing, 17(2):151–178, 2020.