# Efficient and Explicit Modelling of Image Hierarchies for Image Restoration

Yawei Li1 Yuchen Fan2 Xiaoyu Xiang2 Denis Demandolx2
Rakesh Ranjan2 Radu Timofte1,3 Luc Van Gool1,4
1Computer Vision Lab, ETH Zürich 2Meta Reality Labs 3University of Würzburg
4KU Leuven

###### Abstract

The aim of this paper is to propose a mechanism to efficiently and explicitly
model image hierarchies in the global, regional, and local range for image
restoration. To achieve that, we start by analyzing two important properties
of natural images including cross-scale similarity and anisotropic image
features. Inspired by that, we propose the anchored stripe self-attention
which achieves a good balance between the space and time complexity of self-
attention and the modelling capacity beyond the regional range. Then we
propose a new network architecture dubbed GRL to explicitly model image
hierarchies in the Global, Regional, and Local range via anchored stripe self-
attention, window self-attention, and channel attention enhanced convolution.
Finally, the proposed network is applied to 7 image restoration types,
covering both real and synthetic settings. The proposed method sets the new
state-of-the-art for several of those. Code will be available at
https://github.com/ofsoundof/GRL-Image-Restoration.git.

##  1 Introduction

Image restoration aims at recovering high-quality images from low-quality
ones, resulting from an image degradation processes such as blurring, sub-
sampling, noise corruption, and JPEG compression. Image restoration is an ill-
posed inverse problem since important content information about the image is
missing during the image degradation processes. Thus, in order to recover a
high-quality image, the rich information exhibited in the degraded image
should be fully exploited.

Natural images contain a hierarchy of features at global, regional, and local
ranges which could be used by deep neural networks for image restoration.
First, the local range covers a span of several pixels and typical features
are edges and local colors. To model such local features, convolutional neural
networks (CNNs) with small kernels (_e.g._ $3\times 3$) are utilized. Second,
the regional range is characterized by a window with tens of pixels. This
range of pixels can cover small objects and components of large objects (pink
squares in Fig. 1). Due to the larger range, modelling the regional features
(consistency, similarity) explicitly with large-kernel CNNs would be
inefficient in both parameters and computation. Instead, transformers with a
window attention mechanism are well suited for this task. Third, beyond local
and regional, some features have a global span (cyan rectangles in Fig. 1),
incl. but not limited to symmetry, multi-scale pattern repetition (Fig. 1a),
same scale texture similarity (Fig. 1b), and structural similarity and
consistency in large objects and content (Fig. 1c). To model features at this
range, global image understanding is needed.

\begin{overpic}[width=433.62pt]{images/motivation.png}
\put(0.0,-3.0){\scriptsize(a) {bridge} from ICB, $2749\times 4049$}
\put(53.0,37.0){\scriptsize(b) {0848x4} from DIV2K, $1020\times 768$}
\put(53.0,-3.0){\scriptsize(c) {073} from Urban100, $1024\times 765$}
\end{overpic} Figure 1: Natural images show a hierarchy of features in a
global, regional, and local range. The local (edges, colors) and regional
features (the pink squares) could be well modelled by CNNs and window self-
attention. By contrast, it is difficult to efficiently and explicitly model
the rich global features (cyan rectangles). Figure 2: The proposed GRL
achieves state-of-the-art performances on various image restoration tasks.
Details provided in Sec. 5.

Different from the local and regional range features, there are two major
challenges to model the global range features. Firstly, existing image
restoration networks based on convolutions and window attention could not
capture long-range dependencies explicitly by using a single computational
module. Although non-local operations are used in some works, they are either
used sparsely in the network or applied to small image crops. Thus, global
image understanding still mainly happens via progressive propagation of
features through repeated computational modules. Secondly, the increasing
resolution of today’s images poses a challenge for long-range dependency
modelling. High image resolution leads to a computational burden associated
with pairwise pixel comparisons and similarity searches.

The aforementioned discussion leads to a series of research questions: 1) how
to efficiently model global range features in high-dimensional images for
image restoration; 2) how to model image hierarchies (local, regional, global)
explicitly by a single computational module for high-dimensional image
restoration; 3) and how can this joint modelling lead to a uniform performance
improvement for different image restoration tasks. The paper tries to answer
these questions in Sec. 3, Sec. 4, and Sec. 5, resp.

First, we propose anchored stripe self-attention for efficient dependency
modelling beyond the regional range. The proposed self-attention is inspired
by two properties of natural images including cross-scale similarity and
anisotropic image features. Cross-scale similarity means that structures in a
natural image are replicated at different scales. Inspired by that, we propose
to use anchors as an intermediate to approximate the exact attention map
between queries and keys in self-attention. Since the anchors summarize image
information into a lower-dimensional space, the space and time complexity of
self-attention can be significantly reduced. In addition, based on the
observation of anisotropic image features, we propose to conduct anchored
self-attention within vertical and horizontal stripes. Due to the anisotropic
shrinkage of the attention range, a further reduction of complexity is
achieved. And the combination of axial stripes also ensures a global view of
the image content. When equipped with the stripe shift operation, the four
stripe self-attention modes (horizontal, vertical, shifted horizontal, shifted
vertical) achieves a good balance between computational complexity and the
capacity of global range dependency modelling. Furthermore, the proposed
anchored stripe self-attention is analyzed from the perspective of low-
rankness and similarity propagation.

Secondly, a new transformer network is proposed to explicitly model global,
regional, and local range dependencies in a single computational module. The
hierarchical modelling of images is achieved by the parallel computation of
the proposed anchored stripe self-attention, window self-attention, and
channel-attention enhanced convolution. And the transformer architecture is
dubbed GRL.

Thirdly, the proposed GRL transformer is applied to various image restoration
tasks. Those tasks could be classified into three settings based on the
availability of data including real image restoration, synthetic image
restoration, and data synthesis based real image restoration. In total, seven
tasks are explored for the proposed network including image super-resolution,
image denoising, JPEG compression artifacts removal, demosaicking, real image
super-resolution, single image motion deblurring, and defocus deblurring. As
shown in Fig. 2, the proposed network shows promising results on all
investigated image restoration tasks.

##  2 Related Works

Convolution for local range modelling. One of the basic assumptions for
example and learning-based image restoration is that repetitive patterns could
exist in either the same or different images [19] and that the redundant
information they carry could help to restore the local patches. Thus, it helps
if repetitive patterns could be detected and modelled [14, 36, 68, 92, 43,
50]. This intuition matches the computational procedure of convolution well,
which slides the kernel across the image and detects local patterns similar to
the learnable kernels. By stacking multiple convolutional layers, the
receptive field of a CNN gets progressively enlarged and rich image features
are captured. Since the advent of deep learning, great efforts have been made
to design CNNs for image restoration [42, 72, 97, 28, 82, 81, 96, 46, 31, 93].

Non-local and global priors. Besides the local features, it is also important
to model the non-local and global image priors. The early work of non-local
means serves this purpose, which computes an output pixel as the weighted sum
of all the pixels within the image [4]. Inspired by that, later works have
been developed to utilize the repetitive patterns in a non-local range for
image denoising [11] and super-resolution [24]. Apart from the traditional
methods, non-local operations are also introduced into deep neural networks
for video classification [80] and image SR [51, 96].

Besides the non-local operations, self-attention has been developed to model
the global range dependencies [77, 13]. Inspired by the success in natural
language processing, transformers have been applied in both visual perception
tasks [15, 73, 5, 98, 79, 48] and image restoration tasks [6]. However, the
computational complexity of global self-attention grows quadratically with the
number of tokens. Thus, the increase in efficiency of global self-attention is
investigated by several works [8, 34, 10, 38, 78].

Regional self-attention. Among the methods for accelerating transformers,
regional self-attention appears to be promising. The idea is proposed in the
pioneering works [63, 61] and improved as shifted window attention [54, 53].
Inspired by the success of shifted window attention for visual recognition and
perception, this method is also used for image restoration [49, 47, 7].
Despite the good performance of the window attention mechanism, it is pointed
out in recent works that a wider range of pixel involvement could lead to
better image restoration [25, 7]. Thus, in this paper, we try to propose a
method that efficiently brings the modelling capacity of self-attention beyond
the regional range.

##  3 Motivation

###  3.1 Self-attention for dependency modelling

Self-attention is good at modelling long-range dependencies explicitly and it
facilitates the propagation of information across the modelled dependencies.
This operation allows a token to be compared with all the other tokens. The
output token is computed as a weighted sum of all the tokens based on a
similarity comparison, _i.e._ ,

$\mathbf{Y}=\mathrm{Softmax}\left(\mathbf{Q}\cdot\mathbf{K}^{T}/\sqrt{d}\right)\cdot\mathbf{V},$
(1)

where $\mathbf{Q}=\mathbf{W}_{Q}\cdot\mathbf{X}$,
$\mathbf{K}=\mathbf{W}_{K}\cdot\mathbf{X}$,
$\mathbf{V}=\mathbf{W}_{V}\cdot\mathbf{X}$,
$\mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V}\in\mathbb{R}^{d\times d}$, and
$\mathbf{X},\mathbf{Y}\in\mathbb{R}^{d\times N}$. $N$ and $d$ denote the
number of tokens and the dimension of one token, respectively. Additionally,
$\mathbf{M}$ denotes the attention map, _i.e._
$\mathbf{M}=\mathrm{Softmax}(\mathbf{Q}\cdot\mathbf{K}^{T}/\sqrt{d})$.

The time complexity of self-attention is $\mathcal{O}(N^{2}d)$ and the space
complexity is dominated by the term $\mathcal{O}(N^{2})$ of the attention map
$\mathbf{M}$. The computational complexity and memory footprint of self-
attention grow quadratically with the number of tokens. Thus, self-attention
can easily become a computation bottleneck for images where the number of
tokens is the multiplication of the two dimensions of the feature map. To
overcome this problem, it is proposed to apply self-attention within a window.
In this way, the number of tokens that participate in self-attention is
significantly reduced and the computational burden is also lifted.

The problem of window self-attention is that the modelling capacity of the
operation is limited to a regional range due to the small window size
($8\times 8$ [49]). On the other hand, it is shown in recent works [25, 7]
that even a slight increase in window size can lead to better image
restoration. Thus, it can be conjectured that modelling dependencies beyond
the regional range is still important for image restoration. Hence, it
remained to be investigated how to maintain the ability for long-range
dependency modelling under a controlled computational budget.

###  3.2 Motivation I: cross-scale similarity

Figure 3: Cross-scale similarity. (c) and (d) shows the attention map between
the selected pixels and the example high-resolution image. Although the cyan
pixel in (a) and the red pixel in (b) are from images with different
resolutions, their attention map with respect to the high-resolution image
shows very similar structures.

The attention map $\mathbf{M}$ plays an essential role in self-attention as it
captures the similarity between every paired pixels in the image. Thus,
improving the efficiency of the self-attention in Eq. 1 needs one to analyze
the property of the attention map. And we are inspired by a property of
images, _i.e._ cross-scale similarity. That is, the basic structure such as
lines and edges of an image is kept in the different versions of the image
with different scaling factors. In Fig. 3, the attention map between pixels in
an image is shown. Particularly, the attention map between a pixel and the
whole image is visualized as a gray-scale heat map. As shown, no matter
whether the pixel comes from the high-resolution image or the down-scaled
version, the heat map between the pixel and the high-resolution image shows
the basic structure of the image. And the heat maps in Fig. 3(c) and Fig. 3(d)
are very similar to each other.

Anchored self-attention. Inspired by the cross-scale similarity shown in Fig.
3, we try to reduce the complexity of the global self-attention in Eq. 1 by
operating on images with different resolutions and manipulating the number of
tokens, _i.e._ the $N^{2}$ term in $\mathcal{O}(N^{2}d)$. To achieve that, we
introduce an additional concept named anchors besides the triplets of queries,
keys, and values. The set of anchors is a summary of the information in the
image feature map and has a lower dimensionality. Instead of conducting the
similarity comparison between the queries and keys directly, the anchors act
as an intermediate for the similarity comparison. Formally, the anchored self-
attention is proposed as in the following equation

$\displaystyle\mathbf{Y}$
$\displaystyle=\mathbf{M}_{e}\cdot\mathbf{Z}=\mathbf{M}_{e}\cdot\left(\mathbf{M}_{d}\cdot\mathbf{V}\right),$
(2) $\displaystyle\mathbf{M}_{d}$
$\displaystyle=\mathrm{Softmax}({\mathbf{A}\cdot\mathbf{K}^{T}}/\sqrt{d}),$
(3) $\displaystyle\mathbf{M}_{e}$
$\displaystyle=\mathrm{Softmax}(\mathbf{Q}\cdot\mathbf{A}^{T}/\sqrt{d}),$ (4)

where $M\ll N$, $\mathbf{A}\in\mathbb{R}^{M\times d}$ is the anchor,
$\mathbf{M}_{e}\in\mathcal{R}^{N\times M}$ and
$\mathbf{M}_{d}\in\mathcal{R}^{M\times N}$ denotes the attention map between
the query-anchor pair and anchor-key pair. The choice of the operations to
derive the anchors is investigated in the implementation details of the
ablation study of the paper.

Since the number of anchors is much smaller than the number of the other
tokens, the size of the resulting two attention maps $\mathbf{M}_{e}$ and
$\mathbf{M}_{d}$ are much smaller than the original attention map $\mathbf{M}$
in Eq. 1. Then the matrix multiplication in Eq. 2 is computed from the right
hand. The self-attention is first done for the anchors and keys. The attention
map $\mathbf{M}_{d}$ distills the tokens $\mathbf{V}$ into an intermediate
feature $\mathbf{Z}$. Then the self-attention is done between the queries and
the anchors. The second attention map $\mathbf{M}_{e}$ expands the size of the
feature $\mathbf{Z}$ and recovers the information in $\mathbf{V}$. The
computational complexity of the anchored self-attention is reduced to
$\mathcal{O}(NMd)$. And the space complexity is reduced to $\mathcal{O}(NM)$.

###  3.3 Motivation II: anisotropic image features

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
|  |  |
(e) | (f) | (g) | (h)
Figure 4: The image features in natural images are anisotropic. Thus, it is
not always necessary to employ the uniform global range attention in all parts
of the image.

The anchored self-attention could reduce the space and time complexity of the
self-attention in Eq. 1 significantly by removing the quadratic term $N^{2}$.
Yet, for image restoration tasks, the remaining term $N$ is the multiplication
of the width and height of the image. Thus, the complexity of the anchored
self-attention in Eq. 2 could still be unaffordable due to the large term $N$.
Thus, it is desirable to further reduce the complexity of the anchored self-
attention.

To achieve that goal, we resort to another characteristic of natural images,
_i.e._ , the anisotropic image features. As shown in Fig. 4, the natural image
features such as the single object in Fig. 4(c)&(d), the multi-scale
similarity in Fig. 4(h), symmetry in Fig. 4(e)&(g) span in an anisotropic
manner. Thus, isotropic global range attention across the entire image is
redundant to capture the anisotropic image features. And in response to that,
we propose to conduct attention within the anisotropic stripes shown in Fig.
4.

Figure 5: The visualization of the (a) queries, (b) anchors, and (c) keys from
the different layers of the proposed network. (d) shows the attention map
approximated by Eq. 2, _i.e._ $\mathbf{M}_{e}\cdot\mathbf{M}_{d}$. (e) shows
the exact attention map $\mathbf{M}$ computed in Eq. 1.

Stripe attention mechanism. The proposed stripe attention mechanism consists
of four modes including the horizontal stripe, the vertical stripe, the
shifted horizontal stripe, and the shifted vertical stripe. The horizontal and
vertical stripe attention mechanisms could be employed alternately across a
transformer network. In this way, a trade-off is made between maintaining the
global range modelling capacity and controlling the computation complexity of
global self-attention. Thus, in combination with the concept of anchors, we
propose the anchored stripe self-attention. For this attention mechanism,
efficient self-attention is conducted inside the vertical and horizontal
stripes with the help of the introduced anchors.

Figure 6: The architecture of the proposed network. The main component is the
(a) representation learning module which is composed of several stages of
transformer layers. (b) The transformer layer is equipped with global,
regional, and local modelling blocks. (c) The anchored stripe attention is the
key component that helps the network to attend beyond regional ranges.

###  3.4 Discussion

The proposed anchored stripe self-attention mechanism is closely related to
two other concepts including low-rankness and similarity propagation. And we
detail the relationship in this subsection as follows.

Low-rankness of attention map. By comparing the self-attention mechanisms in
Eq. 1 and Eq. 2, we can easily found out that the original attention map
$\mathbf{M}$ is decomposed into small attention maps $\mathbf{M}_{d}$ and
$\mathbf{M}_{e}$ whose rank is no larger than $M$. And the essence here is to
provide the low-rank approximation without calculating the original attention
map first. For the success of the anchored self-attention, it is important to
ensure that with the anchors as the intermediate, the approximated attention
map is similar to the original attention map. Thus, an additional analysis is
provided in Fig. 5.

First, by observing the queries, anchors, and keys, we can conclude that the
anchors have a very similar structure to the query and key. Thus, the anchors
are a good summary of the information in the queries and keys. And
approximating self-attention with anchors as intermediate seems to be
plausible. Additionally, the approximate attention map
$\mathbf{M}_{e}\cdot\mathbf{M}_{d}$ and the exact attention map $\mathbf{M}$
are also compared in Fig. 5. As shown, the approximate attention map keeps the
major structure in the exact attention map, which is confirmed by the large
Pearson correlation coefficients (0.9505) between the two attention maps.
Thus, the quality of the anchored self-attention is guaranteed.

Metric and similarity propagation. From another perspective, in the proposed
anchored self-attention, the queries and keys are first compared with the
anchors and then the query-key similarity is computed. Thus, this computation
procedure needs to propagate the query-anchor and key-anchor similarity to the
query-key pair. And similarity propagation is related to the triangle
inequality in a metric space [29, 83, 21].

A mathematical metric needs to satisfy several conditions including the
essential triangle inequality,

$d(\mathbf{q},\mathbf{k})\leq
d(\mathbf{a},\mathbf{q})+d(\mathbf{a},\mathbf{k}),$ (5)

where $d(\cdot,\cdot)$ defines a metric between two entities. Thus, the
$\mathbf{q}$ / $\mathbf{k}$ distance is upper-bounded by the sum of the
$\mathbf{a}$ / $\mathbf{q}$ distance and the $\mathbf{a}$ / $\mathbf{k}$
distance. This implies that if $\mathbf{a}$ is similar (close) to both
$\mathbf{q}$ and $\mathbf{k}$, then $\mathbf{q}$ and $\mathbf{k}$ should also
be similar (close) to each other. Yet, the similarity measure in Eq. 1 and Eq.
2 is defined by the dot product instead of the distance between tokens, which
does not satisfy the triangle inequality. Thus, similarity propagation could
not be theoretically guaranteed. In order to study the influence of the
similarity measure, an ablation study is conducted and the results are shown
in Sec. 5. Dot product and distance are compared as a similarity measure.
According to the results, although the dot product does not strictly obey the
triangle inequality, it still guarantees better image restoration results.
Thus, we can conclude empirically that the dot product is enough for
similarity propagation in the current setting.

##  4 Modelling Image Hierarchies

In this section, we answer the second research question described in the
introduction, that is, how to explicitly model image hierarchies by a single
computational module. In response to that, we propose the GRL network
architecture that incorporates global range, regional range, and local range
image modelling capacities.

Network architecture. The overall architecture of the proposed network is
shown in Fig. 6. The network takes a degraded low-quality image as input,
processes the image inside the network, and outputs a recovered high-quality
image. In detail, the network contains three parts. 1) The feature extraction
layer is implemented as a simple convolution and converts the input image into
feature maps. 2) The representation learning component enriches the
information extracted in the previous operation. The transformer stage
consists of several transformer layers and ends with a convolution layer. The
dimension of the feature map is maintained across the whole representation
learning module. Skip connection is applied to both the transformer stage and
the representation learning module. 3) The image reconstruction module takes
the rich features calculated by the previous operations and estimates a
recovered image.

Transformer Layer. The transformer layer shown in Fig. 6b is the key component
that provides the hierarchical image modelling capacity in the global,
regional, and local range. This layer first processes the input feature map by
the parallel self-attention module and channel attention enhanced
convolutions. The convolution branch serves to capture local structures in the
input feature map. On the other hand, the self-attention module contains the
window attention proposed in Swin transformer V2 [53] and the anchored stripe
attention proposed in this paper. The feature map is split equally along the
channel dimension and concatenated along the channel dimension again after the
parallel processing within the two attention modules. The window attention
provides the mechanism to capture the regional range dependencies. Then the
feature maps outputted by the convolution module and the attention module are
added to the input feature map, which is processed by the following MLP
module.

Table 1: Single-image motion deblurring results. Top-2 results are highlighted
in red and blue, respectively.

|  GoPro [58] |  HIDE [66]
---|---|---
Method | PSNR$\uparrow$ / SSIM$\uparrow$ | PSNR$\uparrow$ / SSIM$\uparrow$
DeblurGAN [40] | 28.70 / 0.858 | 24.51 / 0.871
Nah  _et al._ [58] | 29.08 / 0.914 | 25.73 / 0.874
DeblurGAN-v2 [41] | 29.55 / 0.934 | 26.61 / 0.875
SRN [71] | 30.26 / 0.934 | 28.36 / 0.915
Gao _et al._[22] | 30.90 / 0.935 | 29.11 / 0.913
DBGAN [91] | 31.10 / 0.942 | 28.94 / 0.915
MT-RNN [60] | 31.15 / 0.945 | 29.15 / 0.918
DMPHN [89] | 31.20 / 0.940 | 29.09 / 0.924
Suin _et al._[70] | 31.85 / 0.948 | 29.98 / 0.930
SPAIR [62] | 32.06 / 0.953 | 30.29 / 0.931
MIMO-UNet+ [9] | 32.45 / 0.957 | 29.99 / 0.930
IPT [6] | 32.52 / - | \- / -
MPRNet [87] | 32.66 / 0.959 | 30.96 / 0.939
Restormer [86] |  32.92 / 0.961 |  31.22 / 0.942
GRL-B (ours) |  33.93 / 0.968 |  31.65 / 0.947

Table 2: Single-image motion deblurring results on RealBlur [64] dataset. The
networks are trained and tested on the same dataset. Top-2 results are
highlighted in red and blue, respectively.

|  RealBlur-R [64] |  RealBlur-J [64]
---|---|---
Method | PSNR$\uparrow$ / SSIM$\uparrow$ | PSNR$\uparrow$ / SSIM$\uparrow$
DeblurGAN-v2 [41] | 36.44 / 0.935 | 29.69 / 0.870
SRN [71] | 38.65 / 0.965 | 31.38 / 0.909
MPRNet [87] | 39.31 / 0.972 | 31.76 / 0.922
MIMO-UNet+ [9] | \- / - | 32.05 / 0.921
MAXIM-3S [76] | 39.45 / 0.962 |  32.84 / 0.935
BANet [75] | 39.55 / 0.971 | 32.00 / 0.923
MSSNet [37] | 39.76 / 0.972 | 32.10 / 0.928
Stripformer [74] |  39.84 / 0.974 | 32.48 / 0.929
GRL-B (ours) |  40.20 / 0.974 |  32.82 / 0.932

Table 3: Defocus deblurring results. S: single-image defocus deblurring. D:
dual-pixel defocus deblurring.

Method | Indoor Scenes | Outdoor Scenes | Combined
---|---|---|---
PSNR$\uparrow$ / SSIM$\uparrow$ / MAE$\downarrow$ / LPIPS$\downarrow$ | PSNR$\uparrow$ / SSIM$\uparrow$ / MAE$\downarrow$ / LPIPS$\downarrow$ | PSNR$\uparrow$ / SSIM$\uparrow$ / MAE$\downarrow$ / LPIPS$\downarrow$
EBDBS [33] | 25.77 / 0.772 / 0.040 / 0.297 | 21.25 / 0.599 / 0.058 / 0.373 | 23.45 / 0.683 / 0.049 / 0.336
DMENetS [44] | 25.50 / 0.788 / 0.038 / 0.298 | 21.43 / 0.644 / 0.063 / 0.397 | 23.41 / 0.714 / 0.051 / 0.349
JNBS [67] | 26.73 / 0.828 / 0.031 / 0.273 | 21.10 / 0.608 / 0.064 / 0.355 | 23.84 / 0.715 / 0.048 / 0.315
DPDNetS [1] | 26.54 / 0.816 / 0.031 / 0.239 | 22.25 / 0.682 / 0.056 / 0.313 | 24.34 / 0.747 / 0.044 / 0.277
KPACS [69] | 27.97 / 0.852 / 0.026 / 0.182 | 22.62 / 0.701 / 0.053 / 0.269 | 25.22 / 0.774 / 0.040 / 0.227
IFANS [45] | 28.11 / 0.861 / 0.026 / 0.179 | 22.76 / 0.720 / 0.052 / 0.254 | 25.37 / 0.789 / 0.039 / 0.217
RestormerS [86] | 28.87 / 0.882 / 0.025 / 0.145 | 23.24 / 0.743 / 0.050 / 0.209 | 25.98 / 0.811 / 0.038 / 0.178
GRLS-B | 29.06 / 0.886 / 0.024 / 0.139 | 23.45 / 0.761 / 0.049 / 0.196 | 26.18 / 0.822 / 0.037 / 0.168
DPDNetD [1] | 27.48 / 0.849 / 0.029 / 0.189 | 22.90 / 0.726 / 0.052 / 0.255 | 25.13 / 0.786 / 0.041 / 0.223
RDPDD [2] | 28.10 / 0.843 / 0.027 / 0.210 | 22.82 / 0.704 / 0.053 / 0.298 | 25.39 / 0.772 / 0.040 / 0.255
UformerD [84] | 28.23 / 0.860 / 0.026 / 0.199 | 23.10 / 0.728 / 0.051 / 0.285 | 25.65 / 0.795 / 0.039 / 0.243
IFAND [45] | 28.66 / 0.868 / 0.025 / 0.172 | 23.46 / 0.743 / 0.049 / 0.240 | 25.99 / 0.804 / 0.037 / 0.207
RestormerD [86] | 29.48 / 0.895 / 0.023 / 0.134 | 23.97 / 0.773 / 0.047 / 0.175 | 26.66 / 0.833 / 0.035 / 0.155
GRLD-B | 29.83 / 0.903 / 0.022 / 0.114 | 24.39 / 0.795 / 0.045 / 0.150 | 27.04 / 0.847 / 0.034 / 0.133

Table 4: Color and grayscale image denoising results. Model complexity and
prediction accuracy are shown for better comparison. The best two results are
highlighted in red and blue, respectively.

Method | # Params [M] | Color | Grayscale
---|---|---|---
CBSD68 [55] |  Kodak24 [18] |  McMaster [94] |  Urban100 [30] |  Set12 [92] |  BSD68 [55] |  Urban100 [30]
$\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$ |  $\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$ |  $\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$ |  $\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$ |  $\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$ |  $\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$ |  $\sigma$$=$$15$ |  $\sigma$$=$$25$ |  $\sigma$$=$$50$
DnCNN [35] | 0.56 | 33.90 | 31.24 | 27.95 | 34.60 | 32.14 | 28.95 | 33.45 | 31.52 | 28.62 | 32.98 | 30.81 | 27.59 | 32.86 | 30.44 | 27.18 | 31.73 | 29.23 | 26.23 | 32.64 | 29.95 | 26.26
RNAN [96] | 8.96 | - | - | 28.27 | - | - | 29.58 | - | - | 29.72 | - | - | 29.08 | - | - | 27.70 | - | - | 26.48 | - | - | 27.65
IPT [6] | 115.33 | - | - | 28.39 | - | - | 29.64 | - | - | 29.98 | - | - | 29.71 | - | - | - | - | - | - | - | - | -
EDT-B [47] | 11.48 | 34.39 | 31.76 | 28.56 | 35.37 | 32.94 | 29.87 | 35.61 | 33.34 | 30.25 | 35.22 | 33.07 | 30.16 | - | - | - | - | - | - | - | - | -
DRUNet [90] | 32.64 | 34.30 | 31.69 | 28.51 | 35.31 | 32.89 | 29.86 | 35.40 | 33.14 | 30.08 | 34.81 | 32.60 | 29.61 | 33.25 | 30.94 | 27.90 | 31.91 | 29.48 | 26.59 | 33.44 | 31.11 | 27.96
SwinIR [49] | 11.75 | 34.42 | 31.78 | 28.56 | 35.34 | 32.89 | 29.79 | 35.61 | 33.20 | 30.22 | 35.13 | 32.90 | 29.82 | 33.36 | 31.01 | 27.91 | 31.97 | 29.50 | 26.58 | 33.70 | 31.30 | 27.98
Restormer [86] | 26.13 | 34.40 | 31.79 | 28.60 | 35.47 | 33.04 | 30.01 | 35.61 | 33.34 | 30.30 | 35.13 | 32.96 | 30.02 | 33.42 | 31.08 | 28.00 | 31.96 | 29.52 | 26.62 | 33.79 | 31.46 | 28.29
GRL-T | 0.88 | 34.30 | 31.66 | 28.45 | 35.24 | 32.78 | 29.67 | 35.49 | 33.18 | 30.06 | 35.08 | 32.84 | 29.78 | 33.29 | 30.92 | 27.78 | 31.90 | 29.43 | 26.49 | 33.66 | 31.23 | 27.89
GRL-S | 3.12 | 34.36 | 31.72 | 28.51 | 35.32 | 32.88 | 29.77 | 35.59 | 33.29 | 30.18 | 35.24 | 33.07 | 30.09 | 33.36 | 31.02 | 27.91 | 31.93 | 29.47 | 26.54 | 33.84 | 31.49 | 28.24
GRL-B | 19.81 | 34.45 | 31.82 | 28.62 | 35.43 | 33.02 | 29.93 | 35.73 | 33.46 | 30.36 | 35.54 | 33.35 | 30.46 | 33.47 | 31.12 | 28.03 | 32.00 | 29.54 | 26.60 | 34.09 | 31.80 | 28.59

Table 5: Classical image SR results. Results of both lightweight models and accurate models are summarized. Method | Scale | # Params [M] |  Set5 [3] |  Set14 [88] |  BSD100 [55] |  Urban100 [30] |  Manga109 [56]
---|---|---|---|---|---|---|---
PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
RCAN [95] | ×2 | 15.44 | 38.27 | 0.9614 | 34.12 | 0.9216 | 32.41 | 0.9027 | 33.34 | 0.9384 | 39.44 | 0.9786
SAN [12] | ×2 | 15.71 | 38.31 | 0.9620 | 34.07 | 0.9213 | 32.42 | 0.9028 | 33.10 | 0.9370 | 39.32 | 0.9792
HAN [59] | ×2 | 63.61 | 38.27 | 0.9614 | 34.16 | 0.9217 | 32.41 | 0.9027 | 33.35 | 0.9385 | 39.46 | 0.9785
NLSA [57] | ×2 | 42.63 | 38.34 | 0.9618 | 34.08 | 0.9231 | 32.43 | 0.9027 | 33.42 | 0.9394 | 39.59 | 0.9789
DBPN [27] | ×2 | 1.26 | 38.09 | 0.9600 | 33.85 | 0.9190 | 32.27 | 0.9000 | 32.55 | 0.9324 | 38.89 | 0.9775
IPT [6] | ×2 | 115.48 | 38.37 | - | 34.43 | - | 32.48 | - | 33.76 | - | - | -
SwinIR [49] | ×2 | 0.88 | 38.14 | 0.9611 | 33.86 | 0.9206 | 32.31 | 0.9012 | 32.76 | 0.9340 | 39.12 | 0.9783
SwinIR [49] | ×2 | 11.75 | 38.42 | 0.9623 | 34.46 | 0.9250 | 32.53 | 0.9041 | 33.81 | 0.9427 | 39.92 | 0.9797
EDT [47] | ×2 | 0.92 | 38.23 | 0.9615 | 33.99 | 0.9209 | 32.37 | 0.9021 | 32.98 | 0.9362 | 39.45 | 0.9789
EDT [47] | ×2 | 11.48 | 38.63 | 0.9632 | 34.80 | 0.9273 | 32.62 | 0.9052 | 34.27 | 0.9456 | 40.37 | 0.9811
GRL-T (ours) | ×2 | 0.89 | 38.27 | 0.9627 | 34.21 | 0.9258 | 32.42 | 0.9056 | 33.60 | 0.9411 | 39.61 | 0.9790
GRL-S (ours) | ×2 | 3.34 | 38.37 | 0.9632 | 34.64 | 0.9280 | 32.52 | 0.9069 | 34.36 | 0.9463 | 39.84 | 0.9793
GRL-B (ours) | ×2 | 20.05 | 38.67 | 0.9647 | 35.08 | 0.9303 | 32.67 | 0.9087 | 35.06 | 0.9505 | 40.67 | 0.9818
RCAN [95] | ×4 | 15.59 | 32.63 | 0.9002 | 28.87 | 0.7889 | 27.77 | 0.7436 | 26.82 | 0.8087 | 31.22 | 0.9173
SAN [12] | ×4 | 15.86 | 32.64 | 0.9003 | 28.92 | 0.7888 | 27.78 | 0.7436 | 26.79 | 0.8068 | 31.18 | 0.9169
HAN [59] | ×4 | 64.20 | 32.64 | 0.9002 | 28.90 | 0.7890 | 27.80 | 0.7442 | 26.85 | 0.8094 | 31.42 | 0.9177
NLSA [57] | ×4 | 44.99 | 32.59 | 0.9000 | 28.87 | 0.7891 | 27.78 | 0.7444 | 26.96 | 0.8109 | 31.27 | 0.9184
DBPN [27] | ×4 | 2.21 | 32.47 | 0.8980 | 28.82 | 0.7860 | 27.72 | 0.7400 | 26.38 | 0.7946 | 30.91 | 0.9137
IPT [6] | ×4 | 115.63 | 32.64 | - | 29.01 | - | 27.82 | - | 27.26 | - | - | -
RRDB [82] | ×4 | 16.70 | 32.73 | 0.9011 | 28.99 | 0.7917 | 27.85 | 0.7455 | 27.03 | 0.8153 | 31.66 | 0.9196
SwinIR [49] | ×4 | 0.90 | 32.44 | 0.8976 | 28.77 | 0.7858 | 27.69 | 0.7406 | 26.47 | 0.7980 | 30.92 | 0.9151
SwinIR [49] | ×4 | 11.90 | 32.92 | 0.9044 | 29.09 | 0.7950 | 27.92 | 0.7489 | 27.45 | 0.8254 | 32.03 | 0.9260
EDT [47] | ×4 | 0.92 | 32.53 | 0.8991 | 28.88 | 0.7882 | 27.76 | 0.7433 | 26.71 | 0.8051 | 31.35 | 0.9180
EDT [47] | ×4 | 11.63 | 33.06 | 0.9055 | 29.23 | 0.7971 | 27.99 | 0.7510 | 27.75 | 0.8317 | 32.39 | 0.9283
GRL-T (ours) | ×4 | 0.91 | 32.56 | 0.9029 | 28.93 | 0.7961 | 27.77 | 0.7523 | 27.15 | 0.8185 | 31.57 | 0.9219
GRL-S (ours) | ×4 | 3.49 | 32.76 | 0.9058 | 29.10 | 0.8007 | 27.90 | 0.7568 | 27.90 | 0.8357 | 32.11 | 0.9267
GRL-B (ours) | ×4 | 20.20 | 33.10 | 0.9094 | 29.37 | 0.8058 | 28.01 | 0.7611 | 28.53 | 0.8504 | 32.77 | 0.9325

Anchored stripe self-attention. The operation of the proposed anchored stripe
attention is conducted according to Eq. 2 and visualized in Fig. 6c. The
dimension of different features is also shown. The triplet of $\mathbf{Q}$,
$\mathbf{K}$, $\mathbf{V}$ is derived by plain linear projections. To
summarize the information into anchors, the anchor projection is implemented
as an average pooling layer followed by a linear projection. After the anchor
projection, the resolution of the image feature map is down-scaled by a factor
of $s$ along both directions. As shown in Fig. 6, the two attention maps
$\mathbf{M}_{d}$ and $\mathbf{M}_{e}$ play a similar role as the original
attention map $\mathbf{M}$ but with less space and time complexity.

##  5 Experimental Results

The experimental results are shown in this section. We answer the third
research question raised in the introduction by investigating the performance
of the proposed network on different image restoration tasks. Based on the
data type, the investigated tasks are classified into three commonly used
settings including 1) real image restoration (single-image motion deblurring,
defocus deblurring), 2) image restoration based on synthetic data (image
denoising, single image SR, JPEG compression artifact removal, demosaicking),
and 3) real image restoration based on data synthesis. We provide three
networks with different model sizes including the tiny, small, and base
versions (GRL-T, GRL-S, GRL-B). For real and synthetic image restoration, Adam
optimizer and $L_{1}$ loss are used to train the network with an initial
learning rate $2\times 10^{-4}$. More details about the training dataset,
training protocols, and additional visual results are shown in the
supplementary material.

Table 6: Grayscale image JPEG compression artifact removal results. As a
comparison metric, the parameter count of FBCNN [32] GRL-S are 71.92M and
3.12M.

Set | QF | JPEG |  DnCNN [92] |  DCSC [20] |  QGAC [16] |  MWCNN [52] |  FBCNN [32] | GRL-S
---|---|---|---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Classic5 [17] | 10 | 27.82 | 0.760 | 29.40 | 0.803 | 29.62 | 0.810 | 29.84 | 0.812 | 30.01 | 0.820 | 30.12 | 0.822 | 30.20 | 0.829
20 | 30.12 | 0.834 | 31.63 | 0.861 | 31.81 | 0.864 | 31.98 | 0.869 | 32.16 | 0.870 | 32.31 | 0.872 | 32.49 | 0.878
30 | 31.48 | 0.867 | 32.91 | 0.886 | 33.06 | 0.888 | 33.22 | 0.892 | 33.43 | 0.893 | 33.54 | 0.894 | 33.72 | 0.899
| 40 | 32.43 | 0.885 | 33.77 | 0.900 | 33.87 | 0.902 | 34.05 | 0.905 | 34.27 | 0.906 | 34.35 | 0.907 | 34.53 | 0.911
BSD500 [55] | 10 | 27.80 | 0.768 | 29.21 | 0.809 | 29.32 | 0.813 | 29.46 | 0.821 | 29.61 | 0.820 | 29.67 | 0.821 | 29.74 | 0.823
20 | 30.05 | 0.849 | 31.53 | 0.878 | 31.63 | 0.880 | 31.73 | 0.884 | 31.92 | 0.885 | 32.00 | 0.885 | 32.05 | 0.885
30 | 31.37 | 0.884 | 32.90 | 0.907 | 32.99 | 0.908 | 33.07 | 0.912 | 33.30 | 0.912 | 33.37 | 0.913 | 33.43 | 0.912
| 40 | 32.30 | 0.903 | 33.85 | 0.923 | 33.92 | 0.924 | 34.01 | 0.927 | 34.27 | 0.928 | 34.33 | 0.928 | 34.38 | 0.928

Table 7: Color image JPEG compression artifact removal results.

Set | QF | JPEG |  QGAC [16] |  FBCNN [32] | GRL-S
---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
LIVE1 [65] | 10 | 25.69 | 0.743 | 27.62 | 0.804 | 27.77 | 0.803 | 28.13 | 0.814
20 | 28.06 | 0.826 | 29.88 | 0.868 | 30.11 | 0.868 | 30.49 | 0.878
30 | 29.37 | 0.861 | 31.17 | 0.896 | 31.43 | 0.897 | 31.85 | 0.905
| 40 | 30.28 | 0.882 | 32.05 | 0.912 | 32.34 | 0.913 | 32.79 | 0.920
BSD500 [55] | 10 | 25.84 | 0.741 | 27.74 | 0.802 | 27.85 | 0.799 | 28.26 | 0.808
20 | 28.21 | 0.827 | 30.01 | 0.869 | 30.14 | 0.867 | 30.57 | 0.875
30 | 29.57 | 0.865 | 31.33 | 0.898 | 31.45 | 0.897 | 31.92 | 0.903
| 40 | 30.52 | 0.887 | 32.25 | 0.915 | 32.36 | 0.913 | 32.86 | 0.919

Table 8: Image demosaicking results.

Datasets | Matlab |  DDR [85] |  DeepJoint [23] |  MMNet [39] |  RLDD [26] |  RNAN [96] |  DRUNet [90] |  GRL-S (ours)
---|---|---|---|---|---|---|---|---
Kodak [18] | 35.78 | 41.11 | 42.00 | 40.19 | 42.49 | 43.16 | 42.68 | 43.57
McMaster [94] | 34.43 | 37.12 | 39.14 | 37.09 | 39.25 | 39.70 | 39.39 | 40.22

###  5.1 Image deblurring

We first investigate the performance of the proposed network on two real image
restoration tasks including single-image motion deblurring, and motion
deblurring.

Single image motion deblurring. Tab. 1 and Tab. 2 shows the experimental
results for single image motion deblurring on synthetic datasets (GoPro [58],
HIDE [66]) and real dataset (RealBlur-R [64]), respectively. Compared with the
previous state-of-the-art Restormer [86], the proposed GRL achieves
significant PSNR improvement of 1.01 dB on the GoPro dataset. On the HIDE
dataset, the PSNR improvement is 0.43 dB. Please note that the improvement is
achieved under fewer parameter budget. As shown in Tab. 4, GRL-B saves 24%
parameters compared with Restormer. As shown in Tab. 2, GRL-B sets the new
state-of-the-art performance of 40.20 PSNR on RealBlur-R dataset.

Defocus deblurring. Tab. 3 shows the experimental results for defocus
deblurring using single image and dual-pixel images. Our GRL outperforms the
previous methods for all three scene types. Compared with Restormer on the
combined scenes, our GRL achieves an elegant performance boost of 0.20 dB and
0.38 dB for single and dual-pixel defocus deblurring. Compared with Uformer
[84] and IFAN [45], GRL achieves PSNR gain of 1.39 dB and 1.05 dB for the
dual-pixel setting.

###  5.2 Image restoration based on synthetic data

Investigating image restoration with synthetic data is also valuable to reveal
the network capacity of restoration methods. Besides the experiments on the
real data, we also study the performance of the network on synthetic data.

Image denoising. First, the experimental results on Gaussian image denoising
are shown in Tab. 4. For a fair comparison between different models, both the
network complexity and accuracy are shown in the table. And several key
findings are observed. I. The tiny version GRL-T is extremely efficient,
reducing model complexity by two orders of magnitude (only 0.76% of [6] and
2.7% of DRUNet [90]) while not sacrificing network accuracy. II. The small
version GRL-S performs competitive with the previous state-of-the-art SwinIR
[49] and Restormer [86]. II. On Urban100, the base version outperforms
Restormer by a large margin (_e.g._ 0.44dB PSNR gain for color image and noise
level 50).

Image SR. Experimental results for classical images are shown in Tab. 5. Both
lightweight models and accurate SR models are summarized. A similar conclusion
could be drawn from the results. I. Among the lightweight networks, GRL-T
outperforms both convolution and self-attention-based networks including DBPN
[27], SwinIR [49] and EDT [47]. Compared with EDT, Significant improvements
are obtained on Urban100 and Manga109 datasets (0.44 dB and 0.22 dB for
$\times 4$ SR). II. GRL-B sets the new state-of-the-art for accurate image SR.
III. GRL-S achieves a good balance between network complexity and SR image
quality.

JPEG compression artifact removal. The experimental results for color and
grayscale images are shown in Tab. 7. Four image quality factors ranging from
10 to 40 for JPEG compression are studied. As shown in the table, the proposed
GRL-S network outperforms the previous state-of-the-art method elegantly
across different datasets and quality factors. Notably, GRL-S has a much
smaller model complexity than FBCNN.

Demosaicking. Results for image demosaicking is shown in Tab. 8. The proposed
method outperforms the previous methods RNAN [96] and DRUNet [90]
significantly.

###  5.3 Real image restoration based on data synthesis

Finally, we also investigate the performance of the network for real-world
image restoration. The aim is to super-resolve a low-quality image by an
upscaling factor of 4. Since there are no ground-truth images for this task,
only the visual comparison is given in Fig. 7. Compared with the other
methods, the proposed GRL is able to remove more artifacts in the low-
resolution images.

|  |  |
---|---|---|---
|  |  |
BSRGAN | Real ESRGAN | SwinIR | GRL
Figure 7: Visual results for real-world image SR.

###  5.4 Ablation study

In this section, the various design choices of the proposed method are
ablated.

Influence of the similarity comparison method. As mentioned in Sec. 3.4, for
theoretical guarantee of similarity propagation, a mathematical metric rather
than a dot product should be used. To study the difference between, image
restoration with the two operations are compared and the results are shown in
Tab. 9. As revealed by the table, the dot product is very competitive compared
with a metric and it outperforms a distance metric for a couple of settings.
Considering this, the dot product is still used in other experiments.

Influence of the anchor projections. The anchor projection operation helps to
summarize the information in the feature map. The ablation study is shown in
Tab. 10. Considering both the accuracy performance and parameter budget,
Avgpool followed by linear projection is finally used. Additionally, the
influence of window and stripe sizes is also ablated in Tab. 11 for image SR.
Increasing the window and anchor size could lead to significant performance
improvement on Urban100 and Manga109.

Table 9: Ablation study on similarity comparison operation.

Operation | Color denoising | Gray denoising | Image SR
---|---|---|---
$\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\times 2$ | $\times 3$ | $\times 4$
Distance | 35.16 | 32.87 | 29.71 | 33.14 | 30.76 | 27.62 | 38.09 | 34.57 | 32.45
Dot product | 35.18 | 32.93 | 29.76 | 33.16 | 30.78 | 27.62 | 38.08 | 34.59 | 32.43

Table 10: Ablation study on anchor projection operation.

Anchor projection operation | # Params [m] | PSNR on Set 5
---|---|---
Depthwise Conv | 3.17 | 35.03
Conv | 4.19 | 35.03
Patch merging | 3.53 | 34.98
Maxpool + Linear Projection | 3.12 | 35.02
Avgpool + Linear Projection | 3.12 | 35.03

Table 11: Ablation study on window and stripe size.

Scale | Window size | Stripe size | Set5 | Set14 | BSD100 | Urban100 | Manga109
---|---|---|---|---|---|---|---
4 | 8 | 16$\times$16 | 32.47 | 28.77 | 27.70 | 26.67 | 30.65
4 | 16 | 64$\times$64 | 32.61 | 28.88 | 27.76 | 26.94 | 31.24
4 | 16 | 64$\times$128 | 32.56 | 28.93 | 27.77 | 27.15 | 31.57

##  6 Conclusion

In this paper, we proposed GRL, a network with efficient and explicit
hierarchical modelling capacities for image restoration. The proposed network
was mainly inspired by two image properties including cross-scale similarity
and anisotropic image features. Based on that, we proposed the efficient
anchored stripe self-attention module for long-range dependency modelling.
Then a versatile network architecture was proposed for image restoration. The
proposed network can model image hierarchies in the global, regional, and
local ranges. Owing to the advanced computational mechanism, the proposed
network architecture achieves state-of-the-art performances for various image
restoration tasks.

## References

  * [1] Abdullah Abuolaim and Michael S Brown.  Defocus deblurring using dual-pixel data.  In European Conference on Computer Vision, pages 111–126. Springer, 2020.
  * [2] Abdullah Abuolaim, Mauricio Delbracio, Damien Kelly, Michael S. Brown, and Peyman Milanfar.  Learning to reduce defocus blur by realistically modeling dual-pixel data.  In ICCV, 2021.
  * [3] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In Proceedings of the British Machine Vision Conference, 2012.
  * [4] Antoni Buades, Bartomeu Coll, and J-M Morel.  A non-local algorithm for image denoising.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 60–65. IEEE, 2005.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proc. ECCV, pages 213–229. Springer, 2020.
  * [6] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299–12310, 2021.
  * [7] Xiangyu Chen, Xintao Wang, Jiantao Zhou, and Chao Dong.  Activating more pixels in image super-resolution transformer.  arXiv preprint arXiv:2205.04437, 2022.
  * [8] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating long sequences with sparse transformers.  arXiv preprint arXiv:1904.10509, 2019.
  * [9] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko.  Rethinking coarse-to-fine approach in single image deblurring.  In ICCV, 2021.
  * [10] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  arXiv preprint arXiv:2009.14794, 2020.
  * [11] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
  * [12] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 11065–11074, 2019.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [14] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In Proceeding of the European Conference on Computer Vision, pages 184–199. Springer, 2014.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [16] Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava.  Quantization guided jpeg artifact correction.  In European Conference on Computer Vision, pages 293–309. Springer, 2020.
  * [17] Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Pointwise shape-adaptive dct for high-quality denoising and deblocking of grayscale and color images.  IEEE Transactions on Image Processing, 16(5):1395–1411, 2007.
  * [18] Rich Franzen.  Kodak lossless true color image suite.  source: http://r0k. us/graphics/kodak, 4(2), 1999.
  * [19] William T Freeman, Thouis R Jones, and Egon C Pasztor.  Example-based super-resolution.  IEEE Computer graphics and Applications, 22(2):56–65, 2002.
  * [20] Xueyang Fu, Zheng-Jun Zha, Feng Wu, Xinghao Ding, and John Paisley.  Jpeg artifacts reduction via deep convolutional sparse coding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2501–2510, 2019.
  * [21] Siddhartha Gairola, Mayur Hemani, Ayush Chopra, and Balaji Krishnamurthy.  Simpropnet: Improved similarity propagation for few-shot image segmentation.  arXiv preprint arXiv:2004.15014, 2020.
  * [22] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia.  Dynamic scene deblurring with parameter selective sharing and nested skip connections.  In CVPR, 2019.
  * [23] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand.  Deep joint demosaicking and denoising.  ACM Transactions on Graphics (ToG), 35(6):1–12, 2016.
  * [24] Daniel Glasner, Shai Bagon, and Michal Irani.  Super-resolution from a single image.  In Proceedings of the International Conference on Computer Vision, pages 349–356. IEEE, 2009.
  * [25] Jinjin Gu and Chao Dong.  Interpreting super-resolution networks with local attribution maps.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9199–9208, 2021.
  * [26] Yu Guo, Qiyu Jin, Gabriele Facciolo, Tieyong Zeng, and Jean-Michel Morel.  Residual learning for effective joint demosaicing-denoising.  arXiv preprint arXiv:2009.06205, 2020.
  * [27] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita.  Deep back-projection networks for super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1664–1673, 2018.
  * [28] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita.  Deep backprojection networks for super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [29] Cheng-Kang Hsieh, Longqi Yang, Yin Cui, Tsung-Yi Lin, Serge Belongie, and Deborah Estrin.  Collaborative metric learning.  In Proceedings of the 26th international conference on world wide web, pages 193–201, 2017.
  * [30] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2015.
  * [31] Jiaxi Jiang, Kai Zhang, and Radu Timofte.  Towards flexible blind jpeg artifacts removal.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4997–5006, 2021.
  * [32] Jiaxi Jiang, Kai Zhang, and Radu Timofte.  Towards flexible blind jpeg artifacts removal.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4997–5006, 2021.
  * [33] Ali Karaali and Claudio Rosito Jung.  Edge-based defocus blur estimation with adaptive scale selection.  TIP, 2017.
  * [34] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020.
  * [35] Daisuke Kiku, Yusuke Monno, Masayuki Tanaka, and Masatoshi Okutomi.  Beyond color difference: Residual interpolation for color image demosaicking.  IEEE Transactions on Image Processing, 25(3):1288–1300, 2016.
  * [36] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1646–1654, 2016.
  * [37] Kiyeon Kim, Seungyong Lee, and Sunghyun Cho.  MSSNet: Multi-scale-stage network for single image deblurring.  In Proceedings of the European Conference on Computer Vision Workshops, pages 524–539. Springer, 2022.
  * [38] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  arXiv preprint arXiv:2001.04451, 2020.
  * [39] Filippos Kokkinos and Stamatios Lefkimmiatis.  Iterative joint image demosaicking and denoising using a residual denoising network.  IEEE Transactions on Image Processing, 28(8):4177–4188, 2019.
  * [40] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas.  DeblurGAN: Blind motion deblurring using conditional adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [41] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang.  DeblurGAN-v2: Deblurring (orders-of-magnitude) faster and better.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [42] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Deep laplacian pyramid networks for fast and accurate super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 624–632, 2017.
  * [43] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4681–4690, 2017.
  * [44] Junyong Lee, Sungkil Lee, Sunghyun Cho, and Seungyong Lee.  Deep defocus map estimation using domain adaptation.  In CVPR, 2019.
  * [45] Junyong Lee, Hyeongseok Son, Jaesung Rim, Sunghyun Cho, and Seungyong Lee.  Iterative filter adaptive network for single image defocus deblurring.  In CVPR, 2021.
  * [46] Runde Li, Jinshan Pan, Zechao Li, and Jinhui Tang.  Single image dehazing via conditional generative adversarial network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8202–8211, 2018.
  * [47] Wenbo Li, Xin Lu, Jiangbo Lu, Xiangyu Zhang, and Jiaya Jia.  On efficient transformer and image pre-training for low-level vision.  arXiv preprint arXiv:2112.10175, 2021.
  * [48] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool.  LocalViT: Bringing locality to vision transformers.  arXiv preprint arXiv:2104.05707, 2021.
  * [49] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 1833–1844, 2021.
  * [50] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1132–1140, 2017.
  * [51] Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang.  Non-local recurrent network for image restoration.  Advances in Neural Information Processing Systems, 31, 2018.
  * [52] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo.  Multi-level wavelet-cnn for image restoration.  In IEEE conference on computer vision and pattern recognition workshops, pages 773–782, 2018.
  * [53] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.  Swin transformer v2: Scaling up capacity and resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12009–12019, 2022.
  * [54] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [55] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings of the IEEE International Conference on Computer Vision, volume 2, pages 416–423. IEEE, 2001.
  * [56] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Sketch-based manga retrieval using manga109 dataset.  Multimedia Tools and Applications, 76(20):21811–21838, 2017.
  * [57] Yiqun Mei, Yuchen Fan, and Yuqian Zhou.  Image super-resolution with non-local sparse attention.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3517–3526, 2021.
  * [58] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3883–3891, 2017.
  * [59] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  In European Conference on Computer Vision, pages 191–207, 2020\.
  * [60] Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun.  Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training.  In ECCV, 2020.
  * [61] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In International Conference on Machine Learning, pages 4055–4064. PMLR, 2018.
  * [62] Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishnu Naresh Boddeti.  Spatially-adaptive image restoration using distortion-guided networks.  In ICCV, 2021.
  * [63] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.  Stand-alone self-attention in vision models.  Advances in Neural Information Processing Systems, 32, 2019.
  * [64] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho.  Real-world blur dataset for learning and benchmarking deblurring algorithms.  In European Conference on Computer Vision, pages 184–201. Springer, 2020.
  * [65] HR Sheikh.  Live image quality assessment database release 2.  http://live. ece. utexas. edu/research/quality, 2005.
  * [66] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao.  Human-aware motion deblurring.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5572–5581, 2019.
  * [67] Jianping Shi, Li Xu, and Jiaya Jia.  Just noticeable defocus blur detection and estimation.  In CVPR, 2015.
  * [68] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1874–1883, 2016.
  * [69] Hyeongseok Son, Junyong Lee, Sunghyun Cho, and Seungyong Lee.  Single image defocus deblurring using kernel-sharing parallel atrous convolutions.  In ICCV, 2021.
  * [70] Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan.  Spatially-attentive patch-hierarchical network for adaptive motion deblurring.  In CVPR, 2020.
  * [71] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia.  Scale-recurrent network for deep image deblurring.  In CVPR, 2018.
  * [72] Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao.  Image super-resolution using dense skip connections.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4809–4817, 2017.
  * [73] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  arXiv preprint arXiv:2012.12877, 2020.
  * [74] Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin.  Stripformer: Strip transformer for fast image deblurring.  In Proceedings of the European Conference on Computer Vision, pages 146–162. Springer, 2022.
  * [75] Fu-Jen Tsai, Yan-Tsung Peng, Chung-Chi Tsai, Yen-Yu Lin, and Chia-Wen Lin.  BANet: A blur-aware attention network for dynamic scene deblurring.  IEEE Transactions on Image Processing, 31:6789–6799, 2022.
  * [76] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxim: Multi-axis mlp for image processing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5769–5780, 2022.
  * [77] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [78] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  arXiv preprint arXiv:2006.04768, 2020.
  * [79] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [80] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018.
  * [81] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 606–615, 2018.
  * [82] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018.
  * [83] Xin-Jing Wang, Wei-Ying Ma, Gui-Rong Xue, and Xing Li.  Multi-model similarity propagation and its application for web image retrieval.  In Proceedings of the 12th annual ACM international conference on Multimedia, pages 944–951, 2004.
  * [84] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A general u-shaped transformer for image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17683–17693, 2022.
  * [85] Jiqing Wu, Radu Timofte, and Luc Van Gool.  Demosaicing based on directional difference regression and efficient regression priors.  IEEE Transactions on Image Processing, 25(8):3862–3874, 2016.
  * [86] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728–5739, 2022.
  * [87] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14821–14831, 2021.
  * [88] Roman Zeyde, Michael Elad, and Matan Protter.  On single image scale-up using sparse-representations.  In Proceedings of International Conference on Curves and Surfaces, pages 711–730. Springer, 2010.
  * [89] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz.  Deep stacked hierarchical multi-patch network for image deblurring.  In CVPR, 2019.
  * [90] Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte.  Plug-and-play image restoration with deep denoiser prior.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [91] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li.  Deblurring by realistic blurring.  In CVPR, 2020.
  * [92] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising.  IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
  * [93] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  * [94] Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding.  Journal of Electronic imaging, 20(2):023016, 2011.
  * [95] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In European Conference on Computer Vision, pages 286–301, 2018\.
  * [96] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu.  Residual non-local attention networks for image restoration.  arXiv preprint arXiv:1903.10082, 2019.
  * [97] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [98] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.


# Data Processing Matters: SRPH-Konvergen AI’s Machine Translation System for
WMT’21

Lintang Sutawika
Konvergen AI
Jakarta, Indonesia
<EMAIL_ADDRESS>
&Jan Christian Blaise Cruz11footnotemark: 1
Samsung Research Philippines
Manila, Philippines
<EMAIL_ADDRESS>
Equal contribution. Order determined via coinflip.

###### Abstract

In this paper, we describe the submission of the joint Samsung Research
Philippines-Konvergen AI team for the WMT’21 Large Scale Multilingual
Translation Task - Small Track 2. We submit a standard Seq2Seq Transformer
model to the shared task without any training or architecture tricks, relying
mainly on the strength of our data preprocessing techniques to boost
performance. Our final submission model scored 22.92 average BLEU on the
FLORES-101 devtest set, and scored 22.97 average BLEU on the contest’s hidden
test set, ranking us sixth overall. Despite using only a standard Transformer,
our model ranked first in Indonesian $\rightarrow$ Javanese, showing that data
preprocessing matters equally, if not more, than cutting edge model
architectures and training techniques.

##  1 Introduction

This paper describes the machine translation system submitted by the joint
team of Samsung Research Philippines and Konvergen AI for the WMT’21 Large
Scale Multilingual Translation Task. Our team participated in Small Track #2,
where the task is to produce a multilingual machine translation system for
five Southeast-Asian languages: Javanese, Indonesian, Malay, Tagalog, and
Tamil111Tamil is considered an official language in Singapore, a Southeast
Asian country, plus English, in all 30 directions.

We will first describe the filtering heuristics that we used to preprocess the
data, and then outline the steps we took to train and evaluate our models.
Specific hyperparameters, preprocessing decisions, and other training
parameters will be listed in their corresponding sections. Finally, we report
our results on the FLORES-101 Goyal et al. (2021) devtest set, as well as on
the competition’s hidden test set.

##  2 Parallel Text Preprocessing Heuristics

The contest dataset comprises of various bitext sources, including: bible-
uedin Christodouloupoulos and Steedman (2015), CCAligned El-Kishky et al.
(2020), ELRC 2922222https://elrc-share.eu/, MultiCCAligned El-Kishky et al.
(2020), ParaCrawl 333https://www.paracrawl.eu/, TED2020 Reimers and Gurevych
(2020), WikiMatrix Schwenk et al. (2019), tico-19, Ubuntu, OpenSubtitles, QED,
Tanzil, Tatoeba, GlobalVoices, GNOME, KDE4, and WikiMedia Tiedemann (2012).

We preprocess the datasets before training in order to minimize spurious
relations that originate from incorrect text pairs. Our preprocessing removes
samples based on a few heuristics that we developed based on our observation
on the datasets. Each bitext file is applied a different set of preprocessing
based on observation. For example we filter by number content for datasets
such as CCAAligned while TED2020 is not applied that same filter.

In this section, we will cover the decisions made during preprocessing. We
observe a score increase of 1.91 BLEU on our submission model when the
preprocessing is applied. We report the total number of lines filtered from
the bitext for all language pairs on Table 1.

ISO | Language Pair | Before Preprocessing | After Preprocessing | Reduction
---|---|---|---|---
en-id | English - Indonesian | 54,075,891 | 27,186,074 | 49.73%
en-ms | English - Malaysian | 13,437,727 | 7,674,956 | 42.89%
en-tl | English - Tagalog | 13,612,403 | 5,302,768 | 61.04%
en-jv | English - Javanese | 3,044,920 | 388,766 | 87.23%
en-ta | English - Tamil | 2,115,925 | 1,420,827 | 32.85%
id-ms | Indonesian - Malaysian | 4,857,321 | 3,371,777 | 30.58%
id-tl | Indonesian - Tagalog | 2,743,305 | 1,823,140 | 33.54%
id-jv | Indonesian - Javanese | 780,119 | 432,734 | 44.53%
id-ta | Indonesian - Tamil | 500,898 | 393,336 | 21.47%
ms-tl | Malaysian - Tagalog | 1,358,486 | 985,493 | 27.46%
ms-jv | Malaysian - Javanese | 434,710 | 250,070 | 42.47%
ms-ta | Malaysian - Tamil | 372,623 | 351,416 | 5.69%
tl-jv | Tagalog - Javanese | 817,146 | 544,233 | 33.40%
tl-ta | Tagalog - Tamil | 563,337 | 482,618 | 14.33%
jv-ta | Javanese - Tamil | 65,997 | 48,806 | 26.05%
Table 1: Number of parallel text lines per language pair before and after
applying preprocessing

###  2.1 Filter by Duplicate

Duplication is present throughout the dataset. Table 2 outlines samples of
duplication based on three distinct types:

  * •

Duplicates within the same language Within a subset file of a designated
language, multiple lines have the same string while the its counterpart may
feature different translations.

  * •

Partial duplication The whole string of text in one language is present in its
counterpart translation.

  * •

Duplication among parallel text Both source and target text line feature
exactly the same string. While this may be correct for named entities, most of
these duplication are short and can be non-informative.

Duplicates within the same file
---
GNOME.en-tl.en | GNOME.en-tl.tl
Error reading from file: %s | Error sa pagbasa ng talaksang ’%s’: %s
Error seeking in file: %s | Error sa pagbasa ng talaksang ’%s’: %s
Error closing file: %s | Error sa pagbasa ng talaksang ’%s’: %s
Partial duplication
WikiMatrix.en-jv.en | WikiMatrix.en-jv.jv
CJ E&M Corporation. | Drama iki diprodhuksi déning CJ E&M Corporation.
New Orleans, Louisiana. | Lair ing New Orleans, Louisiana.
Edward Thomas Hardy. | Jeneng dawané ya iku Edward Thomas Hardy.
Duplication among parallel text
OpenSubtitles.en-ta.en | OpenSubtitles.en-ta.ta
Those who are invited will find the way. | Those who are invited will find the way.
Gazelle, whose face the full moon forms: | Gazelle, whose face the full moon forms:
Time has warned us never to approach her. | Time has warned us never to approach her.
Table 2: Examples of duplication based on three types

###  2.2 Filtering by Language and Letters

In algorithmically-aligned datasets such as CCAligned, some training examples
are not in the list of contest languages. We find full text lines that are in
Azerbaijani, Turkish, Arabic, and Japanese. To identify these languages, we
use langdetect444https://pypi.org/project/langdetect/. This filter works for
sentences that are fully foreign. It is also the case that foreign letters
that may refer to named-entity can be found in the dataset. We consider this
to be allowable so long as the the foreign character string is present in both
source and target text line. To filter this, we use
AlphabetDetector555https://pypi.org/project/alphabet-detector/ and check if
detected foreign letters are present in both text line.

###  2.3 Filter by Specific Keywords and Symbols

There are a number of cases where the translations are generally correct but
also feature extra keywords that have no relation to the parallel text. These
keywords are generally in English and are consistently present in a number of
bitext datasets such as KDE4, GNOME, and Ubuntu.

KDE4.en-id.en | KDE4.en-id.id
---|---
Task Scheduler | Penjadwal TugasComment
Configure and schedule tasks | Atur dan jadwal tugasName
Table 3: Example of translations that also have an extra keyword. Underlined
text are keywords that are misplaced in correct translations.

Bitexts such as OpenSubtitles feature secondary information that relates to a
particular scene (for example "(loud music playing)"). These secondary
information may be in parentheses to denote an action being done or to signify
a song being played. These secondary information are not always available for
each language. We opt to remove all lines that have these specific symbols.

###  2.4 Filtering Number Content

We apply a filter to remove incorrect text lines in the bitext by checking if
both source and target text lines feature the same numeric values such as date
and quantities. Table 4 shows that filtering by number can remove text lines
that do not relate to one another as numeric values tend to translate the
same. Due to the limited time allotted for the shared task, we opt to remove
entirely parallel sentences that do not have matching numbers. We filter this
by using regular expressions.

###  2.5 Filtering by Length

Text lines with very long lengths are generally not informative, we find most
of these text lines consists of a list of names that would normally be found
in a bibliography. We set an arbitrary max length of 500 characters for both
source and target sentences.

| MultiCCAligned.id-tl.id | MultiCCAligned.id-tl.tl
---|---|---
Removed | Di. 13:00 - 17:30 | Mo. 13:00 - 18:00
Di 24 nov. 10h – 18h | Sa 23 nov. 10h – 18h
Kept | (Terakhir diperbarui saat: 24/03/2020) | (Huling nai-update Sa: 24/03/2020)
Harga / $: 1,2835 | presyo / $: 1.2835
Table 4: Incorrect translations can be easily identified by checking whether
numeric values in both strings match. In the first example, the sentence pair
was removed due to differing date and time. In the second example, the
sentence pair was kept as we do not check punctuation for numerical values.

##  3 Experiments

###  3.1 Model Architecture

For our submission, we wish to measure how much performance can be boosted by
heuristics-based data preprocessing alone. Given that we anticipate most, if
not all, submissions to the shared task will be transformer-based models, we
opt to use the standard “vanilla” Sequence-to-Sequence Transformer Vaswani et
al. (2017) model with little-to-no changes. This lets us more clearly compare
the performance boost of our filtering heuristics against the boost provided
by a number of architecture augmentations and training tricks that other
submissions might have.

In addition to using a standard Transformer model, we only train the model
directly on our filtered bitext and do not make use of Backtranslation
Sennrich et al. (2015a) for data augmentation. We also start from-scratch with
models initialized using Glorot Uniform Glorot and Bengio (2010), opting not
to use massively-pretrained translation models such as M2M-100 Fan et al.
(2021) as our starting checkpoint.

Following Vaswani et al. (2017), we produce two models: a base model and a
large model. For the sake of simplicity, for the rest of the paper, we will
refer to our models trained with our filtered data as
$\text{Base}_{\text{Heuristics}}$ and $\text{Large}_{\text{Heuristics}}$.

The hyperparameters used for our models are presented in Table 5.

| Base | Large
---|---|---
Vocab Size | 37,000 | 37,000
Encoder Layers | 6 | 6
Decoder Layers | 6 | 6
Attention Heads | 8 | 16
Embedding Dim. | 512 | 1024
Feedforward Dim. | 2048 | 4096
Dropout | 0.1 | 0.3
Attention Dropout | 0.1 | 0.3
Pos. Embeddings | Sinusoid | Sinusoid
Parameters | 63M | 214M
Table 5: Model hyperparameter choices for the base and large Transformer
variants.

###  3.2 Data Preformatting and Tokenization

Our models employ one single shared vocabulary for all languages and
directions. We train our tokenizer using the
SentencePiece666https://github.com/google/sentencepiece library, limiting our
vocabulary to 37,000 BPE Sennrich et al. (2015b) tokens, and training with a
character coverage of $0.995$.

Before training the tokenizer, we first preformat the dataset into the format
to be used for training later on. We append the source and target language’s
ISO-639-1 code enclosed in square brackets at the beginning of each sentence.
For example:

[en] [tl] Today is a sunny day.

is the preformatted version of "Today is a sunny day." when translating from
English to Tagalog.

This preformatting is only done for the source sentences in the training
dataset, while the target sentences are untouched.

For the purpose of training the tokenizer, the six language tokens ([en],
[id], [jv], [ms], [ta], and [tl]) are treated as special tokens to ensure that
they will not be segmented later on.

###  3.3 Training Setup

We then compile our filtered, preformatted bitext and train our base and large
models. During training, we limit all source and target sentences to a maximum
sequence length of 150 subword tokens. All sentences that are much longer are
truncated.

Our models are trained using the Adam Kingma and Ba (2014) optimizer.
Following Vaswani et al. (2017), we also use the “Noam“ learning rate
scheduler, linearly increasing the learning rate from $0$ for the first 8000
steps, then decaying afterward. We also set Adam’s $\beta_{2}=0.998$ and use a
label smoothing factor of $0.1$.

For batching, we accumulate tokens until we reach a maximum size of
approximately 32,000 tokens per batch, an increase over the 25,000 tokens used
in Vaswani et al. (2017). We then train the base model and the large model for
100,000 steps and 300,000 steps, respectively. All our models are trained on 8
NVIDIA Tesla P100 GPUs in parallel using the OpenNMT-py Klein et al. (2017)
toolkit.

###  3.4 Translation

To generate translations using the model, we use Beam Search with beam size
$5$ and apply an average length penalty of $0.6$. During generation, we limit
all outputs to a maximum sequence length of 100, preemptively terminating
generation if it begins to exceed this maximum length. We do not use sampling
during translation, nor increase the temperature parameter as this induces
randomness Lopez et al. (2020).

We test our experimental models on the FLORES-101 devtest set. We report our
BLEU scores using the SPM-BLEU variant of
SacreBLEU777BLEU+case.mixed+numrefs.1+smooth.exp+tok.spm
+version.1.5.0 Post (2018).

##  4 Results

After training our models and producing sample translations from the
FLORES-101 devtest set, we compare the results of our two models with a number
of baselines:

  * •

Transformers with No Heuristics – These models are essentially identical with
our Transformer models in terms of architecture, hyperparameters, and training
setups, except the bitext they are training on are the raw training corpus
given in the competition (i.e. the filtering heuristics were not applied on
them). We train these models as an ablation experiment to be able to identify
how much of the final performance is attributable to the filtering heuristics.

  * •

M2M-100 615M – This is the baseline given for the WMT’21 Large-scale
Multilingual Translation Task Small Track 2 competition. This M2M-100 Fan et
al. (2021) model was trained on CCMatrix and CCaligned with no further
finetuning on the contest dataset.

  * •

DeltaLM+ZCode – This is the best performing model for the Small Track 2. The
model is a finetuned version of the DeltaLM Ma et al. (2021) encoder-decoder
pretrained model.

All analyses and results within this section are based on the public devtest
set and not the contest’s hidden test set, unless specified. A summary of the
BLEU scores for all models and baselines are available on Table 6.

| $\text{Base}_{\text{Heuristics}}$ | $\text{Large}_{\text{Heuristics}}$ | Base | Large | M2M100 | DeltaLM
---|---|---|---|---|---|---
|  |  |  |  | Baseline | +ZCode
en$\rightarrow$id | 35.94 | 39.29 | 35.12 | 36.51 | 36.34 | 50.90
id$\rightarrow$en | 31.20 | 33.40 | 29.22 | 30.93 | 33.33 | 47.35
en$\rightarrow$jv | 21.53 | 23.57 | 16.95 | 20.98 | 15.06 | 27.70
jv$\rightarrow$en | 22.09 | 24.61 | 18.85 | 21.26 | 21.38 | 39.44
en$\rightarrow$ms | 31.36 | 36.93 | 36.63 | 38.60 | 32.63 | 46.77
ms$\rightarrow$en | 31.92 | 33.16 | 30.31 | 32.97 | 33.63 | 47.86
en$\rightarrow$ta | 9.15 | 10.64 | 8.78 | 9.68 | 4.24 | 35.48
ta$\rightarrow$en | 17.00 | 19.55 | 15.83 | 18.47 | 7.52 | 35.29
en$\rightarrow$tl | 26.91 | 33.23 | 27.87 | 27.56 | 9.95 | 40.52
tl$\rightarrow$en | 31.22 | 33.65 | 26.51 | 29.61 | 26.59 | 48.55
id$\rightarrow$jv | 23.18 | 23.91 | 21.41 | 22.30 | 15.86 | 23.35
jv$\rightarrow$id | 25.45 | 27.10 | 24.15 | 25.15 | 23.21 | 34.64
id$\rightarrow$ms | 30.58 | 33.94 | 28.38 | 33.01 | 29.32 | 38.30
ms$\rightarrow$id | 30.94 | 33.68 | 31.29 | 32.54 | 31.44 | 40.36
id$\rightarrow$ta | 7.04 | 7.88 | 6.78 | 7.09 | 1.44 | 29.61
ta$\rightarrow$id | 13.74 | 16.46 | 13.35 | 14.87 | 4.99 | 28.56
id$\rightarrow$tl | 23.32 | 25.27 | 22.30 | 23.23 | 9.32 | 33.56
tl$\rightarrow$id | 25.31 | 27.76 | 23.40 | 25.03 | 20.76 | 38.70
jv$\rightarrow$ms | 23.36 | 25.08 | 19.92 | 23.63 | 19.57 | 33.14
ms$\rightarrow$jv | 21.08 | 21.29 | 12.33 | 20.97 | 14.22 | 23.91
jv$\rightarrow$ta | 4.70 | 4.97 | 3.85 | 4.62 | 3.52 | 24.19
ta$\rightarrow$jv | 9.25 | 11.13 | 7.54 | 9.22 | 2.51 | 18.35
jv$\rightarrow$tl | 17.43 | 19.61 | 15.79 | 17.31 | 11.96 | 28.50
tl$\rightarrow$jv | 16.96 | 18.82 | 14.56 | 17.00 | 12.31 | 23.17
ms$\rightarrow$ta | 7.01 | 7.87 | 6.65 | 7.23 | 2.38 | 28.83
ta$\rightarrow$ms | 15.09 | 16.64 | 14.54 | 16.44 | 4.70 | 26.83
ms$\rightarrow$tl | 23.30 | 24.97 | 22.17 | 23.01 | 11.04 | 32.81
tl$\rightarrow$ms | 25.86 | 27.10 | 23.19 | 25.85 | 18.16 | 36.15
ta$\rightarrow$tl | 15.26 | 18.43 | 14.98 | 16.05 | 3.15 | 26.64
tl$\rightarrow$ta | 6.27 | 7.65 | 5.89 | 6.60 | 3.10 | 28.80
Average | 20.78 | 22.92 | 19.28 | 21.01 | 15.46 | 33.94
Table 6: Summary of BLEU scores on the FLORES-101 devtest set. The first two
columns show the performance of our Transformer models trained with the data
filtering heuristics. The next two columns show the same Transformer models,
but trained on an unpreprocessed version of the training dataset. We also show
the scores of the M2M-100 615M baseline model, as well as the best performing
model (DeltaLM+ZCode) for the Small Track 2.
$\text{Large}_{\text{Heuristics}}$ (column 2) is our final submission model
for the contest.

###  4.1 Transformer + Heuristics vs. Baselines

We report the results of our $\text{Base}_{\text{Heuristics}}$ and
$\text{Large}_{\text{Heuristics}}$ models against the M2M-100 615M model
baseline as well as the best performing model for the shared task.

$\text{Base}_{\text{Heuristics}}$ scored an average BLEU of 20.78 on all 30
directions. On the other hand, $\text{Large}_{\text{Heuristics}}$ scored 22.92
average BLEU on all 30 directions, which is 2.14 BLEU points higher than the
base model. Both models outperformed the M2M-100 615M baseline, with the base
model giving a 5.32 BLEU improvement, and the large model giving a 7.46 BLEU
improvement.

It is worth noting that, while the $\text{Base}_{\text{Heuristics}}$
outperforms the baseline on average, it fails to outperform it on four
specific translation directions: en$\leftrightarrow$id and
en$\leftrightarrow$ms. Note that it is these two language pairs that have the
most number of training sentences in the training corpus.

The language pairs that benefit significantly from training on the contest
dataset are language pairs that are of less volume than en$\leftrightarrow$id
and en$\leftrightarrow$ms. This is likely due to these pairs being less-
sampled in M2M100’s training dataset, and thus were not as learned by the
model compared to pairs with a higher volume of training data.

The same observations can be found when comparing the performance of
$\text{Large}_{\text{Heuristics}}$ against the baseline model.
$\text{Large}_{\text{Heuristics}}$ only marginally outperformed the baseline
in one direction (id$\rightarrow$en, +0.07 BLEU), and marginally
underperformed against the baseline in one direction (ms$\rightarrow$en, -0.47
BLEU). This higher performance for M2M-100 is likely due to the training
method used in the model in addition to the size of the training corpora used.
While M2M-100 is advantageous in these translation directions, the difference
is only marginal, most likely owing to $\text{Large}_{\text{Heuristics}}$’s
size which gives it higher capacity.

Both our transformer models and the baseline model are significantly
outperformed by the DeltaLM+ZCode model, which is the best performing model in
the competition. The best model outperforms our best model
($\text{Large}_{\text{Heuristics}}$) by a significant 11.02 average BLEU, and
the baseline model by 18.48 average BLEU.

While DeltaLM+ZCode outperforms our model in terms of average performance, it
is worth noting that our model – a standard Transformer without any
augmentations and training tricks – managed to outperform DeltaLM+ZCode in one
translation direction: id$\rightarrow$jv.

$\text{Large}_{\text{Heuristics}}$ scored 23.91 BLEU while DeltaLM+ZCode
scored 23.35 BLEU. While the difference is marginal (+0.56 BLEU), our model
still outperforms the best model in this direction, which we attribute to the
quality of our data preprocessing and filtering heuristics.

###  4.2 Heuristics vs. No Heuristics

To quantify how much our filtering heuristics contributed to the final
performance of our models, we trained two additional models: both identical to
our base and large transformer variants, except the training corpus used was
not processed using our filtering heuristics. For these ablation experiments,
we use the same BPE tokenizer that is used for our main transformer models
(trained on the filtered data). This is to ensure full model equivalency. To
prevent confusion, we will refer to these ablation models simply as Base and
Large to differentiate them from our contest models
$\text{Base}_{\text{Heuristics}}$ and $\text{Large}_{\text{Heuristics}}$.

On average, both sizes of models performed worse when trained without the
filtering heuristics. Base scored 19.28 average BLEU on the devtest set, 1.5
points lower than $\text{Base}_{\text{Heuristics}}$. On the other hand, Large
scored average 21.01 BLEU, which is 1.91 points lower than
$\text{Large}_{\text{Heuristics}}$.

It is interesting, however, that Base outperformed
$\text{Base}_{\text{Heuristics}}$ in two translation directions:
en$\rightarrow$ms and ms$\rightarrow$id. This may indicate that the filtering
heuristics work better for a certain subset of languages. We look towards
exploring how filtering methods such as ours affect multilingual translation
datasets in terms of balance and informativeness in the future.

On the other hand, Large performed worse than
$\text{Large}_{\text{Heuristics}}$ in all 30 directions. This may be due to
the increase in total trainable parameters, as larger models need more data
with higher quality to be effectively trained.

###  4.3 The Case of Tamil

We observe that our models, including the other models on the shared task
leaderboard, struggled with Tamil. X$\leftrightarrow$ta translation is on
average much worse in terms of BLEU score compared to the other translation
directions that do not involve it.

We hypothesize that this is due to two things.

First, Tamil is the most underrepresented language in the shared task dataset,
with X$\leftrightarrow$ta having the least amount of parallel text for every
language X in the training set. This causes the model, to a certain extent, to
underfit on directions that translate to or from Tamil.

Second, Tamil is the only language in the shared task dataset that does not
use the latin alphabet. Combined with the fact that it is the most
underrepresented language in the dataset, there is a possibility that the
model may have treated Tamil as noise during training. The observation that
X$\rightarrow$ta performs worse on average compared to its inverse direction
ta$\rightarrow$X lends more credence to this hypothesis. The model is not
trained well to represent sentences in Tamil, and thus, struggles when
generating Tamil translations.

Part of our planned future work includes identifying methods to improve
translation in multilingual datasets where the alphabets used may be more than
one. This is to improve translation to non-latin alphabet languages in future
methods.

###  4.4 Hidden Test Set Performance

We also report the performance of our models on the shared task’s hidden test
set. We once more compare our results against the baseline M2M-100 model as
well as the best performing DeltaLM+ZCode model.

Our final submission for the shared task was our
$\text{Large}_{\text{Heuristics}}$ model, which performed with an average BLEU
of 22.97 on the shared task’s hidden test set. This is a marginal difference
from it’s devtest set score (+0.05 average BLEU).

$\text{Large}_{\text{Heuristics}}$, unsurprisingly, still outperformed
$\text{Base}_{\text{Heuristics}}$ (20.73 average BLEU, +2.24 improvement) and
the baseline M2M-100 model (14.02 average BLEU, +8.95 improvement) in the
hidden test set. The shared task’s best performing model, DeltaLM+ZCode, still
outperforms all other models in the hidden test set, scoring 33.89 average
BLEU, a 10.92 improvement over our best model.

On the hidden test set, $\text{Large}_{\text{Heuristics}}$ still ranked first
in the id$\rightarrow$jv translation direction, scoring 24.05 BLEU. This
outperforms DeltaLM+ZCode’s 23.79 BLEU (+0.26) and M2M-100’s 15.33 BLEU
(+8.72).

A summary of our model’s performance on the hidden test set, as well as the
baseline and best performing model, can be found on Table 7

| Public | Hidden | Rank
---|---|---|---
| Test | Test |
M2M-100 615M | 15.46 | 14.02 | 8
DeltaLM+ZCode | 33.94 | 33.89 | 1
$\text{Base}_{\text{Heuristics}}$ | 20.78 | 20.73 | -
$\text{Large}_{\text{Heuristics}}$ | 22.92 | 22.97 | 6
Table 7: Average BLEU scores on the contest’s hidden test set. The
$\text{Base}_{\text{Heuristics}}$ model is unranked as it was not submitted as
our final model.

##  5 Conclusion

In this paper, we described the translation systems submitted by the joint
Samsung Research Philippines-Konvergen AI team for the WMT’21 Large Scale
Multilingual Translation Small Track 2 shared task. We outline the filtering
heuristics that we took to preprocess our data. We then train two models with
a bitext preprocessed using our filtering heuristics, with our best model
reaching an average BLEU score of 22.92 on the devtest set, and outperforming
the baseline model by 7.46 BLEU points. In addition, we rank sixth in the
contest leaderboard overall, scoring 22.97 BLEU on the hidden test set.

We also reached first place for the id$\rightarrow$jv translation direction,
beating all other more complex models, despite only using a standard
transformer without any special augmentations and training tricks. This
provides empirical evidence that data quality and preprocessing decisions
weigh just as much, if not even more, than cutting edge model architectures
and training techniques do.

## References

  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: the bible in 100 languages.  _Language resources and evaluation_ , 49(2):375–395.
  * El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020\.  CCAligned: A massive collection of cross-lingual web-document pairs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)_ , pages 5960–5969, Online. Association for Computational Linguistics.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021.  Beyond english-centric multilingual machine translation.  _Journal of Machine Learning Research_ , 22(107):1–48.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings.
  * Goyal et al. (2021) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _arXiv preprint arXiv:2106.03193_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Lopez et al. (2020) Luis Enrico Lopez, Diane Kathryn Cruz, Jan Christian Blaise Cruz, and Charibeth Cheng. 2020.  Simplifying paragraph-level question generation via transformer language models.  _arXiv preprint arXiv:2005.01107_.
  * Ma et al. (2021) Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021.  Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders.  _arXiv preprint arXiv:2106.13736_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Schwenk et al. (2019) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019.  Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia.  _arXiv preprint arXiv:1907.05791_.
  * Sennrich et al. (2015a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_.
  * Sennrich et al. (2015b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.