# TransForensics: Image Forgery Localization with Dense Self-Attention

Jing Hao111Equal contribution. Zhixin Zhang111Equal contribution. Shicai Yang
Di Xie Shiliang Pu222Corresponding author.
Hikvision Research Institute
{haojing, yangshicai, xiedi<EMAIL_ADDRESS>
###### Abstract

Nowadays advanced image editing tools and technical skills produce tampered
images more realistically, which can easily evade image forensic systems and
make authenticity verification of images more difficult. To tackle this
challenging problem, we introduce TransForensics, a novel image forgery
localization method inspired by Transformers. The two major components in our
framework are dense self-attention encoders and dense correction modules. The
former is to model global context and all pairwise interactions between local
patches at different scales, while the latter is used for improving the
transparency of the hidden layers and correcting the outputs from different
branches. Compared to previous traditional and deep learning methods,
TransForensics not only can capture discriminative representations and obtain
high-quality mask predictions but is also not limited by tampering types and
patch sequence orders. By conducting experiments on main benchmarks, we show
that TransForensics outperforms the state-of-the-art methods by a large
margin.

##  1 Introduction

Image is an important medium for information transmission. Recently, tampered
images generated by image editing techniques are commonly confused to be real
ones, and are increasingly used in fake news creation, academic fraud, and
criminal offenses. When tampering occurs in a digital image, we usually expect
that the tampered regions can be found through image forensic analyses.
However, capturing discriminative features of tampered regions with multiple
forgery types (_e.g_. splicing, copy-move, removal) is still a challenge and
often requires exploiting the characteristics of different tampering
artifacts.

Figure 1: Examples from three common image forgery datasets. Four authentic
images (top) with their corresponding tampered images (medium) and ground-
truth masks (bottom).

Unlike semantic object segmentation methods [33, 43] that do predictions of
all meaningful object regions, image forensics makes predictions of tampering
locations only. The former focuses on analyzing the content of different
regions to understand visual concepts, while the latter needs to generalize
some other artifacts (_e.g_. inconsistent local noise variances) created by
different manipulation techniques. As shown in Fig.1, the well-manipulated
images are usually realistic, where the content of fake and genuine regions is
likely to be similar. If we directly use semantic segmentation network for
image forensics, the network would localize both original and manipulated
regions, while the original ones are the wrong predictions for image
forensics. The work of Bappy _et al_. [3] also have demonstrated that semantic
segmentation approaches do not perform well for image manipulations.

The key to image forensics is characterizing different tempering artifacts
that are often hidden in tiny details of the images. Previous methods mainly
employ traditional hand-crafted features, such as error level analysis (ELA)
[20], discrete cosine transform (DCT) [5], and steganalysis rich model (SRM)
[15], to learn local inconsistencies from invisible traces, but they usually
apply only to a specific manipulation type. In fact, the boundary formation of
tampered (smoother) and authentic regions (sharper) within an image is
different [3, 4]. With the success of deep learning approaches, recent works
focus on checking feature consistency or learning boundary discrepancy via
convolutional neural networks (CNNs) [10, 28, 35, 42] or recurrent neural
networks (RNNs) [3, 4], which allow to capture tampering features and perform
better than traditional methods.

However, the major shortcoming of deep learning methods is that they heavily
depend on hand-designed patch sequence orders and manipulation types.
Specifically, RNNs based methods split an image into a series of patches and
use a long-short term memory (LSTM) network to learn the correlations between
them. These networks can receive the sequential inputs, but cannot retain the
spatial location information. In contrast, the methods combining hand-crafted
features with deep features [2, 38, 41, 42] achieve the state-of-the-art
(SOTA) performance, but they usually assume that tampering type is known
beforehand. Taking these facts into consideration, here we show how a spatial
attention network can be used within an image forgery localization framework
to model all pairwise interactions between patches (including rich statistical
features) of an image, yet maintain the global structure and alleviate
ordering techniques and manipulation types limitation.

Framework overview In this paper, the goal of our system is to predict binary
masks for image forgery localization. Firstly, we use a fully convolutional
network (FCN) as backbone for feature extraction. Then, self-attention
encoders are used to model rich interactions between points in feature maps at
different scales. For improving the performance, dense correction modules are
used in our network, which helps to learn more discriminative representations
from the early layers and performs results correction.

Main contributions In this work, the main contributions are as follows. First,
we propose a novel image forgery localization method, called TransForensics.
To the best of our knowledge, it is the first attempt in image forensics to
model all pairwise relations, yet maintain the spatial structure between
patches with the self-attention mechanism. Second, we introduce a dense
correction architecture, which adds the direct supervision for the hidden
layers, and corrects the outputs from different branches by multiplication.
Experiments show that our method outperforms the SOTA methods by a large
margin.

Structure of the paper The paper is organized as follows. We first review
related work in Section 2. Then, Section 3 introduces the proposed dense
attention network for image forgery localization in detail. Section 4 shows
the experimental datasets, details, results and analysis. Finally, Section 5
gives the conclusion of this paper.

##  2 Related work

Prior work on which our work build contains several domains: image forensics,
self-attention mechanism, and deep supervision. Image forensics focuses on
detecting tampering artifacts, and research on this domain contains various
traditional and deep learning approaches for image forgery classification,
detection, and localization. The self-attention mechanism is the core
component of Transformers, which are widely used in nature language processing
(NLP) and computer vision (CV). Deep supervision attempts to enforce direct
supervision for the hidden layers, where the learned features are sensible and
discriminative. In this section, we briefly review prior work.

Figure 2: Architecture of image forgery localization network. The whole image
is the input signal. First, a FCN backbone is applied to extract
discriminative features. Then, the features from four blocks ($C_{2}$,
$C_{3}$, $C_{4}$ and $C_{5}$) combining with positional encodings are input
into the self-attention encoders separately, which captures rich interactions
between ‘patches’ in the input image. Finally, a feature fusion strategy by
multiplication corrects the mask predictions. In this work, we do not split
the whole image into a series of patches, and the points in feature maps is
equivalent to the invisible patches in the input image (see Fig. 3).

Image forensics The development of image editing techniques makes tampered
images widely available and more realistic. The most common tampering
techniques are splicing, copy-move, and removal. Splicing means copying
regions from an image to another image. Copy-move copies and pastes regions
within the same image. Removal means removing regions from the current image.
Image forensics aims at detecting these tampering artifacts, and involves
binary (real or fake) classification and tampered regions detection or
localization tasks. At first, many studies in this filed are traditional
methods based on hand-crafted features, such as local noise analysis [9, 15,
26, 31], CFA artifacts [14], illumination variance analysis [12, 32], and
double JPEG compression [5, 7, 20]. With the revolutionary advance of deep
learning, some methods try to bring deep neural networks into this realm
(_e.g_. RNNs [3, 4], CNNs [10, 17, 22, 35] and GANs [18]). There are also many
papers that combine hand-crafted features and deep features for image
forensics [1, 2, 36, 38, 41, 42].

Local noise variances estimation is used for image splicing detection [31].
This is because different regions within an authentic image containing
intrinsic noise have similar noise variances, and image splicing can be
exposed with inconsistent local noise variances. Similarly, SRM [15, 41, 42]
uses local noise residuals to capture the inconsistency between tampered and
authentic regions. For example, Fridrich _et al_. [15] propose steganalyzers
to construct rich models of the noise component to capture numerous
quantitative relationships between pixels in an image, and Zhou _et al_. [41]
combine SRM features with RGB features by a two-stream Faster R-CNN to perform
manipulation detection. Furthermore, Bammey _et al_. [2] design a CNN
structure based on demosaicing algorithms to point out local mosaic
inconsistencies. Amerini _et al_. [1] combines a spatial domain CNN with a
frequency domain CNN for splicing forgery detection, which is inspired by the
fact that the artifacts of single and double JPEG compression are different.
Bappy _et al_. [3] propose to exploit the interdependency between patches,
which is efficient for various types of manipulation detection, and then they
present a hybrid CNN-LSTM network [4] utilizing resampling features to improve
the detection performance. The studies combining hand-crafted features with
deep features are: CNN and CFA [2], CNN and steganalysis [38, 41, 42], CNN and
double JPEG compression [1, 36].

The commonality among all traditional algorithms discussed above is that they
usually apply only to a certain manipulation type, and the disadvantage of
deep learning methods is that the performance heavily depends on patch
ordering techniques. Motivated by these works, here we present a novel spatial
attention network with the self-attention mechanism for modeling rich
interactions between patches at different scales.

Self-attention mechanism The self-attention mechanism is the core component of
Transformers [34], which has been successfully used in CV [19]. Here, self-
attention can capture ‘long-term’ dependencies between set elements (_e.g_.
pixels, image patches or video frames) to aggregate global information of the
input signal. A recent framework, called DETR [8], views object detection as a
direct set prediction problem and uses Transformers with parallel decoding to
produce unique predictions. Semantic segmentation is a dense prediction task,
where Transformers can be used to model relations between pixels. For example,
Ye _et al_. [39] propose a cross-modal self-attention to learn long-range
dependencies between linguistic and visual features, and Zheng _et al_. [40]
deploy a pure Transformer to encode an image as a sequence of patches.
However, the usual solution for image forensics to learn the correlations
between patches is using LSTM cells, in which the existing orderings (_e.g_.
horizontal, vertical or Hilbert curve [4]) cannot correlate well between
patches (_i.e_. neighbor each other being separated). In contrast, self-
attention mechanisms can aggregate information from the entire input image,
and their global computations make them more suitable than RNNs in this
domain. There are also few studies which make attempts to bring attention
mechanism to image forensics [17, 18]. In [17], a spatial pyramid attention
network (with different dilation distances) is designed, where RGB, Bayer [6]
and SRM [15, 41] features are extracted. [18] using a dual-order (channel)
attention module [11] in GAN, which applies only to a specific manipulation
type. In this work, we try to utilize self-attention to model relations
between ‘patches’ only based on RGB features for exploiting rich statistical
features of different tampering artifacts.

Deep supervision Deep supervision aims at enforcing direct supervision for the
hidden layers. Deeply-supervised net (DSN) [21] makes the learning process of
hidden layers transparent, which boosts the classification performance and
effectively avoids the exploding and vanishing gradients. Based on this, Zhou
_et al_. [43] present UNet++ for medical image segmentation. These
segmentation networks share a key similarity: using skip connections to
combine semantic feature maps from the decoder with shallow feature maps from
the encoder, which helps to improve the segmentation performance. Inspired by
this, we propose a dense correction architecture for capturing both coarse-
grained (high-level, semantic) and fine-grained (low-level, statistical)
predictions and correcting details from different branches by multiplication
(see 3.3). The architecture enables network pruning and producing better
results.

##  3 Method

Image forensics, aiming at capturing tampered regions, is different from
semantic object segmentation. For example, copy-move copies one object region
to another region within the same image. In this case, the main goal of image
forensics is to localize the pasted object region whereas semantic
segmentation needs to segment all object regions including the original one
and the manipulated one. Considering that tampering techniques can create
artifacts (_e.g_. local noise variance, boundary discrepancy), we need to
design a network to capture discriminative features for finding suspicious
regions in a potentially forged image.

Previous works just used hand-crafted ordering techniques to model patches
relations, which cannot retain spatial information. To solve this issue,
inspired by Transformer, we propose to use self-attention to learn invisible
tempering artifacts hidden in tiny details of an image, which is the first
attempt in image forensics. Moreover, we propose a dense correction
architecture to re-correct the output, yielding excellent performance
improvement.

###  3.1 Self-attention for interaction modeling

The self-attention mechanism can be used to model rich interactions between
pixels or patches in an image, which provides more comprehensive and useful
information for dense visual tasks. In this work, we use self-attention
encoders in image forensics, which is motivated by the following observations:
first, the tampering artifacts produced by different manipulation types are
different, and they are commonly hidden in the details of the image; second,
modeling patches relations with hand-designed patch sequence orders cannot
keep spatial information of patches. If we ‘split’ the image into $H\times W$
patches and then fed them into the self-attention encoder, all pairwise
relations between patches can be extracted. This is the theoretical foundation
for using self-attention encoders in tampering localization.

Each point in a feature map is equivalent to the corresponding patch in the
input image, which is called respective field in deep learning (see Fig. 3).
So the discriminative features between patches can be extracted by modeling
the relations between points in feature maps. In this paper, we do not split
the whole image into a series of patches. We use ResNet50 [16] as backbone
(including five stages) for feature extraction, and then we feed the outputs
of the last four stages into self-attention encoders, each responsible for
learning the patch relations at a different scale.

Here, we use a standard transformer encoder architecture to learn attention
maps, and the details are described below. First, a $1\times 1$ convolution
layer is used to reduce the channel dimension of stages’ output from $C$ to
$d$, where $C\in[256,512,1024,2048]$ and $d=256$. Second, in order to maintain
the spatial location of patches, we supplement the features with a sine
positional encoding [34] before passing them into transformer encoders. Third,
a transformer encoder has $6$ encoder layers, and each one consists of a
multi-head self-attention module and a feed forward network (FFN). The
dimension of feed forward is $2048$ and the dropout during training is $0.1$.
In this way, we can capture the difference at the boundary location between
tampered and authentic regions.

Figure 3: The correspondence between the feature map and the input image. The
interactions between points in a feature map are equivalent to the
relationships between patches in a digital image. Figure 4: Network pruning.
In the training phrase, comparing the results between
${\bf{\hat{y}}}_{i}(i=2,...,5)$ and choosing the optimal, where the final
result is produced in the test phrase.

###  3.2 Deep supervision for network pruning

In general, highly discriminative features will produce powerful performance.
If the outputs of the hidden layers can be directly used for the final
classification, the network will obtain both semantic, coarse-grained and low-
level, fine-grained predictions, which contributes to the final performance.
In this work, we consider adding deep supervision in the tampering
localization system.

As shown in Fig. 2, the network has four branches, and each output is used to
calculate the localization loss separately. The advantages of this
architecture are: providing more expressive features (semantic and shallow)
for feature fusion (see Fig. 5) and obtaining more efficient architecture by
network pruning (see Fig. 4). Deep supervision in this work enables the model
to choose a suitable mode from all localization branches
${\bf{\hat{y}}}_{i}(i=2,...,5)$, and the choice determines the extent of
network pruning and speed gain (see Tab. 4 and Tab. 5).

###  3.3 Feature fusion for prediction correction

The fully convolutional network (FCN) was introduced by Shelhamer _et al_.
[33] for semantic segmentation, where the upsampled features are summed with
the features skipped from the encoder. Experiments show that it is effective
in helping recover the full spatial resolution at the model’s output. In this
work, we bring a similar but new feature fusion strategy to the system.
Instead of adding the features, we opt to use the multiplication operation
(see Fig. 2 and Fig. 5), and results show that it is a better choice for the
tampering localization task (see Tab. 6).

We fuse the upsampled output from current block with the output from adjacent
previous block, where they have the same size. As shown in Fig. 5, $B$
represents the output of the high-level block, $A$ represents the output of
the adjacent low-level block, and $C$ is the result of fusing $A$ and $B$ by
multiplication. Specifically, in feature fusion modules, a $1\times 1$
convolution is used to change the dimension of features from different
branches, and the upsampling operation followed by a sigmoid function with
threshold $0.5$ produces the fusing weights. The final mask prediction is
computed by a $3\times 3$ convolution with stride $1$ and padding $1$ after
feature fusion by multiplication.

###  3.4 Prediction loss

In this work, we use DICE loss [29] and Focal loss [24] to supervise each mask
prediction:

$L_{dice}({\bf{y,\hat{y}}})=1-\frac{2\cdot\sum_{k}(y_{k}\cdot\hat{y}_{k})}{\sum_{k}(y_{k}+\hat{y}_{k})}$
(1)

$L_{focal}(p_{t})=-\alpha_{t}(1-p_{t})^{\gamma}\log(p_{t})$ (2)

where ${\bf{y}}$ and ${\bf{\hat{y}}}$ are ground-truth (GT) and predicted
mask, and $k$ denotes the point of the mask. Let $\\{\pm 1\\}$ be the GT class
and $p\in[0,1]$ be the probability ($p_{t}=p$ for class $1$ and $p_{t}=1-p$
for class $-1$). A weighting factor $\alpha\in[0,1]$ is introduced for
addressing class imbalance ($\alpha_{t}=\alpha$ for class $1$ and
$\alpha_{t}=1-\alpha$ for class $-1$). $\gamma$ is the tunable focusing
parameter. Like [24], we set $\alpha=0.25$ and $\gamma=2$. We compute the
joint loss of all branches during training:

$Loss=\sum_{i}\lambda_{i}(L_{dice}({\bf{y}}_{i},{\bf{\hat{y}}}_{i})+L_{focal}(p_{t}))$
(3)

here, the loss for the branch $i$ is
$L_{dice}({\bf{y}}_{i},{\bf{\hat{y}}}_{i})+L_{focal}(p_{t})$ and the
corresponding ratio is $\lambda_{i}$. There are two ways to calculate the
branch loss: upsampling predicted mask or downsampling GT mask.
${\bf{\hat{y}}}_{i}$ can be used to compute the branch loss directly in which
the corresponding ${\bf{y}}_{i}$ can be obtained by downsampling the original
GT mask. We can also employ upsampled mask prediction, in which
${\bf{\hat{y}}}_{i}$ is upsampled to have the same size as the original GT
mask. Note that these two computation types are different. In upsampling, the
higher the layer, the more noise may be introduced into the network via
‘nearest’ interpolating, and in this way, the loss between GT and predicted
mask needs more attention, whereas downsampling does the opposite. For
producing more precise predictions, we choose different coefficients for four
branches to control the correction amplitude during training. Thus, the ratio
set is written as follows.

${Upsampling}:\ \lambda_{i}<\lambda_{j},i<j$ (4)

${Downsampling}:\ \lambda_{i}>\lambda_{j},i<j$ (5)

here, we denote the low-level branch as $i$ and the high-level branch as $j$.
$\lambda_{i}$ and $\lambda_{j}$ are the corresponding ratio of the branch $i$
and $j$ for calculating the joint loss.

##  4 Experiments

Figure 5: Feature fusion strategy. $A$ and $B$ are the outputs of two adjacent
blocks, and $C$ is the result of feature fusion by multiplication. $B$ is the
semantic prediction and can help to correct $A$.

###  4.1 Datasets

In this work, we choose three common datasets in image forensics: CASIA [13],
COVERAGE [37] and IMD2020 [30]. The details are as follows.

CASIA focuses on splicing and copy-move. It provides binary GT masks of
tampered regions. The dataset includes CASIA v1.0 about 921 tampered images
and CASIA v2.0 about 5123 tampered images.

COVERAGE is a relatively small dataset designed for copy-move. It provides 100
manipulated images and corresponding GT masks.

IMD2020 is a ‘real-life’ manipulated dataset made by unknown people and
collected from the Internet. The images having obvious traces of digital
manipulation are discarded and binary masks localizing tampered regions are
created manually. It includes 2010 examples.

In data preparation, we split the entire dataset into three subsets and the
ratio of training, validation and testing is 8:1:1. Note that these subsets
are chosen randomly.

###  4.2 Details

The proposed network contains three main components: a FCN backbone for
feature extraction, dense self-attention encoders for relations modeling
between patches at different scales, and dense correction modules for further
performance optimization. Specifically, we use ResNet-50 as backbone. We train
the network with Adam setting the initial learning rate to 1e-4. We resize the
input images to $512\times 512$, and use random horizontal flipping as the
only data augmentation method. To train the model, we use Pytorch $1.6.0$ to
define the localization network and utilize multi-GPU setting. We set the
batch size to 2 and train the model on two NVIDIA Tesla V100 GPUs over 50
epochs for different datasets. We choose the loss weights within
$[0.1,0.2,0.3,0.4]$ during training.

Note that some previous works chose to finetune their models trained on other
bigger image forgery datasets to obtain the performance on COVERAGE, because
the number of COVERAGE is small. In our experiments, we do not perform such
finetuning on COVERAGE, and the results are still comparable to the SOTA
methods (see the COVER column in Tab. 1 and Tab. 2).

Evaluation We evaluate our model at pixel level with the benchmark metrics:
$F_{1}$ score and area under curve (AUC). The higher value indicates that the
performance is better.

Baseline models In this paper, we compare our work with various baseline
methods. Some methods are described in [42], such as ELA [20], NOI1 [27], CFA1
[14], J-LSTM [3], RGB-N [42], and other methods such as BLK [23], ADQ1 [25],
ManTra-Net [38], LSTM-EnDec [4] and SPAN [17] are described below.

  * •

BLK: The work focuses on extracting block artifact grids caused by the
blocking processing during JPEG compression, and then detecting them with a
marking procedure.

  * •

ADQ1: The method aims to detect tampered images by examining the double
quantization effect hidden among the DCT coefficients in JPEG images. It is
insensitive to different kinds of forgery methods.

  * •

ManTra-Net: A unified deep neural architecture performing both detection and
localization can handle many known forgery types.

  * •

LSTM-EnDec: A manipulation localization architecture utilizing resampling
features, LSTM cells and encoder-decoder modules is performed to segment out
manipulated regions from non-manipulated ones.

  * •

SPAN: The paper presents a spatial pyramid attention network, where Bayer and
SRM features are extracted.

###  4.3 Results

In this subsection, we compare results of various works on three standard
datasets quantitatively and qualitatively.

Quantitative analysis We compare our work with other various SOTA models on
the datasets mentioned above with the benchmark metric AUC and $F_{1}$. From
Tab. 1 and Tab. 2, we can see that our proposed network outperforms baseline
models by a large margin. The results of our method through upsampling and
downsampling are comparable in AUC metric and the former is better in $F_{1}$
score. Note that IMD2020 was released in 2020, and we struggled to find the
published literatures reporting $F_{1}$ score on this dataset, but in vain
until we submitted the paper. The $F_{1}$ score of our method on IMD2020 is
0.545. About cross-dataset results: training our network with IMD2020, the AUC
performance on CASIA and COVER are 0.652 and 0.758, respectively. It proves
the generalizability of our method.

Method | CASIA | COVER | IMD2020
---|---|---|---
ELA [20] | 0.613 | 0.583 | -
NOI1 [27] | 0.612 | 0.587 | -
CFA1 [14] | 0.522 | 0.485 | 0.586
J-LSTM [3] | - | 0.614 | 0.487
RGB-N [42] | 0.795 | 0.817 | -
BLK [23] | - | - | 0.596
ADQ1 [25] | - | - | 0.579
ManTra-Net [38] | 0.817 | 0.819 | 0.748
LSTM-EnDec [4] | - | 0.712 | -
Ours (downsample) | 0.850 |  0.884* | 0.847
Ours (upsample) | 0.837 | 0.883* | 0.848
Table 1: AUC performance comparison against different works on image forgery localization. ‘*’ denotes that our experiments on COVERAGE do not perform finetuning, which is different from the other methods in table. ‘-’ denotes that the result is not available in the literature. Method | CASIA | COVER
---|---|---
ELA [20] | 0.214 | 0.222
NOI1 [27] | 0.263 | 0.269
CFA1 [14] | 0.207 | 0.190
RGB-N [42] | 0.408 | 0.437
SPAN [17] | 0.382 | 0.558
Ours (downsample) | 0.479 | 0.648*
Ours (upsample) | 0.627 |  0.674*
Table 2: $F_{1}$ score performance comparison against different works. ‘*’ and
‘-’ have the same meaning as Tab. 1. Figure 6: Examples of mask predictions
using the proposed dense attention network. Images are taken from three common
datasets mentioned above. From left to right: (a) authentic images, (b)
tampered images, (c) ground-truth masks, and (d) mask predictions.

Qualitative analysis After training, our model can generate high quality mask
predictions that depict tampering locations. Here, we provide some qualitative
examples (see Fig. 6), which are taken from the datasets mentioned above.
These examples are generated by copy-move, splicing, and ‘real-life’ tampering
from the Internet. As we can see, our method can find tampering regions with
different types of manipulation. Note that the predictions are produced
directly by our model without any post-processing.

FCN | Self-attention | Positional encoding | Dense correction | CASIA | COVER | IMD2020
---|---|---|---|---|---|---
$\surd$ |  |  |  | 0.669 | 0.715 | 0.735
$\surd$ | $\surd$ |  |  | 0.655 | 0.674 | 0.725
$\surd$ |  | $\surd$ |  | 0.650 | 0.707 | 0.729
$\surd$ | $\surd$ | $\surd$ |  | 0.809 | 0.862 | 0.815
$\surd$ | $\surd$ | $\surd$ | $\surd$ | 0.837 | 0.883 | 0.848
Table 3: AUC performance in ablations.

###  4.4 Ablations

In this subsection, we conduct ablation experiments to study how the
components of our proposed architecture influence the localization
performance. For the ablation analysis, we use a FCN (_i.e_. ResNet50) as
backbone and evaluate the importance of the self-attention, sine spatial
positional encoding, and dense correction by adding the corresponding module
respectively. We also provide a detailed study to show how the choice of
feature fusion and network pruning affect the final performance.

Architecture analysis The self-attention mechanism is the key component for
modeling rich interactions between set elements, where the positional encoding
is very important. As shown in Tab. 3, performance is significantly improved
only when self-attention and positional encoding are used at the same time.
Dense correction improves the transparency of the hidden layers, which
minimizes the loss error more effectively, and re-corrects the mask prediction
based on semantic dependencies between different attention maps. From the last
line in Tab. 3, we can observe that dense correction is helpful for the final
performance.

Network pruning Dense correction makes network pruning possible. As shown in
see Fig. 4, we can compare the results between ${\bf{\hat{y}}}_{i}(i=2,...,5)$
in the training phase and choosing the optimal, where the final result is
produced in the test phrase. Tab. 4 shows the AUC performance of different
branches of the proposed architecture, and Tab. 5 provides the information of
time spent and GPU memory occupancy. As can be seen in Tab. 4, the low-level
features contribute more to the final localization performance than deep ones,
which means that our model can learn invisible traces that are often hidden in
tiny details of the images. Combining Tab. 4 and Tab. 5, we can see that
$C_{3}$ branch (_i.e_. ${\bf{\hat{y}}}_{3}$) outperforms the other branches
both in performance and time-consuming relatively.

Method | CASIA | COVER | IMD2020
---|---|---|---
Ours ($C_{2}$) | 0.837 | 0.883 | 0.848
Ours ($C_{3}$) | 0.835 | 0.887 | 0.847
Ours ($C_{4}$) | 0.835 | 0.880 | 0.837
Ours ($C_{5}$) | 0.826 | 0.877 | 0.827
Table 4: AUC performance comparison using different branches as final output (see Fig. 4). Method | GPU memory (M) | Inference time (s)
---|---|---
Ours ($C_{2}$) | 6687 | 0.11
Ours ($C_{3}$) | 774 | 0.09
Ours ($C_{4}$) | 368 | 0.08
Ours ($C_{5}$) | 353 | 0.07
Table 5: Comparisons of results of different branches as final output in GPU
memory and inference time.

During training, with the help of dense correction, the features with low-
spatial resolution can get feedback including edges from low-level high-
resolution features. This is the most significant difference from networks
with single direction connection. Thus, although the prediction of $C_{3}$
branch is block-wise, it does not lack refinement on edges.

Type of feature fusion As described above, feature fusion has two types:
multiplication and addition. In our experiments, we try both addition and
multiplication (see Tab. 6), and experimental results demonstrate that the
multiplication type is more suitable for the tampering localization task,
which is consistent with expectations.

Method | CASIA | COVER | IMD2020
---|---|---|---
Ours (Mul) | 0.837 | 0.883 | 0.848
Ours (Add) | 0.739 | 0.857 | 0.827
Table 6: AUC comparison using different types for feature fusion.

##  5 Conclusion

We present TransForensics, which uses dense self-attention encoders to model
global context and all pairwise interactions between patches at different
scales. It is the first work that introduces self-attention mechanisms of
transformers to localizing tampered regions. Further, dense correction modules
re-correct mask predictions by multiplication for nicer results. We
demonstrate the system’s ability to detect tampering artifacts for diverse
realistic tampered images and to achieve a balance between performance and
time-consuming. Experiments show that the proposed system can provide a
powerful model for image forgery. Our method can be served as a solid but
simple-to-implement baseline for image forensics, and can also be employed as
a defense against data poisoning attacks to protect our learning system. In
future work, the dense self-attention architecture would be a novel approach
in other tasks, such as object detection and semantic segmentation.

## References

  * [1] Irene Amerini, Tiberio Uricchio, Lamberto Ballan, and Roberto Caldelli.  Localization of jpeg double compression through multi-domain convolutional neural networks.  In 2017 IEEE Conference on computer vision and pattern recognition workshops (CVPRW), pages 1865–1871. IEEE, 2017.
  * [2] Quentin Bammey, Rafael Grompone von Gioi, and Jean-Michel Morel.  An adaptive neural network for unsupervised mosaic consistency analysis in image forensics.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14194–14204, 2020.
  * [3] Jawadul H Bappy, Amit K Roy-Chowdhury, Jason Bunk, Lakshmanan Nataraj, and BS Manjunath.  Exploiting spatial structure for localizing manipulated image regions.  In Proceedings of the IEEE international conference on computer vision, pages 4970–4979, 2017.
  * [4] Jawadul H Bappy, Cody Simons, Lakshmanan Nataraj, BS Manjunath, and Amit K Roy-Chowdhury.  Hybrid lstm and encoder–decoder architecture for detection of image forgeries.  IEEE Transactions on Image Processing, 28(7):3286–3300, 2019.
  * [5] Mauro Barni, Luca Bondi, Nicolò Bonettini, Paolo Bestagini, Andrea Costanzo, Marco Maggini, Benedetta Tondi, and Stefano Tubaro.  Aligned and non-aligned double jpeg detection using convolutional neural networks.  Journal of Visual Communication and Image Representation, 49:153–163, 2017.
  * [6] Belhassen Bayar and Matthew C Stamm.  A deep learning approach to universal image manipulation detection using a new convolutional layer.  In Proceedings of the 4th ACM workshop on information hiding and multimedia security, pages 5–10, 2016.
  * [7] Tiziano Bianchi, Alessia De Rosa, and Alessandro Piva.  Improved dct coefficient analysis for forgery localization in jpeg images.  In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2444–2447. IEEE, 2011.
  * [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In 2020 IEEE European Conference on Computer Vision (ECCV). IEEE, 2020.
  * [9] Davide Cozzolino, Diego Gragnaniello, and Luisa Verdoliva.  Image forgery localization through the fusion of camera-based, feature-based and pixel-based techniques.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 5302–5306. IEEE, 2014.
  * [10] Davide Cozzolino and Luisa Verdoliva.  Noiseprint: a cnn-based camera model fingerprint.  arXiv preprint arXiv:1808.08396, 2018.
  * [11] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11065–11074, 2019.
  * [12] Tiago José De Carvalho, Christian Riess, Elli Angelopoulou, Helio Pedrini, and Anderson de Rezende Rocha.  Exposing digital image forgeries by illumination color classification.  IEEE Transactions on Information Forensics and Security, 8(7):1182–1194, 2013.
  * [13] Jing Dong, Wei Wang, and Tieniu Tan.  Casia image tampering detection evaluation database.  In 2013 IEEE China Summit and International Conference on Signal and Information Processing, pages 422–426. IEEE, 2013.
  * [14] Pasquale Ferrara, Tiziano Bianchi, Alessia De Rosa, and Alessandro Piva.  Image forgery localization via fine-grained analysis of cfa artifacts.  IEEE Transactions on Information Forensics and Security, 7(5):1566–1577, 2012.
  * [15] Jessica Fridrich and Jan Kodovsky.  Rich models for steganalysis of digital images.  IEEE Transactions on Information Forensics and Security, 7(3):868–882, 2012.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [17] Xuefeng Hu and Zhihan Zhang.  Span: Spatial pyramid attention network for image manipulation localization.  In ECCV, 2020.
  * [18] Ashraful Islam, Chengjiang Long, Arslan Basharat, and Anthony Hoogs.  Doa-gan: Dual-order attentive generative adversarial network for image copy-move forgery detection and localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4676–4685, 2020.
  * [19] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  arXiv preprint arXiv:2101.01169, 2021.
  * [20] Neal Krawetz and Hacker Factor Solutions.  A picture’s worth.  Hacker Factor Solutions, 6(2):2, 2007.
  * [21] Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu.  Deeply-supervised nets.  In Artificial intelligence and statistics, pages 562–570. PMLR, 2015.
  * [22] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo.  Face x-ray for more general face forgery detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5001–5010, 2020.
  * [23] Weihai Li, Yuan Yuan, and Nenghai Yu.  Passive detection of doctored jpeg image via block artifact grid extraction.  Signal Processing, 89(9):1821–1829, 2009.
  * [24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [25] Zhouchen Lin, Junfeng He, Xiaoou Tang, and Chi-Keung Tang.  Fast, automatic and fine-grained tampered jpeg image detection via dct coefficient analysis.  Pattern Recognition, 42(11):2492–2501, 2009.
  * [26] Siwei Lyu, Xunyu Pan, and Xing Zhang.  Exposing region splicing forgeries with blind local noise estimation.  International journal of computer vision, 110(2):202–221, 2014\.
  * [27] Babak Mahdian and Stanislav Saic.  Using noise inconsistencies for blind image forensics.  Image and Vision Computing, 27(10):1497–1503, 2009.
  * [28] Aniruddha Mazumdar, Jaya Singh, Yosha Singh Tomar, and Prabin Kumar Bora.  Universal image manipulation detection using deep siamese convolutional neural network.  arXiv preprint arXiv:1808.06323, 2018.
  * [29] Fausto Milletari, Nassir Navab, and Seyed Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 Fourth International Conference on 3D Vision (3DV), 2016\.
  * [30] Adam Novozamsky, Babak Mahdian, and Stanislav Saic.  Imd2020: A large-scale annotated dataset tailored for detecting manipulated images.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pages 71–80, 2020.
  * [31] Xunyu Pan, Xing Zhang, and Siwei Lyu.  Exposing image splicing with inconsistent local noise variances.  In 2012 IEEE International Conference on Computational Photography (ICCP), pages 1–10. IEEE, 2012.
  * [32] Christian Riess and Elli Angelopoulou.  Scene illumination as an indicator of image manipulation.  In International Workshop on Information Hiding, pages 66–80. Springer, 2010.
  * [33] Evan Shelhamer, Jonathan Long, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):640–651, April 2017.
  * [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [35] Sebastiano Verde, Paolo Bestagini, Simone Milani, Giancarlo Calvagno, and Stefano Tubaro.  Focal: A forgery localization framework based on video coding self-consistency.  arXiv preprint arXiv:2008.10454, 2020.
  * [36] Qing Wang and Rong Zhang.  Double jpeg compression forensics based on a convolutional neural network.  EURASIP Journal on Information Security, 2016(1):1–12, 2016.
  * [37] Bihan Wen, Ye Zhu, Ramanathan Subramanian, Tian-Tsong Ng, Xuanjing Shen, and Stefan Winkler.  Coverage—a novel database for copy-move forgery detection.  In 2016 IEEE international conference on image processing (ICIP), pages 161–165. IEEE, 2016.
  * [38] Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan.  Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9543–9552, 2019.
  * [39] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang.  Cross-modal self-attention network for referring image segmentation.  In 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 10502–10511, 2019.
  * [40] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  arXiv preprint arXiv:2012.15840, 2020.
  * [41] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis.  Two-stream neural networks for tampered face detection.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1831–1839. IEEE, 2017.
  * [42] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis.  Learning rich features for image manipulation detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1053–1061, 2018.
  * [43] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  CoRR, abs/1807.10165, 2018.


# You Only Recognize Once: Towards Fast Video Text Spotting

Zhanzhan Cheng12∗, Jing Lu2∗, Yi Niu2, Shiliang Pu2, Fei Wu1+, Shuigeng Zhou3
1Zhejiang University, Hangzhou, China
chengzhanzhan<EMAIL_ADDRESS>2Hikvision Research Institute,
China<EMAIL_ADDRESS>3Fudan University, Shanghai,
China<EMAIL_ADDRESS>
(2019)

###### Abstract.

Video text spotting is still an important research topic due to its various
real-applications. Previous approaches usually fall into the four-staged
pipeline: text detection in individual images, frame-wisely recognizing
localized text regions, tracking text streams and generating final results
with complicated post-processing skills, which might suffer from the huge
computational cost as well as the interferences of low-quality text. In this
paper, we propose a fast and robust video text spotting framework by only
recognizing the localized text one-time instead of frame-wisely recognition.
Specifically, we first obtain text regions in videos with a well-designed
spatial-temporal detector. Then we concentrate on developing a novel text
recommender for selecting the highest-quality text from text streams and only
recognizing the selected ones. Here, the recommender assembles text tracking,
quality scoring and recognition into an end-to-end trainable module, which not
only avoids the interferences from low-quality text but also dramatically
speeds up the video text spotting process. In addition, we collect a larger
scale video text dataset (LSVTD) for promoting the video text spotting
community, which contains 100 text videos from 22 different real-life
scenarios. Extensive experiments on two public benchmarks show that our method
greatly speeds up the recognition process averagely by 71 times compared with
the frame-wise manner, and also achieves the remarkable state-of-the-art.

video text spotting, detection, tracking, quality scoring

*Both authors contributed equally to this research.

+Corresponding author (wufei@cs.zju.edu.cn).

††submissionid: 10††copyright: acmcopyright††journalyear: 2019††conference:
Proceedings of the 27th ACM International Conference on Multimedia; October
21–25, 2019; Nice, France††booktitle: Proceedings of the 27th ACM
International Conference on Multimedia (MM ’19), October 21–25, 2019, Nice,
France††price: 15.00††doi: 10.1145/3343031.3351093††isbn:
978-1-4503-6889-6/19/10††ccs: Computing methodologies Object detection††ccs:
Computing methodologies Object recognition††ccs: Computing methodologies
Tracking

##  1\. Introduction

Video text spotting is still an important research topic due to the large
amount of video text reading applications such as port container number
identification in industrial monitoring and license plate recognition in
intelligent transportation system.

In the past few years, several methods (Merino-Gracia and Mirmehdi, 2014;
Nguyen et al., 2014; Wang et al., 2017) are proposed to read video scene text
with a multi-stage pipeline strategy: that is, firstly localizing text regions
in individual frames, recognizing localized text regions one-by-one, tracking
recognized regions as text streams, and applying post-precessing techniques
for generating the final results. Figure 1.(a) shows such process.

Figure 1.  Comparison of existing and proposed pipeline in video text
spotting. (a) denotes the traditional framework which contains four steps:
text detection in individual frames, frame-wisely recognizing text regions,
tracking recognized text regions into streams and post-processing for
obtaining final results. (b) denotes our proposed framework which only
contains two steps: text detection among consecutive frames and recognizing
the highest-quality text from text streams as the final results.
$Fr\text{:}(.)$ means the frame ID, and the numbers next to their
corresponding boxes in each frame mean the tracked IDs. The white arrows in
(b) represent the temporal relation built between consecutive frames in
detection process. [Best viewed in color]

However, above methods still suffer from two problems: 1) Excessive
computational cost for directly reading text regions one-by-one, which may be
impractical especially when working on front-end devices such as surveillance
video cameras. 2) Much unstable recognition results due to the overwhelming of
_low-quality_ (_eg._ blurring, perspective distortion, rotation and poor
illumination etc.) text. For example, the low-quality text region in ’stream
3’ of Figure 1.(a) leads to the wrong result ’SERLUE’. In practical, it is
unnecessary to recognize each text region in text streams, which will result
in huge time cost and also bring various interferences of low-quality text.

To circumvent these problems, an idea is to select the _highest-quality_
(_eg._ clear and horizontal) text region from each text stream to be
recognized once instead of one-by-one, which needs us design a robust text
quality scorer to assign a quality score for each candidate text region, as
shown in Figure 1.(b). For further speeding up the process of video text
spotting, we also attempt to assemble the text tracking, text quality scoring
and recognition into an end-to-end trainable module (denoted by text
recommender). The text recommender will benefit from the complementarity among
tracking, scoring and recognition, and then improve the text spotting
performance again.

In this paper, we propose a fast and robust video text spotting approach named
as YORO by combining a well-designed spatial-temporal video text detector and
a novel text recommender into a whole framework. Concretely, the spatial-
temporal detector is designed to recall more text by referring to temporal
relationship among different frames. The text recommender assembles the text
tracking for generating text streams, the text quality scoring for assessing
the quality of each text region and the recognition into an end-to-end
trainable network, as shown in Figure 2 (b), in which the quality scores are
learnt in a weakly-supervised manner (detailed in Methodology Section). In
this way, we can ignore low-quality text regions and only reserve highest-
quality text regions for recognition, which not only improves the recognition
performance but also greatly decreases the computational cost.

Last but not least, we also note that the scales of existing video text
benchmarks are very limited. For example, the largest video scene text dataset
‘Text in Videos’ (Karatzas et al., 2015) only contains 49 videos from 7
different scenarios, which may limit the video text recognition research
progress. Therefore, we collect a larger-scale video text dataset (_abbr_.
LSVTD) containing 100 videos from 22 natural scenarios, and hope to help the
research of video text understanding.

Main contributions of this paper are as follows: (1) We design a novel text
recommender for selecting the highest-quality text from text streams and then
only recognizing the selected text regions once, which significantly speeds up
the recognition process, and also improves the video text spotting
performance. (2) We integrate a well-designed spatial-temporal text detector
and a text recommender into an unified two-stage framework YORO for fast end-
to-end video text spotting. (3) In order to promote the progress of video text
spotting, we collect and annotate a larger scale video text dataset, which
contains 100 videos from 22 different real-life situations. (4) Extensive
experiments demonstrate that our method is fast and robust and achieves
impressive performance in video scene text reading.

##  2\. Related work

###  2.1. Text Reading in Single Images

Traditionally, the scene text reading system contains a text detector for
localizing each text region and a text recognizer for generating corresponding
character sequences. For text detection, numerous methods are proposed to
localize regular and irregular (oriented and curved etc.) text regions, which
can be categorized as anchor-based (Hu et al., 2017; Jiang et al., 2017; Liao
et al., 2018; Liu and Jin, 2017; Ma et al., 2018; Shi et al., 2017a) and
direct-regression-based (He et al., 2017; Wang et al., 2018b; Zhou et al.,
2017). For text recognition, the task is now treated as a sequence recognition
problem, in which CTC (Graves et al., 2006)-based (Borisyuk et al., 2018; Shi
et al., 2017b; Wang and Hu, 2017) and attention-based (Bai et al., 2018; Cheng
et al., 2017, 2018; Shi et al., 2016, 2018) methods are designed and have
achieved promising results. Recently, there are several methods (Borisyuk et
al., 2018; He et al., 2018; Li et al., 2017; Liu et al., 2018) attempting to
spot text end-to-end.

In fact, lots of text reading applications actually work in video scenarios,
in which scene text spotting from multiple frames may be more meaningful.

###  2.2. Text Reading in Videos

In recent years, only a few attention has been drawn to spotting video scene
text in contrast to text reading in still images. For more details of text
detection, tracking and recognition in video, the readers can refer to a
comprehensive survey (Yin et al., 2016). In general, reading text from scene
videos can be roughly categorized into three major modules: 1) text detection,
2) text tracking, and 3) text recognition.

Text detection in videos. In early years (before 2012), most of methods focus
on detecting text in each frame with connected component analysis (Yin et al.,
2014) or sliding window (Kim et al., 2003) strategy. However, the performance
of them is limited due to the low representation of handcrafted features.
Though the recent detection techniques (mentioned in _Section_. 2.1) in still
images can help improve feature representation, detecting text in scene videos
is still challenging because of its complicated temporal characteristics (eg.
motion). Therefore, text tracking strategies are introduced for enhancing the
detection performance, which are further divided into two categories (Yin et
al., 2016): spatial-temporal information based methods (Goto and Tanaka, 2009;
Shiratori et al., 2006; Tanaka and Goto, 2007, 2008) for reducing noise and
fusion based methods (Gómez and Karatzas, 2014; Minetto et al., 2011; Zuo et
al., 2015) for improving detection accuracy. Recently, Wang et al. (Wang et
al., 2018a) employed optical flow based method to refine text locations in the
subsequent frames.

Text tracking in videos. The traditional methods such as template matching
(Fragoso et al., 2011; Na and Wen, 2010; Shiratori et al., 2006; Tanaka and
Goto, 2007) and particle filtering were popular. But these methods failed to
solve the re-initialization problem, especially in scene videos. Then the
tracking-by-detection based methods (Nguyen et al., 2014; Petter et al., 2011;
Rong et al., 2014) were developed to estimate the tracking trajectories and
solve this problem.

Recently, Zuo et al. (Zuo et al., 2015) and Tian et al. (Tian et al., 2016)
attempted to fuse multi-tracking strategies (eg. spatial-temporal context
learning (Zhang et al., 2014), tracking-by-detection etc.) for text tracking,
in which Hungarian (Kuhn, 1955) algorithm was applied for generating the final
text streams. Yang et al. (Yang et al., 2017) also proposed a motion-based
tracking approach in which detected results are directly propagated to the
neighboring frames for recovering missing text regions. In fact, the robust
feature extractor is the most important component for a text tracker.

Text recognition in videos. With the tracked text streams, there are two
strategies for better scene text recognition: selection strategy by selecting
the best text regions from streams (popular before 2010), and results fusion
strategy by combining corresponding recognized character results.
Correspondingly, methods (Shiratori et al., 2006; Tanaka and Goto, 2007, 2008)
selected the region with the longest horizontal length as the most appropriate
region. Then Goto and Tanaka (Goto and Tanaka, 2009) further enhanced the
selection algorithm by taking six different features (eg. Fisher’s
discriminant ratio, text region area etc.) into account. While recent methods
(Greenhalgh and Mirmehdi, 2015; Rong et al., 2014) directly fused recognized
results in text streams for final text prediction by majority voting, CRF or
frame-wise comparison, and these approaches assumed that recognition results
in most frames are trust-worthy, which may not be true in unconstrained
scenarios. In addition, frame-wise text recognition also results in high
computation cost.

End-to-end text recognition in videos. There are several works proposed to
solve the end-to-end video text spotting problem. Nguyen et al. (Nguyen et
al., 2014) first proposed an end-to-end video text reading solution by
extending Wangs’s method (Wang et al., 2011), in which the frame-wise
detection and the tracking with multiple features (eg. the temporal distance,
edit distance etc.) are applied. Merino-Gracia and Mirmehdi (Merino-Gracia and
Mirmehdi, 2014) proposed an end-to-end video scene text reading system by
introducing the unscented Kalman filter (Wan and Van Der Merwe, 2000) , but
mainly focused on large text found in outdoor environments. Recently, Wang et
al. (Wang et al., 2017) firstly utilized an end-to-end deep neural network to
detect and recognize text in each frame, and then employed the _tracking-by-
detection_ strategy to associate text regions, and recovered missed detections
with the tracked results, finally obtain the recognition results by voting the
most frequently appeared text strings.

Different from frame-wise recognition in (Merino-Gracia and Mirmehdi, 2014;
Nguyen et al., 2014; Wang et al., 2017), in this paper we propose a fast and
robust video scene text spotting framework by integrating a spatial-temporal
video text detector and a novel text recommender, which dramatically speeds up
the text spotting process and also improves the final recognition performance.

##  3\. Methodology

Figure 2.  The workflow of YORO, which consists of two submodules: (a) The
spatial-temporal text detector for generating text regions; (b) The text
recommender assembling the tracking, quality scoring and recognition into a
whole end-to-end trainable module, in which the left and right parts
separately denote the training and inference processes. Specifically, the
tracking, scoring and recognition are simultaneously learnt with the metric
learning, the teacher-student strategy and the attention mechanism,
respectively.

###  3.1. Overview

The architecture of YORO is shown in Figure 2, which consists of two modules:
the spatial-temporal video text detector for detecting text regions among
adjacent frames and the text recommender for recognizing the highest-quality
text regions from the tracked text streams.

Concretely, the video text detector is responsible for recalling more text by
referring to temporal relationship among different frames. The recommender is
developed with three submodules: the essential quality scoring component for
directly generating the quality score of each text region by borrowing the
‘teacher-student’ concept (Hinton et al., 2015), the tracking component for
generating text streams with metric learning techniques which can enhance and
simplify the tracking process, and the recognition component for generating
the final results with an attention-based decoder (Shi et al., 2016). Note
that, the recommender is implemented as an end-to-end trainable network.

###  3.2. Spatial-Temporal Detection

The text detection architecture is shown in Figure 2.(a), in which the
backbone of EAST (Zhou et al., 2017) is selected as our backbone. Here, we
learn relations between consecutive frames with a _spatial-temporal
aggregation_ strategy for improving video text detection process, which can be
divided into three steps: 1) enhancing temporal coherence between frames with
a feature warping mechanism (Zhu et al., 2017), 2) spatial matching between
frames with a comparing and matching strategy inspired by (Chen et al., 2018;
Si et al., 2018), and 3) temporal aggregation.

Formally, let $I_{t}$ be the $t\text{-}th$ frame in a video, the detection
results in $I_{t}$ can be refined with the detections of its consecutive
frames $(I_{t-n},...,I_{t+n})$ where the size of refining window is
$2n\text{+}1$.

_Enhancing temporal coherence_. We obtain the corresponding sequence of
feature maps $F\text{=}(F_{t-n},...,F_{t+n})$ by propagating frames through
the EAST backbone. Given a pair of frame features $F_{t+i}$ and $F_{t}$ (the
reference frame), we enhance their temporal coherence by referring to the
estimated flow $flow_{(t+i,t)}$ between $I_{t+i}$ and $I_{t}$ with a flow-
guided warping mechanism

(1) $F_{t+i}^{w}=Warp(F_{t+i},flow_{(t+i,t)}),$

where $flow_{(t+i,t)}$ is pre-computed with TV-L1 algorithm, $Warp(.)$ is the
bilinear warping function applied on each element in the feature maps, and
$F_{t+i}^{w}$ denotes the feature maps warped from frame $I_{t+i}$ to frame
$I_{t}$. Thus $F$ is further transferred as the warped
$F^{w}=(F_{t-n}^{w},...,F_{t+n}^{w})$. Then we generate an enhanced sequence
of _confidence maps_ $C=(C_{t-n},...,C_{t+n})$ by propagating $F^{w}$ into a
classification sub-network, in which each value in $C_{t+i}$ represents the
possibility of being a text region.

_Comparing and matching_. We evaluate the spatial matching degree of two
frames with matching weights. The weights are firstly computed with a
transform module to produce the feature-aware filter which is represented as

(2) $F_{t+i}^{trans}=ReLU(BN(WF_{t+i}^{w}+b)),$

where $W$ and $b$ are learnable parameters, BN and ReLU represent Batch
Normalization and rectified linear unit function, respectively. Given the
transformed feature maps, we compute the similarity energy
$Sim_{t+i,t}=F_{t+i}^{trans}\odot F_{t}^{trans}$ of $I_{t+i}$ and $I_{t}$ as
the matching weights, where $\odot$ means the dot product position-wisely.

_Temporal aggregation_. Then we compute the aggregation weights by

(3) $a_{t+i}=\frac{exp(Sim_{t+i,t}\odot
C_{t+i})}{\sum_{i^{\prime}=-n}^{n}exp(Sim_{t+i^{\prime},t}\odot
C_{t+i^{\prime}})}.$

Here, we multiply $Sim_{t+i,t}$ by $C_{t+i}$ in order to reinforce the
aggregation weights of positive detections.

Finally, the temporal aggregation across the consecutive frames is computed by

(4) $C_{t,agg}=\sum_{i=-n}^{n}a_{t+i}*C_{t+i}$

where ”*” represents element-wise production.

For handling few mis-aggregated situations, we further refine $C_{t,agg}$ as
$C_{t,ref}=C_{t,agg}*M_{t}$ by applying a normalized binary mask $M_{t}$ to
$C_{t,agg}$, where $M_{t}$ is calculated by normalizing $F_{t}^{w}$ as a
binary mask with a pre-set threshold (default by 0.5). The detailed training
process is the same to that in EAST (Zhou et al., 2017).

###  3.3. Text Recommender

The text recommender contains three trainable parts: the quality scoring,
tracking and recognition, as shown in Figure 2.(b).

####  3.3.1. Text Quality Scoring

It is almost impossible to manually annotate the quality score for each text
region because the judgement of imaging quality is subjective for different
annotators, and the annotating cost is also very tremendous. Therefore, we
design an off-line scoring strategy (denoted by teacher) for generating the
quality score labels in a weakly-supervised fashion, and then train the
lightweight quality scoring network (denoted by student) with the teacher’s
scores.

Figure 3. Training mechanism of quality scoring module. (a) Teacher: the off-
line scoring mechanism to generate quality score labels by referring the
estimate templates. Here, different quality of text regions are located at
different positions in feature space, in which those always recognized
correctly will fall into a ‘correct zone’ and its center corresponds to the
estimated position of standard template. (b) Student: the quality scoring
network.

_Teacher_. For teaching the student module, the teacher module should generate
explicit quality score for each text region. Here, we propose the concept of
_standard template_ , which means that the template is the feature
representation of an ideal text image without any interference. Then each text
image is corresponding to a standard template image in the feature space, and
we can define the distance between the template and a candidate text region as
the corresponding quality score. In this assumption, those images close to its
template are more possible to be correctly recognized, vice versa.

However, it’s unrealistic to directly calculate the distance between text
regions in the low-level feature space (_eg._ color and intensity) because it
is hard to directly align two text regions due to the difference on color,
intensity and scales etc. Then, we attempt to represent each text region with
the extracted high-level character features. For example, an image with
content ‘ABC’ will be represented as the concatenation of high-level feature
‘A’, ‘B’ and ‘C’. Here, the character features can be provided by the
recognition model such as the output features of the attention-based decoder.

Specifically, we first feed a text stream into a pre-trained recognition
model, and then represent each of them with the concatenated character
features from the recognition model. We empirically find that features of
correctly recognized text tend to be localized in a compact area while the
features of wrongly recognized ones are usually far from this compact region,
as shown in Figure 3. Therefore, we conduct the estimation of the template
image $r^{t}$ with the correctly recognized feature sets
$\\{r^{cor}_{1},r^{cor}_{1},...,r^{cor}_{k}\\}$ by applying the K-Means
algorithm:

(5) ${r}^{t}\text{=}kmeans(r^{cor}_{1},r^{cor}_{1},...,r^{cor}_{k}),$

Consequentially, we calculate the feature distance between each text region
and its estimated template image as the quality score:

(6) ${s}^{t}_{i}\text{=}\frac{r^{t}\odot r_{i}}{||r^{t}||*||r_{i}||},$

where $\odot$ means the dot product and $||.||$ means the Euclidean distance.

_Student_. Supervised by the generated quality scores from teacher, the
quality scoring network can directly regress the quality score for each text
region, as illustrated in Figure 3.(b). Then the loss function for quality
scoring network is as follows:

(7)
$\mathcal{L}_{S}\text{=}\frac{1}{N}\sum_{i=0}^{N}||s^{t}_{i}\text{-}s^{s}_{i}||$

where $s^{t}$ is the quality score generated by teacher, and $s^{s}$ is the
regressed quality score by student.

####  3.3.2. Text tracking

The tracking task aims to group corresponding text regions into text streams,
shown in the left part of Figure 2.(b). Intuitively, the tracker should have
the ability to ensure that the features of a text region in one stream must
remain closer distance to those in the same stream than others, which implies:
1) the features must be discriminative enough to tolerate various
interferences in unconstrained scenes, and 2) the module may be better if
trained with a good distance measure.

_Robust feature extraction_. Thanks to the studies in deep neural network and
metric learning, we extract robust features for the tracker by applying metric
learning technique. Concretely, we firstly select three regions from localized
candidate regions as an image triplet $(R^{a},R^{p},R^{n})$, in which
$R^{a}\text{and}R^{p}$ are corresponding to the same text instance while
$R^{n}$ is randomly selected from other text instances. Secondly, an image
triplet is fed into a deep CNN for generating its L2 Normalized high-level
representation $(q^{a},q^{p},q^{n})$. The tracker is then trained with two
metric learning loss: contrastive loss (Hadsell et al., 2006)
$\mathcal{L}_{contra}$ and triplet loss (Schroff et al., 2015)
$\mathcal{L}_{triplet}$, i.e.

(8) $\mathcal{L}_{T}=\mathcal{L}_{contra}+\lambda_{t}\mathcal{L}_{triplet},$

where $\lambda_{t}$ is default by 1.

_Text stream generation_. With the trained tracking model, for a pair of
candidate text regions $(R^{1},R^{2})$, we calculate its matching cost by

(9) $MC(R^{1},R^{2})=\frac{1}{q^{1}\odot q^{2}+\epsilon}.$

For avoiding division by zero error, $\epsilon$ is set as $10^{-7}$. Then
those pairs with $MC$ larger than a threshold are considered as invalid
matching pairs and filtered out. Finally, we employ Hungarian algorithm (Kuhn,
1955) to generate the text streams.

####  3.3.3. Text Recognition

The text recognition module is not our focus, and we select attention-based
method as our decoder just like in (Cheng et al., 2017; Shi et al., 2016,
2018).

####  3.3.4. End-to-end training

Since the above three submodules are complementary for extracting
discriminative features, we jointly train all the three tasks with same
backbone in an end-to-end trainable network, which is supervised by:

(10)
$\mathcal{L}=\lambda_{1}\mathcal{L}_{T}+\lambda_{2}\mathcal{L}_{S}+\lambda_{3}\mathcal{L}_{R},$

where $\mathcal{L}_{R}$ is the recognition part loss and $\lambda$i (_i_
=1,2,3) denotes the loss weight for different tasks. Settings are detailed in
_Experiment_ section.

##  4\. The Larger-Scale Video Text Dataset

In recent years, research in video scene text still remains unpopular in
contrast to its promising application prospect. The existing video scene text
datasets such as _IC13_ or _IC15_ (detailed in Table 1) are limited on the
scale of video items and scenarios, which may restrain research of video scene
text spotting. Therefore, we collect and annotate a larger-scale video text
dataset (LSVTD), which contains 100 scene videos acquired from 22 typical
real-life scenarios.

Datasets | #scenarios | #videos | #frames | #instances | quality?
---|---|---|---|---|---
Merino (Merino and Mirmehdi, 2007) | 4 | – | – | – |
Minetto (Minetto et al., 2011) | – | 5 | 3599 | 8706 |
IC13 (Karatzas et al., 2013) | 7 | 28 | 15277 | 93934 | ✓
YVT (Nguyen et al., 2014) | – | 30 | 13500 | – |
IC15 (Karatzas et al., 2015) | 7 | 49 | 27824 | – | ✓
LSVTD | 22 | 100 | 66700 | 569300 | ✓

Table 1.  Comparison with existing video text datasets. ‘ _quality?_ ’ means
if each text region is labeled with quality score.

Dataset characteristics. LSVTD is detailed in Table 1, and mainly
characterized by 1) Much larger scale, which is more than twice the scale of
IC15. 2) More diversified scenarios. LSVTD covers a wide range of 13 indoor
(eg. bookstore, shopping mall) and 9 outdoor (eg. highway, city road)
scenarios, which is more than 3 times the diversity of IC15. The variety of
scenarios challenges text spotting algorithms to achieve robust performance.
3) Multilingual text instances. LSVTD contains text with more than three kinds
of languages (English, Spanish and Chinese etc.), which are further divided
into 2 major categories: Latin and Non-Latin. We additionally label this
attribute for the convenience of evaluation for different algorithms. More
details of LSVTD are shown in the supplementary materials1.

Annotation details. Following the annotation details adopted by IC15, we
annotate the following attributes for each text: 1) _Polygon coordinate_
represents text location; 2) _ID_ means the unique identification for each
text among consecutive frames; That is, the same text in consecutive frames
shares the same ID; 3) _Language_ is categorized as Latin and Non-Latin as
mentioned above; 4) _Quality_ indicates the image quality of each text region,
which is labelled as ‘high’,‘moderate’ or ‘low’. 5) _Transcripts_ means text
string for each text region.

These videos are collected by our 6 colleagues and range from 5 seconds to 1
minute. It took more than 3500 man-hours for annotating these frames. Here, we
hope that the availability111Available at https://davar-
lab.github.io/opensource/dataset/lsvtd of LSVTD may spur more interests in
corresponding areas.

##  5\. Experiments

###  5.1. Evaluation Protocols

The evaluation protocols for text detection, tracking and recognition in
videos have been declared in (Yin et al., 2016). We only select several
mainstream evaluation metrics in this paper.

Detection metrics. Following detection methods (Jiang et al., 2017; Zhong et
al., 2018; Zhou et al., 2017), _precision_ (denoted by PRE), _recall_ (denoted
by REC) and _F-measure_ are selected as the evaluation metrics.

Tracking metrics. The tracking metrics should maximize the sum of overlap
between the tracking results and ground truth. In general, multiple object
tracking precision (_abbr_. MOTP), multiple object tracking accuracy (_abbr_.
MOTA), and the average tracking accuracy (_abbr_. ATA) are used to evaluate
the performance of tracking.

Quality scoring metrics. Naturally, the better quality frames are selected,
the higher recognition accuracy we will get. To evaluate the performance of
the quality scoring mechanism, we first define the quality selection hitting
rate(_abbr_. QSHR) for evaluating the selection accuracy

(11) $QSHR\text{=}\sum_{i=0}^{N}\frac{\bar{q}_{i}}{N},$

where $N$ denotes the number of text streams, and $\bar{q}_{i}\in\\{0,1\\}$.
In the $i\text{-}th$ text stream, $\bar{q}_{i}\text{=}1$ means the region
annotated with ”high” is hit, 0 otherwise.

Based on the selection mechanism, we further define the rate of correctly
recognizing selected text regions (_abbr_. RCR) for evaluating sequence-level
recognition accuracy

(12) $RCR\text{=}\sum_{i=0}^{N}\frac{\bar{a}_{i}}{N},$

where $\bar{a}_{i}\in\\{0,1\\}$. In the $i\text{-}th$ text stream,
$\bar{a}_{i}\text{=}1$ means the selected text region is correctly recognized,
0 otherwise.

End-to-end metrics. In previous methods, MOTP, MOTA and ATA are generally used
in end-to-end evaluation, which evaluate performance in word-level
recognition. That is, a predict word is considered as a true positive if its
IoU over ground truth is larger than 0.5 and the word recognition is correct.

However, in our task, we just score and recognize a tracked text stream one-
time. According to the selection-and-recognition strategy, we redefine the
end-to-end metrics by considering two constrains: 1) The recognized result of
selected region should match the corresponding text transcription. 2) The
temporal location (frame ID) of selected region should fall into the interval
between the annotated starting and ending frame. In addition, the selected
candidate should have a spatial overlap ratio (default by over 0.5) with the
annotated bounding box. Thus we define the sequence-level recall $REC_{s}$ and
precision ($PRE_{s}$)

(13) $REC_{s}=\frac{N_{r}}{N_{g}},~{}~{}~{}PRE_{s}=\frac{N_{r}}{N_{d}},$

for _constrain 1_ and _constrain 2_ , in which ${N}$r, ${N}$g and ${N}$d
separately denote the number of valid recalled streams, the number of total
ground truth streams and the number of detected text streams. Correspondingly,
the sequence-level F-score ($F\text{-}Score$) is denoted as

(14) $F\text{-}Score=\frac{2\ast{PRE_{s}}\ast{REC_{s}}}{PRE_{s}+REC_{s}}$

by simultaneously considering $PRE_{s}$ and $REC_{s}$.

It’s worth to note that we only match a given ground truth stream once, which
also penalize the stream fragmentation problem occurred in text tracking. In
all, the evaluation protocol measures the accuracy and efficiency to extract
useful text information from videos.

###  5.2. Implementation Details

All of our work is built on the CAFFE framework.

_Detection Network_. The EAST backbone is pre-trained on the ‘Incidental Scene
Text’ dataset (Karatzas et al., 2015) and ‘COCO-Text’ dataset (Veit et al.,
2016) by following (Liu et al., 2018), and then the model is fine-tuned on
corresponding video training set such as IC13 or IC15. Images are randomly
cropped and resized to 512$\times$512 and then fed into the network. In
training stage, we set _batch-size_ =4 and learn the network by adopting
‘Adam’ with _learning rate= $10^{-4}$, decay rate=0.94_ for every $10^{4}$
iterations, in which text regions with short side less than 10 pixels are
ignored during training. While in testing stage, we only conduct the single-
scale testing. In the post-processing stage, we adopt NMS on predicted
geometric shapes with _threshold=0.2_.

_Text Recommender Network_. The ‘ResNet Backbone’+‘Conv Blocks’ used in
tracking, scoring and recognition is adopted from an image encoder used in
(Cheng et al., 2017), and the ‘BLSTM’+‘ATT’ module in scoring and recognition
is an attention decoder used in (Cheng et al., 2017; Shi et al., 2016). The
network is pre-trained on the 8-million synthetic data (Jaderberg et al.,
2014) using ‘Adadelta’ by following (Shi et al., 2016), and further fine-tuned
on IC13 or IC15 using SGD with the fixed learning rate of $10^{-4}$. The loss
weights $\lambda$1, $\lambda$2, $\lambda$3 in Equation. 10 are all set to 1,
and the margin $\alpha$ used in triplet loss is set to 0.8. In text tracking
process, the threshold $MC$ for filtering out invalid text pairs is set to
0.92.

###  5.3. Ablation Study

Effects of text recommender. In IC13 and IC15, text regions are annotated as 3
quality levels (‘low’, ‘moderate’ and ‘high’). Those streams containing at
least two types of quality annotations are treated as our testing dataset.

To evaluate the proposed text recommender (_abbr._ TR), we compare our method
with two commonly used selection-and-recognition strategies: 1) Using the
predicted confidence (the average probability of generating characters) of a
word as the quality score (denoted by PCW). 2) Selecting the text region with
the highest frequency of predicted results as the voted best one (denoted by
HFP), which is similar to the _majority voting_ strategy used in (Wang et al.,
2017).

Methods | $QSHR$ | $RCR$ | $FPS$
---|---|---|---
| (IC13/IC15) | (IC13/IC15)
PCW | 74.55/75.83 | 66.06/66.32 | 4.52
HFP | 75.32/76.34 | 68.30/68.56
TR ($\mathcal{L}_{S}$) | 77.89/79.69 | 68.89/69.41 | 324.58
TR ($\mathcal{L}_{S}$+$\mathcal{L}_{T}$) | 78.64/80.36 | 69.12/69.82
TR ($\mathcal{L}_{S}$+$\mathcal{L}_{R}$) | 81.23/83.03 | 69.92/70.69
TR ($\mathcal{L}$) |  81.74/83.29 |  70.18/70.95

Table 2. Effects of TR on IC13 and IC15 compared with other frame selection
methods. (.) refers to the loss applied in TR.

Table. 2 shows the results. Compared to HFP, TR ($\mathcal{L}$) significantly
improves the _QSHR_ performance by 6.4% on IC13 and 7% on IC15, while improves
the _RCR_ performance by 1.9% on IC13 and 2.4% on IC15. Compared to the
results of other settings (TR ($\mathcal{L}_{S}$), TR
($\mathcal{L}_{S}$+$\mathcal{L}_{R}$) and TR
($\mathcal{L}_{S}$+$\mathcal{L}_{T}$)), the end-to-end training strategy can
further boosting the scoring performance. For simplified representation, we
use TR as TR ($\mathcal{L}$) in the sequel.

Moreover, TR only needs recognize a text stream one-time, which can greatly
decrease the computational cost. As shown in Table. 2, TR speeds up the
recognition process averagely by 71 times compared with the frame-wise manner
on two datasets.

Extreme testing for text recommender. It is worth noticing that TR can still
select the best one when handling text streams with a large proportion of low-
quality text regions, while the voting strategy becomes useless. It implies
that TR is more robust in complex and heavily distorted video scenarios.
Therefore, we conduct extreme testing on a constituted _low-quality text
stream set_ by discarding all streams containing more than 40% highest quality
text regions on IC13 and IC15. We calculate the _QSHR_ and _RCR_ on this set
by checking whether the highest quality of text is hit and whether the
selected text is correctly recognized. Table 3 gives the results and
demonstrates that TR is more robust in complex and low-quality video
scenarios.

Methods | $QSHR$ | $RCR$
---|---|---
| (IC13/IC15) | (IC13/IC15)
PCW | 41.73/45.66 | 59.78/60.62
HFP | 39.37/41.73 | 58.96/60.06
TR |  51.18/54.33 |  66.14/67.71

Table 3. Extreme testing of TR on IC13 and IC15 compared with other frame
selection methods.

End-to-end evaluation. To analyze the contributions of above components, we
conduct the end-to-end evaluation on the popular IC15 dataset, as shown in
Table. 4.

For fair comparison, we here declare some baselines in our framework, i.e. For
detection, we treat the video text detection without our spatial-temporal
strategy as our detection baseline (denoted by D-BASE), and denote the
spatial-temporal detection model as D-ST; While for text recommender, compared
with the trainable TR, we treat TR ($\mathcal{L}_{S}$) as our baseline.

From Table. 4, we find that

  * •

Comparing to the detection baseline, the spatial-temporal strategy can
steadily help the end-to-end recognition.

  * •

In the D-ST case, our end-to-end trainable TR greatly improves the end-to-end
performance of $PRE_{s}$, $REC_{s}$ and $F\text{-}score$ by 3.4%, 5.7% and
4.6%, respectively.

  * •

The proposed method (D-ST + TR) significantly outperform the D-BASE+TR
($\mathcal{L}_{S}$) by 6.6%.

Above results demonstrate the effectiveness of text recommender as well as the
whole framework.

D-BASE | ✓ | ✓ |  |
---|---|---|---|---
D-ST |  |  | ✓ | ✓
TR ($\mathcal{L}_{S}$) | ✓ |  | ✓ |
TR |  | ✓ |  | ✓
$PRE_{s}$ | 69.91 | 72.84 | 64.88 | 68.28
$REC_{s}$ | 54.34 | 61.73 | 61.54 | 67.21
$F\text{-}score$ | 61.15 | 66.83 | 63.17 | 67.74

Table 4.  The end-to-end evaluation on IC15.

###  5.4. Comparison with State-of-The-Arts

Comparison on detection. We only evaluate the detection results on IC13
because there are no results reported on IC15. From Table. 5, we find that the
D-BASE already outperforms existing approaches by a large margin thanking to
the robust EAST, but still suffers from low recall due to the complicated
motion scenarios. As expected, the spatial-temporal strategy can significantly
improves _REC_ by 4% and _F -measure_ by 1.3%, but with a 4.3% drop on _PRE_.
Actually, boosting recall performance is more important when facing very low
recall results, which generally results in the precision decreasing.

Methods | $REC$ | $PRE$ | $F\text{-}measure$
---|---|---|---
Khare et al. (Khare et al., 2015) | 41.40 | 47.60 | 44.30
Zhao et al. (Zhao et al., 2011) | 47.02 | 46.30 | 46.65
Shivakumara (Shivakumara et al., 2012) | 53.71 | 51.15 | 50.67
Yin et al. (Yin et al., 2014) | 54.73 | 48.62 | 51.56
Wang et al. (Wang et al., 2018a) | 51.74 | 58.34 | 54.45
D-BASE | 56.21 | 85.76 | 67.91
D-ST | 60.23 | 81.45 | 69.25

Table 5. Detection results on IC13.

Comparison on tracking. Table 6 shows the comparing results on IC13 and IC15.

_Evaluation on IC13_. T-BASE outperforms the reported results by a large
margin 0.35 on $ATA_{D}$, 0.08 on $MOTP_{D}$ and 0.37 on $MOTA_{D}$. TR can
further separately improve the performance by 0.14 and 0.03 on $ATA_{D}$ and
$MOTA_{D}$, and maintain the $MOTP_{D}$ performance.

_Evaluation on IC15_. T-BASE also achieves comparable results with previous
methods. Comparing to the best reported result, TR significantly improves the
$ATA_{D}$ and $MOTA_{D}$ by 0.12 and 0.03, but falls behind (Yang et al.,
2017) on $MOTP_{D}$. However, (Yang et al., 2017) points out that $ATA_{D}$ is
the most important metric in IC15 because $ATA_{D}$ measures the tracking
performance over all the text.

Dataset | Methods | $ATA_{D}$ | $MOTP_{D}$ | $MOTA_{D}$
---|---|---|---|---
| IC13’s base (Karatzas et al., 2013) | 0.00 | 0.63 | -0.09
| TextSpotter (Neumann and Matas, 2013) | 0.12 | 0.67 | 0.27
IC13 | Nguyen et al. (Nguyen et al., 2014) | 0.15 | - | -
| T-BASE | 0.50 | 0.75 | 0.64
| TR | 0.64 | 0.75 | 0.67
| Stradvision-1 (Karatzas et al., 2015) | 0.32 | 0.71 | 0.48
| Deep2Text-I (Karatzas et al., 2015) | 0.45 | 0.71 | 0.41
| Wang et al. (Wang et al., 2017) | 0.56 | 0.70 | 0.57
IC15 | Yang et al. (Yang et al., 2017) | 0.61 | 0.79 | 0.66
| T-BASE | 0.53 | 0.76 | 0.65
| TR | 0.65 | 0.76 | 0.68

Table 6. Tracking performance evaluation on IC13 and IC15. The suffix ‘D’
means tracking is applied for detection.

Comparison on end-to-end evaluation. Conventionally, we also place the frame-
wise recognition results on IC15 by referring to the previous works, in which
we just treat the recognition output of the selected text region as the final
recognition result in each text stream. Table. 7 shows that this setting also
achieves the state of the art.

Method | $MOTP_{R}$ | $MOTA_{R}$ | $ATA_{R}$
---|---|---|---
Stradvision (Karatzas et al., 2015) | 0.69 | 0.57 | 0.29
Deep2Text (Karatzas et al., 2015) | 0.62 | 0.35 | 0.19
Wang et al. (Wang et al., 2017) | 0.70 | 0.69 | 0.60
Ours | 0.76 | 0.69 | 0.63

Table 7.  The traditional end-to-end evaluation on IC15. The suffix ‘R’ means
tracking is applied for measuring recognition.  Figure 4. Illustration of _F-
score_ performance on 22 scenarios. Indoor/Outdoor scenarios are highlighted
in red/green.

###  5.5. Challenges on LSVTD

We also evaluate the proposed framework on the collected LSVTD. Figure 4 shows
the _F-score_ performance on 22 different scenarios. In brief, we find that
video text spotting in _books opening_ and _metro station_ is easier comparing
to that in _bookstore_ or _shopping mall_. While video text spotting in _city
road_ , _camera surveillance_ and _train watching_ is still very challenging
due to extremely complex background and fierce shaking, which needs more
efforts to boost method’s performance. Overall, spotting text in outdoor is
more challenging than that in indoor. More statistics and visualization
results are shown in supplementary materials.

##  6\. Conclusion

In this paper, we propose a fast and robust video text spotting framework
named as YORO by integrating a well-designed spatial-temporal video text
detector and a novel text recommender. The video text detector is responsible
for recalling more text by referring the relation between different frames.
Moreover, we concentrate on developing the text recommender for selecting the
highest-quality text from tracked text streams and then only recognizing the
selected text region once, which not only significantly speeds up the
recognition process, and also improves the video text spotting performance. In
addition, we release a larger-scale video scene text dataset for better
evaluating video text spotting algorithms. In future, we’ll further explore
the spatial-temporal video text spotting in an end-to-end trainable way.

## References

  * (1)
  * Bai et al. (2018) Fan Bai, Zhanzhan Cheng, Yi Niu, Shiliang Pu, and Shuigeng Zhou. 2018.  Edit Probability for Scene Text Recognition. In _CVPR_. 1508–1516.
  * Borisyuk et al. (2018) Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar. 2018\.  Rosetta: Large Scale System for Text Detection and Recognition in Images. In _SIGKDD_. 71–79.
  * Chen et al. (2018) Dapeng Chen, Hongsheng Li, Tong Xiao, Shuai Yi, and Xiaogang Wang. 2018.  Video Person Re-Identification With Competitive Snippet-Similarity Aggregation and Co-Attentive Snippet Embedding. In _CVPR_. 1169–1178.
  * Cheng et al. (2017) Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, and Shuigeng Zhou. 2017\.  Focusing attention: Towards accurate text recognition in natural images. In _ICCV_. 5086–5094.
  * Cheng et al. (2018) Zhanzhan Cheng, Yangliu Xu, Fan Bai, Yi Niu, Shiliang Pu, and Shuigeng Zhou. 2018\.  AON: Towards Arbitrarily-Oriented Text Recognition. In _CVPR_. 5571–5579.
  * Fragoso et al. (2011) Victor Fragoso, Steffen Gauglitz, Shane Zamora, Jim Kleban, and Matthew Turk. 2011.  TranslatAR: A mobile augmented reality translator. In _WACV_. 497–502.
  * Gómez and Karatzas (2014) Llífs Gómez and Dimosthenis Karatzas. 2014.  MSER-based real-time text detection and tracking. In _ICPR_. 3110–3115.
  * Goto and Tanaka (2009) Hideaki Goto and Makoto Tanaka. 2009.  Text-tracking wearable camera system for the blind. In _ICDAR_. 141–145.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006.  Connectionist Temporal Classification : Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In _ICML_. 369–376.
  * Greenhalgh and Mirmehdi (2015) Jack Greenhalgh and Majid Mirmehdi. 2015.  Recognizing Text-Based Traffic Signs.  _IEEE TITS_ 16, 3 (2015), 1360–1369.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006\.  Dimensionality reduction by learning an invariant mapping. In _CVPR_. 1735–1742.
  * He et al. (2018) Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, and Changming Sun. 2018\.  An End-to-End TextSpotter With Explicit Alignment and Attention. In _CVPR_. 5020–5029.
  * He et al. (2017) Wenhao He, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. 2017\.  Deep Direct Regression for Multi-Oriented Scene Text Detection. In _ICCV_. 745–753.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  (2015).  arXiv:arXiv:1806.00578
  * Hu et al. (2017) Han Hu, Chengquan Zhang, Yuxuan Luo, Yuzhuo Wang, Junyu Han, and Errui Ding. 2017\.  WordSup: Exploiting Word Annotations for Character Based Text Detection. In _ICCV_. 4940–4949.
  * Jaderberg et al. (2014) Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition.  (2014).  arXiv:arXiv:1406.2227
  * Jiang et al. (2017) Yingying Jiang, Xiangyu Zhu, Xiaobing Wang, Shuli Yang, Wei Li, Hua Wang, Pei Fu, and Zhenbo Luo. 2017.  R2CNN: Rotational Region CNN for Orientation Robust Scene Text Detection.  (2017).  arXiv:arXiv:1706.09579
  * Karatzas et al. (2015) Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. 2015\.  ICDAR 2015 competition on robust reading. In _ICDAR_. 1156–1160.
  * Karatzas et al. (2013) Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. 2013\.  ICDAR 2013 robust reading competition. In _ICDAR_. 1484–1493.
  * Khare et al. (2015) Vijeta Khare, Palaiahnakote Shivakumara, and Paramesran Raveendran. 2015.  A new Histogram Oriented Moments descriptor for multi-oriented moving text detection in video.  _Expert Systems with Applications_ 42, 21 (2015), 7627–7640.
  * Kim et al. (2003) Kwang In Kim, Keechul Jung, and Jin Hyung Kim. 2003\.  Texture-based approach for text detection in images using support vector machines and continuously adaptive mean shift algorithm.  _IEEE TPAMI_ 25, 12 (2003), 1631–1639.
  * Kuhn (1955) Harold W Kuhn. 1955\.  The Hungarian method for the assignment problem.  _Naval Research Logistics Quarterly_ 2, 1-2 (1955), 83–97.
  * Li et al. (2017) Hui Li, Peng Wang, and Chunhua Shen. 2017.  Towards End-To-End Text Spotting With Convolutional Recurrent Neural Networks. In _ICCV_. 5238–5246.
  * Liao et al. (2018) Minghui Liao, Zhen Zhu, Baoguang Shi, Gui-song Xia, and Xiang Bai. 2018.  Rotation-Sensitive Regression for Oriented Scene Text Detection. In _CVPR_. 5909–5918.
  * Liu et al. (2018) Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan. 2018.  FOTS: Fast Oriented Text Spotting with a Unified Network. In _CVPR_. 5676–5685.
  * Liu and Jin (2017) Yuliang Liu and Lianwen Jin. 2017.  Deep Matching Prior Network: Toward Tighter Multi-oriented Text Detection. In _CVPR_. 3454–3461.
  * Ma et al. (2018) Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue. 2018.  Arbitrary-oriented scene text detection via rotation proposals.  _IEEE Transactions on Multimedia_ (2018).
  * Merino and Mirmehdi (2007) Carlos Merino and Majid Mirmehdi. 2007.  A framework towards realtime detection and tracking of text. In _CBDAR_. 10–17.
  * Merino-Gracia and Mirmehdi (2014) Carlos Merino-Gracia and Majid Mirmehdi. 2014.  Real-time text tracking in natural scenes.  _IET Computer Vision_ 8, 6 (2014), 670–681.
  * Minetto et al. (2011) Rodrigo Minetto, Nicolas Thome, Matthieu Cord, Neucimar J Leite, and Jorge Stolfi. 2011.  Snoopertrack: Text detection and tracking for outdoor videos. In _ICIP_. 505–508.
  * Na and Wen (2010) Yinan Na and Di Wen. 2010\.  An effective video text tracking algorithm based on sift feature and geometric constraint. In _PRCM_. 392–403.
  * Neumann and Matas (2013) Luka Neumann and Jiri Matas. 2013.  On combining multiple segmentations in scene text recognition. In _ICDAR_. 523–527.
  * Nguyen et al. (2014) Phuc Xuan Nguyen, Kai Wang, and Serge Belongie. 2014\.  Video text detection and recognition: Dataset and benchmark. In _WACV_. 776–783.
  * Petter et al. (2011) Marc Petter, Victor Fragoso, Matthew Turk, and Charles Baur. 2011.  Automatic text detection for mobile augmented reality translation. In _Workshop on ICCV_. 48–55.
  * Rong et al. (2014) Xuejian Rong, Chucai Yi, Xiaodong Yang, and Yingli Tian. 2014\.  Scene text recognition in multiple frames based on text tracking. In _ICME_. 1–6.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  Facenet: A unified embedding for face recognition and clustering. In _CVPR_. 815–823.
  * Shi et al. (2017a) Baoguang Shi, Xiang Bai, and Serge Belongie. 2017a.  Detecting Oriented Text in Natural Images by Linking Segments. In _CVPR_. 2550–2558.
  * Shi et al. (2017b) Baoguang Shi, Xiang Bai, and Cong Yao. 2017b.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  _IEEE TPAMI_ 39, 11 (2017), 2298–2304.
  * Shi et al. (2016) B. Shi, X. Wang, P. Lyu, C. Yao, and X. Bai. 2016.  Robust Scene Text Recognition with Automatic Rectification. In _CVPR_. 4168–4176.
  * Shi et al. (2018) B. Shi, M. Yang, X. Wang, P. Lyu, C. Yao, and X. Bai. 2018.  ASTER: An Attentional Scene Text Recognizer with Flexible Rectification.  _IEEE TPAMI_ (2018), 1–1.
  * Shiratori et al. (2006) Hiroki Shiratori, Hideaki Goto, and Hiroaki Kobayashi. 2006\.  An efficient text capture method for moving robots using DCT feature and text tracking. In _ICPR_ , Vol. 2. 1050–1053.
  * Shivakumara et al. (2012) Palaiahnakote Shivakumara, Rushi Padhuman Sreedhar, Trung Quy Phan, Shijian Lu, and Chew Lim Tan. 2012\.  Multioriented video scene text detection through Bayesian classification and boundary growing.  _IEEE TCSVT_ 22, 8 (2012), 1227–1235.
  * Si et al. (2018) Jianlou Si, Honggang Zhang, Chun-Guang Li, Jason Kuen, Xiangfei Kong, Alex C Kot, and Gang Wang. 2018.  Dual Attention Matching Network for Context-Aware Feature Sequence based Person Re-Identification. In _CVPR_. 5363–5372.
  * Tanaka and Goto (2007) Makoto Tanaka and Hideaki Goto. 2007.  Autonomous text capturing robot using improved DCT feature and text tracking. In _ICDAR_ , Vol. 2. 1178–1182.
  * Tanaka and Goto (2008) Makoto Tanaka and Hideaki Goto. 2008.  Text-tracking wearable camera system for visually-impaired people. In _ICPR_. 1–4.
  * Tian et al. (2016) Shu Tian, Wei-Yi Pei, Ze-Yu Zuo, and Xu-Cheng Yin. 2016\.  Scene Text Detection in Video by Learning Locally and Globally. In _IJCAI_. 2647–2653.
  * Veit et al. (2016) Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. 2016.  COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images.  (2016).  arXiv:arXiv:1601.07140
  * Wan and Van Der Merwe (2000) Eric A Wan and Rudolph Van Der Merwe. 2000.  The unscented Kalman filter for nonlinear estimation. In _AS-SPCC_. 153–158.
  * Wang et al. (2018b) Fangfang Wang, Liming Zhao, Xi Li, Xinchao Wang, and Dacheng Tao. 2018b.  Geometry-Aware Scene Text Detection With Instance Transformation Network. In _CVPR_. 1381–1389.
  * Wang and Hu (2017) Jianfeng Wang and Xiaolin Hu. 2017.  Gated recurrent convolution neural network for OCR. In _NIPS_. 335–344.
  * Wang et al. (2011) Kai Wang, Boris Babenko, and Serge Belongie. 2011.  End-to-end scene text recognition. In _ICCV_. IEEE, 1457–1464.
  * Wang et al. (2018a) Lan Wang, Yang Wang, Susu Shan, and Feng Su. 2018a.  Scene Text Detection and Tracking in Video with Background Cues. In _ICMR_. 160–168.
  * Wang et al. (2017) Xiaobing Wang, Yingying Jiang, Shuli Yang, Xiangyu Zhu, Wei Li, Pei Fu, Hua Wang, and Zhenbo Luo. 2017.  End-to-End Scene Text Recognition in Videos Based on Multi Frame Tracking. In _ICDAR_ , Vol. 1. 1255–1260.
  * Yang et al. (2017) Xue-Hang Yang, Wenhao He, Fei Yin, and Cheng-Lin Liu. 2017\.  A Unified Video Text Detection Method with Network Flow. In _ICDAR_ , Vol. 1. 331–336.
  * Yin et al. (2014) Xu-Cheng Yin, Xuwang Yin, Kaizhu Huang, and Hong-Wei Hao. 2014\.  Robust text detection in natural scene images.  _IEEE TPAMI_ 36, 5 (2014), 970–983.
  * Yin et al. (2016) Xu-Cheng Yin, Ze-Yu Zuo, Shu Tian, and Cheng-Lin Liu. 2016\.  Text detection, tracking and recognition in video: a comprehensive survey.  _IEEE TIP_ 25, 6 (2016), 2752–2773.
  * Zhang et al. (2014) Kaihua Zhang, Lei Zhang, Qingshan Liu, David Zhang, and Ming-Hsuan Yang. 2014.  Fast visual tracking via dense spatio-temporal context learning. In _ECCV_. 127–141.
  * Zhao et al. (2011) Xu Zhao, Kai-Hsiang Lin, Yun Fu, Yuxiao Hu, Yuncai Liu, and Thomas S Huang. 2011\.  Text from corners: a novel approach to detect text and caption in videos.  _IEEE TIP_ 20, 3 (2011), 790–799.
  * Zhong et al. (2018) Zhuoyao Zhong, Lei Sun, and Qiang Huo. 2018.  An Anchor-Free Region Proposal Network for Faster R-CNN based Text Detection Approaches.  (2018).  arXiv:arXiv:1804.09003
  * Zhou et al. (2017) Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017.  EAST: An Efficient and Accurate Scene Text Detector. In _CVPR_. 5551–5560.
  * Zhu et al. (2017) Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. 2017.  Flow-Guided Feature Aggregation for Video Object Detection. In _ICCV_. 408–417.
  * Zuo et al. (2015) Ze-Yu Zuo, Shu Tian, Wei-yi Pei, and Xu-Cheng Yin. 2015\.  Multi-strategy tracking based text detection in scene videos. In _ICDAR_. 66–70.