11institutetext: Paper ID 122511institutetext: S-Lab, Nanyang Technological
University 22institutetext: School of Computer Science and Engineering,
Nanyang Technological University 33institutetext: Sensetime Research and
Tetras AI
33email<EMAIL_ADDRESS>
# Quality Assessment in the Time Dimension:
Inter-frame Distortion and Time-series Attention

Anonymous ECCV submission  Haoning Wu 112233 Chaofeng Chen 1122 Jingwen Hou 22
Liang Liao 1122 Annan Wang 1122 Wenxiu Sun 33 Qiong Yan 33 Weisi Lin 22

# FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment
Sampling

Anonymous ECCV submission  Haoning Wu 112233 Chaofeng Chen 1122 Jingwen Hou 22
Liang Liao 1122 Annan Wang 1122 Wenxiu Sun 33 Qiong Yan 33 Weisi Lin 22

###### Abstract

Current deep video quality assessment (VQA) methods are usually with high
computational costs when evaluating high-resolution videos. This cost hinders
them from learning better video-quality-related representations via end-to-end
training. Existing approaches usually consider naive sampling to reduce the
computational cost, such as resizing and cropping. However, they obviously
corrupt quality-related information in videos and are thus not optimal to
learn good representations for VQA. Therefore, there is an eager need to
design a new quality-retained sampling scheme for VQA. In this paper, we
propose Grid Mini-patch Sampling (GMS), which allows consideration of local
quality by sampling patches at their raw resolution and covers global quality
with contextual relations via mini-patches sampled in uniform grids. These
mini-patches are spliced and aligned temporally, named as fragments. We
further build the Fragment Attention Network (FANet) specially designed to
accommodate fragments as inputs. Consisting of fragments and FANet, the
proposed FrAgment Sample Transformer for VQA (FAST-VQA) enables efficient end-
to-end deep VQA and learns effective video-quality-related representations. It
improves state-of-the-art accuracy by around $10\%$ while reducing $99.5\%$
FLOPs on 1080P high-resolution videos. The newly learned video-quality-related
representations can also be transferred into smaller VQA datasets and boost
the performance on these scenarios. Extensive experiments show that FAST-VQA
has good performance on inputs of various resolutions while retaining high
efficiency. We publish our code at https://github.com/timothyhtimothy/FAST-
VQA.

###### Keywords:

Video Quality Assessment, fragments, Quality-retained Sampling, End-to-End
Learning, State-of-the-Art, High Efficiency

##  1 Introduction

Figure 1: Motivation for fragments: (a) The computational cost (FLOPs&Memory
at Batch Size 4) for existing VQA methods is high especially on high-
resolution videos. (b) Sampling approaches. Naive approaches such as resizing
[17, 43] and cropping [14, 15] cannot preserve video quality well. Zoom in for
clearer view.

More and more videos with a variety of contents are collected in-the-wild and
uploaded to the Internet every day. With the growth of high-definition video
recording devices, a growing proportion of these videos are in high resolution
(e.g. $\geq 1080P$). Classical video quality assessment (VQA) algorithms based
on handcrafted features are difficult to handle these videos with diverse
content and degradation. In recent years, deep-learning-based VQA methods [22,
23, 40, 8, 42, 21] have shown better performance on in-the-wild VQA benchmarks
[32, 12, 38, 40]. However, the computational cost of deep VQA methods
increases quadratically when applied to high resolution videos, and a video of
size $1080\times 1920$ would require $\mathbf{42.5\times}$ floating point
operations (FLOPs) than normal $224\times 224$ inputs (as Fig. 1(a) shows),
limiting these methods from practical applications. It is urgent to develop
new VQA methods that are both effective and efficient.

Meanwhile, with high memory cost noted in Fig. 1(a), existing methods usually
regress quality scores with fixed features extracted from pre-trained networks
for classification tasks [11, 33, 10] to alleviate memory shortage problem on
GPUs instead of end-to-end training, preventing them from learning video-
quality-related representations that better represent quality information and
limiting their accuracy. Existing approaches apply naive sampling on images or
videos by resizing [17, 43] or cropping [14, 15] (as Fig. 1(b) shows) to
reduce this cost and enable end-to-end training. However, they both cause
artificial quality corruptions or changes during sampling, _e.g._ , resizing
corrupts local textures that are significant for predicting video quality,
while cropping causes mismatched global quality with local regions. Moreover,
the severity of these problems increases with the raw resolution of the video,
making them unsuitable for VQA tasks.

Figure 2: Fragments, in spatial view (a) and temporal view (b). Zoom-in views
of mini-patches show that fragments can retain spatial local quality
information (a), and spot temporal variations such as shaking across frames
(b). In (a), spliced mini-patches also keep the global scene information of
original frames.

To improve the practical efficiency and the training effectiveness of deep VQA
methods, we propose a new sampling scheme, Grid Mini-patch Sampling (GMS), to
retain the sensitivity to original video quality. GMS cuts videos into
spatially uniform non-overlapping grids, randomly sample a mini-patch from
each grid, and then splice mini-patches together. In temporal view, we
constrain the position of mini-patches to align across frames, in order to
ensure the sensitivity on temporal variations. We name these temporally
aligned and spatially spliced mini-patches as fragments. As shown in Fig. 2,
The proposed fragments can well preserve the sensitivity on both spatial and
temporal quality. First, it preserves the local texture-related quality
information (_e.g._ , spot blurs happened in video 1/2) by retaining the
original resolution in patches. Second, benefiting from the globally uniformly
sampled grids, it covers the global quality even though different regions have
different qualities (_e.g._ , video 3). Third, by splicing the mini-patches,
fragments retains contextual relations of patches so that the model can learn
global scene information of the original frames. At last, with temporal
alignment, fragments preserve temporal quality sensitivity by retaining the
inter-frame variations in mini-patches from raw resolution, so they can be
used to spot temporal distortions in videos and distinguish between severely
shaking videos (_e.g._ , video 5) from relatively stable shots (_e.g._ , video
6).

However, it is non-trivial to build a network using the proposed fragments as
inputs. The network should follow two principles: 1) It should better extract
the quality-related information preserved in fragments, including the retained
local textures inside the raw resolution patches and the contextual relations
between the spliced mini-patches; 2) It should distinguish the artificial
discontinuity between mini-patches in fragments from the authentic quality
degradation in the original videos. Based on these two principles, we propose
a Fragment Attention Network (FANet) with Video Swin Transformer Tiny (Swin-T)
[27] as the backbone. Swin-T has a hierarchical structure and processes inputs
with patch-wise operations, which is naturally suitable for proceeding with
proposed fragments.

Figure 3: Motivation for the two proposed modules in FANet: (a) Gated Relative
Position Biases (GRPB); (b) Intra-Patch Non-Linear Regression (IP-NLR) head.
The structures for the two modules are illustrated in Fig. 5.

Furthermore, to avoid the negative impact of discontinuity between mini-
patches on quality prediction, we propose two novel modules, _i.e._ , Gated
Relative Position Biases (GRPB) and Intra-Patch Non-Linear Regression (IP-
NLR), to correct for the self-attention computation and the final score
regression in the FANet respectively. Specifically, considering that some
pairs in the same attention window might have the same relative position
(e.g., Fig. 3(a) A-C, D-E, A-B), but the cross-patch attention pairs (A-C,
D-E) are in far actual distances while intra-patch attention pairs (A-B) are
in much nearer actual distances in the original video, we propose GRPB to
explicitly distinguish these two kinds of attention pairs to avoid confusion
of discontinuity between patches and authentic video artifacts. In addition,
due to the discontinuity, different mini-patches contain diverse quality
information (Fig. 3(b)), thus pooling operation before score regression
applied in existing methods may confuse the information. To address this
issue, we design IP-NLR as a quality-sensitive head, which first regresses the
quality scores of mini-patches independently with non-linear layers and pools
them after the regression.

In summary, we propose the FrAgment Sample Transformer for VQA (FAST-VQA),
with the following contributions:

  1. 1.

We propose fragments, a new sampling strategy for VQA that preserves both
local quality and unbiased global quality with contextual relations via
uniform Grid Mini-patch Sampling (GMS). The fragments can reduce the
complexity of assessing 1080P videos by 97.6% and enables effective end-to-end
training of VQA with quality-retained video samples.

  2. 2.

We propose the Fragment Attention Network (FANet) to learn the local and
contextual quality information from fragments, in which the Gated Relative
Position Biases (GRPB) module is proposed to distinguish the intra-patch and
cross-patch self-attention and the Intra-Patch Non-Linear Regression (IP-NLR)
is proposed for better quality regression from fragments.

  3. 3.

The proposed FAST-VQA can learn video-quality-related representations
efficiently through end-to-end training. These quality features help FAST-VQA
to be 10% more accurate than the existing state-of-the-art approaches and 8%
better than full-resolution Swin-T baseline with fixed recognition features.
Through transfer learning, these quality features also significantly improve
the best benchmark performance for small VQA datasets.

##  2 Related Works

#### Classical VQA Methods

Classical VQA methods [31, 29, 20, 36, 35, 25] handcrafted features to
evaluate video quality. Among recent works, TLVQM [20] uses a combination of
spatial high-complexity and temporal low-complexity handcraft features and
VIDEVAL [36] ensembles different handcraft features to model the diverse
authentic distortions. However, the reasons affecting the video quality are
quite complicated and cannot be well captured with these handcrafted features.

#### Fixed-feature-based Deep VQA Methods

Due to the extremely high computational cost of deep networks on high
resolution videos, existing deep VQA methods train only a feature regression
network with fixed deep features. Among them, VSFA [22] uses the features
extracted by pre-trained ResNet-50 [11] from ImageNet-1k [5] and GRU [4] for
temporal regression. MLSP-FF [8] also uses heavier Inception-ResNet-V2 [33]
for feature extraction. Some methods [40, 41] use the features extractor pre-
trained with IQA datasets [13, 39]. PVQ [40] also extracts features pretrained
on action recognition dataset [16] for better perception on inter-frame
distortion. These methods are limited by their high computational cost on high
resolution videos. Additionally, without end-to-end training, fixed features
pretrained by other tasks are not optimal for extracting quality-related
information, which also limits the accuracy of quality assessment.

#### VQA Datasets

Tab. 1 shows common VQA datasets, other video datasets and their sizes. The
early VQA datasets [30, 7] are synthesized with specialized distortion and
have a very small volume. Some recent in-the-wild VQA datasets like KoNViD-1k
[12], YouTube-UGC [38] and LIVE-VQC [32] are still small compared to datasets
for other video tasks such as [16, 2, 9]. Recently, LSVQ[40], a large-scale
VQA dataset with 39,076 videos is publicly available. With end-to-end deep
learning of the proposed FAST-VQA, the video-quality-related features learnt
on large-scale LSVQ dataset can be transferred into smaller VQA datasets to
reach better performance.

Table 1: Common datasets in VQA and other video tasks. Most common VQA
datasets are to small (noted in red) to learn sufficient quality
representations independently.

Dataset | Task | Distortion Type | Size
---|---|---|---
Kinetics-400 [16] | Video Recognition | NA | 306,245
ActivityNet [2] | Video Action Localization | NA | 27,801
AVA [9] | Atomic Action Detection | NA | 386,000
CVD2014 [30] | Video Quality Assessment | Synthetic In-capture | 234
KoNViD-1k [12] | Video Quality Assessment | In-the-wild | 1,200
LIVE-VQC [32] | Video Quality Assessment | In-the-wild | 585
Youtube-UGC [38] | Video Quality Assessment | In-the-wild | 1,147
LSVQ [40] | Video Quality Assessment | In-the-wild | 39,076

#### Vision Transformers

Vision transformers [19, 34, 1, 6, 26] have shown effective on computer vision
tasks. They cut images or videos into non-overlapping patches as input and
perform self-attention operations between them. The patch-wise operations in
vision transformers naturally distinguish the edges of mini-patches and are
suitable for handling with the proposed fragments.

Figure 4: The pipeline for sampling fragments with Grid Mini-patch Sampling
(GMS), including grid partition, patch sampling, patch splicing, and temporal
alignment. After GMS, the fragments are fed into the FANet (Fig. 5).

##  3 Approach

In this section, we introduce the full pipeline of the proposed FAST-VQA
method. An input video is first sampled into fragments via Grid Mini-patch
Sampling (GMS, Sec. 3.1). After sampling, the resultant fragments are fed into
the Fragment Attention Network (FANet, Sec. 3.2) to get the final prediction
of the video’s quality. We introduce both parts in the following subsections.

###  3.1 Grid Mini-patch Sampling (GMS)

To well preserve the original video quality after sampling, we follow several
important principles when designing the sampling process for fragments. We
will illustrate the process along with these principles below.

#### Preserving global quality: uniform grid partition.

To include each region for quality assessment and uniformly assess quality in
different areas, we design the grid partition to cut video frames into uniform
grids with each grid having the same size (as shown in Fig. 4). We cut the
$t$-th video frame $\mathcal{V}_{t}$ into $G_{f}\times G_{f}$ uniform grids
with the same sizes, denoted as
$\mathcal{G}_{t}=\\{g_{t}^{0,0},..g_{t}^{i,j},..g_{t}^{G_{f}-1,G_{f}-1}\\}$,
where $g_{t}^{i,j}$ denotes the grid in the $i$-th row and $j$-th column. The
uniform grid partition process is formalized as follows.

$g_{t}^{i,j}=\mathcal{V}_{t}[\frac{i\times H}{G_{f}}:\frac{(i+1)\times
H}{G_{f}},\frac{j\times W}{G_{f}}:\frac{(j+1)\times W}{G_{f}}]$ (1)

where $H$ and $W$ denote the height and width of the video frame.

#### Preserving local quality: raw patch sampling.

To preserve the local textures (e.g. blurs, noises, artifacts) that are vital
in VQA, we select raw resolution patches without any resizing operations to
represent local textural quality in grids. We employ random patch sampling to
select one mini-patch $\mathcal{MP}_{t}^{i,j}$ of size of $S_{f}\times S_{f}$
from each grid $g_{t}^{i,j}$. The patch sampling process is as follows.

$\mathcal{MP}_{t}^{i,j}=\mathbf{S}_{t}^{i,j}(g_{t}^{i,j})$ (2)

where $\mathbf{S}_{t}^{i,j}$ is the patch sampling operation for frame $t$ and
grid $i,j$.

#### Preserving temporal quality: temporal alignment.

It is widely recognized by early works [18, 20, 40] that inter-frame temporal
variations are influential to video qualities. To retain the raw temporal
variations in videos (with $T$ frames), we strictly align the sample areas
during patch sampling operations $\mathbf{S}$ in different frames, as the
following constraint shows.

$\mathbf{S}_{t}^{i,j}=\mathbf{S}_{\hat{t}}^{i,j}~{}~{}~{}~{}~{}~{}\forall~{}0\leq
t,\hat{t}<T,~{}0\leq i,j<G_{f}$ (3)

#### Preserving contextual relations: patch splicing.

Existing works [24, 22, 8] have shown that the global scene information and
contextual information affects quality predictions. To keep the global scene
information of the original videos, we keep the contextual relations of mini-
patches by splicing them into their original positions, as the following
equation shows:

$\displaystyle\mathcal{F}_{t}^{i,j}$ $\displaystyle=\mathcal{F}_{t}[i\times
S_{f}:(i+1)\times S_{f},j\times S_{f}:(j+1)\times S_{f}]$ (4)
$\displaystyle=\mathcal{MP}_{t}^{i,j},~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}0\leq
i,j<G_{f}$

where $\mathcal{F}$ denote the spliced and temporally aligned mini-patches
after the Grid Mini-patch Sampling (GMS) pipeline, named as fragments.

###  3.2 Fragment Attention Network (FANet)

#### The Overall Framework.

Fig. 5 shows the overall framework of FANet. It uses a Swin-T with four
hierarchical self-attention layers as backbone. We also design the following
modules to adapt it to fragments well.

#### Gated Relative Position Biases.

Swin-T adds relative position bias (RPB) that uses learnable Relative Bias
Table ($\mathbf{T}$) to represent the relative positions of pixels in
attention pairs ($QK^{T}$). For fragments, however, as discussed in Fig. 3(a),
the cross-patch pairs have much large actual distances than intra-patch pairs
and should not be modeled with the same bias table. Therefore, we propose the
gated relative position biases (GRPB, Fig. 5(b)) that uses learnable real
position bias table ($\mathbf{T}^{\text{real}}$) and pseudo position bias
table ($\mathbf{T}^{\text{pseudo}}$) to replace $\mathbf{T}$. The mechanisms
of them are the same as $\mathbf{T}$ but they are learnt separately and used
for intra-patch and cross-patch attention pairs respectively. Denote
$\mathbf{G}$ as the intra-patch gate ($\mathbf{G}_{i,j}=1$ if $i,j$ are in the
same mini-patch else $\mathbf{G}_{i,j}=0$), the self-attention matrix
($M_{A}$) with GRPB is calculated as:

Figure 5: The overall framework for FANet, including the Gated Relative
Position Biases (GRPB) and Intra-Patch Non-Linear Regression (IP-NLR) modules.
The input fragments come from Grid Mini-patch Sampling (Fig. 4).

$\displaystyle B_{\text{In},(i,j)}$
$\displaystyle=\mathbf{T}^{\text{real}}_{\mathrm{FRP}(i,j)};B_{\text{Cr},(i,j)}=\mathbf{T}^{\text{pseudo}}_{\mathrm{FRP}(i,j)}$
(5) $\displaystyle M_{A}$ $\displaystyle=QK^{T}+\mathbf{G}\otimes
B_{\text{In}}+(\mathbf{1}-\mathbf{G})\otimes B_{\text{{Cr}}}$ (6)

where $\mathrm{FRP}(i,j)$ is the relative position of pair $(i,j)$ in
fragments.

#### Intra-Patch Non-Linear Regression.

As illustrated in Fig. 3(b), different mini-patches have diverse qualities due
to discontinuity between them. If we pool features from different patches
before regression, the quality representations of mini-patches will be
confused with each other. To avoid this problem, we design the Intra-Patch
Non-Linear Regression (IP-NLR, Fig. 5(c)) to regress the features via non-
linear layers ($\mathbf{R}_{\textit{NL}}$) first, and perform pooling
following the regression. Denote features as $f$, output score as $s_{pred}$,
pooling operation as $Pool(\cdot)$, the IP-NLR can be expressed as follows:

$\displaystyle s_{pred}$ $\displaystyle=Pool(\mathbf{R}_{\textit{NL}}(f))$ (7)

##  4 Experiments

In the experiment part, we conduct several experiments to evaluate and analyze
the performance of the proposed FAST-VQA model.

###  4.1 Evaluation Setup

#### Implementation Details

We use the Swin-T [27] pretrained on Kinetics-400 [16] dataset to initialize
the backbone in FANet. As Tab. 2 shows, we implement two sampling densities
for fragments: FAST-VQA (normal density) and FAST-VQA-M (lower density &
higher efficiency), and accomodate window sizes in FANet to the input sizes.
Without special notes, all ablation studies are on variants of FAST-VQA. We
use PLCC (Pearson linear correlation coef.) and SRCC (Spearman rank
correlation coef.) as metrics and use differentiable PLCC loss
$l=\frac{(1-\mathrm{PLCC}(s_{pred},s_{gt}))}{2}$ as loss function. We set the
training batch size as 16.

Table 2: Comparison of FAST-VQA and FAST-VQA-M with lower sampling density.

Methods |  Number of Frames ($T$) |  Patch Size ($S_{f}$) |  Number of Grids ($G_{f}$) |  Window Size in FANet | FLOPs | Parameters
---|---|---|---|---|---|---
FAST-VQA | 32 | 32 | 7 | (8,7,7) | 279G | 27.7M
FAST-VQA-M | 16 | 32 | 4 | (4,4,4) | 46G | 27.5M

#### Training & Benchmark Sets

We use the large-scale LSVQ${}_{\text{train}}$[40] dataset with 28,056 videos
for training FAST-VQA. For evaluation, we choose 4 testing sets to test the
model trained on LSVQ. The first two sets, LSVQ${}_{\text{test}}$ and
LSVQ${}_{\text{1080p}}$ are official intra-dataset test subsets for LSVQ,
while the LSVQ${}_{\text{test}}$ consists of 7,400 various resolution videos
from 240P to 720P, and LSVQ${}_{\text{1080p}}$ consists of 3,600 1080P high
resolution videos. We also evaluate the generalization ability of FAST-VQA on
cross-dataset evaluations on KoNViD-1k [12] and LIVE-VQC [32], two widely-
recognized in-the-wild VQA benchmark datasets.

###  4.2 Benchmark Results

Table 3: Comparison with existing methods (classical and deep) and our
baseline (Full-res Swin-T features). The 1st/2nd best scores are colored in
red and blue, respectively.

Type/ | Intra-dataset Test Sets | Cross-dataset Test Sets
---|---|---
Testing Set/ | LSVQ${}_{\text{test}}$ | LSVQ${}_{\text{1080p}}$ | KoNViD-1k | LIVE-VQC
Groups | Methods | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
Existing Classical |  BRISQUE[28] | 0.569 | 0.576 | 0.497 | 0.531 | 0.646 | 0.647 | 0.524 | 0.536
TLVQM[20] | 0.772 | 0.774 | 0.589 | 0.616 | 0.732 | 0.724 | 0.670 | 0.691
VIDEVAL[36] | 0.794 | 0.783 | 0.545 | 0.554 | 0.751 | 0.741 | 0.630 | 0.640
Existing Deep |  VSFA[22] | 0.801 | 0.796 | 0.675 | 0.704 | 0.784 | 0.794 | 0.734 | 0.772
PVQ${}_{\text{wo/ patch}}$[40] | 0.814 | 0.816 | 0.686 | 0.708 | 0.781 | 0.781 | 0.747 | 0.776
PVQ${}_{\text{w/ patch}}$[40] | 0.827 | 0.828 | 0.711 | 0.739 | 0.791 | 0.795 | 0.770 | 0.807
Full-res Swin-T[27] features | 0.835 | 0.833 | 0.739 | 0.753 | 0.825 | 0.828 | 0.794 | 0.809
FAST-VQA-M (Ours) | 0.852 | 0.854 | 0.739 | 0.773 | 0.841 | 0.832 | 0.788 | 0.810
FAST-VQA (Ours) | 0.876 | 0.877 | 0.779 | 0.814 | 0.859 | 0.855 | 0.823 | 0.844
Improvement to PVQ${}_{\text{w/ patch}}$ | +6% | +6% | +10% | +10% | +9% | +8% | +7% | +5%

In Tab. 4.2, we compare with existing classical and deep VQA methods and our
baseline, the full-resolution Swin-T with feature regression instead of end-
to-end training (denoted as ‘Full-res Swin-T features’). With its video-
quality-related representations, FAST-VQA achieves at most 10% improvement to
PVQ, the existing state-of-the-art on LSVQ${}_{\text{1080p}}$. Even the
efficient version FAST-VQA-M can outperform existing state-of-the-art. FAST-
VQA also shows significant improvement to its fixed-feature-based baseline
with the same backbone, demonstrating that the proposed new quality-retained
sampling with end-to-end training scheme for VQA is not only much more
efficient (with only 2.36% FLOPs required on 1080P videos) but also notably
more accurate (with 8.10% improvement on PLCC metric for
LSVQ${}_{\text{1080p}}$) than the existing fixed-feature-based paradigm.

###  4.3 Efficiency of FAST-VQA

Figure 6: The Performance-FLOPs curve of proposed FAST-VQA and baseline
methods. Table 4: FLOPs and running time (on GPU/CPU, average of ten runs)
comparison of FAST-VQA, state-of-the-art methods and our baseline on different
resolutions. We boldface FLOPs $\leq$ 500G and running time $\leq$ 1s.

| 540P | 720P | 1080P
---|---|---|---
Method | FLOPs(G) | Time(s) | FLOPs(G) | Time(s) | FLOPs(G) | Time(s)
VSFA[22] |  1024936.7× | 2.603/92.761 |  1818465.2× | 3.571/134.9 |  40919147× | 11.14/465.6
PVQ[40] |  1464652.5× | 3.091/97.85 |  2202979.0× | 4.143/144.6 |  58501210× | 13.79/538.4
Full-res Swin-T[27] feat. |  303210.9× | 3.226/102.0 |  535719.2× | 5.049/166.2 |  1185242.5× | 8.753/234.9
FAST-VQA (Ours) | 2791× |  0.044/9.019 | 2791× |  0.043/9.530 | 2791× |  0.045/9.142
FAST-VQA-M (Ours) | 460.165× | 0.019/0.729 | 460.165× | 0.019/0.613 | 460.165× | 0.019/0.714

To demonstrate the efficiency of FAST-VQA, we compare the FLOPs and running
times on CPU/GPU (average of ten runs per sample) of the proposed FAST-VQA
with existing deep VQA approaches on different resolutions, see Tab. 4. We
also draw the performance-FLOPs curve on LSVQ1080p and LIVE-VQC in Fig. 6. As
we can see, FAST-VQA reduces up to $210\times$ FLOPs and $247\times$ running
time than PVQ while obtaining notably better performance. The more efficient
version, FAST-VQA-M, only requires $1/1273$ FLOPs of PVQ and $1/258$ FLOPs of
our full-resolution baseline while still achieving slightly better
performance. Moreover, FAST-VQA (especially FAST-VQA-M) also runs very fast
even on CPU, which reduces the hardware requirements for the applications of
deep VQA methods. All these comparisons show the unprecedented efficiency of
proposed FAST-VQA. 111Also, RAPIQUE[35] can also infer rapidly on CPU that
requires 17.3s for 1080P videos. Yet, it is not compatible with GPU Inference
due to its handcrafted branch.

###  4.4 Transfer Learning with Video-quality-related Representations

Table 5: The finetune results on LIVE-VQC, KoNViD, CVD2014 and YouTube-UGC
datasets, compared with existing classical and fixed-backbone deep VQA
methods, and ensemble approaches of classical (C) and deep (D) branches.

Finetune Dataset/ | LIVE-VQC | KoNViD-1k | CVD2014 | LIVE-Qualcomm | YouTube-UGC
---|---|---|---|---|---
Groups | Methods | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
Existing Classical |  TLVQM[20] | 0.799 | 0.803 | 0.773 | 0.768 | 0.83 | 0.85 | 0.77 | 0.81 | 0.669 | 0.659
VIDEVAL[36] | 0.752 | 0.751 | 0.783 | 0.780 | NA | NA | NA | NA | 0.779 | 0.773
RAPIQUE[35] | 0.755 | 0.786 | 0.803 | 0.817 | NA | NA | NA | NA | 0.759 | 0.768
Existing Fixed Deep |  VSFA[22] | 0.773 | 0.795 | 0.773 | 0.775 | 0.870 | 0.868 | 0.737 | 0.732 | 0.724 | 0.743
PVQ[40] | 0.827 | 0.837 | 0.791 | 0.786 | NA | NA | NA | NA | NA | NA
GST-VQA[3] | NA | NA | 0.814 | 0.825 | 0.831 | 0.844 | 0.801 | 0.825 | NA | NA
CoINVQ[37] | NA | NA | 0.767 | 0.764 | NA | NA | NA | NA | 0.816 | 0.802
Ensemble C+D |  CNN+TLVQM[21] | 0.825 | 0.834 | 0.816 | 0.818 | 0.863 | 0.880 | 0.810 | 0.833 | NA | NA
CNN+VIDEVAL[36] | 0.785 | 0.810 | 0.815 | 0.817 | NA | NA | NA | NA | 0.808 | 0.803
Full-res Swin-T[27] features | 0.799 | 0.808 | 0.841 | 0.838 | 0.868 | 0.870 | 0.788 | 0.803 | 0.798 | 0.796
FAST-VQA-M (Ours) | 0.803 | 0.828 | 0.873 | 0.872 | 0.877 | 0.892 | 0.804 | 0.838 | 0.768 | 0.765
FAST-VQA w/o VQ-representations (Ours) | 0.765 | 0.782 | 0.842 | 0.844 | 0.871 | 0.888 | 0.756 | 0.778 | 0.794 | 0.784
FAST-VQA (ours) | 0.849 | 0.865 | 0.891 | 0.892 | 0.891 | 0.903 | 0.819 | 0.851 | 0.855 | 0.852
Improvements led by VQ-representations | +11.0% | +10.6% | +5.8% | +5.7% | +2.3% | +1.7% | +8.3% | +9.4% | +7.7% | +8.7%

FAST-VQA also makes the pretrain-finetune scheme on VQA possible with
affordable computation resources. With FAST-VQA, we can pretrain with large
VQA datasets in end-to-end manner to learn quality related features, and then
transfer to specific VQA scenarios where only small datasets are available.
Note that this manner is not applicable to current methods due to their high
computational load (as discussed in Sec. 4.3). We use LSVQ as the large
dataset and choose four small datasets representing diverse scenarios,
including LIVE-VQC (real-world mobile photography, 240P-1080P), KoNViD-1k
(various contents collected online, all 540P), CVD2014 (synthetic in-capture
distortions, 480P-720P), LIVE-Qualcomm (selected types of distortions, all
1080P) and YouTube-UGC (user-generated contents, including computer graphic
contents, 360P-2160P222Due to privacy reasons, the current public version of
YouTube-UGC is incomplete and only with 1147 videos. The peer comparison is
only for reference.). We divide each dataset into random splits for 10 times
and report the average result on the test splits. As Tab. 4.4 shows, with
video-quality-related representations, the proposed FAST-VQA outperforms the
existing state-of-the-arts on all these scenarios while obtaining much higher
efficiency. Note that YouTube-UGC contains 4K(2160P) videos but FAST-VQA still
performs well. Even without video-quality-related representations, FAST-VQA
also still achieves competitive performance, while these features steadily
improve the performance. It implies that the pretrained FAST-VQA can be able
to serve as a strong backbone that boost further downstream tasks related to
video quality.

###  4.5 Ablation Studies on fragments

For the first part of ablation studies, we prove the effectiveness of
fragments by comparing with other common sampling approaches and different
variants of fragments (Tab. 6). We keep the FANet structure fixed during this
part.

Table 6: Ablation study on fragments: comparison with resizing, cropping
(Group 1) and different variants for fragments (Group 2).

Testing Set/ | LSVQ${}_{\text{test}}$ | LSVQ${}_{\text{1080p}}$ | KoNViD-1k | LIVE-VQC
---|---|---|---|---
Methods/Metric | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
Group 1: Naive Sampling Approaches
bilinear resizing | 0.857 | 0.859 | 0.752 | 0.786 | 0.841 | 0.840 | 0.772 | 0.814
random cropping | 0.807 | 0.812 | 0.643 | 0.677 | 0.734 | 0.776 | 0.740 | 0.773
\- test with 3 crops | 0.838 | 0.835 | 0.727 | 0.754 | 0.841 | 0.827 | 0.785 | 0.809
\- test with 6 crops | 0.843 | 0.844 | 0.734 | 0.761 | 0.845 | 0.834 | 0.796 | 0.817
Group 2: Variants of fragments
random mini-patches | 0.857 | 0.861 | 0.754 | 0.790 | 0.844 | 0.845 | 0.792 | 0.818
shuffled mini-patches | 0.858 | 0.863 | 0.761 | 0.799 | 0.849 | 0.847 | 0.796 | 0.821
w/o temporal alignment | 0.850 | 0.853 | 0.736 | 0.779 | 0.823 | 0.816 | 0.764 | 0.802
fragments (ours) | 0.876 | 0.877 | 0.779 | 0.814 | 0.859 | 0.855 | 0.823 | 0.844

#### Comparing with resizing/cropping

In Group 1 of Tab. 6, we compare the proposed fragments with two common
sampling approaches: bilinear resizing and random cropping. The proposed
fragments are notably better than bilinear resizing on high-resolution
(LSVQ${}_{\text{1080p}}$) (+4%) and cross-resolution (LIVE-VQC) scenarios
(+4%). Fragments still lead to non-trivial 2% improvements than resizing on
lower-resolution scenarios where the problems of resizing is not that severe.
This proves that keeping local textures is vital for VQA. Fragments also
largely outperform single random crop as well as ensemble of multiple crops,
suggesting that retaining the uniform global quality is also critical to VQA.

#### Comparing with variants of fragments

We also compare with three variants of fragments in Tab. 6, Group 2. We prove
the effectiveness of uniform grid partition by comparing with random mini-
patches (ignore grids while sampling), and the importance of retaining
contextual relations by comparing with shuffled mini-patches. Fragments show
notable improvements than both variants. Moreover, the proposed fragments show
much better performance than the variant without temporal alignment especially
on high resolution videos, suggesting that preserving the inter-frame temporal
variations is necessary for fragments.

Table 7: Ablation study on FANet design: the effects for GRPB and IP-NLR
modules.

Testing Set/ | LSVQ${}_{\text{test}}$ | LSVQ${}_{\text{1080p}}$ | KoNViD-1k | LIVE-VQC
---|---|---|---|---
Variants/Metric | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
w/o GRPB | 0.873 | 0.872 | 0.769 | 0.805 | 0.854 | 0.853 | 0.808 | 0.832
semi-GRPB on Layer 1/2 | 0.873 | 0.875 | 0.772 | 0.809 | 0.856 | 0.851 | 0.812 | 0.838
linear Regression | 0.872 | 0.873 | 0.768 | 0.803 | 0.847 | 0.849 | 0.810 | 0.835
PrePool non-linear Regression | 0.873 | 0.874 | 0.771 | 0.805 | 0.851 | 0.850 | 0.813 | 0.834
FANet (ours) | 0.876 | 0.877 | 0.779 | 0.814 | 0.859 | 0.855 | 0.823 | 0.844

###  4.6 Ablation Studies on FANet

#### Effects of GRPB and IP-NLR

In the second part of ablation studies, we analyze the effects of two
important designs in FANet: the proposed Gated Relative Position Biases (GRPB)
and Intra-Patch Non-Linear Regression (IP-NLR) VQA Head as in Tab. 7. We
compare the IP-NLR with two variants: the linear regression layer and the non-
linear regression layers with pooling before regression (PrePool). Both
modules lead to non-negligible improvements especially on high-resolution
(LSVQ${}_{\text{1080p}}$) or cross-resolution (LIVE-VQC) scenarios. As the
discontinuity between mini-patches is more obvious in high-resolution videos,
this result suggests that the corrected position biases and regression head
are helpful on solving the problems caused by such discontinuity.

###  4.7 Reliability and Robustness Analyses

As FAST-VQA is based on samples rather than original videos while a single
sample for fragments only keeps 2.4% spatial information in 1080P videos, it
is important to analyze the reliability and robustness of FAST-VQA
predictions.

#### Reliability of Single Sampling.

We measure the reliability of single sampling in FAST-VQA by two metrics: 1)
the assessment stability of different single samplings on the same video; 2)
the relative accuracy of single sampling compared with multiple sample
ensemble. As shown in Tab. 8, the normalized std. dev. of different sampling
on a same video is only around 0.01, which means the sampled fragments are
enough to make very stable predictions. Compared with 6-sample ensemble,
sampling only once can already be 99.40% as accurate even on the pure high-
resolution test set (LSVQ${}_{\text{1080P}}$). They prove that a single sample
of fragments is enough stable and reliable for quality assessment even though
only a small proportion of information is kept during sampling.

Table 8: Assessment stability and relative accuracy of single sampling of
fragments.

Testing Set/ | LSVQ${}_{\text{test}}$ | LSVQ${}_{\text{1080p}}$ | KoNViD-1k | LIVE-VQC
---|---|---|---|---
Score Range | 0-100 | 0-100 | 1-5 | 0-100
std. dev. of Single Samplings | 0.65 | 0.79 | 0.046 | 1.07
Normalized std. dev. | 0.0065 | 0.0079 | 0.0115 | 0.0107
Relative Pair Accuracy compared with 6-samples | 99.59% | 99.40% | 99.45% | 99.52%

#### Robustness on Different Resolutions

To analyze the robustness of FAST-VQA on different resolutions, we divide the
cross-resolution VQA benchmark set LIVE-VQC into three resolution groups: (A)
1080P (110 videos); (B) 720P (316 videos); (C) $\leq$540P (159 videos) to see
the performance of FAST-VQA on different resolutions, compared with several
variants. As the results shown in Tab. 9, the proposed FAST-VQA shows good
performance ($\geq 0.80$ SRCC&PLCC) on all resolution groups and most superior
improvement than other variants on Group (A) with 1080P high-resolution
videos, proving that FAST-VQA is robust and reliable on different resolutions
of videos.

Table 9: Performance comparison on different resolution groups of LIVE-VQC
dataset.

Resolution | (A): 1080P | (B): 720P |  (C): $\leq$540P
---|---|---|---
Variants | SRCC | PLCC | KRCC | SRCC | PLCC | KRCC | SRCC | PLCC | KRCC
Full-res Swin features (Baseline) | 0.771 | 0.774 | 0.584 | 0.796 | 0.811 | 0.602 | 0.810 | 0.853 | 0.625
bilinear resizing (Sampling Variant) | 0.758 | 0.773 | 0.573 | 0.790 | 0.822 | 0.599 | 0.835 | 0.878 | 0.650
random cropping (Sampling Variant) | 0.765 | 0.768 | 0.565 | 0.774 | 0.787 | 0.581 | 0.730 | 0.809 | 0.535
w/o GRPB (FANet Variant) | 0.796 | 0.785 | 0.598 | 0.802 | 0.820 | 0.608 | 0.834 | 0.883 | 0.649
FAST-VQA (Ours) | 0.807 | 0.806 | 0.610 | 0.803 | 0.825 | 0.610 | 0.840 | 0.885 | 0.654

Figure 7: Spatial-temporal patch-wise local quality maps, where red areas
refer to low predicted quality and green areas refer to high predicted
quality. This sample video is a 1080P video selected from LIVE-VQC [32]
dataset. Zoom in for clearer view.

###  4.8 Qualitative Results: Local Quality Maps

The proposed IP-NLR head with patch-wise independent quality regression
enables FAST-VQA to generate patch-wise local quality maps, which helps us to
qualitatively evaluate what quality information can be learned in FAST-VQA. We
show the patch-wise local quality maps and the re-projected frame quality maps
for a 1080P video (from LIVE-VQC [32] dataset) in Fig. 7. As the patch-wise
quality maps and re-projected quality maps in Fig. 7 (column 2&4) shows, FAST-
VQA is sensitive to textural quality information and distinguishes between
clear (Frame 0) and blurry textures (Frame 12/24). It demonstrates that FAST-
VQA with fragments (column 3) as input is sensitive to local texture quality.
Furthermore, the qualities of the action-related areas are notably different
from the background areas, showing that FAST-VQA effectively learns the global
scene information and contextual relations in the video.

##  5 Conclusions

Our paper has shown that proposed fragments are effective samples for video
quality assessment (VQA) that better retain quality information in videos than
naive sampling approaches, to tackle the difficulties as results of high
computing and memory requirements when high-resolution videos are to be
evaluated. Based on fragments, the proposed end-to-end FAST-VQA achieves
higher efficiency ($-99.5\%$ FLOPs) and accuracy ($+10\%$ PLCC) simultaneously
than existing state-of-the-art method PVQ on 1080P videos. We hope that the
FAST-VQA can bring deep VQA methods into practical use for videos in any
resolutions.

##  6 Acknowledgement

This study is supported under the RIE2020 Industry Alignment Fund – Industry
Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-
kind contribution from the industry partner(s).

## References

  * [1] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., Schmid, C.: Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6836–6846 (October 2021)
  * [2] Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2015)
  * [3] Chen, B., Zhu, L., Li, G., Lu, F., Fan, H., Wang, S.: Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment. IEEE Transactions on Circuits and Systems for Video Technology (2021)
  * [4] Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1724–1734. ACL (2014)
  * [5] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 248–255 (2009)
  * [6] Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., Feichtenhofer, C.: Multiscale vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6824–6835 (October 2021)
  * [7] Ghadiyaram, D., Pan, J., Bovik, A.C., Moorthy, A.K., Panda, P., Yang, K.C.: In-capture mobile video distortions: A study of subjective behavior and objective algorithms. IEEE Transactions on Circuits and Systems for Video Technology 28(9), 2061–2077 (2018)
  * [8] Götz-Hahn, F., Hosu, V., Lin, H., Saupe, D.: Konvid-150k: A dataset for no-reference video quality assessment of videos in-the-wild. In: IEEE Access 9\. pp. 72139–72160. IEEE (2021)
  * [9] Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan, S., Toderici, G., Ricco, S., Sukthankar, R., Schmid, C., Malik, J.: Ava: A video dataset of spatio-temporally localized atomic visual actions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [10] Hara, K., Kataoka, H., Satoh, Y.: Learning spatio-temporal features with 3d residual networks for action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. pp. 3154–3160 (2017)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016)
  * [12] Hosu, V., Hahn, F., Jenadeleh, M., Lin, H., Men, H., Szirányi, T., Li, S., Saupe, D.: The konstanz natural video database (konvid-1k). In: Ninth International Conference on Quality of Multimedia Experience (QoMEX). pp. 1–6 (2017)
  * [13] Hosu, V., Lin, H., Sziranyi, T., Saupe, D.: Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing 29, 4041–4056 (2020)
  * [14] Kang, L., Ye, P., Li, Y., Doermann, D.: Convolutional neural networks for no-reference image quality assessment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2014)
  * [15] Kang, L., Ye, P., Li, Y., Doermann, D.: Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks. IEEE international conference on image processing (ICIP) (2015)
  * [16] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, A., Suleyman, M., Zisserman, A.: The kinetics human action video dataset. ArXiv abs/1705.06950 (2017)
  * [17] Ke, J., Wang, Q., Wang, Y., Milanfar, P., Yang, F.: Musiq: Multi-scale image quality transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5148–5157 (October 2021)
  * [18] Kim, W., Kim, J., Ahn, S., Kim, J., Lee, S.: Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
  * [19] Kolesnikov, A., Dosovitskiy, A., Weissenborn, D., Heigold, G., Uszkoreit, J., Beyer, L., Minderer, M., Dehghani, M., Houlsby, N., Gelly, S., Unterthiner, T., Zhai, X.: An image is worth 16x16 words: Transformers for image recognition at scale (2021)
  * [20] Korhonen, J.: Two-level approach for no-reference consumer video quality assessment. IEEE Transactions on Image Processing 28(12), 5923–5938 (2019)
  * [21] Korhonen, J., Su, Y., You, J.: Blind natural video quality prediction via statistical temporal features and deep spatial features. In: Proceedings of the 28th ACM International Conference on Multimedia. p. 3311–3319. MM ’20, Association for Computing Machinery, New York, NY, USA (2020)
  * [22] Li, D., Jiang, T., Jiang, M.: Quality assessment of in-the-wild videos. In: Proceedings of the 27th ACM International Conference on Multimedia. p. 2351–2359. MM ’19, Association for Computing Machinery, New York, NY, USA (2019)
  * [23] Li, D., Jiang, T., Jiang, M.: Unified quality assessment of in-the-wild videos with mixed datasets training. International Journal of Computer Vision 129(4), 1238–1257 (2021)
  * [24] Li, D., Jiang, T., Lin, W., Jiang, M.: Which has better visual quality: The clear blue sky or a blurry animal? IEEE Transactions on Multimedia 21(5), 1221–1234 (2019)
  * [25] Liao, L., Xu, K., Wu, H., Chen, C., Sun, W., Yan, Q., Lin, W.: Exploring the effectiveness of video perceptual representation in blind video quality assessment. In: Proceedings of the 30th ACM International Conference on Multimedia (ACM MM) (2022)
  * [26] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021)
  * [27] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. arXiv preprint arXiv:2106.13230 (2021)
  * [28] Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 21(12), 4695–4708 (2012)
  * [29] Mittal, A., Saad, M.A., Bovik, A.C.: A completely blind video integrity oracle. IEEE Transactions on Image Processing 25(1), 289–300 (2016)
  * [30] Nuutinen, M., Virtanen, T., Vaahteranoksa, M., Vuori, T., Oittinen, P., Häkkinen, J.: Cvd2014—a database for evaluating no-reference video quality assessment algorithms. IEEE Transactions on Image Processing 25(7), 3073–3086 (2016)
  * [31] Saad, M.A., Bovik, A.C., Charrier, C.: Blind image quality assessment: A natural scene statistics approach in the dct domain. IEEE Transactions on Image Processing 21(8), 3339–3352 (2012)
  * [32] Sinno, Z., Bovik, A.C.: Large-scale study of perceptual video quality. IEEE Transactions on Image Processing 28(2), 612–627 (2019)
  * [33] Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. p. 4278–4284. AAAI’17, AAAI Press (2017)
  * [34] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., J’egou, H.: Training data-efficient image transformers & distillation through attention. In: Proceedings of the International Conference on Machine Learning (ICML) (2021)
  * [35] Tu, Z., Chen, C.J., Wang, Y., Birkbeck, N., Adsumilli, B., Bovik, A.C.: Efficient user-generated video quality prediction. In: 2021 Picture Coding Symposium (PCS). pp. 1–5 (2021)
  * [36] Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., Bovik, A.C.: Ugc-vqa: Benchmarking blind video quality assessment for user generated content. IEEE Transactions on Image Processing 30, 4449–4464 (2021)
  * [37] Wang, Y., Ke, J., Talebi, H., Yim, J.G., Birkbeck, N., Adsumilli, B., Milanfar, P., Yang, F.: Rich features for perceptual quality assessment of ugc videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13435–13444 (June 2021)
  * [38] Yim, J.G., Wang, Y., Birkbeck, N., Adsumilli, B.: Subjective quality assessment for youtube ugc dataset. In: 2020 IEEE International Conference on Image Processing (ICIP). pp. 131–135 (2020)
  * [39] Ying, Z.a., Niu, H., Gupta, P., Mahajan, D., Ghadiyaram, D., Bovik, A.: From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality. arXiv preprint arXiv:1912.10088 (2019)
  * [40] Ying, Z., Mandal, M., Ghadiyaram, D., Bovik, A.: Patch-vq: ’patching up’ the video quality problem. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)). pp. 14019–14029 (June 2021)
  * [41] You, J.: Long short-term convolutional transformer for no-reference video quality assessment. In: Proceedings of the 29th ACM International Conference on Multimedia. p. 2112–2120. MM ’21, Association for Computing Machinery, New York, NY, USA (2021)
  * [42] You, J., Korhonen, J.: Deep neural networks for no-reference video quality assessment. In: Proceedings of the IEEE International Conference on Image Processing (ICIP). pp. 2349–2353 (2019)
  * [43] Zhang, W., Ma, K., Yan, J., Deng, D., Wang, Z.: Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology 30(1), 36–47 (2020)