# Architectural Implications of Neural Network Inference for High Data-Rate,
Low-Latency Scientific Applications

Olivia Weng
Alexander Redding  UC San Diego  Nhan Tran  Fermi National Accelerator
Laboratory  Javier Mauricio Duarte
Ryan Kastner  UC San Diego

###### Abstract

With more scientific fields relying on neural networks (NNs) to process data
incoming at extreme throughputs and latencies, it is crucial to develop NNs
with all their parameters stored on-chip. In many of these applications, there
is not enough time to go off-chip and retrieve weights. Even more so, off-chip
memory such as DRAM does not have the bandwidth required to process these NNs
as fast as the data is being produced (e.g., every 25 ns). As such, these
extreme latency and bandwidth requirements have architectural implications for
the hardware intended to run these NNs: 1) all NN parameters must fit on-chip,
and 2) codesigning custom/reconfigurable logic is often required to meet these
latency and bandwidth constraints. In our work, we show that many scientific
NN applications must run fully on chip, in the extreme case requiring a custom
chip to meet such stringent constraints.

##  I Introduction

As neural networks (NNs) become increasingly capable, disciplines outside of
computer science are looking to NNs to play a part in their research. This is
particularly prevalent in many scientific experiments, including particle
physics, electron microscopy, and X-ray diffraction [1]. As the scientific
instruments used in these experiments become more advanced, they are able to
produce data at extremely high rates, e.g., 40 TB/s [1]. Many of these fields
are grappling with how to best collect and process the data to conduct the
latest research at such extreme rates. Several are using NNs to filter out
data for post-processing later, as traditional algorithms have reached their
limits [2]. NNs are typically built in a brute force way, where researchers
add more and more parameters to increase performance, scaling them out to
enormous sizes, e.g., GPT-3 has 175 billion parameters, which amounts to 350
GB of data when stored with float16 precision [3]. However, this is not
practical when data is coming in at rates of several TB/s because current
computer architectures cannot execute such enormous NNs in time without being
extremely expensive and impractical. As a result, researchers in these
scientific fields are building edge NNs that can be executed on a machine
within the computation budget ($O(\text{ns})$) allowed by the data rate. This
has several implications for the computer architectures that need to meet
these extreme low-latency requirements:

  1. 1.

All NN parameters must fit on-chip.

  2. 2.

Fully on-chip inference often requires hardware-software codesign with
custom/reconfigurable logic to meet latency and bandwidth constraints.

In this paper, we argue for these architectural implications, evaluating on
the LHC sensor benchmark, which exhibits a 40 TB/s data rate, as a case study.
From our results, we can see that on a benchmark with the most extreme data
rate and latency requirements, on-chip inference and custom architectural NN
implementation is the necessary path forward.

Figure 1: Many scientific and edge NNs must process incoming data at a high
rate, requiring on-chip inference to process the data at least as fast as it
arrives [1, 4, 5]. This leads to extreme low-latency and high-bandwidth
requirements.

##  II Architectural Implications

###  II-A All NN parameters must fit on chip

Given the extreme bandwidth and latency requirements of many scientific
machine learning tasks [1, 4], on-chip memories such as flip-flops (FFs) and
block random access memories (BRAMs) are often the only kinds of memory with
both the bandwidth and latency capable of meeting these constraints. To
demonstrate, we present Fig. 1. We chart several scientific and IoT NNs
according to their memory bandwidth for reading NN weights and the latency
they achieve on field-programmable gate arrays (FPGAs) and application-
specific integrated circuits (ASICs), focusing first on latency on the x-axis
and memory bandwidth on the left y-axis. For example, the LHC sensor NN
requires 80 GB/s memory bandwidth since it loads 2000 1-byte weights every 25
ns. This NN has bandwidth requirements that exceed the capabilities of DRAM
(assuming DDR4’s 20 GB/s bandwidth noted by the green horizontal dashed line).
It is therefore infeasible to access weights off-chip from DRAM. As a result,
the LHC sensor NN must rely on on-chip memory technology such as BRAM (red
dashed line) and FFs (blue dashed line), which have significantly higher
bandwidth at the expense of smaller memory capacity (Tab. I).

Next, let us focus on latency on the x-axis and compute performance (in
operations/s) on the right y-axis to understand the diagonally charted lines.
To start, we make some assumptions. First, we assume the NNs for each task are
fully connected and the weights are 1 byte wide, as scientific benchmarks
often work with heavily quantized, fully connected layers. Also, these NNs
process data with batch size 1, as data is usually streamed in one sample at a
time. Second, we assume these NNs execute on a performant FPGA, e.g., a Xilinx
VU13P, using 5 000 digital signal processors (DSPs) operating at 200 MHz,
performs 1 TOP/s. Third, we assume that with 1-byte weights and 1 TOP/s, we
can handle 1 TB/s of bandwidth at a compute performance of 1 TOP/s. Based on
these assumptions, consider a model with 10 MB of weights (the diagonal pink
line). Executing this model at a latency of $10^{-5}$s requires 1 TOP/s with a
memory bandwidth of 1 TB/s (left y-axis). From these assumptions, we can
project how much compute performance and memory bandwidth a 10 MB model needs
to run at different latencies. We also project the compute and bandwidth for a
100 KB model (purple diagonal line) and a 50 MB model (red diagonal line).
Assuming that on-chip storage can only store a maximum of 50 MB of weights, we
shade the region of this chart where on-chip inference is required in red. The
floor is the maximum DRAM bandwidth and the ceiling is the 50 MB model’s
memory bandwidth and compute performance.

As such, NN implementations that fall in this shaded region must be executed
with weights fully on-chip, as demonstrated by the LHC sensor, LHC trigger,
and beam control NNs. NN implementations that reside outside this region can
instead rely on off-chip memory, e.g., the Anomaly Detection IoT NN.

Many of these scientific NN implementations will likely become more bandwidth-
intensive as scientific instruments continue to advance. This implies that the
NNs charted in Fig. 1 will continue to move up and to the left. Thus, on-chip
NN inference will continue to persist in the future.

| DRAM | BRAM | FF
---|---|---|---
Latency | 10 - 100 ns | 1 - 1.4 ns | 1 - 1.4 ns
Bandwidth | 10s GB / s | TB / s | 100s TB / s
Total capacity | GBs | MBs | 100s KBs
TABLE I: On-chip memory storage, i.e., BRAMs and FFs, offers significantly
more bandwidth and speed, making it the ideal option for our high-bandwidth,
low-latency benchmarks [6]. The tradeoff, however, is total capacity, as these
on-chip options can only store megabytes of data as opposed to DRAM’s
gigabytes.

###  II-B Fully on-chip inference often requires hardware-software codesign

To perform on-chip NN inference fast enough, researchers have developed a
couple different architectures, such as spatial dataflow style and lookup
table (LUT) based NN implementations, which heavily rely on hardware-software
codesign to meet accuracy and hardware performance requirements. Spatial
dataflow style architectures, e.g., hls4ml [7] and FINN [8], often target
FPGAs but also target ASICs [2]. These architectures implement the NN layer-
wise, dedicating fabric to each layer, dataflowing them to achieve high
throughput. To get NNs to fit entirely on chip, these designs rely on
quantizing the weights and altering the NN topology to achieve good accuracy
and hardware performance. An even more extreme architecture implements NNs
entirely using LUTs [9]. Rather than implementing multiply-accumulates (MACs),
the NN is executed entirely by using lookups. However, since LUT-based NNs
scale up exponentially ($O(2^{n})$) with respect to neuron fan-in, extreme
sparsity and quantization is required, so this method also leans heavily on
hardware-software codesign techniques to construct, train, and implement a
performant NN.

LHC Sensor Case Study: We compare the performance of the LHC sensor NN [2]
implemented across several different hardware platforms to demonstrate how for
a task with a extreme requirements (80 GB/s memory bandwidth and 25 ns latency
budget) custom hardware is needed. As seen in Tab. II, only the ASIC can meet
the 25 ns latency budget. Our FPGA design was implemented using hls4ml [7].
The results demonstrate that for one of the most latency-sensitive scientific
applications (LHC sensor), we can achieve similar performance to an ASIC by
exercising hardware-software codesign on an FPGA. This relationship is not as
feasible when developing for the traditional high-performance GPU
architectures we evaluted against.

Hardware Platform | Latency (ns) | Throughput (inf/s)
---|---|---
ASIC [2] | 25 | 40 000 000
AMD ZCU104 FPGA | 37.5* | 26 666 667
Nvidia Jetson GPU (batch size = 1) | 17 456 | 52 185
Nvidia Jetson GPU (batch size = 512) | 47 851 | 10 343 270
Nvidia V100 GPU (batch size = 1) | 10 253 | 43 185
Nvidia V100 GPU (batch size = 512) | 17 456 | 22 246 912
CPU | 1 800 000 | 550

TABLE II: LHC sensor NN benchmarked across hardware. The ASIC outperforms all
other platforms and is a must for meeting the 25 ns latency requirement [2]. *
- This latency is from simulation and presents a best case.

##  III Conclusion

Our key takeaways are: 1) To meet the extreme low-latency and high bandwidth
requirements of many scientific tasks, all NN parameters must fit on-chip. 2)
To fit all NN parameters on-chip, hardware-software codesign on custom and
reconfigurable hardware facilitates meeting the requirements of these extreme
tasks.

## References

  * [1] J. Duarte, N. Tran, B. Hawks, C. Herwig, J. Muhizi, S. Prakash, and V. J. Reddi, “Fastml science benchmarks: Accelerating real-time scientific edge machine learning,” _arXiv preprint arXiv:2207.07958_ , 2022.
  * [2] G. Di Guglielmo, F. Fahim, C. Herwig, M. B. Valentin, J. Duarte, C. Gingu, P. Harris, J. Hirschauer, M. Kwok, V. Loncar _et al._ , “A reconfigurable neural network asic for detector front-end data compression at the hl-lhc,” _IEEE Transactions on Nuclear Science_ , vol. 68, no. 8, pp. 2179–2186, 2021.
  * [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [4] Y. Wei, R. F. Forelli, C. Hansen, J. P. Levesque, N. Tran, J. C. Agar, G. Di Guglielmo, M. E. Mauel, and G. A. Navratil, “Low latency optical-based mode tracking with machine learning deployed on fpgas on a tokamak,” Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States), Tech. Rep., 2023.
  * [5] H. Borras, G. Di Guglielmo, J. Duarte, N. Ghielmetti, B. Hawks, S. Hauck, S.-C. Hsu, R. Kastner, J. Liang, A. Meza _et al._ , “Open-source fpga-ml codesign for the mlperf tiny benchmark,” _arXiv preprint arXiv:2206.11791_ , 2022.
  * [6] R. Kastner, J. Matai, and S. Neuendorffer, “Parallel programming for fpgas,” _arXiv preprint arXiv:1805.03648_ , 2018.
  * [7] J. Duarte _et al._ , “Fast inference of deep neural networks in FPGAs for particle physics,” _JINST_ , vol. 13, no. 07, p. P07027, 2018.
  * [8] Y. Umuroglu, N. J. Fraser, G. Gambardella, M. Blott, P. Leong, M. Jahre, and K. Vissers, “Finn: A framework for fast, scalable binarized neural network inference,” in _Proceedings of the 2017 ACM/SIGDA international symposium on field-programmable gate arrays_ , 2017, pp. 65–74.
  * [9] Y. Umuroglu, Y. Akhauri, N. J. Fraser, and M. Blott, “Logicnets: Co-designed neural networks and circuits for extreme-throughput applications,” in _2020 30th International Conference on Field-Programmable Logic and Applications (FPL)_. IEEE, 2020, pp. 291–297.


# Video Coding with Cross-Component Sample Offset

Han Gao, Xin Zhao, , Tianqi Liu, and Shan Liu Manuscript received XX XX, 2024;
revised XX XX, 2024. (Corresponding author: Han Gao)Han Gao, Xin Zhao, Tainqi
Liu, and Shan Liu are with Tencent America, Palo Alto, California, USA
(e-mail: {gaohan, xinzzhao, tainqitliu, shanl}@global.tencent.com)).

###### Abstract

Beyond the exploration of traditional spatial, temporal and subjective visual
signal redundancy in image and video compression, recent research has focused
on leveraging cross-color component redundancy to enhance coding efficiency.
Cross-component coding approaches are motivated by the statistical
correlations among different color components, such as those in the Y’CbCr
color space, where luma (Y) color component typically exhibits finer details
than chroma (Cb/Cr) color components. Inspired by previous cross-component
coding algorithms, this paper introduces a novel in-loop filtering approach
named Cross-Component Sample Offset (CCSO). CCSO utilizes co-located and
neighboring luma samples to generate correction signals for both luma and
chroma reconstructed samples. It is a multiplication-free, non-linear mapping
process implemented using a look-up-table. The input to the mapping is a group
of reconstructed luma samples, and the output is an offset value applied on
the center luma or co-located chroma sample. Experimental results demonstrate
that the proposed CCSO can be applied to both image and video coding,
resulting in improved coding efficiency and visual quality. The method has
been adopted into an experimental next-generation video codec beyond AV1
developed by the Alliance for Open Media (AOMedia), achieving significant
objective coding gains up to 3.5 % and 1.8 % for PSNR and VMAF quality
metrics, respectively, under random access configuration. Additionally, CCSO
notably improves the subjective visual quality.

###### Index Terms:

Video coding, in-loop filter, Alliance for Open Media (AOMedia), AV1, next-
generation video coding, cross-component sample offset (CCSO)

##  I Introduction

In recent years, the development of new-generation video coding standards by
several standardization organizations has made substantial progress. In 2018,
the AOMedia Video 1 (AV1) video coding standard [1, 2, 3, 4] was released,
with substantial bitrate reduction achieved over its predecessor VP9 [5]. The
latest Versatile Video Coding (VVC) [6, 7] standard has been released by the
Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC1/SC29/WG11
in July, 2020. The VVC standard was reported to outperform High Efficient
Video Coding (HEVC) [8, 9] by 25 % and 35 % Bjøntegaard Delta rate (BD-rate)
[10] savings under All-Intra (AI) and RA configuration [5], respectively,
using the traditional PSNR objective quality metrics. The Audio and Video
Coding Standard 3 (AVS3) [11] was finalized in 2019 by the digital audio and
video coding standard workgroup of China. In 2021, AOMedia started research
activities on exploring next-generation video codec with significantly higher
compression efficiency than AV1. A software experimental test model, known as
the AOMedia Video Model (AVM) has been developed and used for evaluating input
technical contributions related to this exploration activity.

The current mainstream video coding schemes still follow a hybrid block-based
processing framework. The prediction mode and residual coding mode for each
coding block are usually determined through Rate-Distortion Optimization
(RDO). Quantization of transform block coefficients introduces loss of
information, causing visual artifacts such as blockiness and ringing. The
quantization step is controlled by the quantization parameter (QP) value, and
it varies from 0 to 255 in AVM. The larger the quantization step, the stronger
the artifacts appear in reconstructed images.

To effectively reduce the artifacts and improve the visual quality of the
reconstructed image, in-loop filters are commonly used in the modern video
coding standards. Deblocking filter (DBF) [12, 13, 14] is a well-developed
filter to address blocking artifacts in AVC, HEVC, VVC, and AV1. The DBF
applies a low-pass filter to smooth out the block boundaries and reduce the
discontinuities. Additionally, the quantization of transform coefficients also
introduces other artifacts, such as blurriness and ringing. In HEVC, Sample
Adaptive Offset (SAO) [15] is used to address these artifacts by classifying
reconstructed pixels into different categories and applying an offset to each
category. The offset is entropy coded into bitstream to enhance the texture.
In the latest VVC standard, an Adaptive Loop Filter (ALF) [13] method is
introduced to further reduce these artifacts after applying DBF and SAO. ALF
utilizes Wiener filter to reduce the distortion between the source signals and
the reconstructed frames. In AV1, new loop filtering tools are also employed
to reduce these artifacts after DBF, including the Constrained Directional
Enhancement Filter (CDEF) [16] and Loop Restoration Filter (LRF) [17]. CDEF is
capable of identifying the edge direction in a block and applying a high-
degree non-linear filter to enhance the edges. LRF is composed of a symmetric
Wiener filter and a self-guided filter with subspace projection. Besides those
loop filtering methods introduced above, many other in-loop filtering tools
[18, 19, 20, 21] have also been studied in recent years to further enhance the
video quality.

In recent years, cross-component video coding techniques have achieved
impressive coding efficiency improvements in many modules within the hybrid
block-based video coding framework, including prediction [22], transform [23],
residual coding [24], and loop filtering [25]. The motivation of cross-
component approaches may be explored by the color spaces and chroma sampling
basics. The well-know Y’CbCr color format is linearly transformed from RGB
color format. Hence, a strong correlation between the Y, Cb, and Cr color
components can be easily identified. The Y component contains luma information
which has finer structures and details than chroma components. Moreover, in
the popular 4:2:0 Y’CbCr sampling format, the chroma components are
downsampled by a factor of two in both horizontal and vertical dimension. As a
result, each chroma component contains one quarter number of samples in the
luma component. Therefore, luma component typically contains richer
information than chroma components. Previous works on cross-component video
coding technology are mainly motivated by the aforementioned observations.

This paper introduces a novel approach called Cross-Component Sample Offset
(CCSO), which employs co-located and neighboring luma samples to derive
correction signals for both luma and chroma reconstructed samples. CCSO is
characterized as a multiplication-free, non-linear mapping process,
implemented via a look-up table. The process takes reconstructed luma samples
as input and produces offset values applied to central luma or co-located
chroma samples. Experimental results indicate that the proposed CCSO method
enhances both image and video coding, leading to superior coding efficiency
and visual quality.

The remaining of the paper is organized as follows: Section II reviews the
related studies on cross-component video coding technology and loop filtering
from both academia and industry. Section III details the proposed CCSO
technology. Section IV discusses the encoder implementation and software
optimization of CCSO. Section V shows the experimental results including
coding performance summary, ablation studies, and visual quality comparison.
Finally, Section VI concludes the work.

##  II Cross-Component Coding Techniques

###  II-A Cross-Component Prediction

Cross Component Linear Mode (CCLM) [22] is a well-studied cross-component
prediction tool leveraging inter-channel statistical redundancy. This
prediction mode utilizes a linear model to predict chroma samples based on the
reconstructed luma samples. The predicted chroma samples, denoted as
$\text{pred}_{C}(i,j)$, are computed using

$\text{pred}_{C}(i,j)=\alpha\cdot\text{rec}_{L}(i,j)+\beta,$ (1)

where $\text{rec}_{L}(i,j)$ represents the downsampled co-located
reconstructed luma samples, $i$ and $j$ denote the coordinate of the sample to
be predicted, $\alpha$ and $\beta$ denote parameters of the linear mode
derived using neighboring chroma samples and the corresponding downsampled
luma samples. These parameters are implicitly derived at both the encoder and
decoder. In [26], a CCLM mode with three linear models is proposed, and the
linear model derivation depends whether left, above or both left and above
neighboring samples are used to derive the linear model. In a subsequent work
[27], an adaptive template selection method was proposed, enabling chroma
prediction not only by the Y component but also by the Cb component or an
adaptive combination of Y and Cb. Furthermore, an improved CCLM approach is
proposed in [28], which is based on univariate polynomial regression to model
available adjacent samples of chroma blocks and predict chroma samples using
reconstructed samples in co-located luma blocks. In [29], a Multi-Directional
Linear Model was proposed including left and top versions. These advancements
enable more efficient chroma intra prediction by leveraging both cross-
component correlation and spatial correlation among image samples.

In [30], a multi-model CCLM (MM-CCLM) approach is introduced, which
incorporates multiple linear models within a coding block. In MM-CCLM,
reconstructed neighboring luma and chroma samples of the current block are
classified into several groups, and different linear models may be derived for
different groups. Additionally, in [31] and [32], an LM-angular prediction
(LAP) method is proposed, which combines traditional intra-prediction and MM-
CCLM prediction using a weighted sum as the prediction samples, exploiting
cross-component and spatial correlations simultaneously to enhance prediction
efficiency. Furthermore, in [32], a multi-filter CCLM method is introduced,
where four additional candidate down-sampling filters are employed on top of
the CCLM prediction mode, and the best filter is selected and signaled to
downsample luma samples for deriving the linear model parameters and
performing cross-component prediction.

In [33], template matching was utilized for chroma prediction by employing the
reconstructed luma block. Meanwhile, a chroma from luma (CfL) prediction mode
[34] is employed in AV1, and the prediction block is derived as the sum of the
chroma DC contribution and the scaled luma AC contribution. The DC
contribution of a block is the average value of the block, while the AC
contribution is derived by removing the DC contribution from the block. In CfL
mode, the model parameters, such as the scaling factor applied to the luma AC
contribution, are calculated during the encoding process and signaled into the
bitstream. Notably, signaling scaling parameters in the bitstream enables
encoder-only fitting of the linear model, reducing decoder complexity and
enhancing prediction accuracy. In [35], a convolutional cross-component model
(CCCM) is proposed, which predicts a chroma sample using multiple downsampled
luma samples located within a cross-shape filter centered at the co-located
luma sample. The filter coefficients used in CCCM mode are computed via least
mean square minimization between the reconstructed chroma and predicted chroma
samples in the reference area of the predicted unit.

###  II-B Cross-Component Residual Coding

Numerous studies have focused on eliminating cross-component redundancy using
linear models for residual pixel values. Kim  _et al._ [36] introduced the
Cross Component Prediction (CCP) method, predicting the chroma residual signal
from the luma residual signal. CCP dynamically switches predictors based on a
linear model to code the residuals of the second and/or the third color
components using the residuals of the first color component. This approach
subtracts the residuals of the remaining color component ($r_{CR}$) from the
residuals of the main color component ($r_{CM}^{\prime}$) before transform and
quantization, employing a linear operation expressed as

$r_{CR}^{\prime}=r_{CR}-\gamma\cdot r_{CM},$ (2)

where $\gamma$ is a weighting factor. An extension of this work [37] allows
the Cb residual to predict the Cr residual. Khairat  _et al._ [38] leveraged
the correlation between residual components in 4:4:4 format with CCP,
predicting the second and the third components from the first component in RGB
and Y’CbCr color spaces, with the model parameters signaled in the bitstream.

In [39], a Joint Coding of Chroma Residual (JCCR) coding mode was introduced
and utilized in VVC standard to further reduce redundancy that exist in the
residual signals between two chroma components. Rather than signaling the
residuals separately, one of three JCCR modes, featuring various weighting
combinations of a single-coded chroma residual, can be selectively applied at
the coding unit level. A related method was also proposed in [24] by applying
a block-wise, rotational Inter-Component Transform (ICT) on top of two
residual signals from intra or inter prediction of Cb and Cr components. By
applying inter-component transformation to joint coding of chroma residual
signals, better energy compaction is achieved.

###  II-C Cross-Component Signaling

For chroma component intra prediction, modern video coding standard, e.g.,
HEVC and VVC, offers the direct mode (DM) [hevc-DM], which utilizes the same
intra prediction mode applied to the co-located luma component for the chroma
components. This mode effectively reduces signaling overhead for the chroma
component, based on the assumption that the texture direction in chroma coding
block is similar to that of the co-located luma coding block, allowing reuse
of luma intra modes by associated chroma coding blocks. In addition, an
enhanced cross-component context modeling technique on top of the AV1 intra
mode coding scheme is proposed in [40], which signals the chroma nominal mode
is based on the co-located luma nominal mode. Additionally, leveraging the
high correlation between chroma and luma delta angles, the context for chroma
delta angles is derived from the delta angles of co-located luma blocks.

###  II-D Cross-Component Loop Filtering

Misra  _et al._ [23] present a cross component adaptive loop filter (CC-ALF),
which leverages the correlation between luma and chroma samples to enhance the
quality of chroma samples. It applies a linearly filtered version of co-
located luma samples to generate corrections for chroma samples, effectively
reducing the compression distortion. This approach extends previous ALF
methods by refining chroma samples using filtered luma samples after a single-
channel process. Unlike traditional ALF, which handles luma and chroma
channels separately, CC-ALF processes them jointly to exploit cross-component
relationships. Additionally, the luma filtering occurs concurrently with the
single-channel ALF luma filtering, avoiding increased latency and line buffer
requirements. The filter coefficients for CC-ALF are determined using a
Wiener-filter approach at the encoder side and signaled in the Adaptation
Parameter Set (APS). This filtering process, selectively enabled and disabled
across the image, aims to refine each chroma component using the corresponding
luma component, enhancing overall image quality.

In scenarios where chroma components contain abundant textures and edges,
using the reconstructed chroma samples to restore blurred textures in the luma
component can improve coding efficiency. Cross-component SAO (CCSAO) has been
proposed in [41], which utilizes collocated luma samples to classify chroma
samples and enhance the quality of reconstructed chroma samples. CCSAO reduces
sample distortion by leveraging the strong correlation between luma and chroma
components to classify reconstructed samples into different categories and
deriving one offset for each category, which is then added to the samples in
that category. The offsets for each category are derived at the encoder and
signaled in the bitstream. To maintain low complexity, only the band
information of reconstructed samples is considered for sample classification
in CCSAO.

##  III Proposed Cross-Component Sample Offset

###  III-A Motivation

Figure 1: Comparison of detailed textures between luma and chroma component;
top row: original Y (left) and Cb (right) channels, bottom row: reconstructed
Y (left) and Cb (right) channels after compression.

As illustrated in Fig. 1, for lossy coding of a camera captured image, the
luma channel preserves a significant amount of texture details, whereas the
chroma channels suffer greater information loss in structural texture
patterns. To exploit the correlations between the luma and chroma channels
during the in-loop filtering process, CC-ALF [23] was introduced in VVC. In
CC-ALF, filters are applied to the reconstructed luma sample values to enhance
the reconstructed chroma sample values, using a typical linear weighted sum of
co-located and neighboring luma reconstruction samples. However, the linear
relationship assumption of CC-ALF between luma and chroma samples may not
always hold true.

To make the cross-component loop filtering more flexible without linear
filtering limitation, a CCSO in-loop filtering method is introduced, which is
designed to improve both the luma and chroma reconstructed signals using
information from the luma reconstructed signal. Different from CC-ALF, CCSO is
featured as a non-linear filtering process. In the subsequent subsections,
technical design of CCSO will be introduced with details.

###  III-B Filtering Process

As illustrated in Fig. 2, CCSO operates concurrently with CDEF. The
reconstructed samples following deblocking are used as input for both CCSO and
CDEF. CCSO produces offset values, which are added to the reconstructed
samples of the luma and chroma components to reduce reconstruction error. This
in-loop filter arrangement offers a better balance between hardware
implementation cost and coding efficiency compared to other alternatives.

DeblockingCDEFCCSOLoop Restoration

Figure 2: Location of the proposed CCSO filter in the loop filtering pipeline
on top of the AV1 codec.

CCSO is designed for improved loop filtering on both luma and chroma
components, and the filtering process of CCSO involves three main steps.
First, the current reconstructed luma samples, i.e., the output of the
deblocking process, are classified using two types of classifiers: the edge-
offset (EO) classifier and the band-offset (BO) classifier. These classifiers
can operate jointly or individually based on indicators signaled at the frame
level. Second, the class associated with the current luma sample is used as an
index to fetch offset values from a lookup table (LUT), which is determined at
the frame level with entries selected from a limited number of predefined
values. This LUT is shared across the entire frame. Finally, the derived
offset values using the LUT and class index are added to the corresponding
luma and chroma components. A filter unit-level on/off flag (non-overlapped
$256\times 256$ luma samples) is signaled to indicate whether CCSO filtering
is applied for the associated filter unit.

###  III-C CCSO classifier

####  III-C1 BO classifier

The BO classifier utilizes only a single input luma sample. When CCSO is
applied to the luma component, the current luma sample is used in the BO
classifier. When CCSO is applied to the chroma components, the co-located luma
sample serves as the input for the BO classifier.

Additionally, the BO classifier can be used in conjunction with the EO
classifier or independently. A frame-level flag is signaled to indicate
whether the current frame uses only the BO classifier or combines both the BO
and EO classifiers. In both scenarios, the BO index ($\text{BO}_{\text{idx}}$)
is derived using the following formula:

$\text{BO}_{\text{idx}}=I_{\text{rl}}(x,y)\gg(\text{bitDepth}-{\log_{2}N_{\text{band}}}),$
(3)

where $I_{\text{rl}}$ denotes the intensity values of the reconstructed luma
samples located at $(x,y)$, bitDepth corresponds to the bit depth of the input
signal, and $N_{\text{band}}$ is the maximum allowed number of bands which are
indicated in the frame header. The supported values of $N_{\text{band}}$ are
limited to 1, 2, 4, and 8 when BO and EO are jointly applied. When BO is
applied alone, the supported values of $N_{\text{band}}$ include 1, 2, 4, 8,
16, 32, 64, and 128. The maximum allowed number of bands is signaled at the
frame level using two or three bits. The computed $\text{BO}_{\text{idx}}$
from (3) is used in the LUT to derive the offset values of CCSO.

####  III-C2 EO classifier

The EO classifier employs a three-tap filter, utilizing three reconstructed
luma samples as input. As illustrated in Fig. 3, the current luma sample (when
CCSO is applied to the luma component) or the co-located luma sample (when
CCSO is applied to the chroma components), denoted as $rl$, along with its
neighboring samples $p_{0}$ and $p_{1}$, whether adjacent or non-adjacent, are
used in this classifier.

Figure 3: The input of CCSO EO classifier are three luma reconstructed samples
$p_{0}$, $p_{1}$ and $rl$; $rc$ is the current chroma sample which applies
CCSO, and $rl$ is the co-located luma sample of $rc$.

To better preserve edges in various scenarios, the locations of the
neighboring input luma samples are adjustable. As illustrated in Fig. 4, six
filter shapes are defined by the positions of $p_{0}$ and $p_{1}$, including
four with adjacent filter taps (filters 1–4) and two with non-adjacent filter
taps (filters 5 and 6). These filter taps are switchable at the frame level,
and the selection of the filter shape is indicated by a 3-bit index. This
design of filter shapes ensures that only one top and one bottom line of
reconstructed samples are involved, thereby minimizing line buffer memory
requirements.

Figure 4: Six switchable filter shapes of CCSO, one filter shape is selected
at frame-level for each color component.

Given the intensity values of $p_{0}$, $p_{1}$ and $rl$, the EO classifier
index $\text{EO}_{\text{idx0,1}}$ as derived as following. The delta values
between $p_{0}$, $p_{1}$ and $rl$ are computed first, denoted as $m_{0}$ and
$m_{1}$, respectively, described as follows,

$m_{0}=p_{0}-rl;$ (4) $m_{1}=p_{1}-rl.$ (5)

The delta values $m_{0}$ and $m_{1}$ are further quantized into two or three
indices depending on EO quantizer types. As illustrated in Fig. 5, two types
of EO quantizers are supported. That is, the edge_clf0, which quantize $m_{0}$
and $m_{1}$ into one of three quantization indices using (6), and edge_clf1,
which quantizes them into one of two quantization indices as in (7),

$\text{EO}_{\text{idx0,1}}=\begin{cases}0,&m_{0,1}<-T,\\\ 1,&-T\leqslant
m_{0,1}\leqslant T,\\\ 2,&m_{0,1}>T;\end{cases}$ (6)
$\text{EO}_{\text{idx0,1}}=\begin{cases}0,&m_{0,1}<-T,\\\
1,&m_{0,1}\geqslant-T,\\\ \end{cases}$ (7)

where $T$ is denoted as quantization step with $T\in\\{8,16,32,64\\}$.

(a) EO quantizer edge_clf0 with three quantizaiton indices.

(b) EO quantizer edge_clf1 with two quantizaiton indices.

Figure 5: Illustration of the two proposed EO quantizer types.

###  III-D Derivation of Offsets

A combined CCSO class index $\text{CCSO}_{\text{idx}}$ is calculated using the
BO index and the EO indices as follows:

$\text{CCSO}_{\text{idx}}=(\text{BO}_{\text{idx}}\ll
4)+(\text{EO}_{\text{idx0}}\ll 2)+\text{EO}_{\text{idx1}}.$ (8)

If only BO classifier is used without EO classifier, the
$\text{EO}_{\text{idx0,1}}$ are set as zero. The $\text{CCSO}_{\text{idx}}$
will be used as the input of the offset values LUT to derive the corresponding
offset values. This LUT is adaptively generated for individual color component
inside one frame at encoder and signaled into the bitstream.

The size of of this LUT depends on the number of possible values of
$\text{CCSO}_{\text{idx}}$. For example, when the maximum number of bands is
eight, and the EO quantizer type provides three quantization intervals, the
LUT size in this case equals to $8\times 3\times 3=72$, which is the largest
possible size for BO and EO classifiers combined case. For BO classifier
standalone case, largest possible LUT size is 128, calculated as $128\times
1\times 1$.

The entries $s_{i}$ with $i=\text{CCSO}_{\text{idx}}$ of this LUT directly
reflect the offset values that will be added on the output of the CDEF. Thus,
the values are carefully selected based on experimental results and limited as

$s_{i}\in\\{0,1,-1,3,-3,7,-7,-10\\}.$ (9)

At the frame level, a truncated unary syntax element is signaled to indicate
each individual $s_{i}$. At the encoder side, the method for determining the
best offset value for a certain class is described in Section IV-A.

Algorithm 1 CCSO Syntax Design

Syntax Descriptor

frame_header()

ccso_frame_flag u(1)∗

if ccso_frame_flag then

for plane = 0 to num_planes do

ccso_enable[plane] u(1)

if ccso_enable[plane] then

ccso_bo_only[plane] u(1)

if ccso_bo_only[plane] then

max_band_log2[plane] u(3)

else

max_band_log2[plane] u(2)

quant_step_idx[plane] u(2)

filter_shape_idx[plane] u(3)

edge_clf[plane] u(1)

end if

max_band = 1 $\ll$ max_band_log2

intervals = ccso_bo_only ? 1 : ($\mathord{\sim}$edge_clf + 2)

for d_0 = 0 to intervals do

for d_1 = 0 to intervals do

for bdn = 0 to max_band do

ccso_idx = bnd $\ll$ 4 + d_0 $\ll$ 2 + d_1

offset_idx tu†

offset_lut[plane][ccso_idx]

= offsets[offset_idx]

end for

end for

end for

end if

end for

end if

  * •

∗ Fixed-length coded syntax elements.

  * •

† Truncated unary coded syntax elements.

###  III-E Applying Offsets

The offset values are separately derived for luma and chroma component. If
CCSO is enabled for luma, the CDEF reconstructed luma samples are corrected by

$rl^{\prime}(x,y)=\text{clip}(rl(x,y)+s_{\text{luma}}(x,y)),$ (10)

where $rl(x,y)$ is the luma reconstructed sample from CDEF,
$s_{\text{luma}}(x,y)$ is the derived CCSO offset values, and
$rl^{\prime}(x,y)$ is the final filtered reconstructed samples by CCSO; and
when CCSO is enabled for Cb and Cr components, the corrected chroma samples
are given by

$rc^{\prime}_{\text{cb}}(x,y)=\text{clip}(rc_{\text{cb}}(x,y)+s_{\text{cb}}(x,y)),$
(11)

and

$rl^{\prime}_{\text{cr}}(x,y)=\text{clip}(rc_{\text{cr}}(x,y)+s_{\text{cr}}(x,y)),$
(12)

where $rc_{\text{cb, cr}}(x,y)$ are reconstructed samples of CDEF,
$rc_{\text{cb,cr}}(x,y)$ are CCSO offset values, and $rc^{\prime}_{\text{cb,
cr}}(x,y)$ are final filtered samples in Cb and Cr components, respectively.

Additionally, a fixed-size non-overlapped filter unit with $256\times 256$
luma samples, i.e., $128\times 128$ chroma samples for 4:2:0 color format, are
introduced. The CCSO filter can be turned on or off at this filter unit level
for each color component, separately.

###  III-F Syntax Elements Design

The syntax elements signaling of CCSO can be categorized into frame-level and
filter-unit-level.

The frame-level signaled syntax elements are given by Algorithm 1, more
specifically,

  * •

One-bit flag ccso_frame_flag flag indicates whether CCSO is applied on any
color component of the current frame;

  * •

One-bit flag ccso_enable indicates whether CCSO is applied in the current
component of the current frame;

  * •

One-bit flag band_only_flag indicates if BO classifier is used alone or
combined with EO classifier;

  * •

Two (or three)-bits syntax max_band_log2 specifies the logarithm base two
value of the maximum number of allowed bands.

  * •

Two-bits index quant_step_idx indicates the selection of quantization step
size;

  * •

Three-bits index filter_shape_idx indicates the selection of filter shape used
in EO classifier;

  * •

One-bit flag edge_clf indicates the EO quantizer type;

  * •

For each CCSO class, offset values indices offset_idx indicate the LUT entries
from eight pre-defined values descried in (9), which are signaled using
truncated unary code (up to 7 bits for each index).

At filter-unit-level, a context coded flag is signaled to indicate whether the
CCSO filter is enabled or not for a specific filer unit. Three contexts are
added for signaling this flag, one for each color component.

##  IV Implementation of Proposed Methods

As described in Section III, the proposed CCSO method is a highly adaptive
loop filtering technique with multiple syntaxes designed to accommodate
various video content. This approach offers a substantial search space for
CCSO filtering parameters. To achieve optimal coding gain, it is essential to
develop an efficient encoder implementation of the proposed method, alongside
an effective Single Instruction/Multiple Data (SIMD) implementation for the
filtering process.

###  IV-A Encoder Search of CCSO Parameters

There are seven frame-level and one filter-unit-level CCSO filtering
parameters to be determined by encoder, as described in Section III-F. The
derivation of a group of five frame-level syntax, including ccso_bo_only,
filter_shape_idx, quant_step_idx, max_band_log2, and edge_clf, involves
evaluating different combinations of model parameters through cascaded loops.
This approach results in a substantial search space if implemented using a
brute-force search method.

For each combination of the five frame-level filtering parameters, the encoder
must determine the filter-unit-level CCSO filter enabling flag for each
filtering unit and derive the offset LUT. Given that the derivation of the
offset LUT is intertwined with determining the filter units applying CCSO, an
iterative search strategy is implemented to optimize both the filter-unit-
level enabling flags and the LUT entries.

Initially, the filter-unit-level CCSO filter enabling flag is set to one for
all filtering units, assuming that CCSO filtering will be applied to all units
in the current frame. The optimization process then proceeds iteratively
through the following steps:

Step 1: Based on the current determination of filtering units with CCSO
enabled, derive the optimal LUT by calculating the difference between the
original samples and the reconstruction samples for each combination of CCSO
classes ($\text{CCSO}_{\text{idx}}$) as given by (8).

Step 2: Using the updated LUT, compare the rate-distortion (R-D) cost of
enabling versus disabling CCSO filtering for each filtering unit. Set the CCSO
filter enabling flag according to the option with the lower R-D cost.

The above steps 1 and 2 are repeated until a maximum of 15 iterations is
reached or the accumulated filter-unit-level R-D cost for the current picture
ceases to decrease. According to the determined filter-unit-level CCSO
enabling flag, the frame-level CCSO enabling flags ccso_enable for individual
component and ccso_frame_flag for the whole frame are set.

###  IV-B SIMD Optimization

The proposed CCSO loop filtering method is highly compatible with SIMD
architecture, and an implementation of SIMD for the CCSO filtering process has
been successfully developed and validated to significantly reduce the runtime
complexity of the software implementation.

The CCSO loop filtering process consists of two primary steps:

Step 1 involves determining the index for the Lookup Table (LUT) entry, which
is calculated based on the BO and EO classes, denoted as the combination of
$\text{BO}_{\text{idx}}$, and $\text{BO}_{\text{idx0,1}}$.

Step 2 entails retrieving the offset from the LUT and adding this offset value
to the reconstruction samples.

In the CCSO encoder design, $\text{BO}_{\text{idx0}}$ and
$\text{BO}_{\text{idx1}}$ for each reconstruction sample are computed on a
frame-level basis before the application of filtering. The SIMD implementation
of step one is straightforward, as determining the LUT entry involves simply
packing the BO and EO classes indices into a single value. For step 2, the
operation of fetching the LUT offset can be efficiently executed using the
_mm256_shuffle_epi8 instruction, taking advantage of the small size of the
LUT, which comprises up to 9 entries for individual $\text{BO}_{\text{idx0}}$,
each limited to 3 bits. The runtime saving achieved by the proposed SIMD
optimization is shown in Table I. For the overall encoder runtime, average 1.2
% and 1.6 % savings are achieved for RA and LD configurations, respectively.
For the overall decoder runtime, average 5.0 % and 6.2 % savings are achieved
for RA and LD configurations, respectively.

TABLE I: SIMD Optimization Runtime Comparison (Anchor: w/o SIMD, Test w/ SIMD)

| AI | RA | LD
---|---|---|---
| Enc-T | Dec-T | Enc-T | Dec-T | Enc-T | Dec-T
A1$\\_$4k | 101.6 % | 96.4 % | 99.9 % | 93.8 % | $-$ | $-$
A2$\\_$2k | 99.5 % | 93.8 % | 98.8 % | 94.7 % | 98.5 % | 93.2 %
A3$\\_$720p | 98.6 % | 92.8 % | 98.4 % | 94.6 % | 98.0 % | 94.8 %
A4$\\_$360p | 97.9 % | 96.2 % | 98.3 % | 98.2 % | 97.8 % | 92.6 %
A5$\\_$270p | 98.9 % | 101.2 % | 98.6 % | 95.3 % | 98.1 % | 93.4 %
B1$\\_$SYN | 99.9 % | 94.7 % | 98.5 % | 95.0 % | 99.1 % | 95.0 %
AVG | 99.5 % | 94.9 % | 98.8 % | 95.0 % | 98.4 % | 93.8 %

##  V Experimental Results

###  V-A Test Condition

The proposed algorithm has been implemented based on tag v6.0.0 of AVM
reference software[42]. The common test condition (CTC) [43] specified by AOM
Testing Subgroup is used to conduct the experiment. The test set has a total
of 56 sequences, including camera captured sequences with resolutions from 4K
to 240p (class A1–A5), synthetic content sequences (class B1), and screen
content sequences (class B2). The QP settings are from 110 to 235 for RA and
LD, and from 85 to 210 for AI. BD-Rate is used to evaluate the coding gain,
which is calculated for each color component (Y/Cb/Cr) using PSNR and VMAF.
The overall BD-rate is also reported using 14/16, 1/16, and 1/16 weightings on
the quality scores of Y, Cb, and Cr components (marked as YCbCr in the results
tables), respectively. The complexity is measured by the ratio between the
encoding and decoding time of the test and anchor.

###  V-B Coding Performance and Complexity Analysis

TABLE II: Overall Coding Performance of CCSO under the AI, RA, and LD
Configurations

|  | Y | Cb | Cr | YCbCr | VMAF(Y) | Enc-Time | Dec-Time
---|---|---|---|---|---|---|---|---
AI | A1_4k |  $-$0.08% |  $-$2.24% |  $-$3.88% |  $-$0.48% |  $-$0.42% | 107% | 109%
A2_2k |  $-$0.16% |  $-$4.20% |  $-$4.68% |  $-$0.62% |  $-$0.71% | 104% | 107%
A3_720p |  $-$0.29% |  $-$4.01% |  $-$4.17% |  $-$0.65% |  $-$0.58% | 102% | 105%
A4_360p |  $-$0.25% |  $-$9.31% |  $-$5.87% |  $-$0.88% |  $-$1.26% | 101% | 109%
A5_270p | 0.02% |  $-$2.23% |  $-$1.23% |  $-$0.10% |  $-$0.50% | 102% | 107%
B1_SYN |  $-$0.45% |  $-$4.58% |  $-$4.93% |  $-$0.88% |  $-$1.10% | 104% | 109%
AVG w/o B2 |  $-$0.22% |  $-$4.37% |  $-$4.41% |  $-$0.64% |  $-$0.75% | 104% | 107%
B2_SCC |  $-$0.25% |  $-$1.33% |  $-$0.93% |  $-$0.32% |  $-$1.15% | 103% | 104%
RA | A1_4k |  $-$0.40% |  $-$3.77% |  $-$5.39% |  $-$0.91% |  $-$0.85% | 106% | 107%
A2_2k |  $-$0.32% |  $-$6.26% |  $-$5.97% |  $-$0.85% |  $-$0.70% | 103% | 104%
A3_720p |  $-$0.27% |  $-$5.28% |  $-$5.43% |  $-$0.75% |  $-$0.33% | 103% | 103%
A4_360p |  $-$0.19% |  $-$7.53% |  $-$6.53% |  $-$0.79% |  $-$0.53% | 101% | 102%
A5_270p | 0.01% |  $-$1.67% |  $-$4.04% |  $-$0.25% |  $-$1.07% | 102% | 100%
B1_SYN |  $-$0.42% |  $-$5.43% |  $-$6.09% |  $-$0.94% |  $-$0.78% | 103% | 104%
AVG w/o B2 |  $-$0.30% |  $-$5.41% |  $-$5.75% |  $-$0.81% |  $-$0.69% | 103% | 104%
B2_SCC |  $-$1.87% |  $-$2.24% |  $-$2.30% |  $-$1.92% |  $-$5.25% | 110% | 105%
LD | A1_4k | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
A2_2k |  $-$0.62% |  $-$7.42% |  $-$7.29% |  $-$1.24% |  $-$1.07% | 104% | 106%
A3_720p |  $-$0.33% |  $-$5.40% |  $-$4.79% |  $-$0.78% |  $-$0.77% | 104% | 107%
A4_360p |  $-$0.02% |  $-$7.06% |  $-$5.45% |  $-$0.55% | 0.20% | 102% | 106%
A5_270p |  $-$0.15% |  $-$4.80% |  $-$6.21% |  $-$0.58% |  $-$1.25% | 101% | 106%
B1_SYN |  $-$0.77% |  $-$6.33% |  $-$8.13% |  $-$1.40% |  $-$0.84% | 103% | 106%
AVG w/o B2 |  $-$0.48% |  $-$6.57% |  $-$6.72% |  $-$1.05% |  $-$0.82% | 103% | 106%
B2_SCC |  $-$3.22% |  $-$3.59% |  $-$4.14% |  $-$3.28% |  $-$7.99% | 109% | 105%

Table II presents the BD-rate improvement achieved by CCSO across AI, RA, and
LD configurations, respectively, wherein the anchor is AVM tag v6.0.0 with
CCSO disabled (achieved by setting runtime flag –enable-ccso=0), and the test
is AVM tag v6.0.0 with CCSO enabled (achieved by setting runtime flag –enable-
ccso=1). A negative number in Table II indicates coding gain (or bitrate
saving under the same quality) when CCSO is applied.

Overall, CCSO demonstrates a promising coding gain, exhibiting improvements of
0.64 %, 0.81 %, and 1.05 % YCbCr coding gain under AI, RA, and LD
configurations,respectively. Notably, substantial enhancements are observed in
the chroma channels, with notable gains chroma component, e.g., 5.41 % and
5.75 % Cb and Cr coding gain under RA configuration. It is worth mentioning
that for individual sequences, such as WestWindEasy in natural video content,
the maximum RA coding gains of 0.88 %, 26.36 %, and 28.83 % for the Y, Cb, and
Cr components, respectively, are achieved. For screen content sequence
Slides2r in class B2, the maximum RA coding gains of 3.45 %, 13.75 %, and
12.16 % for Y, Cb, and Cr components, respectively, are observed.

The BD-rate curves for sequence WestWindEasy is shown in Fig. 6, it is
observed that both Cb and Cr channel BD-rate curves show significantly
improved coding performance with the proposed CCSO.

$0$$2{,}000$$4{,}000$$6{,}000$$8{,}000$$40$$45$$50$bitrate(kbps)Cb-PSNR
(dB)CCSO onCCSO off (a)

$0$$2{,}000$$4{,}000$$6{,}000$$8{,}000$$40$$45$$50$bitrate(kbps)Cr-PSNR
(dB)CCSO onCCSO off (b)

Figure 6: BD-rate curves of sequence WestWindEasy from class A3 under RA
configuration tested using AVM tag v6.0.0; (a) Cb-PSNR vs bitrate curve; (b)
Cr-PSNR vs bitrate curve.

When analyzing the performance of CCSO across different video classes, such as
class A5 (270p), and class B2 (screen content), it becomes evident that CCSO
tends to exhibit relatively different coding gains compared to other test
classes. This discrepancy in performance, particularly observed in class A5,
may be attributed to the minimal block size required for implementing CCSO.
The minimal block size found in class A5 videos contains relatively more
intricate video content, which poses challenges for effective offset
compensation by CCSO. On the other hand, for the luma channel in the class B2
videos, a relatively high coding gain is observed. This may be due to the
screen content videos having less natural noise and clearer structures, which
allows CCSO to better exploit the luma offset compensation.

TABLE III: Fixed EO Classifier of CCSO Only on Chroma Components

|  | Y | Cb | Cr | YCbCr
---|---|---|---|---|---
AI | A1$\\_$4k | 0.10 % |  $-$3.51 % |  $-$3.64 % |  $-$0.33 %
A2$\\_$2k | 0.06 % |  $-$3.15 % |  $-$4.09 % |  $-$0.28 %
A3$\\_$720p | 0.05 % |  $-$3.45 % |  $-$2.47 % |  $-$0.23 %
A4$\\_$360p | 0.08 % |  $-$6.76 % |  $-$9.40 % |  $-$0.58 %
A5$\\_$270p | 0.10 % |  $-$4.37 % |  $-$2.55 % |  $-$0.21 %
B1$\\_$SYN | 0.03 % |  $-$2.58 % |  $-$2.63 % |  $-$0.18 %
AVG | 0.06 % |  $-$3.57 % |  $-$3.95 % |  $-$0.28 %
RA | A1$\\_$4k | 0.21 % |  $-$7.37 % |  $-$9.09 % |  $-$0.73 %
A2$\\_$2k | 0.13 % |  $-$6.64 % |  $-$8.25 % |  $-$0.54 %
A3$\\_$720p | 0.09 % |  $-$4.67 % |  $-$4.44 % |  $-$0.33 %
A4$\\_$360p | 0.17 % |  $-$6.60 % |  $-$7.10 % |  $-$0.48 %
A5$\\_$270p | 0.05 % |  $-$6.25 % |  $-$1.75 % |  $-$0.31 %
B1$\\_$SYN | 0.04 % |  $-$4.92 % |  $-$3.87 % |  $-$0.33 %
AVG | 0.12 % |  $-$6.10 % |  $-$6.43 % |  $-$0.47 %
LD | A1$\\_$4k | $-$ | $-$ | $-$ | $-$
A2$\\_$2k | 0.04 % |  $-$8.84 % |  $-$10.84 % |  $-$0.84 %
A3$\\_$720p | 0.12 % |  $-$5.27 % |  $-$4.56 % |  $-$0.32 %
A4$\\_$360p | 0.15 % |  $-$7.30 % |  $-$7.91 % |  $-$0.56 %
A5$\\_$270p | 0.16 % |  $-$6.89 % |  $-$0.62 % |  $-$0.17 %
B1$\\_$SYN | -0.02 % |  $-$7.09 % |  $-$5.09 % |  $-$0.52 %
AVG | 0.06 % |  $-$7.56 % |  $-$7.42 % |  $-$0.60 %

Additionally, for B2 classes, the coding gains on chroma channels are
relatively lower than natural video contents, except for the class A5. This
suggests that the performance of CCSO may vary depending on the content
characteristics, with screen content videos demonstrating unique behavior
compared to natural video content.

When comparing the relative complexity, excluding class B2, the average
encoding time increases by 4 % and decoding time by 7 % for AI, 3 % and 4 %
for RA, and 3 % and 6 % for LD configuration, respectively. The effective
encoder searching methods and SIMD implementations as described in Section
IV-A contributing the relative low complexity of the proposed CCSO method.
Upon analysis, there is a more noticeable increase in encoding times under the
RA and LD configurations for class B2. This discrepancy underscores the
importance of considering the specific video class characteristics when
assessing the performance of CCSO.

TABLE IV: BO Classifier of CCSO Only on Chroma Components

|  | Y | Cb | Cr | YCbCr
---|---|---|---|---|---
AI | A1$\\_$4k | 0.09 % |  $-$1.45 % |  $-$1.62 % |  $-$0.12 %
A2$\\_$2k | 0.07 % |  $-$1.85 % |  $-$1.70 % |  $-$0.10 %
A3$\\_$720p | 0.04 % |  $-$1.07 % |  $-$0.87 % |  $-$0.04 %
A4$\\_$360p | 0.06 % |  $-$1.53 % |  $-$0.47 % |  $-$0.03 %
A5$\\_$270p | 0.06 % |  $-$0.29 % |  $-$0.12 % | 0.04 %
B1$\\_$SYN | 0.05 % |  $-$0.76 % |  $-$0.82 % |  $-$0.04 %
AVG | 0.06 % |  $-$1.34 % |  $-$1.17 % |  $-$0.07 %
RA | A1$\\_$4k |  $-$0.17 % |  $-$1.82 % |  $-$2.01 % |  $-$0.37 %
A2$\\_$2k | 0.02 % |  $-$2.32 % |  $-$1.46 % |  $-$0.14 %
A3$\\_$720p | 0.04 % |  $-$2.11 % |  $-$0.97 % |  $-$0.10 %
A4$\\_$360p | 0.03 % |  $-$2.41 % |  $-$0.90 % |  $-$0.11 %
A5$\\_$270p | 0.04 % |  $-$0.32 % |  $-$0.40 % | 0.01 %
B1$\\_$SYN | 0.03 % |  $-$1.04 % |  $-$1.18 % |  $-$0.08 %
AVG | 0.00 % |  $-$1.84 % |  $-$1.29 % |  $-$0.14 %
LD | A1$\\_$4k | $-$ | $-$ | $-$ | $-$
A2$\\_$2k | 0.06 % |  $-$2.56 % |  $-$1.65 % |  $-$0.12 %
A3$\\_$720p | 0.03 % |  $-$2.20 % |  $-$1.07 % |  $-$0.12 %
A4$\\_$360p | 0.10 % |  $-$2.71 % |  $-$0.92 % |  $-$0.05 %
A5$\\_$270p | 0.05 % |  $-$1.00 % |  $-$1.12 % |  $-$0.04 %
B1$\\_$SYN | 0.03 % |  $-$0.98 % |  $-$1.87 % |  $-$0.10 %
AVG | 0.05 % |  $-$2.04 % |  $-$1.49 % |  $-$0.10 %

###  V-C Ablation Study

To gain deeper insights into the components of the proposed CCSO method, a
comprehensive set of ablation studies has been conducted.

In the first experiment, we isolate the impact of EO classifier (with only EO
quantizer type edge_clf0) exclusively on the chroma channel to exploring the
use cases of the EO classifier. The results, summarized in Table III, reveal
notable coding gains across different configurations. Specifically, 0.28 %,
0.47 %, and 0.60 % YCbCr overall BD-rate reduction is observed under AI, RA,
and LD configurations. These gains primarily stem from enhancements in the Cb
and Cr channels, whereas the Y channel experiences marginal coding because
CCSO is only applied to chroma in this test.

Furthermore, we conducted another experiment focusing solely on the BO
classifier on chroma components to explain the functionality of BO classifier.
The results are detailed in Table IV. Overall, BO classifier standalone
provides 1.34 % and 1.17 % coding gain for Cb and Cr components under AI
configuration, 1.84 % and 1.29 % Cb and Cr gains under RA configuration, and
2.04 % and 1.49 % Cb and Cr gains under LD configuration. The luma coding loss
is negligible because CCSO is also not applied on luma component in this test.

Lastly, we conducted an experiment to evaluate the efficiency of applying CCSO
exclusively to the luma component. Therefore, a test was designed using both
EO and BO classifiers solely on the luma channel. The simulation results,
presented in Table V, show significant coding gains for the Y component, with
0.39 %, 0.43 %, and 0.53 % luma BD-rate gain under AI, RA, and LD
configurations, respectively.

###  V-D Visual Analysis

Figure 7: Visual quality comparison of the proposed CCSO method; (a) original
images, 8th frame of WestWindEasy, 16th frame of TunnelFlag, 1st frame of
ToddlerFountain, and 1st frame of Boat; (b) CCSO applied regions of the
corresponding reconstructed Cr component, coded with QP 235 under RA
configuration; (c) selected regions (marked with box in original images) with
CCSO disabled, (d) selected regions with CCSO enabled.  TABLE V: EO and BO
Classifiers Only on Luma Component

|  | Y | Cb | Cr | YCbCr
---|---|---|---|---|---
AI | A1$\\_$4k |  $-$0.52 % | 0.10 % | 0.11 % |  $-$0.45 %
A2$\\_$2k |  $-$0.36 % | 0.12 % | 0.13 % |  $-$0.32 %
A3$\\_$720p |  $-$0.39 % | 0.15 % | 0.14 % |  $-$0.34 %
A4$\\_$360p |  $-$0.43 % | 0.17 % | 0.18 % |  $-$0.38 %
A5$\\_$270p |  $-$0.09 % | 0.28,% | 0.29 % |  $-$0.06 %
B1$\\_$SYN |  $-$0.42 % | 0.09 % | 0.09 % |  $-$0.37 %
AVG |  $-$0.39 % | 0.13 % | 0.13 % |  $-$0.34 %
RA | A1$\\_$4k |  $-$0.76 % | 0.07 % | 0.21 % |  $-$0.67 %
A2$\\_$2k |  $-$0.41 % | 0.07 % | 0.08 % |  $-$0.37 %
A3$\\_$720p |  $-$0.29 % | 0.07 % | 0.05 % |  $-$0.25 %
A4$\\_$360p |  $-$0.32 % | 0.17 % | 1.20 % |  $-$0.25 %
A5$\\_$270p | 0.05 % | 0.21 % | 0.50 % | 0.08 %
B1$\\_$SYN |  $-$0.54 % |  $-$0.17 % | 0.30 % |  $-$0.48 %
AVG |  $-$0.43 % | 0.04 % | 0.28 % |  $-$0.37 %
LD | A1$\\_$4k | $-$ | $-$ | $-$ | $-$
A2$\\_$2k |  $-$0.61 % | 0.13 % | 0.37 % |  $-$0.54 %
A3$\\_$720p |  $-$0.56 % |  $-$0.03 % | 0.08 % |  $-$0.50 %
A4$\\_$360p |  $-$0.36 % | 0.05 % |  $-$0.41 % |  $-$0.35 %
A5$\\_$270p |  $-$0.12 % | 0.71 % |  $-$0.60 % |  $-$0.09 %
B1$\\_$SYN |  $-$0.59 % | 0.25 % |  $-$0.26 % |  $-$0.53 %
AVG |  $-$0.53 % | 0.17 % | 0.02 % |  $-$0.47 %

The visual quality comparison is presented in Fig. 7. In column (a), the
source frame of the video sequence is shown. Column (b) highlights the regions
where CCSO is applied. Column (c) shows the specific regions selected from
sequences with CCSO disabled, whereas column (d) displays the same regions
from sequences with CCSO enabled. From column (b), it is observed that the
rate of applying CCSO is very high over the whole picture. Comparing columns
(c) and (d), it is evident that the application of CCSO successfully enhances
and restores the textures of chroma components. The notably sharper edges in
column (d) illustrated the effective of EO classifier, whereas the detailed
textures show the strength BO classifier. In general, it is concluded from the
analysis that, the proposed CCSO method recovers more details in the
reconstruction image.

##  VI Conclusion

A Cross-Component Sample Offset approach is proposed and implemented on top of
a developing AOMedia video codec towards a next-generation video coding
standard beyond AV1. The proposed method is featured as a non-linear
multiplication-free loop filtering method that explores cross-component
statistical redundancy to achieve higher coding efficiency and better subject
quality. Simulation results show that, under the common test conditions
specified by AOMedia, the proposed method achieved significant objective BD-
rate coding gain for several mainstream quality metrics, such as PSNR and
VMAF, and observable subjective quality enhancement, with marginal increased
complexity and hardware friendly decoder implementation.

## References

  * [1] Y. Chen, D. Murherjee, J. Han, A. Grange, Y. Xu, Z. Liu, S. Parker, C. Chen, H. Su, U. Joshi, C.-H. Chiang, Y. Wang, P. Wilkins, J. Bankoski, L. Trudeau, N. Egge, J.-M. Valin, T. Davies, S. Midtskogen, A. Norkin, and P. de Rivaz, “An Overview of Core Coding Tools in the AV1 Video Codec,” in _2018 Picture Coding Symposium (PCS)_ , 2018, pp. 41–45.
  * [2] J. Han, B. Li, D. Mukherjee, C.-H. Chiang, A. Grange, C. Chen, H. Su, S. Parker, S. Deng, U. Joshi, Y. Chen, Y. Wang, P. Wilkins, Y. Xu, and J. Bankoski, “A Technical Overview of AV1,” _Proceedings of the IEEE_ , vol. 109, no. 9, pp. 1435–1462, 2021.
  * [3] Y. Chen, D. Mukherjee, J. Han, A. Grange, Y. Xu, S. Parker, C. Chen, H. Su, U. Joshi, C.-H. Chiang, Y. Wang, P. Wilkins, J. Bankoski, L. Trudeau, N. Egge, J.-M. Valin, T. Davies, S. Midtskogen, A. Norkin, P. de Rivaz, and Z. Liu, “An Overview of Coding Tools in AV1: the First Video Codec from the Alliance for Open Media,” _APSIPA Trans. on Signal and Information Processing_ , vol. 9, no. 1, pp. –, 2020. [Online]. Available: http://dx.doi.org/10.1017/ATSIP.2020.2
  * [4] P. de Rivaz and J. Haughton, “AV1 Bitstream & Decoding Process Specification,” https://aomediacodec.github.io/av1-spec/av1-spec.pdf, accessed: 2023-08-01.
  * [5] X. Zhao, S. Liu, L. Zhao, X. Xu, B. Zhu, and X. Li, “A comparative study of HEVC, VVC, VP9, AV1 and AVS3 video codecs,” vol. 11510, pp. 192–200, 2020.
  * [6] B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (VVC) standard and its applications,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 31, no. 10, pp. 3736–3764, 2021.
  * [7] _Versatile video coding_ , ITU-T. Rec. H.266, ISO/IEC 23090-3, version 3, 2023.
  * [8] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 22, no. 12, pp. 1649–1668, Dec. 2012.
  * [9] _High efficiency video coding_ , ITU-T. Rec. H.265 and ISO/IEC 23008-2:2013, version 1, 2013.
  * [10] G. Bjøntegaard, “Calculation of average PSNR differences between RD-curves,” ITU-T SG16/Q6, Austin, TX, USA, 13th meeting, document VCEG-M33, Apr. 2001.
  * [11] J. Zhang, C. Jia, M. Lei, S. Wang, S. Ma, and W. Gao, “Recent development of AVS video coding standard: AVS3,” in _2019 picture coding symposium (PCS)_. IEEE, 2019, pp. 1–5.
  * [12] A. Norkin, G. Bjontegaard, A. Fuldseth, M. Narroschke, M. Ikeda, K. Andersson, M. Zhou, and G. Van der Auwera, “HEVC deblocking filter,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 22, no. 12, pp. 1746–1754, 2012.
  * [13] M. Karczewicz, N. Hu, J. Taquet, C.-Y. Chen, K. Misra, K. Andersson, P. Yin, T. Lu, E. François, and J. Chen, “VVC in-loop filters,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 31, no. 10, pp. 3907–3925, 2021.
  * [14] Y. Chen, D. Murherjee, J. Han, A. Grange, Y. Xu, Z. Liu, S. Parker, C. Chen, H. Su, U. Joshi _et al._ , “An overview of core coding tools in the AV1 video codec,” in _2018 Picture Coding Symposium (PCS)_. IEEE, 2018, pp. 41–45.
  * [15] C.-M. Fu, C.-Y. Chen, Y.-W. Huang, and S. Lei, “Sample adaptive offset for HEVC,” in _2011 IEEE 13th International Workshop on Multimedia Signal Processing_. IEEE, 2011, pp. 1–5.
  * [16] S. Midtskogen and J.-M. Valin, “The AV1 constrained directional enhancement filter (CDEF),” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 1193–1197.
  * [17] D. Mukherjee, S. Li, Y. Chen, A. Anis, S. Parker, and J. Bankoski, “A switchable loop-restoration with side-information framework for the emerging AV1 video codec,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 265–269.
  * [18] D. Ding, G. Chen, D. Mukherjee, U. Joshi, and Y. Chen, “A CNN-based in-loop filtering approach for AV1 video codec,” in _2019 Picture Coding Symposium (PCS)_. IEEE, 2019, pp. 1–5.
  * [19] D. Ma, F. Zhang, and D. R. Bull, “MFRNet: a new CNN architecture for post-processing and in-loop filtering,” _IEEE J. Sel. Topics in Signal Process._ , vol. 15, no. 2, pp. 378–387, 2020.
  * [20] T. Shao, T. Liu, D. Wu, C.-Y. Tsai, Z. Lei, and I. Katsavounidis, “PTR-CNN for in-loop filtering in video coding,” _Journal of Visual Communication and Image Representation_ , vol. 88, p. 103615, 2022.
  * [21] T. Liu, H. Huang, Z. Lei, R. Fang, and D. Wu, “Texture and motion aware perception in-loop filter for AV1,” _Journal of Visual Communication and Image Representation_ , vol. 98, p. 104025, 2024.
  * [22] S. H. Lee and N. I. Cho, “Intra prediction method based on the linear relationship between the channels for YUV 420 intra coding,” in _2009 16th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2009, pp. 1037–1040.
  * [23] K. Misra, F. Bossen, and A. Segall, “On cross component adaptive loop filter for video compression,” in _2019 Picture Coding Symposium (PCS)_. IEEE, 2019, pp. 1–5.
  * [24] C. Rudat, C. R. Helmrich, J. Lainema, T. Nguyen, H. Schwarz, D. Marpe, and T. Wiegand, “Inter-component transform for color video coding,” in _2019 Picture Coding Symposium (PCS)_. IEEE, 2019, pp. 1–5.
  * [25] X. Zhao, L. Zhao, M. Krishnan, Y. Du, S. Liu, D. Mukherjee, Y. Xu, and A. Grange, “Study on coding tools beyond AV1,” in _2021 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2021, pp. 1–6.
  * [26] X. Zhang, C. Gisquet, E. Francois, F. Zou, and O. C. Au, “Chroma intra prediction based on inter-channel correlation for HEVC,” _IEEE Trans. Image Process._ , vol. 23, no. 1, pp. 274–286, 2013.
  * [27] T. Zhang, X. Fan, D. Zhao, and W. Gao, “Improving chroma intra prediction for HEVC,” in _2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_. IEEE, 2016, pp. 1–6.
  * [28] W. Li, X. Jiang, T. Song, and D. Han, “Adaptive Cross Component Linear Model for Chroma Intra-Prediction in VVC,” in _Proceedings of the 10th International Conference on Communications and Broadband Networking_ , 2022, pp. 52–59.
  * [29] L. Zhang, W.-J. Chien, J. Chen, X. Zhao, and M. Karczewicz, “Multiple direct mode for intra coding,” in _2017 IEEE Visual Communications and Image Processing (VCIP)_ , 2017, pp. 1–4.
  * [30] K. Zhang, J. Chen, L. Zhang, X. Li, and M. Karczewicz, “Multi-model based cross-component linear model chroma intra-prediction for video coding,” in _2017 IEEE Visual Communications and Image Processing (VCIP)_. IEEE, 2017, pp. 1–4.
  * [31] X. Li, R.-L. Liao, J. Chen, and Y. Ye, “EE2-1.2: On chroma intra prediction,” in _JVET-Z0051_. Joint Video Experts Team (JVET), 2022.
  * [32] K. Zhang, J. Chen, L. Zhang, X. Li, and M. Karczewicz, “Enhanced cross-component linear model for chroma intra-prediction in video coding,” _IEEE Trans. Image Process._ , vol. 27, no. 8, pp. 3983–3997, 2018.
  * [33] C. Yeo, Y. H. Tan, Z. Li, and S. Rahardja, “Chroma intra prediction using template matching with reconstructed luma components,” in _2011 18th IEEE International Conference on Image Processing_. IEEE, 2011, pp. 1637–1640.
  * [34] L. Trudeau, N. Egge, and D. Barr, “Predicting chroma from luma in AV1,” in _2018 Data Compression Conference_. IEEE, 2018, pp. 374–382.
  * [35] P. Astola, J. Lainema, R. G. Youvalari, A. Aminlou, and K. Panusopone, “Convolutional cross-component model (CCCM) for intra prediction,” in _JVET-Z0064_. Joint Video Experts Team (JVET), 2022.
  * [36] W.-S. Kim, W. Pu, A. Khairat, M. Siekmann, J. Sole, J. Chen, M. Karczewicz, T. Nguyen, and D. Marpe, “Cross-component prediction in HEVC,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 30, no. 6, pp. 1699–1708, 2015.
  * [37] T. Nguyen, A. Khairat, D. Marpe, M. Siekmann, and T. Wiegand, “Extended cross-component prediction in HEVC,” in _2015 Picture Coding Symposium (PCS)_. IEEE, 2015, pp. 164–168.
  * [38] A. Khairat, T. Nguyen, M. Siekmann, D. Marpe, and T. Wiegand, “Adaptive cross-component prediction for 4: 4: 4 high efficiency video coding,” in _2014 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2014, pp. 3734–3738.
  * [39] B. Bross, J. Chen, J.-R. Ohm, G. J. Sullivan, and Y.-K. Wang, “Developments in international video coding standardization after AVC, with an overview of versatile video coding (VVC),” _Proceedings of the IEEE_ , vol. 109, no. 9, pp. 1463–1493, 2021.
  * [40] Y. Jin, L. Zhao, X. Zhao, S. Liu, and A. C. Bovik, “Improved intra mode coding beyond AV1,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 1580–1584.
  * [41] C.-W. Kuo, X. Xiu, Y.-W. Chen, H.-J. Jhu, W. Chen, N. Yan, and X. Wang, “Cross-component sample adaptive offset,” in _2022 Data Compression Conference (DCC)_. IEEE, 2022, pp. 359–368.
  * [42] “AVM tag research-v6.0.0,” https://gitlab.com/AOMediaCodec/avm/-/tree/research-v6.0.0, accessed: 2024-05-31.
  * [43] X. Zhao, Z. R. Lei, A. Norkin, T. Daede, and A. Tourapis, “AV2 Common Test Conditions v5.0, Document: CWG-D103,” 2023.