###### Abstract

Although the studies of tensor structure of the Higgs boson interactions with
vector bosons and fermions at CMS and ATLAS experiments have established that
the JPC quantum numbers of the Higgs boson should be 0++, small CP violation
in the Higgs sector (up to 10% contribution of the CP-odd state) cannot be
excluded with the current experimental precision. We review possibilities to
measure CP violating mixing angle $\mathrm{\Psi_{CP}}$ between scalar and
pseudoscalar states, at a linear electron-positron collider, at center-of-mass
energy of 1 TeV.

Talk presented at
the International Workshop on Future Linear Colliders (LCWS 2023), 15-19 May
2023.
C23-05-15.3.
This work was carried out in the framework of the ILD concept group

##  1 Introduction

In this study we present a method for the determination of the CP violating
(CPV) mixing angle between scalar and pseudoscalar CP states. We assume that
the SM-like Higgs boson 125 GeV mass eigenstate ($\mathrm{h_{125}}$) is a
superposition of a scalar (CP-even) state H and a pseudoscalar (CP-odd) state
A, via a mixing angle $\mathrm{\Psi_{CP}}$:

$h_{125}=H\cdot cos\Psi_{CP}+A\cdot sin\Psi_{CP}$ (1)

A common framework for interpretation of CPV results from future Higgs
$e^{-}e^{+}$ factories (both linear and circular), as well as from LHC and HL-
LHC is given in the Snowmass CPV white paper [1] for measurements based on
angular observables and Effective Field Theory (EFT). The benchmark CPV
parameter $f_{CP}$ quantifies contributions from CP-even and CP-odd amplitudes
(here in example of a $H\rightarrow X$ decay) as:

$f_{CP}=\frac{\Gamma_{H\rightarrow X}^{CP^{odd}}}{\Gamma_{H\rightarrow
X}^{CP^{odd}}+\Gamma_{H\rightarrow X}^{CP^{even}}}$ (2)

The sensitivity target for factor $f_{CP}$ is set from theory assuming 68% CL
measurement of the mixed Higgs state with up to 10% CP-odd contribution. The
estimates of $f_{CP}$ in the Higgs-vector boson vertices (HVV) are obtained
from EFT theory in the Higgsstrahlung process. This is the first study
addressing the $f_{CP}$ ($\Psi_{CP}$) determination in HVV vertices in Vector
Boson Fusion. CPV in HVV vertices occurs at loop level, differently from
Higgs-fermion vertices where it is realized at the Born level. As can be seen
in Fig. 1 [1], the precision target for factor $f_{CP}$ in HVV vertices is set
to $<10^{-5}$, assuming up to 10% contribution of the CP-odd state.

Figure 1: Current estimates and theoretical target for $f_{CP}$ for different
Higgs boson vertices at present and future colliders.

The ILC (International Linear Collider) is a mature option for a linear
$e^{-}e^{+}$ collider designed to operate from 250 GeV up to 500 GeV [2] in
the center-of-mass, with projected integrated luminosities of 2
$\mathrm{ab^{-1}}$ and 4 $\mathrm{ab^{-1}}$, respectively. It offers
possibility for upgrade to 1 TeV. Beam acceleration will be realized in
superconducting accelerating cavities [3]. The baseline design includes
polarization scenario for $e^{-}$ and $e^{+}$ beams of 80% and 30%,
respectively. We have considered 1 $\mathrm{ab}^{-1}$ of data simulated either
in the full ILD detector simulation (Mokka [4]) or fast ILD detector
simulation (DELPHES [5]) with ILC-gen card, in ZZ-fusion at 1 TeV center-of-
mass energy. Due to the fact that ZZ-fusion is a t-channel process $e^{\pm}$
final states are emmited at low polar angles. In the central tracker it will
be found about 42% of initially produced signal events. However, 1 TeV is
found to be the optimal energy for this measurement in comparison to other
center-of-mass energies, due to the most optimal interplay of centrality
(which decreases with energy) and Higgs boson production cross-section (which
increases with energy). As neutral current processes favor
$e^{-}_{L}e^{+}_{R}$ and $e^{-}_{R}e^{+}_{L}$ polarisation of the initial
state, measurements can be combined to reduce uncertainties. This is planned
to be done, however at this instant we present results obtained with 100%
$e^{-}_{L}e^{+}_{R}$ beam polarization.

The ILD detector model [6] is assumed in this analysis. All detector
subsystems are placed within a magnetic field of B = 3.5 T. ILD detector
comprises: all-silicon vertex detector, gaseous (Time Projection Chamber) as a
central tracker and compact Electromagnetic (ECAL) and Hadronic (HCAL)
calorimeters. Apart from TPC, the tracking system comprises: two barrel
components (the Silicon Inner Tracker SIT and the Silicon External Tracker
SET), one End cap Tracker (ETD) and the Forward Tracker (FTD), covering
between 140 mrad and 3000 mrad in polar angles. Excellent performances of the
tracking system enable measurement of transverse momenta ($\mathrm{p_{T}}$)
with an asymptotic resolution of $\sigma{(1/p_{T})}$ $\sim$ 2 $\cdot$ 10-5
GeV-1 [6]. Particle reconstruction and identification relies on Particle Flow
Algorithm (PFA) [7] using information from all detector subsystems. For
example, PFA provides separation of jets that originate from Higgs boson and
vector bosons ($\mathrm{W^{\pm},Z^{0}}$) with 3-5% jet energy resolution [6].

##  2 CPV sensitive observable

This analysis exploits the CPV-sensitive angular observable ($\Delta\Phi$),
defined as the angle between production $e^{-}$ and $e^{+}$ planes in the
Higgs rest frame, as illustrated in Fig. 2. One should note that the angle
$\Delta\Phi$ is not the only CPV sensitive angular observable, but, as is
shown in [8], it carries the most information on the CP nature of a Higgs
boson.

Figure 2: Schematic view of production planes in ZZ-fusion and the angle
$\Delta\Phi$ between them.

$\Delta\Phi$ is calculated according to the following definition:

$\Delta\Phi=\begin{cases}arccos(cos(\Delta\Phi)),\mathrm{sgn}(sin(\Delta\Phi))\geq
0\\\ 2\pi-arccos(cos(\Delta\Phi)),\mathrm{sgn}(sin(\Delta\Phi))\leq 0\\\
\end{cases}$ (3)

where

$cos(\Delta\Phi)=\overrightarrow{n}_{1}\cdot\overrightarrow{n}_{2}\hskip
28.45274pt\mathrm{and}\hskip
28.45274pt\mathrm{sgn}(sin(\Delta\Phi))=\frac{\overrightarrow{q}_{1}\cdot(\overrightarrow{n}_{1}\times\overrightarrow{n}_{2})}{|\overrightarrow{q}_{1}\cdot(\overrightarrow{n}_{1}\times\overrightarrow{n}_{2})|}$
(4)

Unit vectors $\overrightarrow{n}_{1}$ and $\overrightarrow{n}_{2}$ are defined
by momenta of initial $\overrightarrow{q}_{{e}^{-(+)}_{i}}$ and final state
$\overrightarrow{q}_{{e}^{-(+)}_{f}}$ electron (positron) and they are
orthogonal to the corresponding production planes:

$\overrightarrow{n}_{1}=\frac{q_{e^{-}_{i}}\times
q_{e^{-}_{f}}}{|q_{e^{-}_{i}}\times q_{e^{-}_{f}}|}\hskip
28.45274pt\mathrm{and}\hskip
28.45274pt\overrightarrow{n}_{2}=\frac{q_{e^{+}_{i}}\times
q_{e^{+}_{f}}}{|q_{e^{+}_{i}}\times q_{e^{+}_{f}}|}$ (5)

In Eq. (5) $\overrightarrow{q}_{1}$ stands for momentum of $Z^{0}$ boson in
the electron plane. Sign of the $sin(\Delta\Phi)$ from Eq. (4) actually shows
if the positron plane rotates forward ($\mathrm{sgn}(sin(\Delta\Phi))$ = +1)
or backward ($\mathrm{sgn}(sin(\Delta\Phi))$ = -1) with respect to the
electron plane following the right-handed rule with $Z^{0}$ boson in the
electron plane emitted in the direction of a right-hand thumb.

##  3 Event samples

The exclusive $H\rightarrow b\bar{b}$ final state is considered, to avoid high
cross-section radiative processes ( $e^{+}e^{-}\rightarrow e^{+}e^{-}\gamma$).
In case of signal event samples, about 3,500 events are fully simulated with
the ILD detector and in the presence of the beam induced backgrounds. This
statistics correspond to $\sim$ 1/5 of expected number of events in 1
$\mathrm{ab^{-1}}$ in full physical range of polar angles. About 28,000 events
are simulated with Delphes V3.4.2 fast simulation, with ILD detector model.
The relevant background (Table 1) is also fully simulated.

Table 1:  List of signal and background processes at 1 TeV with corresponding cross section $\sigma$, expected number of events in 1 ab-1 $N_{evt}$, number of simulated events $N_{sim}$ and number of selected events in 1 ab-1 of data $N_{sel}$.  1 TeV | $\sigma(fb)$ | $N_{\mathrm{evt}}@1\mathrm{ab^{-1}}$ |  $N_{\mathrm{sim}}$ |  $N_{\mathrm{sel}}@1\mathrm{ab^{-1}}$
---|---|---|---|---
Signal: $e^{-}e^{+}\rightarrow Hee;H\rightarrow b\bar{b}$ | 16 | 16016/8231tracker |  27911 DELPHES, 3495 MC |  5658
$e^{-}e^{+}\rightarrow q\bar{q}l^{+}l^{-}(l=\mu,\tau)^{\dagger}$ | 255 | 255000 |  5886(1/43) |  $/$
$e^{-}e^{+}\rightarrow q\bar{q}$ | 9375 | 9375000 |  120343(1/78) |  $/$
$e^{-}e^{+}\rightarrow q\bar{q}l\nu^{\dagger}$ | 4116 | 4116000 |  955058(1/4) |  $/$

${\dagger}$ Currently there is only simulation for $l=\mu,\tau$. Electron
state will be additionally added.

1.5(2.3, -2.7) ILD preliminary 1.5(8., -2.7) ILD preliminary

Figure 3: Stacked histogram of the $\Delta\Phi$ distribution for signal and
background, after the preselection (left) and after the overall selection
(right), for 1 $\mathrm{ab^{-1}}$ of data, in the tracking system of polar
angles.

1.5(8.5, -2.7) ILD preliminary

Figure 4: Unbiased event selection for the pure scalar state (left).
$\Delta\Phi$ distributions from the reconstructed data (green) and at the
generator level (blue), where the reconstruction is performed with Delphes
(right).

1.5(1.5, -2.7) ILD preliminary 1.5(7.5, -2.7) ILD preliminary

Figure 5: $\Delta\Phi$ distribution at the generator level in full physical
range (black line) and in the tracker (red line) (left). Comparison of
$\Delta\Phi$ distributions at the generator level in full physical range
(black line) and after correction for the central tracker acceptance effects
(green line) (right).

By applying electron isolation in Delphes, events with exactly one isolated
electron and one isolated positron are preselected. Rest of the particles are
clustered in 2 jets by the Durham algorithm [9]. After preselection, about
1170 background events are left in 1 $\mathrm{ab^{-1}}$ of data (Fig. 3
(left)). In general, $\Delta\Phi$ distribution for background is flat as it is
CP-insensitive, though it is not obvious from Fig. 3 (left) where limitted
number of background events is selected with large scaling factors. Further
event selection requires: cut on di-jet invariant mass: 80 GeV
$<m_{q\bar{q}}<$ 160 GeV, cut on Z boson masses: $m_{Z_{1},Z_{2}}>$ 30 GeV,
cut on transverse momentum of final state system electron and positron:
$p_{T_{ee}}>$ 15 GeV, cut on missing transverse momentum: $p_{T_{miss}}<$ 150
GeV. After applying all these criteria background is fully suppressed while
the total signal efficiency, including preselection, is about 68%.
$\Delta\Phi$ for signal after full event selection is given in the Fig. 3
(right). Event selection doesn’t bias $\Delta\Phi$ distribution and that is
illustrated in the Fig. 4 (left).

In order to quantify the impact of polar angle acceptance and detector
reconstruction effects on the $\Delta\Phi$ observable, Whizard generator
version 2.8.3 [10] is used with the Higgs characterization model [11] within
the UFO framework. As can be seen from Fig. 5 (left), acceptance limited by
the tracker polar angles affects the $\Delta\Phi$ distribution with respect to
the full physical range. On the other hand, comparing generated and
reconstructed information in the tracker, it is obvious that detector effects
are negligible as is illustrated in the Fig. 4 (right). Thus reconstructed
information has to be corrected for the acceptance effects. Corrected
$\Delta\Phi$ distribution is illustrated in Fig. 5 (right). It reasonably
reproduces distribution at the generator level in the full physical range, up
to statistical fluctuations.

##  4 $\mathrm{\Psi_{CP}}$ determination

Unlike $H\rightarrow\tau^{+}\tau^{-}$ decays where the shape of the angular
observable is derived from the differential cross-section [12], dependence of
$\Delta\Phi$ on CP violating mixing angle $\mathrm{\Psi_{CP}}$ is not known.
Due to that we applied a phenomenological approach in extraction of
$\mathrm{\Psi_{CP}}$ from simulated (or experimental) data. We have assumed
CP-odd admixture up to 17%, corresponding to $\mathrm{\Psi_{CP}}$ values up to
0.2 rad. $\Delta\Phi$ distributions are changing for different assumptions on
$\mathrm{\Psi_{CP}}$ value, in a way that the local minimum gets shifted
accordingly (to the left or right for negative or positive
$\mathrm{\Psi_{CP}}$ values, respectively). This is illustrated in Fig. 6
(left and right)).

Figure 6: $\Delta\Phi$ distribution for negative $\mathrm{\Psi_{CP}}$ values
(left) and positive $\mathrm{\Psi_{CP}}$ values (right), illustrates the
corresponding shift of the local minimum.

We perform a fit around local minimum with the fit function to determine its
position:

$f(\Delta\Phi,\mathrm{\Psi_{CP}})=A+B\cdot cos(a\cdot\Delta\Phi-b)$ (6)

where $\mathrm{A,B},a$ and $b$ are free parameters, and the ratio of
coefficients $b$ and $a$ gives the minimum of the $\Delta\Phi$ distribution.
Example of such a fit performed on 105 generated events for
$\mathrm{\Psi_{CP}}$ = 0.1 is illustrated in Fig. 7 (left). Position of the
minimum ($b/a$) over $\mathrm{\Psi_{CP}}$ true value is a linear function of
the true value of $\mathrm{\Psi_{CP}}$. This is illustrated in Fig. 7 (right).

Figure 7: Fit of $\Delta\Phi$ distribution around local minimum for
$\mathrm{\Psi_{CP}}$ = 0.1 in order to determine position of the minimum
(left). Dependence of the local minimum position $b/a$ on true
$\mathrm{\Psi_{CP}}$ values (right).

Thus the following equation holds:

$(b/a)/\mathrm{\Psi_{CP}}=k\cdot\mathrm{\Psi_{CP}}+m$ (7)

Coefficients $k$ and $m$ can be determined from simulation assuming certain
range of $\mathrm{\Psi_{CP}}$ values, while minimum can be measured from the
(experimental) data as is illustrated in Fig. 7 (left). Thus
$\mathrm{\Psi_{CP}}$ can be determined from Eq. (7) by solving this quadratic
equation. Dependence of the extracted values $\mathrm{\Psi_{exp}}$ versus the
true values $\mathrm{\Psi_{true}}$ is given in Fig. 8. For the
$\mathrm{\Psi_{true}}\leq 0.1$ corresponding to up to 9% admixture of the CP-
odd state differences between $\mathrm{\Psi_{true}}$ and $\mathrm{\Psi_{exp}}$
central values is not larger than 5 mrad.

Figure 8: Dependence of extracted $\mathrm{\Psi_{exp}}$ values with respect to
the true ones ($\mathrm{\Psi_{true}}$).

##  5 $\mathrm{\Psi_{CP}}$ from reconstructed data - a pseudo-experiment

Figure 9: Fit of $\Delta\Phi$ local minimum on the corrected reconstructed
data for pure scalar state. Figure 10: Statistical dissipation of the
extracted $\mathrm{\Psi_{CP}}$ value with respect to the
$\mathrm{\Psi_{true}}$ = 0, in 2,000 repeated pseudo-experiments each with 1
$\mathrm{ab^{-1}}$ of 100% $e^{-}_{L}e^{+}_{R}$ polarized data.

Using the parameters ($k$ and $m$) of the linear fit from simulation (Fig. 7
(right)), measurement of the local minimum from the reconstructed data (Fig.
9) gives the value of $\mathrm{\Psi_{CP}}$ = 0.9 mrad for the pure scalar
state. In order to estimate statistical dissipation of the extracted
$\mathrm{\Psi_{CP}}$, we run 2000 pseudo-experiments for the pure scalar
state, each experiment with about 1 $\mathrm{ab^{-1}}$ of generated data. This
is illustrated in Fig. 10. Absolute statistical uncertainty
($\Delta\mathrm{\Psi_{CP}}$) of extracted $\mathrm{\Psi_{CP}}$ value is
determined by RMS of the distribution from Fig. 10 reflecting the
probabilistic nature of the analized sample in the context of statistical
population. RMS is found to be 7 mrad. $\Delta\mathrm{\Psi_{CP}}$ value of 7
mrad corresponds to 68% CL uncertainty to measure $\mathrm{\Psi_{CP}}$ = 0 in
1 ab-1 of data. The latter translates to uncertainty of the CPV factor
$f_{CP}\approx$ 4.9 $\cdot$ 10-5, that is still slightly above the theoretical
target. Further improvement is expected in combination of results obtained on
samples with different beam polarizations, as mentioned in Sec.1. Systematic
uncertainty from the fit $\leq$ 1 mrad is significantly smaller compared to
the statistical one.

##  6 Conclusion

In this paper we present the result of the first CPV mixing angle measurement
based on angular observable, in HVV vertex at an $e^{-}e^{+}$ collider.
Measurement is simulated in ZZ-fusion at 1 TeV ILC with the ILD detector.
Local minimum of the angular observable is sensitive to the
$\mathrm{\Psi_{CP}}$ mixing angle and can be obtained from the
phenomenological fit of the reconstructed (simulated or experimental) data.
Individual measurement on the fully simulated data gives deviation of 0.9 mrad
from the truth value for the pure scalar state. The method is stable for
$\mathrm{\Psi_{CP}}$ variations up to 0.2 rad, corresponding to pseudoscalar
admixture of $\sim$ 17%. At 1 TeV ILC with 1 ab-1 of $e^{-}_{L}e^{+}_{R}$
data, pure scalar should be measured with statistical uncertainty of 7 mrad at
68% CL. Obtained statistical uncertainty corresponds to the precision of the
CPV factor $f_{CP}\approx$ 4.9 $\cdot$ 10-5. The obtained precision is not yet
final, as ILC offers possibility to combine different beam polarizations.

## Acknowledgements

We would like to acknowledge our colleagues from ILC IDT Working Group 3 for
the technical support as well as for comments on the physics content,
especially to Aleksander Filip Żarnecki for useful ideas and discussions
leading to better understanding of the sensitive observable behavior. This
research was funded by the Ministry of Education, Science and Technological
Development of the Republic of Serbia and by the Science Fund of the Republic
of Serbia through the Grant No. 7699827, IDEJE HIGHTONE-P.

## References

  * [1] A. V. Gritsan et. al, _Snowmass White Paper: Prospects of CP-violation measurements with the Higgs boson at future experiments_ ,  arXiv:2205.07715v2 [hep-ex] (2022).
  * [2] T. Barklow et al., _ILC Operating Scenarios_ , ILC-NOTE-2015-068, arXiv:1506.07830v1 [hep-ex] (2015).
  * [3] C. Adolphsen et al., _The International Linear Collider Technical Design Report - Volume 3.II: Accelerator Baseline Design_ , arXiv:1306.6328 [physics.acc-ph] (2013).
  * [4] P. Mora de Freitas and H. Videau, _Detector Simulation with MOKKA/GEANT4: Present and Future, International Workshop on Linear Colliders_ , JeJu Island, Korea, Technical Report No. LC-TOOL-2003-010, (2002).
  * [5] J. de Favereau et al., _DELPHES 3: a modular framework for fast simulation of a generic collider experiment_ , Journal of High Energy Physics 2014, 57, arXiv:1307.6346 [hep-ex] (2014).
  * [6] T. Behnke et al., _The International Linear Collider Technical Design Report - Volume 4: Detectors_ , arXiv:1306.6329 [physics.ins-det] (2013).
  * [7] M. A. Thomson, _Particle flow calorimetry and the pandora PFA algorithm_ , Nucl. Instrum. Methods Phys. Res., Sect. A 611, 25 (2009).
  * [8] S. Bolognesi et al., _Spin and parity of a single-produced resonance at the LHC_ , Phys. Rev. D 86 (2012) 095031, arXiv:1208.4018 (2012).
  * [9] S. Catani et al, _New clustering algorithm for multi - jet cross-sections in $e^{+}e^{-}$ annihilation_, Phys. Lett. B 269 (1991) 432-438 (1991).
  * [10] W. Kilian, T. Ohl, and J. Reuter, _WHIZARD: Simulating multi-particle processes at LHC and ILC_ , Eur. Phys. J. C 71, 1742 (2011).
  * [11] P. Artoisenet et al., _A framework for Higgs characterization_ , Journal of High Energy Physics 2013, 43, arXiv:1306.6464 [hep-ph] (2013).
  * [12] D. Jeans and G. W. Wilson, _Measuring the CP state of tau lepton pairs from Higgs decay at the ILC_ , Phys. Rev. D 98 013007 (2018).


# Content-Adaptive Variable Framerate Encoding Scheme for Green Live Streaming

Vignesh V Menon ID , , Samira Afzal ID , ,
Prajit T Rajendran ID , , Klaus Schoeffmann ID , ,
Radu Prodan ID , and Christian Timmerer ID We acknowledge the financial
support of the Austrian Federal Ministry for Digital and Economic Affairs, the
National Foundation for Research, Technology and Development, and the
Christian Doppler Research Association. Christian Doppler Laboratory ATHENA:
https://athena.itec.aau.at/.Vignesh V Menon is with the Video Coding Systems
research group at the Video Communication and Applications department,
Fraunhofer HHI, Berlin (e-mail: vignesh.menon@hhi.fraunhofer.de).Samira Afzal,
Klaus Schoeffmann, Radu Prodan and Christian Timmerer are with the Institute
of Information Technology, Alpen-Adria-Universität Klagenfurt, Austria
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>christian.timmerer@aau.at).Prajit T Rajendran is with CEA, List, F-91120
Palaiseau, Université Paris-Saclay, France (e-mail:
prajit.thazhurazhikath@cea.fr).

###### Abstract

Adaptive live video streaming applications use a fixed predefined
configuration for the bitrate ladder with constant framerate and encoding
presets in a session. However, selecting optimized framerates and presets for
every bitrate ladder representation can enhance perceptual quality, improve
computational resource allocation, and thus, the streaming energy efficiency.
In particular, low framerates for low-bitrate representations reduce
compression artifacts and decrease encoding energy consumption. In addition,
an optimized preset may lead to improved compression efficiency. To this
light, this paper proposes a Content-adaptive Variable Framerate (CVFR)
encoding scheme, which offers two modes of operation: ecological (ECO) and
high-quality (HQ). CVFR-ECO optimizes for the highest encoding energy savings
by predicting the optimized framerate for each representation in the bitrate
ladder. CVFR-HQ takes it further by predicting each representation’s optimized
framerate-encoding preset pair using low-complexity _discrete cosine
transform_ energy-based spatial and temporal features for compression
efficiency and sustainable storage. We demonstrate the advantage of CVFR using
the x264 open-source video encoder. The results show that CVFR-ECO yields an
average PSNR and VMAF increase of $0.02\text{\,}\mathrm{dB}\text{p}$ and 2.50
points, respectively, for the same bitrate, compared to the fastest preset
highest framerate encoding. CVFR-ECO also yields an average encoding and
storage energy consumption reduction of $34.54\text{\,}\mathrm{\char
37\relax}\text{p}$ and $76.24\text{\,}\mathrm{\char 37\relax}\text{p}$
considering a _Just Noticeable Difference_ (JND) of six VMAF points. In
comparison, CVFR-HQ yields an average increase in PSNR and VMAF of
$2.43\text{\,}\mathrm{dB}\text{p}$ and 10.14 points, respectively, for the
same bitrate. Finally, CVFR-HQ resulted in an average reduction in storage
energy consumption of $83.18\text{\,}\mathrm{\char 37\relax}\text{p}$
considering a JND of six VMAF points.

###### Index Terms:

Energy consumption; variable framerate encoding; low latency encoding;
encoding preset; just noticeable difference.

##  I Introduction

Streaming video has become the most popular online activity, with viewers
accessing content across various devices such as TVs, laptops, tablets, and
smartphones [1]. Optimized delivery requires video encoding into multiple
quality levels adjusted to network conditions and user devices [2]. _HTTP
Adaptive Streaming_ (HAS) has become the de-facto standard in delivering video
content for various clients regarding internet speeds and device types. HAS
divides the video content into segments and encodes each segment at various
bitrates and resolutions, called representations, stored in plain HTTP
servers, which continuously adapt the video delivery to the network conditions
and device capabilities of the client [2]. In the current streaming landscape,
_live-streaming_ is one of the foremost challenges, which necessitates faster
compression and simplified encoding techniques [3] to reduce the time between
video capture and playback [4]. The viewer’s experience improves by the
reduced latency, enabling a more responsive and engaging content consumption
[5, 6].

####  I-1 Low-latency live-streaming

A low-latency live encoder must maintain an encoding speed greater than the
video framerate regardless of the complexity of the video content. A reduced
encoding speed can lead to frames dropping during transmission [7], which may
decrease the QoE. Traditionally, live-streaming sessions use a fixed bitrate
ladder, such as _HTTP Live Streaming_ (HLS) [8]. Per-title encoding schemes
that optimize bitrate ladder according to the specific video content [9, 10]
received wide adoption for only _Video on Demand_(VoD) services due to the
expensive convex-hull computation [11, 12]. Moreover, state-of-the-art per-
title encoding schemes optimize target resolution [9, 10, 11, 13] or bitrate
[12] based on perceptual quality. Noteworthy, these approaches are not
suitable for live-streaming solutions, where (i) predictable switching between
representations and (ii) simplified player logic on the client side are of
significant importance [4]. Hence, per-title encoding optimization using fixed
bitrate-resolution pairs is essential.

####  I-2 Energy consumption

Bitrate ladder encoding for HAS platforms incurs substantial energy
consumption, thereby straining environmentally conscious resource management
in data centers [14]. Encoding energy optimization in this context aims to
reduce the ecological footprint and operational costs associated with adaptive
live-streaming [15]. Additionally, storing the video on the server and
streaming it to devices using content distribution networks (CDNs) consume
energy [16, 17, 15]. Encoding solutions in video streaming directly impact
storage and transmission energy consumption. Consequently, minimizing the
overall energy consumption in video streaming is a significant challenge in
the industry today [15, 18, 19]. Thus, energy-efficient encoding techniques
are necessary to reduce overall consumption without compromising the
perceptual quality of the delivered video content.

####  I-3 Low-latency, minimized perceptual redundancy

This paper targets a low-latency encoding scheme yielding optimized trade-offs
between overall energy consumption and compression efficiency without
significantly changing the streaming architecture. To achieve this goal, we
consider _variable framerate_ (VFR) encoding methods [20, 21, 22, 23]
introduced in the literature to limit the modifications to the encoding server
and client. This approach enhances adaptive streaming mechanisms by aligning
the framerate selection with perceptual optimization, allowing for dynamic
framerate adjustment based on network conditions and viewer devices. Fig. 1
shows an example of VFR encoding for streaming applications that temporarily
downsamples the video of the original framerate of 30 fps to a framerate of 15
fps indicated by the framerate selection module. After decoding, the video is
temporally upscaled to its original framerate. Noteworthy, state-of-the-art
video encoders offer several presets to balance the trade-off between encoding
time and compression efficiency [24] that differ in the encoding tools used.
Faster presets utilize a subset of the tools to reduce encoding time and
energy consumption [25]. Live encoders usually choose faster presets to encode
video frames in real-time, sacrificing some quality and compression ratio.
Hence, this paper also uses variable preset encoding methods in conjunction
with VFR to optimize perceptual quality and energy consumption. Furthermore, a
_Just Noticeable Difference_ (JND)-aware representation elimination optimizes
the allocation of streaming bits based on the perceptual thresholds of human
vision [26]. Ensuring that the adjacent points of the bitrate ladder have a
perceptual quality difference of at least one JND eliminates the perceptual
redundancy between representations and reduces the overall energy consumption.
This paper considers JND as a function of VMAF [27], and future work will
study other functions.

Figure 1: An example scenario of a VFR encoding scheme for adaptive streaming
that encodes a video segment of ultra-high definition resolution and 30fps in
representation (7.8 Mbps, 1080p) with the selected framerate of 15fps. On the
client side, red dashed blocks indicate the additional steps introduced
compared to the traditional bitrate ladder encoding.

####  I-4 Contributions

This paper contains five contributions.

##### Comprehensive compression efficiency and energy consumption analysis

uses a state-of-the-art video encoder at multiple framerates and encoding
presets. This highlights the complexity of this relationship and the need for
adaptive solutions that balance video quality and energy efficiency.

#####  CVFR

_content-adaptive variable framerate encoding_ scheme optimizes the bitrate
ladder by maximizing the perceptual quality and energy efficiency while
maintaining the target encoding speed for low-latency encoding. A JND-based
representation elimination algorithm removes the perceptual redundancy in the
bitrate ladder. CVFR offers two operational modes: _ecological_ (CVFR-ECO) and
_high-quality_ (CVFR-HQ).

#####  CVFR-ECO

predicts the optimized framerate for the fastest encoding preset for each
representation of the bitrate ladder, yielding the lowest encoding energy
consumption.

#####  CVFR-HQ

jointly predicts the optimized framerate-encoder preset pairs for each
representation to yield the highest possible compression efficiency while
maintaining the target encoding speed for low-latency encoding.

##### Experimental evaluation

compares the CVFR schemes with state-of-the-art encoding methods in terms of
compression efficiency and energy consumption. CVFR-ECO reduces encoding and
storage energy consumption by $34.54\text{\,}\mathrm{\char 37\relax}\text{p}$
and $76.24\text{\,}\mathrm{\char 37\relax}\text{p}$ during adaptive live video
streaming, respectively. On the other hand, CVFR-HQ significantly improves
compression efficiency, achieving a PSNR increase of $2.43$ dB and VMAF
increase of $10.14$ points for the same bitrate. Additionally, CVFR-HQ reduces
storage energy consumption by $83.18\text{\,}\mathrm{\char 37\relax}\text{p}$
during adaptive live-streaming.

##  II Variable framerate encoding

Several studies highlighted the effectiveness of raising the framerate in
reducing temporal artifacts, including flickering, stuttering, and motion blur
[28, 29, 30]. Encoding each frame involves various computational operations,
including motion estimation, transformation, quantization, and entropy coding.
Higher framerates lead to more frames processed per unit of time, increased
computational workload, and higher energy consumption during encoding [31].
Moreover, encoding at higher framerates might require higher bitrates to
maintain video quality, potentially increasing energy use. Sometimes, the
trade-off between encoding speed, energy consumption, and video quality might
suggest a more energy-efficient encoding at a lower framerate for a given
quality [31, 20].

(a) Average PSNR.

(b) Average VMAF.

(c) Average encoding time.

(d) Average energy consumption.

Figure 2: Average encoding metrics for 7.5 fps, 15 fps, 24 fps, and 30 fps HLS CBR encoding using the veryslow preset of the x264 AVC encoder [32, 33] using VCD dataset [34]. TABLE I: Comparison of the state-of-the-art VFR encoding methods with CVFR. Method | Target scenario | Framerate estimation method | Number of pre-encodings | Scalability
---|---|---|---|---
Bruteforce [9] | VoD | Bruteforce encoding | $\tilde{b}\times\tilde{f}$ | Yes
Huang  _et al._ [35] | VoD | Prediction using support vector regression method | 0 | Yes
Katsenou  _et al._ [23] | VoD | Prediction using bagged decision trees | 0 | No
ViSTRA [22] | VoD | Encoding at two framerates | 2 | No
Herrou  _et al._ [20] | Live | Prediction using random forest classifier models | 0 | No
CVFR | Live | Prediction using random forest regression models | 0 | Yes

###  II-A Motivation

Mackin  _et al._ [36] found that higher framerates are more encoding efficient
at higher bitrates, particularly in simple sequences with camera movements.
This content dependency to select the optimized framerate is the basis for a
VFR coding scheme. Accordingly, the example scenario in Fig. 1 and the results
in Fig. 2(a) and 2(b), demonstrate a similar trend in the average PSNR and
VMAF metrics for 7.5 fps, 15 fps, 24 fps, and 30 fps encodings of the VCD
dataset [34] using the veryslow preset of the x264 encoder [32]. The
assumption is that dropping frames in slow-motion videos has a less noticeable
difference in the perceived moving objects’ quality than in fast-motion. This
coding scheme can considerably reduce the bitrate and encoding energy without
incurring apparent distortions. Additionally, a higher framerate increases the
encoding time (_cf._ Fig. 2(c)) and energy (_cf._ Fig. 2(d)) overheads
measured using the CodeCarbon tool for the x264 encoder on an Intel Xeon Gold
5218R processor using eight threads. Finally, the average perceptual quality
difference (measured using VMAF [27]) between multiple framerate encodings at
low bitrates is insignificant. Closer quality for different framerates at low
bitrates could be due to the perceptual sensitivity of the VMAF metric.
Encoding decisions at low bitrates may prioritize certain video aspects,
leading to similar perceptual quality even for different framerates.

###  II-B State-of-the-art VFR scheme architecture

State-of-the-art VFR schemes include a temporal downsampling step before
encoding and a temporal upsampling step after decoding (_cf._ Fig. 1),
described in this section.

####  II-B1 Temporal downsampling

involves discarding or reducing the number of frames in the video to achieve a
lower framerate, reducing the temporal detail and smoothness of the video.
There are two standard downsampling techniques.

##### Frame dropping

intentionally discards or drops specific frames from the video sequence based
on a regular pattern (_e.g._ , every $n$th frame) or dynamically determined
based on specific criteria (_e.g._ , low visual importance) [37].

##### Temporal filtering

analyzes the temporal redundancy between frames and applies filtering or
motion analysis to generate new frames that are blends or interpolations of
adjacent frames. The resulting frames can reduce the framerate while
maintaining smoother motion and minimizing judder or jerkiness in the video.

(a) Average PSNR.

(b) Average VMAF.

(c) Average encoding time.

(d) Average energy consumption.

Figure 3: Average encoding metrics for HLS CBR encoding at 30 fps using
selected presets of x264 AVC encoder [32, 33] using the VCD dataset [34].

####  II-B2 Temporal upsampling

upsamples the VFR-encoded video to the original framerate at the client side,
as most display devices do not support a variable framerate viewing [20].
There are two standard upsampling techniques.

##### Frame duplication

generates additional replicated frames of the already decoded ones and inserts
them between consecutive frames. This straightforward technique does not
require motion estimation or complex algorithms [38]. However, it provides no
additional temporal information or motion details compared to the original
video sequence.

##### Frame interpolation

generates new frames by estimating the intermediate motion between adjacent
frames. Various algorithms, such as optical flow estimation [39], can estimate
motion vectors and generate new frames [40]. Frame interpolation can
effectively increase the framerate, providing smoother motion in the video
sequence [41].

###  II-C Related VFR works

Many research works have investigated VFR. Some use different motion-related
features with thresholding techniques [22] or machine learning algorithms [35,
23] to select the desired framerate. Table I shows the target scenario,
framerate estimation method, and the number of pre-encodings needed to
optimize the framerate of the state-of-the-art VFR methods. To determine the
optimal framerate for $\tilde{b}$ representations and $\tilde{f}$ supported
framerates, it is necessary to “bruteforce” execute $\tilde{b}\times\tilde{f}$
test encodings [9].

Huang  _et al._ [35] proposed a framerate selection mechanism to meet the
“satisfied user ratio” using a support vector regression method. However, it
uses complex and computationally expensive visual saliency and spatial
randomness map features for each frame unsuitable for real-time dynamic
framerates.

Katsenou  _et al._ [23] trained decision trees to predict the critical
framerate at a sequence level using optical flow as the temporal and gray-
level co-occurrence matrix as the spatial features. However, the feature
extraction needs significant processing time, rendering it unfit for live-
streaming applications.

ViSTRA [22] employs a temporal resolution optimization using a framerate-
dependent quality metric [42] to assess the perceptual quality difference
between a temporally downsampled video frame and its full framerate original.
This method involves encoding at the original framerate and half of the
original framerate; hence, it introduces significant latency.

Herrou  _et al._ [20] proposed a VFR method to determine the minimum framerate
that preserves the perceived video quality using two random forest
classifiers. However, this method has limitations to three framerates (_i.e._
, original and downsampled by a factor of two and four).

###  II-D Summary

Most related works on VFR yield latency unsuitable for live-streaming
applications. The methods use complex and computationally expensive features
or pre-encodings, which introduce significant latency in streaming. The
solution proposed in [20] is not scalable and requires model retraining for
different framerate and bitrate ladder representations. To mitigate these
problems, this paper proposes a low-latency scalable solution using random-
forest-based framerate prediction. Finally, due to the low computational
complexity, a prerequisite for low-latency encoding, this paper considers
frame dropping and frame duplication techniques for temporal downsampling and
upsampling, respectively.

##  III Variable preset encoding

###  III-A Motivation

Traditionally, video encoders provide predefined settings and configurations,
termed encoding presets optimized for specific use cases and target devices
based on empirical data, industry standards, and encoding techniques for
different scenarios. Using presets, video encoders can efficiently handle the
vast possible encoding configurations without specifying each parameter
manually. Presets directly impact the compression efficiency (_cf._ Fig. 3(a)
and 3(b)). Moreover, encoding time and energy consumption increase
exponentially for slower presets (_cf._ Fig. 3(c) and 3(d)). Faster presets
typically employ simpler algorithms and fewer encoding passes, with reduced
computations required for motion estimation, transform coding, and entropy
coding. The result is a reduced overall computational effort and a lower
energy consumption [25]. Faster presets also shorten the processing time,
which saves energy by keeping the hardware components active for shorter
durations.

Figure 4: Low-latency live encoding using CVFR.

The x264 [32] and x265 [43] encoders offer ten pre-defined presets ranging
from the highest compression efficiency and energy consumption (preset 9,
known as placebo) to the fastest encoding speed and lowest energy consumption
(preset 0, known as ultrafast). The encoder parameters depend on the
particular encoding preset configuration [44, 24]. Generally, live content
prefers the fastest encoding preset (ultrafast), independent of its dynamic
complexity. Although this conservative technique achieves the intended result
of live encoding with low encoding energy, the resulting visual quality is
sub-optimal, especially for dynamically changing content [45]. Furthermore,
when the content becomes easier to encode (_i.e._ , slow-motion videos,
predictable frames with simple textures), the encoder achieves a higher speed
than the target encoding speed. Configuring the preset to reduce this higher
speed while still being compatible with the target live encoding speed
improves the quality of the encoded content. However, when the content becomes
complex again, the encoder preset must return to the faster configuration that
achieves live encoding speed [46].

###  III-B Related variable preset encoding works

Ramachandran  _et al._ [7] proposed an architecture using a proportional
integral derivate (PID) module leveraged for content-adaptive live encoding.
The proposed PID controller dynamically monitors the encoder’s achieved
framerate and adjusts parameters to maintain the framerate at the expected
level while maximizing quality. However, the implementation details are
proprietary and limited.

Nasiri  _et al._ [44] proposed an ML method to estimate bitrate ladders of
multi-preset encoders for VoD applications. This method offers a reference
content-adaptive bitrate ladder that exhaustively uses the fastest known
preset of a given encoder. Then, an offline trained regressor transforms the
fast-preset ladder into the slow-preset ladder needed for the encoding pass.
However, the features used for prediction (_i.e._ , GLCM and temporal
coherency) are computationally intensive and unsuitable for real-time live-
streaming.

Our previous work CAPS [24] determines the optimized preset for a given target
bitrate representation to maintain the target encoding speed for low-latency
encoding using XGBoost-based models [47]. This method improves the visual
quality of representations at lower bitrates.

###  III-C Summary

There are a few related works on variable preset encoding. Ramachandran  _et
al._ [7] is proprietary, while Nasiri  _et al._ [44] applies only to VoD
applications. This paper considers two common approaches to low-latency
encoding:

##### Low encoding energy

referred to as ecological CVFR-ECO mode, opts for the fastest preset while
meeting encoding time constraints in scenarios where real-time or near-real-
time encoding is critical.

##### Highest compression efficiency

referred to as high-quality CVFR-HQ mode, maximizes the compression efficiency
by encoding at slower presets and maintaining the encoding speed higher than
the video framerate.

##  IV Content-adaptive Variable Framerate (CVFR) encoding scheme

Fig. 4 presents the architecture of the proposed CVFR scheme for live video
streaming applications. CVFR receives the video segment initially and extracts
its complexity features. CVFR then offers two modes of operation:

##### Ecological

CVFR-ECO determines the optimized framerate for each bitrate ladder
representation [48] using the fastest available preset.

##### High-quality

CVFR-HQ predicts the representation’s optimized framerate and preset using the
video complexity features extracted for every segment and the set of pre-
defined framerates and presets supported by the service provider.

The JND-based representation elimination algorithm ensures that the adjacent
rate-distortion (RD) points of the bitrate ladder have a perceptual quality
difference of at least one JND. Prediction for every segment is necessary
because of the reasonably uniform frame-to-frame spatiotemporal content of the
frames within a segment [48]. The JND-based representation elimination uses
the average JND quality, assuming VMAF as the “optimal” measure of perceptual
quality, predicted for each bitrate-resolution and framerate-preset
configuration. The constant bitrate (CBR) encoding process uses the selected
optimized bitrate-resolution and framerate-preset configurations.

TABLE II: Notations used in CVFR.

_Notation_ | _Description_
---|---
_Video complexity features_
$E_{\text{S}}$ | Average luma texture energy of segment
$h_{\text{S}}$ | Average gradient of the luma texture energy of segment
$L_{\text{S}}$ | Average luminescence of segment
_Input parameters_
$\mathcal{R}$ | Set of bitrate ladder representations
$\mathcal{F}$ | Set of supported framerates
$\mathcal{P}$ | Set of supported presets
$f_{\text{T}}$ | Target encoding speed [fps]
$v_{\text{J}}$ | Target JND
$v_{\text{T}}$ | Maximum VMAF threshold
$r_{t},b_{t},\hat{f}_{t},\hat{p}_{t},\hat{v}_{t}$ | Resolution, bitrate and predicted framerate, preset and
VMAF of the $t$th representation

Therefore, CVFR encoding has four connected phases depicted in Fig. 4:

  1. 1

Video complexity feature extraction (Section IV-A);

  2. 2

Optimized framerate and preset prediction (Section IV-B);

  3. 3

JND-based representation elimination (Section IV-C);

  4. 4

CBR encoding of the segment using the predicted bitrate-resolution and
framerate-preset configurations.

Table II summarizes the model notation for convenience.

###  IV-A Video complexity feature extraction

In video streaming applications, convolutional neural networks [49] is an
intuitive method for feature extraction but cause inherent disadvantages for
live-streaming applications [12], such as longer training time, inference
time, and storage requirements. Although such methods can result in rich
features, simpler models yielding significant prediction accuracy are more
suitable for video streaming. Two popular state-of-the-art video complexity
features are spatial information and temporal information [50], offering low
feature correlation with the encoding output features such as bitrate and
encoding time, which is insufficient for encoding parameter prediction in
streaming applications [51].

This paper uses three _discrete cosine transform_ energy-based features, the
average luma texture energy $E_{\text{Y}}$, the average gradient of luma
texture energy $h$, and the average luminescence $L_{\text{Y}}$ extracted
using the open-source _video complexity analyzer (VCA)_ , and use them as
spatial and temporal complexity [51, 12]. These energy features averaged
across all segment frames represent the average complexity segment
characteristics $E_{\text{S}}$, $h_{\text{S}}$, and $L_{\text{S}}$.

###  IV-B Optimized framerate and preset prediction

The optimized framerate and preset prediction method comprises two steps:
modeling and optimization.

####  IV-B1 Modeling

We define a set of video representation
$\mathcal{R}=\left\\{\left({r}_{i},{b}_{i}\right)|1\leq i\leq q\right\\}$ for
a pair of encoding resolution $r_{i}$ and bitrate $b_{i}$. The perceptual
quality $v_{\left(r_{t},b_{t},f_{t},p_{t}\right)}$ and encoding speed
$s_{\left(r_{t},b_{t},f_{t},p_{t}\right)}$ of the $t$th representation in
$\mathcal{R}$ relies on video complexity features $E_{\text{S}}$,
$h_{\text{S}}$, $L_{\text{S}}$, encoding resolution $r_{t}$, target bitrate
$b_{t}$, framerate $f_{t}$, and preset $p_{t}$ parameters:

$\displaystyle v_{\left(r_{t},b_{t},f_{t},p_{t}\right)}$
$\displaystyle=f_{V}\left(E_{\text{S}},h_{\text{S}},L_{\text{S}},r_{t},b_{t},f_{t},p_{t}\right);$
(1) $\displaystyle s_{\left(r_{t},b_{t},f_{t},p_{t}\right)}$
$\displaystyle=f_{S}\left(E_{\text{S}},h_{\text{S}},L_{\text{S}},r_{t},b_{t},f_{t},p_{t}\right).$
(2)

Spatiotemporal features encapsulate intricate spatial details and temporal
dynamics within the video segment and help assess the video fidelity [52].
Including resolution, bitrate, framerate, and preset parameters in the models
acknowledges the interplay between compression efficiency, temporal
smoothness, and spatial clarity in shaping perceived quality. A higher
resolution, bitrate, or framerate may improve the quality and increase the
file size of the video segment. A slower preset at the same target bitrate can
reduce the file size of the video segment. Similarly, a higher resolution,
bitrate, framerate, or a slower preset can reduce the encoding speed.

####  IV-B2 Optimization

of the framerate-preset pairs for each target representation utilizes the
perceptual quality and encoding speed models developed in the previous step.

#####  CVFR-ECO

involves a dual commitment to perceptual fidelity and encoding energy
conservation. First, it chooses the fastest preset supported by the encoder
$p_{1}$ for all representations to save energy. Second, it predicts the
optimized framerate $\hat{f}_{t}$ from the supported framerates $\mathcal{F}$
for the $t$th representation of $\mathcal{R}$ based on video complexity
features $E_{\text{S}}$, $h_{\text{S}}$, $L_{\text{S}}$, and the target
bitrate-resolution pair $\left(r_{t},b_{t}\right)$ maintaining the encoding
speed above the threshold $f_{\text{T}}$;

$\displaystyle\hat{f}_{t}$
$\displaystyle=\operatorname*{arg\,max}_{f\in\mathcal{F}}\hat{v}_{\left(r_{t},b_{t},f,p_{1}\right)}$
$\displaystyle c.t.\hskip
10.00002pt\hat{s}_{\left(r_{t},b_{t},f,p_{1}\right)}\geq f_{\text{T}};$ (3)
$\displaystyle\hat{v}_{t}$
$\displaystyle=\max_{f\in\mathcal{F}}\hat{v}_{\left(r_{t},b_{t},f,p_{1}\right)}$
$\displaystyle c.t.\hskip 10.00002pt\hat{s}_{(r_{t},b_{t},f,p_{1})}\geq
f_{\text{T}}.$ (4)

where $\hat{v}_{(r_{t},b_{t},f,p_{1})}$ and $\hat{s}_{(r_{t},b_{t},f,p_{1})}$
are the predicted VMAF and encoding speed of the $t$th representation in
$\mathcal{R}$, using framerate $f$ and preset $p_{1}$. Thus, CVFR-ECO
estimates the framerates synergizing the perceptual quality enhancement,
energy reduction, and real-time processing imperatives. Therefore, the
encoding configuration for the $t$th representation in $\mathcal{R}$ is
$\left(r_{t},b_{t},\hat{f}_{t},p_{1}\right)$.

#####  CVFR-HQ

represents a strategic amalgamation of perceptual quality enhancement and
computational efficiency. This scheme selects specific combinations of
framerates and presets that maximize VMAF for each representation. By
integrating temporal coherence, content complexity, and encoding efficiency,
CVFR-HQ optimizes the perceptual quality of encoded videos while adhering to
real-time processing constraints. It jointly predicts the optimized framerate
and preset of the $t$th representation to maximize the compression efficiency
while maintaining the encoding speed below the threshold $f_{\text{T}}$. CVFR-
HQ ensures that the encoding speed exceeds the video framerate and maintains
real-time feasibility. The optimization function is:

$\displaystyle(\hat{f}_{t},\hat{p}_{t})$
$\displaystyle=\operatorname*{arg\,max}_{f\in\mathcal{F}\land
p\in\mathcal{P}}\hat{v}_{\left(r_{t},b_{t},f,p\right)}\hskip
10.00002ptc.t.\hskip 10.00002pt\hat{s}_{\left(r_{t},b_{t},f,p\right)}\geq
f_{\text{T}};$ (5) $\displaystyle\hat{v}_{t}$
$\displaystyle=\max_{f\in\mathcal{F}\land
p\in\mathcal{P}}\hat{v}_{\left(r_{t},b_{t},f,p\right)}\hskip
10.00002ptc.t.\hskip 10.00002pt\hat{s}_{\left(r_{t},b_{t},f,p\right)}\geq
f_{\text{T}}.$ (6)

Thus, the encoding configuration for the $t$th representation in $\mathcal{R}$
is $\left(r_{t},b_{t},\hat{f}_{t},\hat{p}_{t}\right)$.

Input:

$q$ : number of representations in $\mathcal{R}$

$\mathcal{R}=\bigcup_{t=1}^{q}\left\\{\left(r_{t},b_{t},\hat{f}_{t},\hat{p}_{t}\right)\right\\}$:
representations with predicted framerate and preset $\hat{v}_{t}~{};1\leq
t\leq q$: predicted VMAF

$v_{\text{T}}$ : maximum VMAF threshold

$v_{J}$ : average target JND

Output: $\hat{\mathcal{R}}=\left(r,b,\hat{f},\hat{p}\right)$: set of encoding
configurations

1:
$\hat{\mathcal{R}}\leftarrow\left\\{\left(r_{1},b_{1},\hat{f}_{1},\hat{p}_{1}\right)\right\\}$

2: $u\leftarrow 1$

3: if $\hat{v}_{1}\geq v_{\text{T}}$ then

4: return $\hat{\mathcal{R}}$

5: end if

6: $t\leftarrow 2$

7: while $t\leq q$ do

8: if $\hat{v}_{t}-\hat{v}_{u}\geq$ $v_{\text{J}}$ then

9:
$\hat{\mathcal{R}}\leftarrow\hat{\mathcal{R}}\cup\left\\{\left(r_{t},b_{t},\hat{f}_{t},\hat{p}_{t}\right)\right\\}$

10: $u\leftarrow t$

11: if $\hat{v}_{t}\geq v_{\text{T}}$ then

12: return $\hat{\mathcal{R}}$

13: end if

14: end if

14: $t\leftarrow t+1$

15: end while

16: return $\hat{\mathcal{R}}$

Algorithm 1 JND-based representation elimination.

###  IV-C JND-based representation elimination

In practice, the VMAF scores of consecutive representations are similar and
introduce perceptual redundancy in the bitrate ladder. To address this issue,
CVFR leverages the minimum JND threshold required by the human eye to perceive
differences in quality [53, 54, 55] since perceptually redundant
representations yield diminishes perceptual gains relative to their
computational and energy costs. Eliminating redundant representations obviates
associated encoding tasks, reducing processing time and energy consumption,
and contributes to a more resource-efficient encoding workflow, conserving
energy and reducing operational costs. While [56, 57] explored complex VMAF-
based JND thresholds unsuitable for live-streaming applications, CVFR adopts a
fixed JND threshold $v_{\text{J}}$ as input from the streaming service
provider.

TABLE III: Experimental parameters of CVFR used in this paper. _Parameter_ | _Symbol_ | _Values_
---|---|---
_Set of representations_ | _Resolution height [pixels]_ | $\mathcal{R}$ | 234 | 360 | 432 | 432 | 540 | 720 | 720 | 1080 | 1080
_Bitrate [Mbps]_ | 0.145 | 0.365 | 0.730 | 1.100 | 2.000 | 3.000 | 4.500 | 6.000 | 7.800
_Set of framerates [fps]_ | $\mathcal{F}$ | 7.5, 15, 24, 30
_Set of presets [x264]_ | $\mathcal{P}$ | 0 (ultrafast) – 8 (veryslow)
_Encoding speed threshold [fps]_ | $f_{\text{T}}$ | 30
_Average target JND_ | $v_{\text{J}}$ | 2 | 4 | 6
_Maximum VMAF threshold_ | $v_{\text{T}}$ | 98 | 96 | 94

Algorithm 1 implements the JND-based representation elimination. This
algorithm receives the number of representations $q$ in $\mathcal{R}$
comprising their predicted framerate and preset
${(r_{t},b_{t},\hat{f}_{t},\hat{p}_{t})}$ and VMAF $\hat{v}_{t}$ (from Section
IV-B), maximum VMAF threshold $v_{\text{T}}$, and average target JND
$v_{\text{J}}$. The first representation of $\mathcal{R}$ is always part of
the encoding representation set $\hat{\mathcal{R}}$ (line 1). When the
predicted VMAF of the first representation in $\mathcal{R}$, _i.e._ ,
$\hat{v}_{1}$ is greater than $v_{\text{T}}$ (above which the representation
is perceptually lossless), it eliminates all other representations from the
bitrate ladder(lines 3–4). If the predicted VMAF difference between the
current representation $\hat{v}_{t}$ and the previously selected
representation in $\hat{\mathcal{R}}$ is greater than (or equal to)
$v_{\text{J}}$, $\hat{\mathcal{R}}$ includes the current representation (lines
8–10). The algorithm terminates when the predicted VMAF of the current
representation is higher than $v_{\text{T}}$ (lines 11–12). The algorithm
loops between lines 7 and 15 until it analyzes all representations in
$\mathcal{R}$. Finally, $\hat{\mathcal{R}}$ is the algorithm’s output.

##  V Experimental Design

This section illustrates our experimental design to assess the performance of
CVFR. We evaluate CVFR, comparing it with five distinct benchmark schemes
regarding energy consumption and compression efficiency.

###  V-A Experimental setup

We run experiments on a dual server with Intel Xeon Gold 5218R processors (80
cores, operating at 2.10 GHz). We execute the VCA and x264 encoder using eight
threads, using x86 SIMD optimizations [58]. We use the video complexity
dataset (VCD) [34] consisting of five hundred 2160p resolution video sequences
(segments) and consider the x264 version r3107 [32] as the target encoder. We
adopt the bitrate-resolution pairs specified in the Apple HLS authoring
specifications [8] as the set of representations $\mathcal{R}$. We extract the
spatiotemporal features, $E_{\text{S}}$, $h_{\text{S}}$, and $L_{\text{S}}$,
using VCA v2.0 [51] to assess video complexity. We select the average target
JND $v_{\text{J}}$ according to current industry practices [59, 60] and set
the maximum VMAF threshold $v_{\text{T}}$ accordingly, _i.e._ ,
$v_{\text{T}}=100-v_{\text{J}}$. Table III summarizes the list of experimental
parameters.

###  V-B Prediction models

To ensure the robustness and generalization of the prediction models, we
perform a five-fold cross-validation scheme for video sequences and average
the results. The scheme also ensures splitting the test and training segments.
Our approach to designing prediction models is scalable by training models for
each preset supported by the streaming service provider, avoiding retraining
the entire network when adding a new preset. This work uses three prediction
models for comparing their accuracy in terms of the coefficient of
determination ($R^{2}$) and mean absolute error (MAE): (i) linear regression
[61], (ii) XGBoost [47] and (iii) random forest regression [62].

TABLE IV: Prediction accuracy of VMAF and encoding speed prediction models for ultrafast preset for VCD dataset [34] using x264 AVC encoder. | VMAF prediction | Encoding speed prediction
---|---|---
Method | $R^{2}$ | MAE | $R^{2}$ | MAE
Linear regression | 0.748 | 7.988 | 0.716 | 252.744
XGBoost | 0.882 | 4.873 | 0.946 | 51.325
Random forest | 0.895 | 4.552 | 0.949 | 39.200

Table IV shows the results of the VMAF prediction and the encoding speed of
the ultrafast preset using the default hyperparameters of the models. We
observe that the $R^{2}$ score is the maximum, and MAE is the minimum for
random forest models. Therefore, we use random forest for VMAF and encoding
speed prediction for each encoding preset in our experiments. We perform the
hyperparameter tuning on the prediction models on the ultrafast preset to
balance the size and prediction accuracy of the models. The selected
hyperparameters [63] are as follows:

  1. 1.

minimum number of samples required to be at a leaf node (min_samples_leaf) set
to 1;

  2. 2.

minimum number of samples required to split an internal node
(min_samples_split) set to 2;

  3. 3.

number of trees in forest (n_estimators) set to 100;

  4. 4.

maximum depth of the tree (max_depth) set to 14.

###  V-C Benchmarks

We compare CVFR with five schemes carefully selected according to the
following paragraphs:

##### Default

adopts a fixed framerate of 30 fps and the ultrafast preset for the CBR
encoding of the HLS bitrate ladder [8].

##### Bruteforce-ECO

[9] determines the optimized framerate for each representation by encoding all
framerates with the ultrafast preset, representing the ideal bitrate ladder
constructed using CVFR-ECO, _i.e._ , when the prediction models are
$100\text{\,}\mathrm{\char 37\relax}\text{p}$ accurate.

##### Bruteforce-HQ

[9] determines the optimized framerate-preset pair for each representation by
encoding all framerates and presets, indicating the ideal bitrate ladder
constructed using CVFR-HQ, _i.e._ , when the prediction models are
$100\text{\,}\mathrm{\char 37\relax}\text{p}$ accurate.

##### Herrou  _et al._

[20] determine the lowest framerate that does not affect the perceived
original video quality. Combining two successive binary random forest
classifiers uses 32 features, including the pixel luminance map, frame
difference magnitude, and horizontal and vertical coordinates of motion
vectors. [20] is the only work applied to live-streaming essential for our
evaluation, as presented in Table I.

#####  CAPS

[24] determines the optimized preset for each representation for a fixed
framerate of 30 fps.

###  V-D Evaluation metrics

We evaluate the CVFR encoding using the following metrics:

BDRP and BDRV

[64] refer to the average increase in bitrate of the representations compared
to the reference bitrate ladder encoding scheme to maintain the same PSNR and
VMAF. A negative BDR suggests a boost in the coding efficiency of the
considered encoding scheme compared to the reference bitrate ladder encoding
scheme.

BD-PSNR and BD-VMAF

refer to the average increase in PSNR and VMAF at the same bitrate compared
with the reference bitrate ladder encoding scheme. Positive BD-PSNR and BD-
VMAF denote an increase in the coding efficiency of the considered encoding
scheme compared to the reference bitrate ladder encoding.

Relative storage space difference

between the considered encoding scheme $b_{\text{opt}}$ and the reference
encoding scheme $b_{\text{ref}}$ to store all bitrate ladder representations:

$\Delta S=\frac{\sum b_{\text{opt}}}{\sum b_{\text{ref}}}-1.$ (7)

Encoding energy consumption

measures the CPU energy consumption on Linux using the Running Average Power
Limit (RAPL) interface and the CodeCarbon tool [65].

Storage energy consumption

of all server data [17]:

$E_{\text{sto}}=S_{\text{d}}\cdot P_{\text{b}}\cdot T_{\text{s}},$ (8)

where $S_{\text{d}}$ is the video data size in $\mathrm{bit}\text{p}$,
$P_{\text{b}}$ is the power consumption per bit in
$\mathrm{W}\text{p}\mathrm{bit}$, and $T_{\text{s}}$ is the time duration
taken for data to be stored in $\mathrm{h}\text{p}$. In our experiments, we
measure $T_{\text{s}}=$1.9\text{\,}\mathrm{GB}\text{p}\mathrm{s}$$ using the
Unix dd command and set
$P_{\text{b}}=7.84\cdot${10}^{-12}\text{\,}\mathrm{W}\text{p}\mathrm{bit}$$
[17].

##  VI Experimental Results

We conduct three experiments to evaluate CVFR in four areas:  i) prediction
models,  ii) coding efficiency analysis,  iii) storage consumption analysis,
and  iv) energy consumption.

###  VI-A Prediction models

In this section, we a) assess the accuracy of our VMAF and encoding speed
prediction models, b) explore the relative importance of features, c) analyze
average framerate-preset predictions, and d) discuss latency considerations.

(a) VMAF.

(b) Encoding speed.

(c) Feature relevance for VMAF prediction.

(d) Feature relevance for encoding speed prediction.

(e) Average CVFR-ECO framerate.

(f) Average CVFR-HQ framerate and preset.

Figure 5: Prediction results.

(a) Bunny_s000

(b) Characters_s000

(c) Eldorado_s000

(d) Eldorado_s005

(e) HoneyBee_s000

(f) RushHour_s000

(g) Runners_s000

(h) Wood_s000

Figure 6: RD curves of representative video sequences (segments) in Table V.

##### Accuracy

We evaluate the accuracy of our VMAF and encoding speed prediction models
against the ground truth values recorded within the training dataset (_cf._
Fig. 5(a) and Fig. 5(b)). The analysis reveals an average $R^{2}$ score of
0.886 for VMAF and 0.968 for encoding speed prediction models, showing a
strong positive correlation of the prediction. Furthermore, the average MAE of
the VMAF and encoding speed prediction models are 4.765 and 45.848,
respectively. Noteworthy, the minor differences in VMAF scores may not be
perceptible or bothersome to the audience. Viewers are generally more
concerned about smooth and uninterrupted streaming than small quality
fluctuations. Hence, the prediction errors are acceptable for live-streaming
applications.

##### Relative feature importance

Impurity-based feature importance measures the contribution of each input
feature to accurate predictions within the ensemble [66], critical for model
interpretability in random forest regression. In this paper, we measure
impurity using MAE. Features consistently leading to the most significant
impurity reduction across all the forest decision trees are the most
important. Fig. 5(c) and Fig. 5(d) show the impurity-based feature importance
measures corresponding to the features utilized in the VMAF and encoding speed
prediction models. The target bitrate in the logarithmic scale
($\log\left(b_{t}\right)$) is the most influential feature for VMAF
prediction, followed by the $h_{\text{S}}$, $E_{\text{S}}$, framerate, and
$L_{\text{S}}$. Similarly, the order of importance in encoding speed
prediction is $\log\left(b_{t}\right)$, $h_{\text{S}}$, $L_{\text{S}}$,
$E_{\text{S}}$, and framerate.

##### Framerate-preset prediction

We analyze the average framerate-preset predictions of CVFR in Figs. 5(e) and
5(f). On average, CVFR-ECO chooses 15 fps at 0.145 Mbps and 30 fps at 7.8
Mbps, as it always selects the fastest preset for encoding (ultrafast in x264)
as discussed in Section IV-B. Like CVFR-ECO, CVFR-HQ selects a lower framerate
when the target bitrate decreases. However, it tends towards slower presets as
the target bitrate drops because slower preset encodings at lower bitrates can
yield higher VMAF while satisfying the encoding speed constraint for low-
latency encoding.

##### Latency

We evaluate the pre-processing latency ($\tau_{\text{p}}$) in encoding
introduced by the video complexity feature extraction and the model inference
to predict the optimized bitrate-resolution-framerate-preset configurations.
We extract the features at an average rate of 352 fps over the entire dataset
(2160p resolution). This result is critical in future-proofing the system by
handling evolving content requirements (_e.g._ , 8K resolution or high
framerate content). The time to predict the framerate-preset for each
representation is $5\text{\,}\mathrm{ms}\text{p}$. As video complexity feature
extraction and the optimized framerate and preset prediction can execute
concurrently in real applications, the overall latency introduced by CVFR is
negligible.

###  VI-B Coding efficiency analysis

####  VI-B1 Rate-distortion (RD) analysis

Fig. 6 analyzes the RD curves of default encoding, bruteforce encoding [9],
CVFR-ECO encoding, and CVFR-HQ encoding for selected video sequences
(segments) of various video content complexities displayed in Table V. In most
cases, CVFR-HQ yields higher VMAF than the other encoding schemes at the same
target bitrates for all video complexity classes because it is specifically
optimized to maximize VMAF within the bounds of the target encoding speed.
Moreover, the RD curve of CVFR-HQ is very close to Bruteforce-HQ encoding [9],
demonstrating the high accuracy of the VMAF and encoding speed prediction.
CVFR-ECO yields higher VMAF than the default scheme at low bitrates due to the
selection of lower framerates. Furthermore, the VMAF difference between
consecutive RD points of CVFR-ECO and CVFR-HQ is at least the target JND,
assumed as six VMAF points in the figure.

TABLE V: Representative video sequences. Sequence | $E_{\text{S}}$ | $h_{\text{S}}$ | $L_{\text{S}}$
---|---|---|---
Bunny_s000 | 22.40 | 4.70 | 129.21
Characters_s000 | 45.42 | 36.88 | 134.56
Eldorado_s000 | 15.28 | 49.76 | 140.54
Eldorado_s005 | 100.37 | 9.23 | 109.06
HoneyBee_s000 | 42.93 | 7.91 | 103.00
RushHour_s000 | 47.75 | 19.70 | 101.66
Runners_s000 | 105.85 | 22.48 | 126.60
Wood_s000 | 124.72 | 47.03 | 119.57

####  VI-B2 Bjøntegaard delta rates (BDR)

We further evaluate the coding efficiency using BDRP, BDRV, BD-PSNR, and BD-
VMAF compared to the default encoding, as shown in Table VI. Bruteforce
encoding [9] (with and without JND-based representation elimination) yields
$100\text{\,}\mathrm{\char 37\relax}\text{p}$ accurate results representing
the highest bound of the compression efficiency improvement (in VMAF) compared
to the default encoding.

TABLE VI: Average encoding performance compared to the default encoding.

Method | $v_{\text{J}}$ | BDRP | BDRV | BD-PSNR | BD-VMAF | $\Delta S$ | $\Delta E_{\text{enc}}$ | $\Delta E_{\text{sto}}$
---|---|---|---|---|---|---|---|---
|  |  |  | [dB] |  |  |  |
Bruteforce-ECO |  | -17.81% | -16.41% | 0.73 | 5.94 | -10.94% | 267.11% | -20.69%
Bruteforce-ECO | 2 | -19.61% | -18.11% | 0.78 | 6.66 | -23.93% | 267.11% | -22.43%
Bruteforce-ECO | 4 | -20.13% | -19.00% | 0.79 | 6.86 | -42.79% | 267.11% | -61.30%
Bruteforce-ECO | 6 | -20.15% | -19.13% | 0.79 | 6.89 | -54.59% | 267.11% | -75.59%
Bruteforce-HQ |  | -52.33% | -37.90% | 2.39 | 9.89 | -10.94% | 19174.15% | -20.69%
Bruteforce-HQ | 2 | -51.96% | -39.18% | 2.39 | 10.33 | -38.80% | 19174.15% | -62.55%
Bruteforce-HQ | 4 | -52.56% | -40.04% | 2.44 | 10.62 | -52.87% | 19174.15% | -77.78%
Bruteforce-HQ | 6 | -53.06% | -40.41% | 2.44 | 10.74 | -59.01% | 19174.15% | -83.20%
Herrou  _et al._ [20] |  | -8.75% | -8.81% | 0.38 | 3.90 | -1.35% | -1.29% | -2.72%
CAPS [24] |  | -43.24% | -30.46% | 2.12 | 8.57 | -10.73% | 746.70% | -20.37%
CVFR-ECO |  | -16.37% | -11.17% | 0.52 | 3.72 | -1.18% | -10.33% | -20.38%
CVFR-ECO | 2 | -18.38% | -12.05% | 0.51 | 4.23 | -21.70% | -23.12% | -38.70%
CVFR-ECO | 4 | -17.25% | -13.22% | 0.53 | 4.43 | -40.15% | -39.54% | -64.18%
CVFR-ECO | 6 | -17.91% | -13.54% | 0.52 | 4.50 | -51.26% | -48.64% | -76.24%
CVFR-HQ |  | -51.69% | -37.44% | 2.28 | 9.13 | -21.11% | 638.84% | -40.99%
CVFR-HQ | 2 | -53.42% | -38.81% | 2.38 | 9.85 | -44.41% | 426.15% | -69.10%
CVFR-HQ | 4 | -54.00% | -39.53% | 2.42 | 10.09 | -55.83% | 322.03% | -80.49%
CVFR-HQ | 6 | -54.25% | -39.63% | 2.43 | 10.14 | -58.98% | 282.94% | -83.18%

Figure 7: Average cumulative bitrate of each video segment using the
considered encoding schemes.

(a) Encoding energy.

(b) Storage energy.

Figure 8: Comparison of average encoding and storage energy consumption for
each $4\text{\,}\mathrm{s}\text{p}$ video segment using the considered
encoding schemes.

###  VI-C Storage consumption

We evaluate the relative difference in the storage space between the
considered encoding schemes and the default encoding scheme to store all
bitrate ladder representations. CVFR-HQ achieved a remarkable reduction in
data size during video encoding (up to $58.98\text{\,}\mathrm{\char
37\relax}\text{p}$) compared to the alternative schemes, as observed in Table
VI, and ensures more efficient utilization of network resources. Higher
$\Delta S$ translates to reduced storage requirements and lower delivery
costs, which can be significant for large-scale streaming platforms. Lower
cumulative bitrates place less strain on the network infrastructure, reducing
the risk of network congestion.

###  VI-D Energy consumption

We conduct a comprehensive evaluation of encoding schemes by analyzing the
relative differences in energy consumption during encoding ($\Delta
E_{\text{enc}}$) and storage ($\Delta E_{\text{sto}}$) of the bitrate ladder
compared to the reference encoding scheme.

####  VI-D1 Encoding energy consumption

Predictably, CVFR-ECO outperforms the other schemes in encoding energy, as
shown in Fig. 8(a). Table VI shows a reduction in encoding energy consumption
reduction for CVFR-ECO compared to the default encoding by
$10.33\text{\,}\mathrm{\char 37\relax}\text{p}$, which scales further up to
$48.64\text{\,}\mathrm{\char 37\relax}\text{p}$, using JND-based
representation elimination ($v_{\text{J}}$=6). CVFR-ECO encodes video using
ultrafast preset, which yields the lowest encoding time and, subsequently, the
lowest encoding energy consumption, in contrast to CAPS and CVFR-HQ. Moreover,
lower framerates at low bitrates further reduce energy consumption. CAPS
results in the highest encoding energy, owing to the choice of slower presets
at lower target bitrates. CVFR-HQ yields lower energy consumption than CAPS,
attributed to the optimized framerate selection and preset selection. On
average, CAPS and CVFR-HQ consume $746.70\text{\,}\mathrm{\char
37\relax}\text{p}$ and $638.84\text{\,}\mathrm{\char 37\relax}\text{p}$ more
energy than the default encoding, respectively. However, the encoding energy
of CVFR-HQ is only $282.94\text{\,}\mathrm{\char 37\relax}\text{p}$ higher
than the default encoding, using JND-based representation elimination
($v_{\text{J}}$=6).

####  VI-D2 Storage energy consumption

Fig. 8(b) shows that CVFR-HQ and all the other state-of-the-art methods yield
reductions in storage energy compared to the default encoding, directly
influenced by the size and the time required to store the data on the disk, as
described in Eq. 8. Consequently, despite its emphasis on maximizing coding
efficiency and eliminating perceptually redundant representations, CVFR-HQ
achieves significant reductions in data size during video encoding, resulting
in remarkable storage energy savings and, hence, streaming/transmission
savings. According to Table VI, across various JND values, CVFR-HQ
demonstrates a noteworthy range of reductions, ranging from
$40.99\text{\,}\mathrm{\char 37\relax}\text{p}$ to
$83.18\text{\,}\mathrm{\char 37\relax}\text{p}$.

####  VI-D3 Summary

To summarize, CVFR-HQ is advantageous in scenarios where bandwidth
conservation is a primary concern, such as in regions with limited network
capacity or for streaming providers aiming to reduce data delivery costs.
Despite higher encoding energy consumption, CVFR-HQ delivers superior video
quality due to its efficient utilization of available bits, contributing to a
more satisfying streaming experience. Meanwhile, CVFR-ECO prioritizes energy
efficiency, lowering computational resource consumption, which is suitable for
reducing energy consumption in data centers. CVFR-ECO can still offer
bandwidth optimization, albeit to a lesser extent, compared to CVFR-HQ. The
choice of the encoding scheme should align with the overarching objectives and
constraints of the streaming service provider and its target audience.

##  VII Conclusions

This paper proposes a content-adaptive variable framerate encoding scheme
(CVFR) for adaptive live-streaming applications. CVFR includes a random
forest-based model, which predicts optimized framerate and encoding preset for
each bitrate ladder representation of a given video segment based on its
spatiotemporal characteristics. Furthermore, a JND-based representation
elimination algorithm is proposed to minimize the perceptual redundancy of the
representations. Two variations of CVFR: CVFR-ECO and CVFR-HQ are presented,
where the former predicts an optimized framerate for each representation using
the fastest preset. The experimental results show that, on average, CVFR-ECO
yields bitrate savings of $17.91\text{\,}\mathrm{\char 37\relax}\text{p}$ and
$13.54\text{\,}\mathrm{\char 37\relax}\text{p}$ to maintain the same PSNR and
VMAF, respectively, compared to the fixed framerate-fastest preset CBR
encoding of the reference HLS bitrate ladder using x264 encoder. This is
accompanied by a cumulative decrease of $48.64\text{\,}\mathrm{\char
37\relax}\text{p}$ in encoding energy needed for various representations and
$76.24\text{\,}\mathrm{\char 37\relax}\text{p}$ in storage energy, considering
a JND of six VMAF points. However, CVFR-HQ predicts the optimized framerate-
preset pair for each representation, which yields the highest possible
compression efficiency within the low latency encoding speed threshold. The
experimental results show that, on average, CVFR-HQ yields bitrate savings of
$54.25\text{\,}\mathrm{\char 37\relax}\text{p}$ and
$39.63\text{\,}\mathrm{\char 37\relax}\text{p}$ to maintain the same PSNR and
VMAF, respectively, compared to the fixed framerate-fastest preset CBR
encoding of the reference HLS bitrate ladder. Although the encoding energy
consumption increases to $282.94\text{\,}\mathrm{\char 37\relax}\text{p}$,
CVFR-HQ yields an $83.18\text{\,}\mathrm{\char 37\relax}\text{p}$ decrease in
storage energy, respectively, considering a JND of six VMAF points.

In the future, CVFR can be extended to high-framerate video streaming
applications, where the original video framerates are up to 120 fps. CVFR-ECO
can be extended for VoD applications using the slowest encoding preset and
sophisticated temporal filtering and frame interpolation methods, as the low-
latency constraint can be compromised.

## References

  * [1] Cisco, “Cisco Annual Internet Report (2018–2023) White Paper,” Mar. 2020. [Online]. Available: https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.pdf
  * [2] T. Stockhammer, “Dynamic Adaptive Streaming over HTTP –: Standards and Design Principles,” in _Proceedings of the Second Annual ACM Conference on Multimedia Systems_ , 2011, p. 133–144.
  * [3] Bitmovin, “The 6th Annual Bitmovin Video Developer Report.” [Online]. Available: https://bitmovin.com/wp-content/uploads/2022/12/bitmovin-6th-video-developer-report-2022-2023.pdf
  * [4] J. Giertz, W. Weiger, M. Törhönen, and J. Hamari, “Understanding the what and how of successful social live streaming,” Jul. 2020. [Online]. Available: http://ceur-ws.org/Vol-2637/paper17.pdf
  * [5] J. Žádník, M. Mäkitalo, J. Vanne, and P. Jääskeläinen, “Image and Video Coding Techniques for Ultra-Low Latency,” in _ACM Comput. Surv._ , vol. 54, no. 11s. New York, NY, USA: Association for Computing Machinery, Sep. 2022.
  * [6] M. Vijayaratnam, M. Cagnazzo, G. Valenzise, A. Trioux, and M. Kieffer, “Towards zero-latency video transmission through frame extrapolation,” in _IEEE International Conference on Image Processing, ICIP 2022_ , Bordeaux, France, Oct. 2022.
  * [7] P. Ramachandran, S. Yang, P. Tiwari, and G. Akisetty, “Content Adaptive Live Encoding with Open Source Codecs,” in _Proceedings of the 11th ACM Multimedia Systems Conference_ , 2020, p. 345–348.
  * [8] Apple Inc., “HLS Authoring Specification for Apple Devices.” [Online]. Available: https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices
  * [9] J. De Cock, Z. Li, M. Manohara, and A. Aaron, “Complexity-based consistent-quality encoding in the cloud,” in _2016 IEEE International Conference on Image Processing (ICIP)_ , 2016, pp. 1484–1488.
  * [10] D. Silhavy, C. Krauss, A. Chen, A.-T. Nguyen, C. Müller, S. Arbanowski, S. Steglich, and L. Bassbouss, “Machine Learning for Per-Title Encoding,” in _SMPTE Motion Imaging Journal_ , vol. 131, no. 3, 2022, pp. 42–50.
  * [11] A. V. Katsenou, J. Sole, and D. R. Bull, “Content-gnostic Bitrate Ladder Prediction for Adaptive Video Streaming,” in _2019 Picture Coding Symposium (PCS)_ , 2019, pp. 1–5.
  * [12] V. V. Menon, P. T. Rajendran, C. Feldmann, K. Schoeffmann, M. Ghanbari, and C. Timmerer, “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023, pp. 1–1.
  * [13] V. V. Menon, H. Amirpour, M. Ghanbari, and C. Timmerer, “OPTE: Online Per-Title Encoding for Live Video Streaming,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 1865–1869.
  * [14] M. G. Koziri, P. K. Papadopoulos, N. Tziritas, T. Loukopoulos, S. U. Khan, and A. Y. Zomaya, “Efficient cloud provisioning for video transcoding: Review, open challenges and future opportunities,” in _IEEE Internet Computing_ , vol. 22, no. 5. IEEE, 2018, pp. 46–55.
  * [15] A. Stephens, C. Tremlett-Williams, L. Fitzpatrick, L. Acerini, M. Anderson, and N. Crabbendam, “The Carbon Impacts of Video Streaming,” Jun. 2021. [Online]. Available: https://www.carbontrust.com/our-work-and-impact/guides-reports-and-tools/carbon-impact-of-video-streaming
  * [16] R. Farahani, A. Bentaleb, E. Çetinkaya, C. Timmerer, R. Zimmermann, and H. Hellwagner, “Hybrid p2p-cdn architecture for live video streaming: An online learning approach,” in _GLOBECOM 2022-2022 IEEE Global Communications Conference_. IEEE, 2022, pp. 1911–1917.
  * [17] A. Bianco, R. Mashayekhi, and M. Meo, “Energy consumption for data distribution in content delivery networks,” in _2016 IEEE International Conference on Communications (ICC)_. IEEE, 2016, pp. 1–6.
  * [18] M. A. Hoque, M. Siekkinen, J. K. Nurminen, M. Aalto, and S. Tarkoma, “Mobile multimedia streaming techniques: QoE and energy saving perspective,” in _Pervasive and Mobile Computing_ , vol. 16. Elsevier BV, Jan. 2015, pp. 96–114.
  * [19] J. Zhang, G. Fang, C. Peng, M. Guo, S. Wei, and V. Swaminathan, “Profiling Energy Consumption of DASH Video Streaming over 4G LTE Networks,” in _Proceedings of the 8th International Workshop on Mobile Video_. New York, NY, USA: Association for Computing Machinery, 2016.
  * [20] G. Herrou, C. Bonnineau, W. Hamidouche, P. Dumenil, J. Fournier, and L. Morin, “Quality-driven Variable Frame-Rate for Green Video Coding in Broadcast Applications,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , 2020, pp. 1–1.
  * [21] Z. Ma, M. Xu, Y.-F. Ou, and Y. Wang, “Modeling of Rate and Perceptual Quality of Compressed Video as Functions of Frame Rate and Quantization Stepsize and Its Applications,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 22, no. 5, 2012, pp. 671–682.
  * [22] M. Afonso, F. Zhang, and D. R. Bull, “Video Compression Based on Spatio-Temporal Resolution Adaptation,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 1, 2019, pp. 275–280.
  * [23] A. V. Katsenou, D. Ma, and D. R. Bull, “Perceptually-Aligned Frame Rate Selection Using Spatio-Temporal Features,” in _2018 Picture Coding Symposium (PCS)_ , 2018, pp. 288–292.
  * [24] V. V. Menon, H. Amirpour, P. T. Rajendran, M. Ghanbari, and C. Timmerer, “Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming,” in _2022 Picture Coding Symposium (PCS)_ , 2022, pp. 253–257.
  * [25] D. Silveira, M. Porto, and S. Bampi, “Performance and energy consumption analysis of the X265 video encoder,” in _2017 25th European Signal Processing Conference (EUSIPCO)_ , 2017, pp. 1519–1523.
  * [26] D. Yuan, T. Zhao, Y. Xu, H. Xue, and L. Lin, “Visual JND: A Perceptual Measurement in Video Coding,” in _IEEE Access_ , vol. 7, 2019, pp. 29 014–29 022.
  * [27] Z. Li _et al._ , “VMAF: The journey continues,” in _Netflix Technology Blog_ , vol. 25, 2018.
  * [28] P. C. Madhusudana, X. Yu, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “Subjective and Objective Quality Assessment of High Frame Rate Videos,” 2020\.
  * [29] J. Y. C. Chen and J. E. Thropp, “Review of Low Frame Rate Effects on Human Performance,” in _IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans_ , vol. 37, no. 6, 2007.
  * [30] Q. Huynh-Thu and M. Ghanbari, “Impact of jitter and jerkiness on perceived video quality,” in _2nd International Workshop on Video Processing for Consumer Electronics (VPQM 2006)_ , Jan. 2006.
  * [31] O. Alaoui-Fdili, Y. Fakhri, P. Corlay, F.-X. Coudoux, and D. Aboutajdine, “Energy consumption analysis and modelling of a H. 264/AVC intra-only based encoder dedicated to WVSNs,” in _2014 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2014, pp. 1189–1193.
  * [32] VideoLAN, “x264.” [Online]. Available: https://www.videolan.org/developers/x264.html
  * [33] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 13, no. 7, 2003, pp. 560–576.
  * [34] H. Amirpour, V. V. Menon, S. Afzal, M. Ghanbari, and C. Timmerer, “VCD: Video Complexity Dataset,” in _Proceedings of the 13th ACM Multimedia Systems Conference_ , 2022, p. 234–239.
  * [35] Q. Huang, S. Y. Jeong, S. Yang, D. Zhang, S. Hu, H. Y. Kim, J. S. Choi, and C.-C. J. Kuo, “Perceptual Quality Driven Frame-Rate Selection (PQD-FRS) for High-Frame-Rate Video,” in _IEEE Transactions on Broadcasting_ , vol. 62, no. 3, 2016, pp. 640–653.
  * [36] A. Mackin, F. Zhang, M. A. Papadopoulos, and D. Bull, “Investigating the impact of high frame rates on video compression,” in _2017 IEEE International Conference on Image Processing (ICIP)_ , 2017, pp. 295–299.
  * [37] Y.-L. Chang, T.-L. Lin, and P. C. Cosman, “Network-Based H.264/AVC Whole-Frame Loss Visibility Model and Frame Dropping Methods,” in _IEEE Transactions on Image Processing_ , vol. 21, no. 8, 2012, pp. 3353–3363.
  * [38] V. V. Menon, H. Amirpour, M. Ghanbari, and C. Timmerer, “CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming,” in _2022 Data Compression Conference (DCC)_ , 2022, pp. 475–475.
  * [39] W. Bao, X. Zhang, L. Chen, and Z. Gao, “KalmanFlow 2.0: Efficient Video Optical Flow Estimation via Context-Aware Kalman Filtering,” in _IEEE Transactions on Image Processing_ , vol. 28, no. 9, 2019, pp. 4233–4246.
  * [40] H. Lim and H. W. Park, “A Symmetric Motion Estimation Method for Motion-Compensated Frame Interpolation,” in _IEEE Transactions on Image Processing_ , vol. 20, no. 12, 2011, pp. 3653–3658.
  * [41] R.-Y. Tseng, Y.-K. Liu, J.-C. Chen, and K. W. Lin, “Adaptive Frame Interpolation using an End-to-End Deep Net with High Quality Flow Estimation,” in _2019 International Conference on Technologies and Applications of Artiﬁcial Intelligence (TAAI)_ , 2019, pp. 1–4.
  * [42] F. Zhang, A. Mackin, and D. R. Bull, “A frame rate dependent video quality metric based on temporal wavelet decomposition and spatiotemporal pooling,” in _2017 IEEE International Conference on Image Processing (ICIP)_ , 2017, pp. 300–304.
  * [43] VideoLAN, “x265.” [Online]. Available: https://www.videolan.org/developers/x265.html
  * [44] F. Nasiri, W. Hamidouche, L. Morin, N. Dhollande, and J.-Y. Aubié, “Multi-Preset Video Encoder Bitrate Ladder Prediction,” in _Proceedings of the 2nd International Workshop on Design, Deployment, and Evaluation of Network-Assisted Video Streaming_ , 2022, p. 8–13.
  * [45] Q. Huangyuan, L. Song, Z. Luo, X. Wang, and Y. Zhao, “Performance evaluation of H.265/MPEG-HEVC encoders for 4K video sequences,” in _Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific_ , 2014, pp. 1–8.
  * [46] S. Zvezdakov, D. Kondranin, and D. Vatolin, “Machine-Learning-Based Method for Content-Adaptive Video Encoding,” in _2021 Picture Coding Symposium (PCS)_ , 2021, pp. 1–5.
  * [47] T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , Aug. 2016, pp. 785–794.
  * [48] A. Bentaleb, B. Taani, A. C. Begen, C. Timmerer, and R. Zimmermann, “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP,” in _IEEE Communications Surveys Tutorials_ , vol. 21, no. 1, 2019, pp. 562–585.
  * [49] J. You and J. Korhonen, “Deep Neural Networks for No-Reference Video Quality Assessment,” in _2019 IEEE International Conference on Image Processing (ICIP)_ , 2019, pp. 2349–2353.
  * [50] ITU-T, “P.910 : Subjective video quality assessment methods for multimedia applications,” Nov. 2021. [Online]. Available: https://www.itu.int/rec/T-REC-P.910-202111-I/en
  * [51] V. V. Menon, C. Feldmann, K. Schoeffmann, M. Ghanbari, and C. Timmerer, “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming,” in _Proceedings of the First International Workshop on Green Multimedia Systems_ , 2023, p. 16–18.
  * [52] Q. Cai, Z. Chen, D. O. Wu, and B. Huang, “Real-Time Constant Objective Quality Video Coding Strategy in High Efficiency Video Coding,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 7, 2020, pp. 2215–2228.
  * [53] J. Y. Lin, L. Jin, S. Hu, I. Katsavounidis, Z. Li, A. Aaron, and C.-C. J. Kuo, “Experimental design and analysis of JND test on coded image/video,” in _Applications of digital image processing XXXVIII_ , vol. 9599. SPIE, 2015, pp. 324–334.
  * [54] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset,” in _2016 IEEE international conference on image processing (ICIP)_. IEEE, 2016, pp. 1509–1513.
  * [55] H. Wang, I. Katsavounidis, J. Zhou, J. Park, S. Lei, X. Zhou, M.-O. Pun, X. Jin, R. Wang, X. Wang _et al._ , “VideoSet: A large-scale compressed video quality dataset based on JND measurement,” in _Journal of Visual Communication and Image Representation_ , vol. 46. Elsevier, 2017, pp. 292–302.
  * [56] J. Zhu, S. Ling, Y. Baveye, and P. Le Callet, “A Framework to Map VMAF with the Probability of Just Noticeable Difference between Video Encoding Recipes,” in _2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)_ , 2022, pp. 1–5.
  * [57] V. V. Menon, J. Zhu, P. T. Rajendran, H. Amirpour, P. L. Callet, and C. Timmerer, “Just Noticeable Difference-Aware Per-Scene Bitrate-Laddering for Adaptive Video Streaming,” in _2023 IEEE International Conference on Multimedia and Expo (ICME)_ , 2023, pp. 1673–1678.
  * [58] P. K. Tiwari, V. V. Menon, J. Murugan, J. Chandrasekaran, G. S. Akisetty, P. Ramachandran, S. K. Venkata, C. A. Bird, and K. Cone, “Accelerating x265 with Intel® Advanced Vector Extensions 512,” in _White Paper on the Intel Developers Page_ , 2018. [Online]. Available: https://www.intel.com/content/dam/develop/external/us/en/documents/mcw-intel-x265-avx512.pdf
  * [59] A. Kah, C. Friedrich, T. Rusert, C. Burgmair, W. Ruppel, and M. Narroschke, “Fundamental relationships between subjective quality, user acceptance, and the VMAF metric for a quality-based bit-rate ladder design for over-the-top video streaming services,” in _Applications of Digital Image Processing XLIV_ , vol. 11842, 2021, p. 118420Z.
  * [60] J. Ozer, “Finding the Just Noticeable Difference with Netflix VMAF,” Sep. 2017\. [Online]. Available: https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html
  * [61] D. C. Montgomery, E. A. Peck, and G. G. Vining, _Introduction to Linear Regression Analysis (4th ed.)_. Wiley & Sons, 2006.
  * [62] L. Breiman, “Random Forests,” in _Machine Learning_ , vol. 45, 2001. [Online]. Available: https://doi.org/10.1023/A:1010933404324
  * [63] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine Learning in Python,” in _Journal of Machine Learning Research_ , vol. 12, 2011, pp. 2825–2830.
  * [64] HSTP-VID-WPOM, “Working practices using objective metrics for evaluation of video coding efficiency experiments,” in _International Telecommunication Union_ , 2020. [Online]. Available: http://handle.itu.int/11.1002/pub/8160e8da-en
  * [65] BCG-GAMMA and MILA, “CodeCarbon.” [Online]. Available: https://codecarbon.io/
  * [66] J. Kazemitabar, A. Amini, A. Bloniarz, and A. S. Talwalkar, “Variable Importance Using Decision Trees,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017.

|  Vignesh V Menon is a researcher at Fraunhofer HHI, Berlin, and a Ph.D.
candidate at the Institute of Information Technology (ITEC), Alpen-Adria-
Universität Klagenfurt (AAU). He received a B.Tech. in Electronics and
Communication Engineering from Amrita Vishwa Vidyapeetham University, India,
and an M.Sc. in Information and Network Engineering from KTH Royal Institute
of Technology, Sweden, in 2016 and 2020, respectively. He was a software
engineer developing video encoding software solutions in MulticoreWare Inc.,
India, between 2016-2018 and Divideon, Sweden, between 2018-2020. His research
interests are video streaming, image, and video compression. Further
information at https://vigneshvijay94.com.
---|---
|  Samira Afzal is a Postdoctoral researcher at Alpen-Adria-Universität
Klagenfurt (AAU), currently working on the GAIA project. She received her
Ph.D. in Electrical Engineering at the School of Electrical and Computer
Engineering (FEEC), University of Campinas (UNICAMP), in November 2019. She
received her master’s degree in IT engineering at Sharif University of
Technology (SUT) in 2011.
---|---
|  Prajit T Rajendran is a Ph.D. candidate at Université Paris-Saclay, working
on his doctoral research project in collaboration with CEA LIST and LNE
France. He received his B.Engg degree in Electronics and Communication
Engineering from Ramaiah Institute of Technology, Bangalore, India, in 2018
and subsequently an M.Sc. degree in Information and Network Engineering from
KTH Royal Institute of Technology, Sweden, in 2020. His research interests
include computer vision, deep learning, active learning, and human-in-the-loop
artificial intelligence.
---|---
|  Klaus Schoeffmann is an associate professor at the Institute of Information
Technology (ITEC). He received his Ph.D. degree and Habilitation (venia
docendi) from Klagenfurt University in both 2009 and 2015, respectively, in
computer science. He is currently an Associate Professor with the Institute of
Information Technology (ITEC), Klagenfurt University, Klagenfurt, Austria. His
research focuses on video analytics and interactive multimedia systems,
particularly in medicine. He has co-authored more than 110 publications on
various topics in multimedia. He has co-organized several international
conferences, workshops, and special sessions in the field of multimedia.
Furthermore, he is a co-founder of the Video Browser Showdown (VBS), a member
of the ACM, and a regular reviewer for international conferences and journals
in multimedia.
---|---
|  Radu Prodan is a professor in distributed systems at ITEC, University of
Klagenfurt, Austria. Previously, he was an associate professor at the
University of Innsbruck, Austria. He received his Ph.D. in 2004 from the
Vienna University of Technology. His research interests are performance,
optimization, and resource management tools for parallel and distributed
systems. He participated in numerous projects and coordinated the European
Union projects ARTICONF and Graph-Massivizer. He co-authored over 200
publications and received three IEEE best paper awards.
---|---
|  Christian Timmerer (M’08-SM’16) is a full professor of computer science at
Alpen-Adria-Universität Klagenfurt (AAU), Institute of Information Technology
(ITEC), and he is the director of the Christian Doppler (CD) Laboratory ATHENA
(https://athena.itec.aau.at/). His research interests include multimedia
systems, immersive multimedia communication, streaming, adaptation, and
quality of experience, where he co-authored seven patents and more than 300
articles. He was the general chair of WIAMIS 2008, QoMEX 2013, MMSys 2016, and
PV 2018 and has participated in several EC-funded projects, notably DANAE,
ENTHRONE, P2P-Next, ALICANTE, SocialSensor, COST IC1003 QUALINET, ICoSOLE, and
SPIRIT. He also participated in ISO/MPEG work for several years, notably in
MPEG-21, MPEG-M, MPEG-V, and MPEG-DASH, where he served as standard editor. In
2012, he cofounded Bitmovin (http://www.bitmovin.com/) to provide professional
services around MPEG-DASH, where he holds the position of the Chief Innovation
Officer (CIO) –- Head of Research and Standardization. Further information at
http://timmerer.com.
---|---