# Video Super-Resolution for Optimized Bitrate and Green Online Streaming

Vignesh V Menon1, Prajit T Rajendran2, Amritha Premkumar3, Benjamin Bross1,
Detlev Marpe1  1Video Communication and Applications Department, Fraunhofer
HHI, Berlin, Germany  2Universite Paris-Saclay, CEA, List, F-91120, Palaiseau,
France  3Department of Computer Science, Rheinland-Pfälzische Technische
Universität, Kaiserslautern, Germany

###### Abstract

Conventional per-title encoding schemes strive to optimize encoding
resolutions to deliver the utmost perceptual quality for each bitrate ladder
representation. Nevertheless, maintaining encoding time within an acceptable
threshold is equally imperative in online streaming applications. Furthermore,
modern client devices are equipped with the capability for fast deep-learning-
based video super-resolution (VSR) techniques, enhancing the perceptual
quality of the decoded bitstream. This suggests that opting for lower
resolutions in representations during the encoding process can curtail the
overall energy consumption without substantially compromising perceptual
quality. In this context, this paper introduces a video super-resolution-based
latency-aware optimized bitrate encoding scheme (ViSOR) designed for online
adaptive streaming applications. ViSOR determines the encoding resolution for
each target bitrate, ensuring the highest achievable perceptual quality after
VSR within the bound of a maximum acceptable latency. Random forest-based
prediction models are trained to predict the perceptual quality after VSR and
the encoding time for each resolution using the spatiotemporal features
extracted for each video segment. Experimental results show that ViSOR
targeting fast super-resolution convolutional neural network (FSRCNN) achieves
an overall average bitrate reduction of $24.65\text{\,}\mathrm{\char
37\relax}\text{p}$ and $32.70\text{\,}\mathrm{\char 37\relax}\text{p}$ to
maintain the same PSNR and VMAF, compared to the HTTP Live Streaming (HLS)
bitrate ladder encoding of 4 s segments using the x265 encoder, when the
maximum acceptable latency for each representation is set as two seconds.
Considering a just noticeable difference (JND) of six VMAF points, the average
cumulative storage consumption and encoding energy for each segment is reduced
by $79.32\text{\,}\mathrm{\char 37\relax}\text{p}$ and
$68.21\text{\,}\mathrm{\char 37\relax}\text{p}$, respectively, contributing
towards greener streaming.

###### Index Terms:

Video super-resolution, dynamic resolution encoding, reduced latency, green
streaming.

##  I Introduction

_HTTP Adaptive Streaming_(HAS) has become the de-facto standard in delivering
video content for various clients regarding internet speeds and device types.
The main idea behind HAS is to divide the video content into segments and
encode each segment at various bitrates and resolutions, called
representations, which are stored in plain HTTP servers. These representations
are stored to continuously adapt the video delivery to the network conditions
and device capabilities of the client [1]. Traditionally, a fixed bitrate
ladder, _e.g._ , _HTTP Live Streaming_(HLS) bitrate ladder [2], is used in
online streaming applications. However, recently, per-title encoding [3] is
introduced for online streaming, _i.e._ , estimating encoding resolution [4,
5, 6], bitrate [7], framerate [8], encoding preset [9], CPU resources [10],
_etc._ which yield the highest perceptual quality for a given video content.
Notably, in online streaming, where real-time viewing is crucial, encoding
delays are typically expected to be very low, often in a few seconds or less.
The minimal delay ensures viewers receive the action as close to real-time as
possible [11]. Moreover, reducing encoding time is also essential, resulting
in lower energy consumption at the data centers during the encoding process.
This is particularly important as environmental consciousness grows and
companies seek to adopt greener practices to reduce energy consumption in the
streaming chain, _i.e._ , encoding (server-side), transmission (network-side),
and decoding (client-side). Fig. 1 shows the rate versus encoding time curve
of two representative sequences encoded at various encoding resolutions. It is
observed that lowering encoding resolution can contribute to lowering the
encoding latency.

#### Video super-resolution for green streaming

In recent years, client devices have increased processing capability owing to
the increasing RAM capacity, CPU power, and, more importantly, the
introduction of powerful GPUs. Hence, it is possible to execute deep neural
network (DNN)-based video super-resolution (VSR) approaches in the client
devices with generic graphics processing unit (GPU) to enhance low-resolution
bitstreams into high-quality streams, improving viewer retention and
satisfaction [12]. Some of the popular VSR models include FSRCNN [13], ESPCN
[14], EDSR [15], EVSRNet [16], CARN [17], SRGAN [18], and RBPN [19]. Even when
the encoding resolution is reduced to control the encoding latency, client-
side VSR can reduce the associated quality drop. As shown in Fig. 1, using
EDSR-based client-side VSR for 360p, 720p, and 1080p representations improves
visual quality (in terms of VMAF [20]) at lower bitrates. Moreover, client-
side VSR helps optimize bitrate allocation, which lowers the data transmission
costs of streaming. It complements traditional server-side adaptive streaming
techniques and contributes to a more energy-efficient and customized video
delivery strategy, ultimately improving the overall viewing experience for the
audience.

(a) Characters_s000

(b) Dolls_s000

Figure 1: Encoding results (encoding time and VMAF [20]) of Characters_s000
and Dolls_s000 sequences of VCD dataset [21] encoded at various resolutions,
with and without client-side VSR using EDSR [15].

Various related works in the literature explored estimating the encoding
resolution to yield maximum perceptual quality [4, 3, 5, 6, 22]. However, as
observed in comparing the rate-distortion curves in Fig. 1, using VSR
approaches pushes the upper-quality boundary of the convex-hull [3],
especially at lower bitrates. It is also observed that the cross-over points
of the RD curves of the considered resolutions change when VSR is applied.
Hence, predicting the perceptual quality of representations after VSR is
mandatory to select optimized encoding resolutions for the bitrate ladder
representations, which, to our knowledge, has not been investigated yet.
Furthermore, it is also observed that the perceptual redundancy between
representations increases with VSR. Hence, JND-aware representation
elimination [8] can save encoding energy and streaming bandwidth costs.

#### Contributions

This paper proposes ViSOR, a VSR-based encoding solution for green online
video streaming. It includes an encoding latency-aware dynamic resolution
encoding scheme for online adaptive streaming applications where the clients
are assumed to have GPU capability for real-time VSR. To the best of our
knowledge, ViSOR presents the pioneering effort in estimating optimized
encoding resolution for each target bitrate, which yields the maximum
perceptual quality after client-side VSR techniques, constrained to a maximum
acceptable encoding latency. A random forest-based model is trained to predict
the VMAF score of the representation viewed by the client. Furthermore, based
on the predicted VMAF scores, perceptually redundant representations are
eliminated from the bitrate ladder, saving the encoding energy and streaming
costs.

Figure 2: Encoding architecture using ViSOR for green online streaming.

##  II ViSOR architecture

The architecture of ViSOR is shown in Fig. 2. ViSOR receives the set of
supported resolutions ($\mathcal{R}$) and set of bitrates ($\mathcal{B}$),
maximum acceptable encoding latency ($\tau_{\text{L}}$), target JND
($v_{\text{J}}$), the maximum quality threshold ($v_{\text{T}}$) and the
target VSR model as input. It extracts spatiotemporal complexity features of
the input video segment and uses them to predict optimized encoding resolution
for each target bitrate. Furthermore, it eliminates perceptually redundant
representations based on the predicted quality metric (_i.e._ , VMAF in this
paper) of each representation. ViSOR comprises three phases (_cf._ Fig. 2):

  1. 1

spatiotemporal complexity feature extraction;

  2. 2

optimized resolution prediction;

  3. 3

JND-aware representation elimination.

The encoding process is performed exclusively for the selected bitrate-
resolution combinations for every video segment. On the client side, VSR is
applied to only low-resolution decoded representations before display.

###  II-A Spatiotemporal complexity feature extraction

ViSOR leverages three _discrete cosine transform_ energy-based features: (i)
the mean texture energy ($E_{\text{Y}}$), (ii) the mean gradient of the
texture energy ($h$), and (iii) the mean brightness ($L_{\text{Y}}$) extracted
using _video complexity analyzer_ (VCA) as the spatiotemporal features of
video segments [23, 22].

###  II-B Optimized resolution estimation

The optimized resolution selection in ViSOR based on bitrate, target latency,
and video complexity features is decomposed into two parts, as discussed
below.

####  II-B1 Modeling

The perceptual quality after VSR, $v_{(r_{t},b_{t})}$ and encoding time
$\tau_{(r_{t},b_{t})}$ of $t^{th}$ representation relies on video complexity
features $E_{\text{Y}}$, $h$, $L_{\text{Y}}$, encoding resolution $r_{t}$, and
target bitrate $b_{t}$ [24, 8]:

$\displaystyle v_{\left(r_{t},b_{t}\right)}$
$\displaystyle=f_{\text{V}}\left(E_{\text{Y}},h,L_{\text{Y}},r_{t},b_{t}\right);$
(1) $\displaystyle\tau_{\left(r_{t},b_{t}\right)}$
$\displaystyle=f_{\tau}\left(E_{\text{Y}},h,L_{\text{Y}},r_{t},b_{t}\right).$
(2)

####  II-B2 Optimization

ViSOR optimizes the perceptual quality of encoded video segments after VSR
while adhering to real-time processing constraints. It predicts the optimized
resolution of the $t$th representation to maximize the compression efficiency
while maintaining the encoding time below the threshold $\tau_{\text{L}}$. The
optimization function is:

$\displaystyle\hat{r}_{t}$
$\displaystyle=\operatorname*{arg\,max}_{r\in\mathcal{R}}\hat{v}_{(r,b_{t})}$
$\displaystyle c.t.\hskip
10.00002pt\hat{\tau}_{(r,b_{t})}\leq\tau_{\text{L}},$ (3)

where $\hat{v}_{(r,b_{t})}$ and $\hat{\tau}_{(r,b_{t})}$ are the predicted
VMAF after VSR and encoding speed of the representation $(r,b_{t})$.

###  II-C JND-aware representation elimination

In practice, empirical evidence demonstrates a striking similarity in VMAF
scores across various representations, indicating perceptual redundancy within
the bitrate ladder (as shown in Fig 1(a)). As a result, this redundancy
signifies inefficient energy utilization during data encoding, storage, and
transmission, failing to augment the quality of experience (QoE). To mitigate
this perceptual redundancy, ViSOR employs the notion of the _just noticeable
difference_ (JND) threshold. The JND threshold signifies the minimum threshold
at which the human eye can discern differences in quality [25, 26, 27, 28].
ViSOR uses the JND-aware representation elimination algorithm proposed in our
previous work [8]. However, it is described in this paper (_cf._ Algorithm 1)
to make it self-contained.

Input:

$m$ : number of representations in $\mathcal{B}$

$\mathcal{Q}=\bigcup_{t=1}^{m}\left\\{\left(\hat{r}_{t},b_{t},\hat{v}_{t}\right)\right\\}$:representations
with predicted resolution

$\hat{v}_{t}~{};1\leq t\leq m$: predicted VMAF

$v_{\text{T}}$ : maximum VMAF threshold

$v_{\text{J}}$ : average target JND

Output: $\hat{\mathcal{Q}}=\left(\hat{r},b\right)$: set of encoding
configurations

1:
$\hat{\mathcal{Q}}\leftarrow\left\\{\left(\hat{r}_{1},b_{1}\right)\right\\}$

2: $u\leftarrow 1$

3: if $\hat{v}_{1}\geq v_{\text{T}}$ then

4: return $\hat{\mathcal{Q}}$

5: end if

6: $t\leftarrow 2$

7: while $t\leq m$ do

8: if $\hat{v}_{t}-\hat{v}_{u}\geq v_{\text{J}}$ then

9:
$\hat{\mathcal{Q}}\leftarrow\hat{\mathcal{Q}}\cup\left\\{\left(\hat{r}_{t},b_{t}\right)\right\\}$

10: $u\leftarrow t$

11: if $\hat{v}_{t}\geq v_{\text{T}}$ then

12: return $\hat{\mathcal{Q}}$

13: end if

14: end if

14: $t\leftarrow t+1$

15: end while

16: return $\hat{\mathcal{Q}}$

Algorithm 1 JND-based representation elimination.

Assume the predicted VMAF difference between two representations is below
$v_{\text{J}}$; consequently, the higher bitrate representation within this
scope is removed. Moreover, if the predicted VMAF is greater than the maximum
VMAF threshold value ($v_{\text{T}}$) at which the representation achieves
perceptual losslessness, said representation is excluded from the bitrate
ladder. In this manner, ViSOR effectively reduces the overall encoding
latency.

##  III Experimental results

###  III-A Test methodology

We use 400 sequences ($80\text{\,}\mathrm{\char 37\relax}\text{p}$ of the
sequences) from the Video Complexity Dataset [21] as the training dataset, and
the remaining (_i.e._ , $20\text{\,}\mathrm{\char 37\relax}\text{p}$) is used
as the test dataset. The experimental parameters used to evaluate ViSOR are
shown in Table I. We encode the sequences at 30 fps with the fastest encoding
preset supported by the x265 [29] encoder, _i.e._ , _ultrafast_ preset, on a
dual-processor server with Intel Xeon Gold 5218R (80 cores, frequency at 2.10
GHz). VCA and x265 are run using eight CPU threads with x86 SIMD optimization
[30]. We classify the maximum acceptable encoding times ($\tau_{\text{L}}$) as
ultra-low latency ($1\text{\,}\mathrm{s}\text{p}$), low latency
($2\text{\,}\mathrm{s}\text{p}$), moderate latency
($4\text{\,}\mathrm{s}\text{p}$), and standard latency
($8\text{\,}\mathrm{s}\text{p}$). We measure VSR results on an NVIDIA GeForce
GT 710 GPU (frequency at 954 MHz). Though EDSR, ESPCN, FSRCNN, and LapSRN
models are available in OpenCV, we use FSRCNN for evaluation, as it yields the
shortest throughput time. It is noteworthy that EDSR yields the highest VMAF
improvements but with a throughput time of 430x compared to that of FSRCNN.

We compare ViSOR with the following state-of-the-art dynamic resolution
estimation schemes:

  1. 1.

Default: This scheme employs a fixed bitrate ladder, _i.e._ , a fixed set of
bitrate-resolution pairs. This paper uses the HLS bitrate ladder specified in
the Apple authoring specifications [2] as the fixed bitrate ladder.

  2. 2.

OPTE [22]: This scheme predicts optimized resolution, which yields the highest
VMAF for a given target bitrate.

Other state-of-the-art methods in the literature [3, 6, 4, 5] are not
considered for evaluation, as they are unsuitable for live streaming
applications.

TABLE I: Experimental parameters used to evaluate ViSOR.

_Parameter_ | _Values_
---|---
$\mathcal{R}$ | { 360, 720, 1080, 2160 }
$\mathcal{B}$ | 0.145 | 0.300 | 0.600 | 0.900 | 1.600 | 2.400
| 3.400 | 4.500 | 5.800 | 8.100 | 11.600 | 16.800
$\tau_{\text{L}}$ | 1s, 2s, 4s, 8s, $\infty$
$v_{\text{J}}$ | 2 | 4 | 6
$v_{\text{T}}$ | 98 | 96 | 94
Target encoder | x265 (ultrafast preset)
Target VSR | no VSR, FSRCNN

(a) Beauty_s000

(b) Park_s000

(c) RaceNight_s000

(d) Wood_s000

Figure 3: Rate-distortion (RD) curves and encoding times of the representative
video sequences (segments) using Default encoding (blue line), OPTE (purple
line), ViSOR without VSR (red line), and ViSOR with FSRCNN-based VSR (green
line) for $\tau_{\text{L}}$=$2\text{\,}\mathrm{s}\text{p}$. TABLE II: Average
results of ViSOR compared to Default encoding [2].

Parameters | Without VSR | With FSRCNN-based VSR
---|---|---
Remark | $\tau_{\text{L}}$ | $v_{\text{J}}$ | $BDR_{\text{P}}$ | $BDR_{\text{V}}$ | BD-PSNR | BD-VMAF | $\Delta E$ | $\Delta S$ | $\overline{\tau_{\text{E}}}$ | $BDR_{\text{P}}$ | $BDR_{\text{V}}$ | BD-PSNR | BD-VMAF | $\Delta E$ | $\Delta S$ | $\overline{\tau_{\text{E}}}$
Highest quality (OPTE [22]) | $\infty$ | - | -20.49% | -27.96% | 1.37 dB | 6.25 | 91.41% | 1.74% | 2.54 s | -28.33% | -37.05% | 1.57 dB | 7.73 | -15.86% | -0.58% | 1.11 s
Ultra-low latency | $1\text{\,}\mathrm{s}\text{p}$ | 2 | -15.57% | -22.97% | 0.54 dB | 4.64 | -46.47% | -71.88% | 0.71 s | -24.75% | -33.06% | 0.84 dB | 6.55 | -50.49% | -67.87% | 0.66 s
4 | -14.22% | -21.43% | 0.52 dB | 4.72 | -54.18% | -80.32% | 0.61 s | -22.81% | -32.14% | 0.88 dB | 6.83 | -59.36% | -78.49% | 0.54 s
6 | -10.91% | -19.04% | 0.39 dB | 4.63 | -61.08% | -85.35% | 0.52 s | -19.78% | -30.30% | 0.73 dB | 6.76 | -66.93% | -83.96% | 0.44 s
Low latency | $2\text{\,}\mathrm{s}\text{p}$ | 2 | -15.96% | -24.26% | 0.64 dB | 5.81 | -31.66% | -53.80% | 0.77 s | -27.19% | -34.54% | 0.97 dB | 6.58 | -46.25% | -58.87% | 0.71 s
4 | -14.97% | -23.53% | 0.53 dB | 5.39 | -44.41% | -69.05% | 0.69 s | -26.32% | -33.84% | 0.98 dB | 6.88 | -59.09% | -71.70% | 0.54 s
6 | -13.46% | -22.36% | 0.48 dB | 4.56 | -51.67% | -77.17% | 0.57 s | -24.65% | -32.70% | 0.93 dB | 6.81 | -68.21% | -79.32% | 0.42 s
Moderate latency | $4\text{\,}\mathrm{s}\text{p}$ | 2 | -17.76% | -26.93% | 0.94 dB | 5.67 | -3.49% | -46.49% | 1.51 s | -26.27% | -34.91% | 1.22 dB | 6.59 | -44.12% | -50.29% | 0.74 s
4 | -18.64% | -26.34% | 0.85 dB | 5.83 | -15.95% | -62.87% | 1.12 s | -25.09% | -34.60% | 1.19 dB | 6.89 | -56.59% | -66.80% | 0.58 s
6 | -17.38% | -26.22% | 0.91 dB | 5.61 | -36.09% | -73.98% | 0.85 s | -23.28% | -33.28% | 1.17 dB | 6.82 | -65.28% | -77.93% | 0.46 s
Standard latency | $8\text{\,}\mathrm{s}\text{p}$ | 2 | -19.76% | -27.87% | 1.13 dB | 6.15 | 22.32% | -45.02% | 1.62 s | -26.81% | -35.05% | 1.36 dB | 7.25 | -43.64% | -49.14% | 0.75 s
4 | -19.35% | -27.39% | 1.22 dB | 6.25 | -8.73% | -61.80% | 1.21 s | -25.86% | -34.82% | 1.27 dB | 7.47 | -56.11% | -65.64% | 0.58 s
6 | -18.37% | -26.19% | 1.17 dB | 6.03 | -30.22% | -72.20% | 0.93 s | -24.15% | -33.54% | 1.21 dB | 7.19 | -64.80% | -74.58% | 0.47 s

###  III-B Prediction analysis

The average mean absolute error (MAE) of VMAF and encoding time prediction
models for all resolutions are 4.47 and $0.22\text{\,}\mathrm{s}\text{p}$,
while the average standard deviation are 3.43 and
$0.47\text{\,}\mathrm{s}\text{p}$, respectively, which is acceptable for
online streaming applications. We extract spatiotemporal features (_cf._
Section II-A) at a rate of 352 fps. The overall inference time (including the
feature extraction time, VMAF prediction time, and inference time) for a
$4\text{\,}\mathrm{s}\text{p}$ video segment of 2160p resolution is
$0.37\text{\,}\mathrm{s}\text{p}$. However, since the feature extraction and
encoding are carried out as concurrent processes, the additional latency
introduced by ViSOR is negligible.

###  III-C Rate-distortion analysis

Fig. 3 shows the rate-distortion (RD) curves and the encoding resolution
selected for the target bitrates specified in $\mathcal{B}$ of selected video
sequences (segments) encoded using ViSOR. At lower bitrates, ViSOR reduces the
compression artifacts, especially the blocking artifacts and ringing effects,
improving visual quality. As the compressed video quality improves, the
incremental quality gain from VSR becomes less significant. Moreover, lower
resolutions are chosen when the target maximum latency is stringent since
lower resolutions yield lower encoding latency. Similarly, lower resolutions
are selected for lower target bitrates. Client-side VSR further enables the
selection of lower encoding resolutions, which can be efficiently upscaled to
higher quality without substantially compromising encoding latency or bitrate
constraints.

Table II shows the Bjøntegaard Delta values [31] of ViSOR compared to
_Default_ encoding. It is observed that the BD-rates in terms of PSNR
($BDR_{\text{P}}$) and VMAF ($BDR_{\text{V}}$) increase, while BD-PSNR and BD-
VMAF values decrease as the encoding time limit decreases.

###  III-D Encoding latency and energy consumption analysis

Since the encodings are assumed to be carried out concurrently, the total
encoding time for each segment is determined to be the highest encoding time
yielded among the representations [32]. Table II shows the average encoding
time for each segment ($\overline{\tau_{\text{E}}}$) using the considered
encoding schemes, and relative energy consumption difference compared to
Default encoding ($\Delta E$). The CodeCarbon tool [33] is used to calculate
the encoding energy. Notably, we observe a linear relationship between the
encoding time and the encoding energy of every representation.

We observe that OPTE yields the highest encoding time. The encoding time for a
video segment decreases as $\tau_{\text{L}}$ is lowered. This reduction can be
attributed to the adoption of lower resolutions as $\tau_{\text{L}}$
decreases. Furthermore, encoding time and energy consumption decreases as
$v_{\text{J}}$ increases. This is due to eliminating more representations with
an increase in $v_{\text{J}}$. Notably, in the current setup, energy savings
on the server side are offset by greater losses on the client side (using
VSR). However, it is expected that VSR approaches will evolve to use lower
computational power.

###  III-E Storage consumption analysis

Higher absolute relative storage difference ($\Delta S$) translates to reduced
storage requirements and lower delivery costs, which can be significant for
large-scale streaming platforms. Lower cumulative bitrates place less strain
on the network infrastructure, reducing the risk of network congestion [34].
When the maximum acceptable encoding latency constraint is lowered, high
bitrate representations, which typically require more computational resources
and time to encode, are often sacrificed due to the impracticality of meeting
the time constraint while maintaining high bitrates. Hence, we observe a
decrease in storage consumption when $\tau_{\text{L}}$ decreases (_cf._ Table
II). Representations upscaled using VSR often exhibit improved perceptual
quality, reducing the need for high-resolution encodings at high bitrates.
Consequently, bitstreams with lower bitrates can be selected to represent the
same quality, which reduces the storage footprint. Hence, storage consumption
decreases when using VSR.

##  IV Conclusions

This paper proposed ViSOR, an online energy-efficient VSR-based bitrate ladder
estimation scheme for adaptive streaming applications, where optimized
resolution is predicted for a given target bitrate and maximum acceptable
encoding latency. ViSOR includes an algorithm to determine an optimized
encoding bitrate ladder, where redundant representations are eliminated based
on the JND threshold. ViSOR, on average, yields a bitrate reduction of
$24.65\text{\,}\mathrm{\char 37\relax}\text{p}$ and
$32.70\text{\,}\mathrm{\char 37\relax}\text{p}$ to maintain the same PSNR and
VMAF, respectively, compared to the state-of-the-art HLS bitrate ladder
encoding for an HEVC streaming session with the client using an FSRCNN super-
resolution model, considering a JND of six VMAF points, and a maximum
acceptable encoding latency of $2\text{\,}\mathrm{s}\text{p}$. This is
accompanied by encoding energy and storage consumption savings of
$68.21\text{\,}\mathrm{\char 37\relax}\text{p}$ and
$79.32\text{\,}\mathrm{\char 37\relax}\text{p}$, respectively.

Future iterations of ViSOR are poised to yield even more advantages with
emerging codecs such as _Versatile Video Coding_ (VVC) [35], given that the
selection of lower resolutions implies significantly reduced computational
complexity. In the future, ViSOR can generate customized bitrate ladders
optimized for various client device types (with and without GPU) according to
their ability to perform VSR. This aligns with the principle of common media
client data (CMCD) [36] enriching users’ quality of experience.

## References

  * [1] A. Bentaleb _et al._ , “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP,” in _IEEE Communications Surveys Tutorials_ , vol. 21, no. 1, 2019, pp. 562–585.
  * [2] Apple Inc., “HLS Authoring Specification for Apple Devices.” [Online]. Available: https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices
  * [3] J. De Cock _et al._ , “Complexity-based consistent-quality encoding in the cloud,” in _2016 IEEE International Conference on Image Processing (ICIP)_ , 2016, pp. 1484–1488.
  * [4] A. V. Katsenou, J. Sole, and D. R. Bull, “Content-gnostic Bitrate Ladder Prediction for Adaptive Video Streaming,” in _2019 Picture Coding Symposium (PCS)_ , 2019, pp. 1–5.
  * [5] M. Bhat, J.-M. Thiesse, and P. L. Callet, “Combining Video Quality Metrics To Select Perceptually Accurate Resolution In A Wide Quality Range: A Case Study,” in _2021 IEEE International Conference on Image Processing (ICIP)_ , 2021, pp. 2164–2168.
  * [6] A. Zabrovskiy _et al._ , “FAUST: Fast Per-Scene Encoding Using Entropy-Based Scene Detection and Machine Learning,” in _2021 30th Conference of Open Innovations Association FRUCT_ , 2021, pp. 292–302.
  * [7] V. V. Menon _et al._ , “All-Intra Rate Control Using Low Complexity Video Features for Versatile Video Coding,” in _2023 IEEE International Conference on Image Processing (ICIP)_ , 2023, pp. 2760–2764.
  * [8] V. V. Menon _et al._ , “Content-Adaptive Variable Framerate Encoding Scheme for Green Live Streaming,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2311.08074
  * [9] F. Nasiri _et al._ , “Multi-Preset Video Encoder Bitrate Ladder Prediction,” in _Proceedings of the 2nd International Workshop on Design, Deployment, and Evaluation of Network-Assisted Video Streaming_ , 2022, p. 8–13.
  * [10] V. V. Menon _et al._ , “Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding,” in _Proceedings of the 3rd Mile-High Video Conference_ , 2024.
  * [11] P. Ramachandran _et al._ , “Content Adaptive Live Encoding with Open Source Codecs,” in _Proceedings of the 11th ACM Multimedia Systems Conference_ , 2020, p. 345–348.
  * [12] H. Liu _et al._ , “Video Super-Resolution Based on Deep Learning: A Comprehensive Survey,” in _Artif. Intell. Rev._ , vol. 55, no. 8, Dec. 2022, p. 5981–6035.
  * [13] C. Dong, C. C. Loy, and X. Tang, “Accelerating the Super-Resolution Convolutional Neural Network,” in _Computer Vision – ECCV 2016_. Cham: Springer International Publishing, 2016, pp. 391–407.
  * [14] W. Shi _et al._ , “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2016, pp. 1874–1883.
  * [15] B. Lim _et al._ , “Enhanced deep residual networks for single image super-resolution,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , Jul. 2017.
  * [16] S. Liu _et al._ , “EVSRNet: Efficient Video Super-Resolution with Neural Architecture Search,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2021, pp. 2480–2485.
  * [17] C. Ledig _et al._ , “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 105–114.
  * [18] N. Ahn, B. Kang, and K.-A. Sohn, “Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network,” in _Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X_ , 2018, p. 256–272.
  * [19] M. Haris, G. Shakhnarovich, and N. Ukita, “Recurrent Back-Projection Network for Video Super-Resolution,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2019, pp. 3892–3901.
  * [20] Z. Li _et al._ , “VMAF: The journey continues,” in _Netflix Technology Blog_ , vol. 25, 2018.
  * [21] H. Amirpour _et al._ , “VCD: Video Complexity Dataset,” in _Proceedings of the 13th ACM Multimedia Systems Conference_ , 2022, p. 234–239.
  * [22] V. V. Menon _et al._ , “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023, pp. 1–1.
  * [23] V. V. Menon _et al._ , “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming,” in _First International ACM Green Multimedia Systems Workshop (GMSys ’23)_ , 2023.
  * [24] Q. Cai _et al._ , “Real-Time Constant Objective Quality Video Coding Strategy in High Efficiency Video Coding,” in _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 7, 2020, pp. 2215–2228.
  * [25] J. Y. Lin _et al._ , “Experimental design and analysis of JND test on coded image/video,” in _Applications of Digital Image Processing XXXVIII_ , vol. 9599, 2015, pp. 324–334.
  * [26] H. Wang _et al._ , “VideoSet: A large-scale compressed video quality dataset based on JND measurement,” in _Journal of Visual Communication and Image Representation_ , vol. 46, 2017, pp. 292–302.
  * [27] J. Zhu _et al._ , “A Framework to Map VMAF with the Probability of Just Noticeable Difference between Video Encoding Recipes,” in _2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)_ , 2022, pp. 1–5.
  * [28] V. V. Menon _et al._ , “Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming,” in _2023 IEEE International Conference on Visual Communications and Image Processing (VCIP)_ , 2023.
  * [29] VideoLAN, “x265.” [Online]. Available: https://www.videolan.org/developers/x265.html
  * [30] P. K. Tiwari _et al._ , “Accelerating x265 with Intel® Advanced Vector Extensions 512,” in _White Paper on the Intel Developers Page_ , 2018.
  * [31] HSTP-VID-WPOM, “Working practices using objective metrics for evaluation of video coding efficiency experiments,” in _International Telecommunication Union_ , 2020.
  * [32] V. V. Menon _et al._ , “EMES: Efficient Multi-Encoding Schemes for HEVC-Based Adaptive Bitrate Streaming,” in _ACM Trans. Multimedia Comput. Commun. Appl._ , vol. 19, no. 3s, Mar. 2023.
  * [33] BCG-GAMMA and MILA, “CodeCarbon.” [Online]. Available: https://codecarbon.io/
  * [34] R. Farahani _et al._ , “Hybrid P2P-CDN architecture for live video streaming: An online learning approach,” in _GLOBECOM 2022-2022 IEEE Global Communications Conference_ , 2022, pp. 1911–1917.
  * [35] B. Bross _et al._ , “Overview of the Versatile Video Coding (VVC) Standard and its Applications,” vol. 31, no. 10, 2021, pp. 3736–3764.
  * [36] A. Bentaleb _et al._ , “Common Media Client Data (CMCD): Initial Findings,” in _Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video_ , Jul. 2021, p. 25–33.


#  Spontaneous interstitial (anti)merons in D2d symmetric Mn-Pt(Pd)-Sn-In
system

Bimalesh Giri1 Dola Chakrabartty1 S. S. P. Parkin2 and Ajaya K. Nayak1,∗

###### Abstract

Interstitial topological objects, such as skyrmions, within a natural 1-D
helix are predicted to be free from ambiguous ’skyrmion Hall effect’. The
helical ambience precipitate an additional potential that counteract the
Magnus force arising from the gyrotropic motion of skyrmion. Here, we present
the observation of $\pm$ $\frac{1}{2}$ topological charge objects (anti)merons
within the 1-D helical stripes in D2d symmetric Mn1.4Pt0.9Pd0.1Sn1-xInx
system. With the help of Lorentz transmission electron microscopy study we
demonstrate that the pair-wise meron and antimeron chains can be spontaneously
stabilized for a critical In concentration in the system. The exchange
frustration induced proportionate fragmentation of the magnetic moment in the
in-plane and easy-axis directions acts as a basic ingredient for the formation
of (anti)merons within the helical stripe. A constrained drift motion of
(anti)merons along the stripe makes this system an ideal platform for the
realization of skyrmion Hall free track motion. Moreover, the observation of
(anti)merons in addition to the skyrmion and antiskyrmion in D2d materials
makes them a suitable horizon for zoo of topology.

School of Physical Sciences, National Institute of Science Education and
Research, An OCC of Homi Bhabha National Institute, Jatni-752050, India

Max Planck Institute of Microstructure Physics, Weinberg 2, 06120 Halle
(Saale), Germany
$\ast$<EMAIL_ADDRESS>
Topologically protected magnetic whirls, such as (anti)skyrmions,
(anti)merons, etc., have attracted significant research interest owing to
their distinctive spin textures that possess enormous possibilities in future
spintronic devices. The stabilization mechanism of these complex magnetic
objects relies on the underlying magnetic interactions complying with the
inherent crystal symmetry of the system. For instance, antisymmetric
Dzyaloshinskii– Moriya interaction (DMI) originating from the relativistic
spin-orbit coupling is an essential ingredient for the stabilization of
skyrmion (skx)/antiskyrmion (askx) in bulk crystals [1, 2, 3, 4] and layered
thin films [5, 6, 7, 8] system with broken inversion symmetry. On the other
hand magnetic frustration[9, 10], higher order exchange interaction[14],
dipolar interaction[12, 13], can stabilize skyrmions in certain
centrosymmetric materials. Although independent microscopic mechanisms are
responsible for the stabilization of different topological spin textures, it
is interesting to study how these objects transform between two different
topological states. In this direction, it has been demonstrated that
application of in-plane magnetic field[16, 17] and thermal energy modification
of magnetic anisotropy[18] can enable overcoming the energy barrier imposed
between two topological phases. Moreover, it is also proposed that the co-
existence of two distinct topological objects can have significant future
prospectives [36]. Hence, material engineering is an important pathway to
achieve a delicate balance between the different energy landscapes for the
realization of emergent topological magnetic phases [21, 16].

The topological character of skyrmions attracts a significant research
interest, in particular, to study their stabilization mechanism and electrical
manipulation for the application in spintronic devices [5, 22]. However, all
the topological magnetic objects with finite topological charge (nsk) showcase
an unwanted ’skyrmion Hall effect’, irrespective of nature of the driving
field. Nonetheless, several ideas have been proposed to mitigate the
intriguing perplexity for the finite nsk objects[28, 29, 30, 31]. Moreover,
topological objects consisting of opposite nsk can intrinsically compensate
the reverse gyro-tropic response, e.g. antiferromagnetic skx [23, 24, 25] and
skyrmionium [27, 26] are believed to be exhibit zero ’skyrmion Hall effect’.
Interestingly, it has been demonstrated that the interstitial skx in a natural
1-D spin helical stripe is free from such ambiguity [32, 33, 34]. The
confinement within the helical background accomplishes an additional potential
barrier that balance the Magnus force, thereby imposing the motion along the
stripe. The distinct characteristics of skx motion within helical stripes and
ferromagnetic (FM) environment shift the paradigm for searching topological
objects within the helical background. It has been also proposed that the
meron, skyrmion, and bimerons can stabilize along with the helical stripes as
natural intermediate states at the phase transition from skyrmion to helical
state [35]. Later, Müller et al., have demonstrated the coexistence of skx
with the broken helical stripe experimentally[32]. However, a real
interstitial topological object in true helical ordered state is never
observed.

The present manuscript demonstrates the first observation of natural one-
dimensional helical stripes that spontaneously host a chain of merons and
antimerons. In this regard, we present an extensive study on the phase
stability of different topological objects in the $D_{2d}$ symmetric system.
Especially, we address the effect of exchange frustration on the ground state
spin modification. To realize our goal, we consider doping of In atom in place
of Sn in Mn1.4Pt0.9Pd0.1Sn1-xInx system. It is to be noted here that the first
observation of antiskyrmions has been demonstrated in Mn1.4Pt0.9Pd0.1Sn, an
inverse tetragonal Heusler system that belongs to D2d crystal class [4]. The
underlying $D_{2d}$ crystal symmetry restricts the Dzyaloshinskii– Moriya (DM)
vector within the tetragonal basal plane with Dx= - Dy, where Dx, Dy are DM
vectors along x and y directions, respectively. Hence, the helical modulation
are found only along [100] and [010] directions. Consequently, the askx
lattice gets stabilized in the $ab$ plane above the spin re-orientation
temperature ($T_{SR}$ $\sim$130 K) [4, 17, 36]. Powder neutron diffraction
(ND) study shows the presence of a collinear and a non-collinear spin ordering
above and below the $T_{SR}$ [37]. Furthermore, the dipolar energy associated
with the large magnetic moment of about $\sim$ 5 $\mu$ B/f.u. in this system
also plays a vital role in controlling the shape and size of the askx in this
system [38]. In our previous study it has been shown that replacing Sn by In
results into a non-collinear magnetic state in Mn1.5PtIn with a small magnetic
moment of $\sim$1.6 $\mu$ B/f.u. [39]. Moreover, a comparison between the low
temperature ND data of Mn1.4PtSn and Mn1.5PtIn sample indicates that a large
magnetization can be induced in the basal plane with In doping in place of Sn.
Previous theoretical study proposes that the formation of (anti)meron lattice
can be energetically favorable with in-plane anisotropy [40]. Therefore, an
effective change of the magnetization from out of plane (OP) to in-plane (IP)
direction or compatible fractionation of the same in presence of the
anisotropic DMI may showcase exotic topological phases like (anti)meron in the
present system.

Figure 1:  Evolution of magnetic ground states and related spin textures with
different In doping in Mn1.4PtSn1-xInx. a A collinear spin ordering with only
out of plane magnetization component $M_{z}$. b A noncollinear magnetic
ordering with dominant $M_{z}$ in comparison to the in plane magnetization
$M_{x}$. The magnetic states shown in a and b correspond to the spin alignment
above and below the spin re-orientation transition $T_{SR}$ for $x$=0.2,
respectively. c A noncollinear spin structure for $x$=0.4 with comparable
$M_{x}$ and $M_{z}$. d Noncollinear ordering with dominating in-plnae magnetic
component i.e., $M_{z}$ $<$ $M_{x}$. The blue and red balls corresponds to the
Mn atoms at 4a and 8d Wyckoff positions, respectively. Evolution of the
canting angle of the Mn sublattice at 8d position (red balls) is shown in each
figures. All the depicted magnetic ordering are obtained from the powder
neutron diffraction experiment. Schematic spin textures of e an antiskyrmion,
f a nontopological bubble (NT), and g a meron, generally found for the ground
state magnetic ordering shown in figures a, b and c, respectively. The
topological charge (nsk) for each spin texture is mentioned in the inset of
the corresponding figure. h Variation of effective anisotropy (Keff) with In
concentration ($x$) and saturation magnetization $M_{S}$ . Figure 2:  Real
space mapping of different spin textures using over focused (OF) Lorentz
transmission electron microscopy (LTEM) study of Mn1.4Pt0.9Pd0.1Sn1-xInx
samples for $x$= 0.2 to 0.5. a Evidence of askx lattice in $x$=0.2 sample at
T$>$ TSR. b The coexistence of the askx and skx in $x$=0.3 sample. The
elliptical Bloch skx and askx is marked with the elliptical yellow and blue
circles, respectively. c Formation of nontopological (NT) bubble phase in
$x=$0.4. d The coexistence of NT bubbles and elliptical Bloch skx in $x$=0.4
for at T$<$TSR. e No LTEM contrast is evident for $x=$0.5 sample indicating
absence of any spin texture. f Appearance of the few (anti)merons as indicated
with the yellow arrows for $x$=0.4. g The Formation of (anti)meron chain
within the helical ambiance. h and i The real space spin texture of the black
and white dots contained within the blue and violet dashed rectangle in g,
respectively. The inset color wheel corresponds to the in-plane magnetic field
directions. Fig. h and i deduced by performing transport of intensity equation
(TIE) from the selected region of interest.

With the above discussed ideas, we present a scheme for the stability of
different topological phases in the $D_{2d}$ symmetric Mn1.4Pt0.9Pd0.1Sn1-xInx
system as shown in Fig. 1. The ground state magnetic ordering obtained from
the neutron diffraction (ND) study for samples with different In concentration
is presented in Fig. 1a-d. The magnetic moments of Mn atoms sitting in two
different sublattices are shown by blue and red arrows. For $x=$ 0.2, Mn
moments at both the magnetic sublattices align along the c-axis, resulting in
only out of plane component moment Mz above the spin-reorientation transition
TSR [Fig. 1a]. In general, this type of spin arrangement favors the
antiskyrmion phase as shown in Fig. 1e. Below the TSR, both the magnetic
sublattices exhibit a small in-plane moment Mx, however, the major component
of the moment is along the c-axis, i.e., Mz $>$Mx [Fig. 1b]. Formation of
nontopological bubbles (NT) [Fig. 1f] is favorable in this region. For $x\geq$
0.4, a dominant non-collinear magnetic ordering prevails throughout the
magnetic ordered phase [see Fig. 1c$\&$d]. The Mn moments at 8d position for
$x=$ 0.4 is tilted more towards the in-plane direction compared to the $x=$
0.2 sample [see Fig. 1b & c]. Interestingly, for $x=$ 0.4 sample both the
magnetization components are comparable, i.e Mz $\sim$Mx. The formation of
(anti)meron [Fig. 1g] is expected in this sample. It is worth mentioning that
for all the cases, i.e. $x=$ 0.2-0.4, the skyrmion phase can be stabilized
based on the lamellae thickness and experimental protocols. For $x=$ 0.6, the
Mn moments at the 8d position align completely in the in-plane direction. In
this case, Mz $<$Mx and no signature of any topological phase is noticed. The
variation of effective magnetic anisotropy for different $x$ values is shown
in Fig. 1h. A local minima can be visible around $x=$ 0.4. A detailed
structural and magnetic characterization along with the ND results for the
present samples is presented in the Supplementary Information [41].

##  1 LTEM results

For the real space visualization of the above-mentioned topological spin
textures, we have carried out a detailed Lorentz transmission electron
microscopy (LTEM) imaging using thin lamellae of Mn1.4Pt0.9Pd0.1Sn1-xInx
samples. Figure 2a-g depicts the over focused (OF) LTEM images recorded on the
(001) oriented thin plates for $x=$ 0.2 to 0.5. The formation of triangular
askx lattice for $x=$ 0.2 at $T=$ 250 K ($>$TSR) can be seen from Fig. 2a. We
also find the formation of hexagonal lattice of elliptical Bloch skx in this
sample at TSR $<$ T $<$ 250 K [41]. A detailed temperature and field evolution
of the skx and askx lattices is provided in the supplementary information
[41]. Below $T_{SR}$, we notice that the formation of nontopological bubbles
(NT) is more favorable. In fact, mixed states of skx and NT are also found in
the case of thicker lamellae ($\sim$ 200 nm )[see supplementary][41]. However,
it is not possible to stabilize askx phase below $T_{SR}$. This can be
understood from the fact that the dominating dipole interaction causes near
degenerate energy associated with the skx, askx, and NT bubble phase in D2d
symmetric materials. As it can be seen from Fig. 2b, the skx and askx phases
coexists in case of $x=$0.3. The weak energy barrier among them can be
overcome and envisaged through the application of an in-plane magnetic field,
hence, a reversible transformation between skx, NT, and askx can be
realized[17]. In case of $x=$0.4, the appearance of triangular lattice of the
NT bubbles at $T>$$T_{SR}$ and the conjunction of NT bubbles and skx at
$T<$$T_{SR}$ is shown in Fig. 2c $\&$ d, respectively. In the present system,
topologically trivial and non-trivial magnetic textures are present for In
concentration up to $x=$0.4. For $x=$0.5, we do not find any magnetic contrast
( Fig. 2e).

###  1.1 Spontaneous (anti)merons:

Now, we concentrate on the $x=$0.4 sample that exhibits a non-collinear
magnetic ordering throughout the ordered phase and satisfies the criteria of
$M_{z}$ $\sim$ $M_{x}$. At zero magnetic field this sample shows the usual
helical phase, consistent with the D2d symmetric system. Interestingly, at
temperature $T=$112 K, nanometric size local dark and bright dots start
appearing in the interstitial position of the helical stripes [Fig. 2f]. With
further lowering the temperature to $T=$100 K (the lowest possible temperature
achievable in our system), the magnetic contrast of these dots become very
prominent as can be seen in Fig. 2g. In addition, these interstitial dots
appear with alternate black and white chain without any periodicity, as the
distance between two pair of chains is not equal (Fig. 2g). These dark and
bright dots may refer to the formation of nanometric size swirling spin
textures. To confirm the nature of these interstitial spin textures, we have
performed transport of intensity equation (TIE) mapping of the LTEM images
taken in in-focus, under-focus and over-focus conditions. Figure 2h$\&$i show
the in-plane magnetization mapping and spin curling of the selected dots
marked in the blue and violet dashed rectangles in Fig. 2g. The nature of the
local magnetic field variation clearly establish that these interstitial
objects are meron and antimerons. The (anti)merons are vortex-like topological
objects with a core up/down and in-plane peripheral spin distribution that
carry a topological charge nSk= $\pm$ $\frac{1}{2}$. They are different from
half (anti)skyrmions which also hold nSk=$\pm$ $\frac{1}{2}$ and are found at
the edge of the thin lamellae of Mn1.4Pt0.9Pd0.1Sn [42]. Although it is
difficult to determine the exact core spin direction of the(anti)merons from
the LTEM contrast, it can be safely said that if the meron is with upward
core, the antimeron must be a downward core spin as the z-component of the
moment is maximum/minimum (upward/downward) at the middle of the dark and
bright helical stripe. Therefore, the bright spots are meron with downward
core spins, and the black spots are antimerons with upward core spins. We
reach this conjecture by analogy with the LTEM contrast of the simulated spin
texture, which will be discuss later.

Figure 3:  Field evolution of the (anti)meron chains. a-e Over-focused images
taken at different magnetic fields at $T=$ 100 K for $x=$0.4 sample. The width
of each figure is 2 $\mu$m. f Average distance between the meron-antimeron
chain with respect to the applied magnetic field. The line is guide to the
eye. The average distance is taken by measuring the separation at three
different positions: bottom, middle and top region.

###  1.2 Field evolution of (anti)merons and their motion

In the present system the (anti)merons appear spontaneously by cooling the
sample from the paramagnetic state. Therefore, it is important to understand
their stability under the external magnetic field. Figure 3a-e depicts the
field evolution of the pair of meron and antimeron chains. The interstitial
(anti)merons are stable up to a field of 0.07 T [see Fig. 3a-d]. For higher
fields (anti)merons disappear, resulting in only the helical stripe phase [see
Fig. 3e]. We observe that the (anti)meron chains are not reversible under
field reversal. Interestingly, the (anti)merons chains start drifting with the
application of external magnetic field. It can be seen that the (anti)meron
chain with bright contrast remains almost fixed in its position, whereas a
gradual shift of the meron chain with dark spots is visible. Figure 3f
describes the field-dependent separation length of the pair of (anti)merons
chains. Importantly, the drift is constrained along the stripe only. All the
images were taken close to the pole positions so that the effect of the in-
plane field is negligible in their drift motion. This study clearly
demonstrates the motion of (anti)merons along the stripes.

##  2 Micromagnetic Simulation

All our experimental results show that the present Mn1.4Pt0.9Pd0.1Sn1-xInx
samples with $x=$ 0 to 0.4, can host different topological magnetic states
depending on the temperature and magnetic field application. The stability of
askx in D2d symmetric materials is well described by the Hamiltonian written
as-

$\displaystyle H={}$
$\displaystyle-J\sum_{i}m_{i}\>.\>(m_{i+\hat{a}}+m_{i+\hat{b}}+m_{i+\hat{c}})$
(1) $\displaystyle+$ $\displaystyle D\sum_{i}(m_{i}\times
m_{i+\hat{a}}\>.\>\hat{a}+m_{i}\times m_{i+\hat{b}}\>.\>\hat{b})$
$\displaystyle+$ $\displaystyle K\sum_{i}(m_{c,i})^{2}-H\sum_{i}m_{c,i}\>,$

where the first and second terms are the conventional isotropic exchange and
Dzyaloshinskii-Moriya interaction; the third and fourth terms are the uniaxial
anisotropy and Zeeman energy, respectively. $\hat{a}$ , $\hat{b}$, $\hat{c}$
are the unit vectors in the Cartesian co-ordinate system. It is well
understood that the competition between anisotropic DMI and isotropic exchange
interaction governs the propagation of helical state along [100] and [010]
directions. The application of the magnetic field along the anisotropic
magnetic axis converts the helical state into askx lattices. The large
magnetization in these materials causes dominant dipolar interaction depending
on the lamellae thickness and temperature[38]. Moreover, the dipolar energy
can overtake the anisotropic DMI and compete with the anisotropy to stabilize
elliptical skx at lower temperature region [17, 36]. The elongation direction
of these elliptical objects remains to constrain along [100] and [010] only.
Theoretically, the skyrmion lattice can be stabilized in an easy-plane
magnetic system under an external field and can be transformed into the
(anti)meron lattice state by increasing the easy-plane anisotropy [40]. Most
of the system show the (anti)merons as a lattice [16], inside a domain
wall[15, 18], and as isolated one in an easy plane magnetic system [45, 44].

In the present In doped samples, the magnetic moment partially aligns in the
plane direction with a large magnetization component in the basal plane [see
Fig. 1b-d]. To understand the origin of the (anti)merons formation in the
present system, we have carried out a detailed micromagnetic simulations by
implementing easy-plane anisotropy. We set up $H=0$ to echo the experimental
observation. Therefore, the effective Hamiltonian contains ferromagnetic
exchange, anisotropic DMI, and easy-plane anisotropy. We take random
magnetization as an initial state and relax the system to stabilize the
(anti)meron pairs formed only after cooling the system to a low temperature
from a paramagnetic state [see Fig. 2f $\&$ g]. Figure 4a-c shows the
magnetization configuration for different values of in-plane anisotropy
constant (Ku). For low values of Ku, some paired and isolated anitskyrmions
are formed [see Fig. 4a]. Further increasing the in-plane anisotropy in the
presence of DMI mainly results in a non-collinear in-plane magnetic state. For
Ku = 2.0 $\times$10 5 J/m3, local meron-antimeron chains are stabilized within
the non-collinear spin configurations [Fig. 4b]. We find the formation of both
core up and down merons and antimerons as marked in red and blue color circles
in Fig. 4b. By further increasing the Ku, mostly collinear in-plane magnetic
state gets stabilized with some isolated chains and single (anti)meron [Fig.
4c]. The simulated results in Fig. 4b closely corroborate our experimental
finding of spontaneous (anti)merons in the system. For a better correspondence
of the experimental results with the simulated one, the LTEM contrast of the
magnetization configuration in Fig. 4b is shown in Fig. 4d. The LTEM contrast
is sensitive only to the twisting of the spins, e.g. the domain wall,
vortices, etc. The bright spots appearing within the fainted bright stripes in
Fig. 4d corresponds to the core down Bloch-type meron [shown within blue
circle in Fig. 4e]. Similarly, core up merons appear as dark spots within the
fainted dark stripes [shown in red circle of Fig. 4e]. These simulated spin
textures fully corroborate our experimental results. To justify the particle-
like nature of these local objects, we calculate the topological charge
density (TCD) [53], as shown in Fig. 4f. The TCD of these localized objects
follows the definition given as nsk= $\frac{pw}{2}$, where, $p=$ polarization
of the core spin (+1 for up, -1 for down) and $w$ is the winding number
($w=$+1 for vortex,$w=$ -1 for antivortex) [43]. Therefore, a meron-antimeron
pair with opposite core spin polarity can lead to a total topological charge
of +1 or - 1.

Figure 4:  Micromagnetic simulations of (anti)merons. a-c Magnetization
configuration for different values of in-plane anisotropy constant (Ku)
obtained from Object Oriented MicroMagnetic Framework (OOMMF) simulation. The
color bar represent the mz component. d The LTEM contrast of the magnetic
configuration in b. e An enlarge view of the chain of meron/antimeron pair
enclosed in a green rectangle in b. Bloch type merons with core up and down
encircled in red and blue color, respectively. f Topological charge density
(TCD) of b. TCD of merons with core up and down is highlighted in f. The
average magnitude of nsk= 0.435 for each of the (anti)meron in f.

##  3 Discussions and Conclusion

In the earlier studies, Gao et al. have shown that a delicate fractionation of
the magnetization results in the spontaneous formation of domain wall
(anti)merons in the ferromagnetic in Fe5-xGeTe2 [15]. Similarly, in the
amorphous easy-plane ferrimagnetic Gd-Fe-Co system, the periodic appearance of
Bloch lines within the Bloch domain wall is responsible for the formation of
meron-antimeron chains [18]. In case of $\beta$-Mn type Co8Zn9Mn3, the origin
of square lattice (anti)merons is assigned to the overlapping between two
orthogonal helices propagating in the plane perpendicular to the magnetic
field [16]. Nonetheless, several theoretical studies have indicated the
variation in magnetic anisotropy can lead to the emergence of a topological
(anti)merons [40, 46].

In the present case the doping of In in place of Sn serves two purposes.
First, since the magnetic Mn concentration does not change, the effect of
magnetic disorder in the system can be avoided. Secondly, the addition or
reduction in the valence electron count maximally impacts the magnetic
interactions, which is linked to the higher power of moments, such as
Ruderman-Kittel-Kasuya-Yosida (RKKY), biquadratic ($\propto$
$(S_{i}.S_{j})^{2}$)[47], and topological-chiral interaction ($\propto$
$[S_{i}.(S_{j}\times S_{k})]^{2}$)[48]. In a metallic system, the effective
exchange strength may not be limited to the nearest neighbor site only. The
itinerant electron can give rise to an oscillatory RKKY-type interaction which
is extended beyond the nearest neighbor. For instance, a spin-canted state
arises in Mn2Y(=Rh, Pt, Ir)Z due to the exchange frustration mediated through
RKKY interaction [49]. Swekis et al. have shown that the origin of the spin
re-orientation transition in Mn1.5PtSn can be associated with the exchange
frustration in the system [50]. Hence, the presence of spin-canted state in
the present system can be connected to the exchange frustrations arising due
to addition of extra electron to the system. As a result, the net canted
magnetic moment in the OP/IP direction systematically decreases/increases with
In concentration. Therefore, in the present system a gradual shift in the
major magnetic component is attributed to the re-alignment of the magnetic
anisotropy. The proportionate IP and OP moment in presence of anisotropic DMI
in the D2d symmetric system can result in interstitial (anti)meron chains.
This is also supported by our micromagnetic simulations.

In summary, we present a comprehensive study on the finding of different
topological magnetic phases in the $D_{2d}$ symmetric tetragonal Heusler
system Mn1.4Pt0.9Pd0.1Sn1-xInx. The stabilization of these topological objects
are well connected with the systematic change in the magnetic structure with
different degrees of non-collinearity arising due to exchange frustration in
the system. The antiskyrmion phase stabilizes only when the underlying spins
are collinear in nature, whereas the non-topological bubbles are more
favorable in the noncollinear background spin. At a critical noncollinear
magnetic state where $M_{z}$ $\sim$ $M_{x}$, we find the simultaneous presence
of helical stripes and (anti)meron chains. Most importantly, the drift motion
of the (anti)meron chains along the helical stripe under an external magnetic
field mimics the 1-D-like deflection-free movement. Therefore, our proposition
of interstitial (anti)merons with $\pm$ $\frac{1}{2}$ topological charge can
be a potential alternative for the future development of ’race track memory’.

## Methods

Experimental methods Polycrystalline samples of Mn1.4Pt0.9Pd0.1Sn1-xInx were
prepared by arc melting the stoichiometric amount of high-purity constituent
elements. The samples were kept at 750∘C for seven days after being sealed in
high vacuum quartz tubes. The structural phase of the samples was confirmed by
room temperature powder X-ray diffraction (XRD) using Cu-Kα radiation
(Rigaku). The composition and homogeneity of the samples were checked in field
emission scanning electron microscopy (FESEM) and energy dispersive X-ray
spectroscopy (EDS). Magnetic measurements were carried out using
superconducting quantum interference device vibrating sample magnometer (SQUID
VSM) (MPMS-3). Electron back scattered diffraction (EBSD) was used to find the
suitable grain for lamellae preparation from the polycrystalline samples. The
transmission electron microscopy (TEM) lamellae were prepared using a 30 kV Ga
ion-based Focused Ion Beam (FIB). A very low dose factor was used to thin down
the lamellae to desired thickness. The (001) orientation of the lamella was
confirmed by performing the selected area electron diffraction (SAED) in 200
kV JEOL TEM (JEM-F200). The magnetic contrast was observed using the Lorentz
TEM (LTEM) mode. A double-tilting liquid nitrogen-based sample holder
(GATAN-636) was used for temperature dependent LTEM imaging.

Transport of intensity equation (TIE) LTEM imaging is a versatile technique to
visualize various nano- to micro-scale magnetic textures in thin lamella
samples. Interaction of the electron with the in-plane magnetic field of of
the sample results in the convergence(divergence) of the electron beam that
produces bright(dark) contrast under different defocus conditions. If a
magnetic image appears as bright in over focus (OF) condition, then it should
be dark in the under focus (UF) and vice versa. Extraction of the actual spin
texture is possible by solving the transport of intensity equation (TIE) from
the intensity map of the series of digital images. The TIE solves the
differential equation between intensity and electrons phase have given as-

$\frac{2\pi}{\lambda}\frac{\partial^{2}I(x,y,z)}{\partial^{2}z}=-\nabla_{xy}\textbf{.}[I(x,y,z)\nabla_{xy}\phi(x,y,z)],$
(2)

where, $\lambda$ is electron wavelength, I is intensity of the electrons beam,
and $\phi$ is the phase of electrons. A series of systematic images can be
acquired to map the in-plane magnetization distribution of the thin lamella.
For example, an equal defocus length of UF and OF, along with an in-focus (IF)
image, can be used to map out the magnetization distribution. First, we
extracted the electron-phase images and then mapped out the magnetization
textures using the software QPt [54].

Micromagnetic simulation Micromagnetic simulations are carried out using
Object Oriented Micro Magnetic Framework OOMMF[51] to understand the possible
mechanism of (anti)merons formation in D2d symmetric materials. We have taken
a slab dimension of 250$\times$250$\times$10 nm3 and the unit cell dimension
of 1$\times$1$\times$1 nm3 . The Gilbert damping parameter ($\alpha$) was set
at 0.5. We have fixed the exchange strength (J) = 1.2 pJ/m, and DMI strength
(D)= 0.5 mJ/m2, saturation magnetisation (Ms)= 7$\times$105 A/m, and applied
magnetic field (H)= 0. The in-plane anisotropy ($K_{u}$) was varied in the
range of 1.0-3.0$\times$ 105 J/m3 to stabilize a meron-antimeron chain within
the helical-like stripes.

## Data availability

All data used to obtain the conclusions in this paper are presented in the
paper and/or the Supplementary Materials. Other data may be requested from the
authors. Please direct all inquiries to A.K.N. (ajaya@niser.ac.in).

## References

## References

  * [1] Mühlbauer, S. et al. Skyrmion lattice in a chiral magnet. Science  323, 915 (2009).
  * [2] Yu, X. Z. et al. Real-space observation of a two-dimensional skyrmion crystal. Nature (London) 465, 901 (2010).
  * [3] Adams, T. et al. Long-Wavelength helimagnetic order and skyrmion lattice phase in Cu2OSeO3. Phys. Rev. Lett. 108, 237204, 2012.
  * [4] Nayak, A. K. et al. Magnetic antiskyrmions above room temperature in tetragonal Heusler materials. Nature (London) 548, 561 (2017).
  * [5] Sampaio, J. et al., Nucleation, stability and current-induced motion of isolated magnetic skyrmions in nanostructures. Nat. Nanotechnol. 8, 839–844 (2013).
  * [6] Jiang, W. et al. Blowing magnetic skyrmion bubbles. Science 349, 283-286, (2015).
  * [7] Moreau-Luchaire, C. et al. Additive interfacial chiral interaction in multilayers for stabilization of small individual skyrmions at room temperature Nat. Nanotechnol.  11, 444–448 (2018).
  * [8] Soumyanarayanan, A. et al. Tunable room-temperature magnetic skyrmions in Ir/Fe/Co/Pt multilayers. Nat. Mater 16, 898–904 (2017).
  * [9] Okubo, T. Chung, S. $\&$ Kawamura, H. Multiple-$q$ States and the Skyrmion Lattice of the Triangular-Lattice Heisenberg Antiferromagnet under Magnetic Fields. Phys. Rev. Lett. 108, 017206 (2012).
  * [10] Leonov A. O. $\&$ Mostovoy, M. Multiply periodic states and isolated skyrmions in an anisotropic frustrated magnet. Nat. Commun. 6, 8275 (2015).
  * [11] Zhang, X. et al. Skyrmion dynamics in a frustrated ferromagnetic film and current-induced helicity locking-unlocking transition. Nat. Commun. 8, 1717 (2017).
  * [12] Yu, X. et al. Magnetic stripes and skyrmions with helicity reversals. Proc. Natl Acad. Sci. 109, 8856–8860 (2012).
  * [13] Chakrabartty, D. et al., Tunable room temperature magnetic skyrmions in centrosymmetric kagome magnet Mn4Ga2Sn. Communications Physics 5, 189 (2022).
  * [14] Heinze, S. et al. Spontaneous atomic-scale magnetic skyrmion lattice in two dimensions. Nat. Phys. 7, 713–718 (2011).
  * [15] Gao, Y. et al. Spontaneous (Anti)meron Chains in the Domain Walls of van der Waals Ferromagnetic Fe5-xGeTe2. Adv. Mater. 32, 2005228, 2020.
  * [16] Yu, X. Z. et al. Transformation between meron and skyrmion topological spin textures in a chiral magnet. Nature 564, 95-98, 2018.
  * [17] Peng, L. Controlled transformation of skyrmions and antiskyrmions in a non-centrosymmetric magnet. Nat. Nanotechnol. 15, 181–186 (2020).
  * [18] Li, Z. et al., Field-free topological behavior in the magnetic domain wall of ferrimagnetic GdFeCo. Nat. Commun. 12, 5604 (2021).
  * [19] Vir, P. et al. Anisotropic topological Hall effect with real and momentum space Berry curvature in the antiskrymion-hosting Heusler compound Mn1.4PtSn. Phys. Rev. B 99, 140406(R) (2019).
  * [20] Jena, J. Evolution and competition between chiral spin textures in nanostripes with D2d symmetry. Sci. Adv. 6 (49), eabc0723 (2020).
  * [21] Karube, K. et al. Disordered skyrmion phase stabilized by magnetic frustration in a chiral magnet. Sci. Adv., 4 (9), eaar7043 (2018).
  * [22] Huang, S. et al. Stabilization and current-induced motion of antiskyrmion in the presence of anisotropic Dzyaloshinskii-Moriya interaction. Phys. Rev. B 96, 144412 (2017).
  * [23] Barker, J. $\&$ Tretiakov, O. A. Static and Dynamical Properties of Antiferromagnetic Skyrmions in the Presence of Applied Current and Temperature. Phys. Rev. Lett. 116, 147203 (2016).
  * [24] Zhang, X. Zhou, Y. $\&$ Ezawa, M. Antiferromagnetic Skyrmion: Stability, Creation and Manipulation. Sci. Rep. 6, 24795 (2016).
  * [25] Zhang, X. Zhou, Y.$\&$ Ezawa, M. Magnetic bilayer-skyrmions without skyrmion Hall effect. Nat. Commun. 7, 10293 (2016).
  * [26] Zhang, X. et al. Control and manipulation of a magnetic skyrmionium in nanostructures. Phys. Rev. B 94, 094420 (2016).
  * [27] Kolesnikov, A. G. et al. Skyrmionium – high velocity without the skyrmion Hall effect. Sci. Rep. 8, 16966 (2018).
  * [28] Upadhyaya, P. Yu, G. Amiri, P. K. $\&$ Wang, K. L. Electric-field guiding of magnetic skyrmions.  Phys. Rev. B 92, 134411, (2015).
  * [29] Hirata, Y. et al. Vanishing skyrmion Hall effect at the angular momentum compensation temperature of a ferrimagnet. Nat. Nanotechnol., 14, 232–236, (2019).
  * [30] Plettenberg, J. Stier, M. $\&$ Thorwart, V. Steering of the Skyrmion Hall Angle by Gate Voltages. Phys. Rev. Lett. 124, 207202 (2020)
  * [31] Guo, J.H. Hou, Y. Zhang, X. Pong, Philip W.T. $\&$ Zhou, Y. Elimination of the skyrmion Hall effect by tuning perpendicular magnetic anisotropy and spin polarization angle. Physics Letters A 456 128497, (2022).
  * [32] Müller, J et al. Magnetic Skyrmions and Skyrmion Clusters in the Helical Phase of Cu2OSeO3. Phys. Rev. Lett. 119 , 137201, (2017).
  * [33] Knapman, R. Rodrigues, D. R. Masell, J $\&$ Everschor-Sitte, K. Current-induced H-shaped-skyrmion creation and their dynamics in the helical phase. J. Phys. D: Appl. Phys. 54, 404003,(2021).
  * [34] Song, D. et al. Experimental observation of one-dimensional motion of interstitial skyrmion in FeGe. arXiv:2212.08991 [cond-mat.mes-hall].
  * [35] Ezawa, M. Compact merons and skyrmions in thin chiral magnetic films. Phys. Rev. B 83, 100408, (2011)
  * [36] Jena, J. Elliptical Bloch skyrmion chiral twins in an antiskyrmion system. Nat. Commun 11, 1115 (2020).
  * [37] Kumar, V. et al. Detection of antiskyrmions by topological Hall effect in Heusler compounds. Phys. Rev. B 101, 014424 (2020)
  * [38] Ma, T. et al. Tunable Magnetic Antiskyrmion Size and Helical Period from Nanometers to Micrometers in a D2d Heusler Compound. Adv. Mater. 32, 2002043, (2020).
  * [39] Giri, B. et al. Robust topological Hall effect driven by tunable noncoplanar magnetic state in Mn-Pt-In inverse tetragonal Heusler alloys. Phys. Rev. B 102, 014449, (2020).
  * [40] Lin, S.-Z. Saxena, A. $\&$ Batista, C. D. Skyrmion fractionalization and merons in chiral magnets with easy-plane anisotropy. Phys. Rev. B 91, 224407 (2015).
  * [41] Supplementary.
  * [42] Jena, J. et al. Observation of fractional spin textures in a Heusler material. Nat. Commun.  13, 2348 (2022).
  * [43] Tretiakov, O. A. $\&$ Tchernyshyov, O. Vortices in thin ferromagnetic films and the skyrmion number. Phys. Rev. B 75, 012408 (2007).
  * [44] Gao, N. et al. Creation and annihilation of topological meron pairs in in-plane magnetized films.  Nat. Commun. 10, 5603 (2019).
  * [45] Jani, H. et al. Antiferromagnetic half-skyrmions and bimerons at room temperature. Nature 590, 74–79 (2021).
  * [46] Vousden, M. et al. Skyrmions in thin films with easy-plane magnetocrystalline anisotropy. Appl. Phys. Lett. 108, 132406 (2016).
  * [47] Okumura, S. Hayami, S. Kato, Y. $\&$ Motome, Y. Magnetic Hedgehog Lattices in Noncentrosymmetric Metals. Phys. Rev. B: Condens. Matter Mater. Phys. 101, 144416, 2020.
  * [48] Grytsiuk, S. et al. Topological- Chiral Magnetic Interactions Driven by Emergent Orbital Magnetism. Nat. Commun. 11, 511, (2020).
  * [49] Meshcheriakova, O. et al. Large Noncollinearity and Spin Reorientation in the Novel Mn2RhSn Heusler Magnet. Phys. Rev. Lett. 113, 087203, (2014).
  * [50] Swekis, P. Role of Magnetic Exchange Interactions in Chiral-Type Hall Effects of Epitaxial MnxPtSn Films. ACS Appl. Electron. Mater. 3, 3, 1323–1333, (2021).
  * [51] Donahue, M. J. $\&$ Porter, D. G. OOMMF Users Guide, Version 1.0, Interagency Report NISTIR, (2009).
  * [52] McCray, A. R. et al. Understanding complex magnetic spin textures with simulation-assisted Lorentz transmission electron microscopy. Phys. Rev. Appl. 15, 044025 (2021).
  * [53] Cortés-Ortuño, D. OOMMF Skyrmion Number. Zenodo doi:10.5281/zenodo.1296536., (2018).
  * [54] Ishizuka, K. $\&$ Allman, B. Phase measurement in electron microscopy using the transport of intensity equation. J. Electron Microsc. 54, 191–197 (2005).

## Acknowledgments

A.K.N. acknowledges the Department of Atomic Energy (DAE), the Department of
Science and Technology (DST)-Ramanujan research grant (No. SB/S2/RJN-081/2016)
and SERB research grant (CRG/2022/002517) of the Government of India for
financial support. A.K.N. acknowledges the Max Plank Society for support under
the Max Planck-India partner group project. B.G. and A.K.N acknowledges Dr. A.
Das of Solid State Physics Division from Bhabha Atomic Research Center,
Mumbai, for providing the neutron diffraction data.

## Author contributions

A.K.N. conceived the idea of the present work and led the project. B.G.
prepared and characterize polycrystalline samples, prepared the TEM lamellae,
carried out the magnetic measurements, and performed the micromagnetic
simulations. B.G. and D.C. performed the LTEM measurement. A.D. carried out
the neutron diffraction measurements. All authors contribute to the data
analysis. B.G., A.K.N., and S.S.P.P. wrote the manuscript.

## Additional information

Competing interests The authors declare that they have no competing interests.