11institutetext:  Shanghai Key Laboratory of Intelligent Information
Processing
School of Computer Science, Fudan University
11email: {hyyu20, libin<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# EAFormer: Scene Text Segmentation with Edge-Aware Transformers

Haiyang Yu  0000-0003-2747-7338 Teng Fu  0009-0009-8413-2185 Bin Li
Corresponding Author 0000-0002-9633-0033 Xiangyang Xue  0000-0002-4897-9209

###### Abstract

Scene text segmentation aims at cropping texts from scene images, which is
usually used to help generative models edit or remove texts. The existing text
segmentation methods tend to involve various text-related supervisions for
better performance. However, most of them ignore the importance of text edges,
which are significant for downstream applications. In this paper, we propose
Edge-Aware Transformers, termed EAFormer, to segment texts more accurately,
especially at the edge of texts. Specifically, we first design a text edge
extractor to detect edges and filter out edges of non-text areas. Then, we
propose an edge-guided encoder to make the model focus more on text edges.
Finally, an MLP-based decoder is employed to predict text masks. We have
conducted extensive experiments on commonly-used benchmarks to verify the
effectiveness of EAFormer. The experimental results demonstrate that the
proposed method can perform better than previous methods, especially on the
segmentation of text edges. Considering that the annotations of several
benchmarks (e.g., COCO_TS and MLT_S) are not accurate enough to fairly
evaluate our methods, we have relabeled these datasets. Through experiments,
we observe that our method can achieve a higher performance improvement when
more accurate annotations are used for training. The code and datasets are
available at https://hyangyu.github.io/EAFormer/.

###### Keywords:

Scene Text Segmentation Edge-Guided Segmentation Dataset Reannotation

##  1 Introduction

In the last decade, scene text segmentation [11, 54, 32, 38] has gained
significant traction, primarily due to advancements in deep learning. The
objective of a text segmentation model is to accurately distinguish between
foregrounds (text areas) and backgrounds (non-text areas) at the pixel level.
Scene text segmentation plays a crucial role in various applications, such as
document analysis [15, 31], scene text image super-resolution [28, 35], scene
understanding [13] and text erasing [50, 24, 12, 10]. For example, TEAN [35]
introduces the text segmentation results as auxiliary information to better
super-resolve scene text images.

In order to promote the development of scene text segmentation, various
methods [41, 1, 55] and datasets [45, 44, 3] have been proposed in recent
years. Previous scene text segmentation methods tend to introduce text-related
supervision, such as text or character recognition supervision, to improve
performance. TexRNet [44] proposes a pre-trained character discriminator to
introduce the supervision of character recognition, which requires additional
annotations of character-level bounding boxes. Similarly, PGTSNet [45] designs
a text perceptual discriminator to enhance the readability of segmentation
results. In addition, both of them utilize various losses for better
segmentation performance, which may make it challenging to select appropriate
hyper-parameters to balance multiple losses. Recently, TextFormer [41] employs
a recognition head to make the model focus on text details and improve its
perception of texts. For the scene text segmentation task, there are several
widely-used benchmarks, such as ICDAR13 FST [19], COCO_TS [3], MLT_S [4],
Total-Text [8], TextSeg [44] and BTS [45]. Although the samples of these
datasets seem to be adequate for deep-learning-based models, the annotation
quality of some datasets (e.g., MLT_S) may not meet the desired standards,
especially in text-edge areas. The annotations of these datasets are obtained
through bounding-box supervision, which cannot provide the same level of
accuracy and precision as the datasets annotated by humans, such as TextSeg
and BTS.

Figure 1: Results comparison of the downstream application (text erasing) with
different text masks as input. More accurate segmentation at text edges is
beneficial to the text erasing task since less text pixels are wrongly
predicted and more background information is reserved for the inpainting
model.

Although previous methods have achieved a certain performance improvement in
text segmentation, they ignore the significance of text edges in practical
applications. For instance, accurate text masks, especially in text-edge
areas, can provide more background information to inpaint text areas in the
text erasing task, as shown in Figure 1. In experiments, we observe that
traditional edge detection algorithms, such as Canny [5], can well distinguish
text edges. To fully exploit the merits of traditional edge detection methods
to improve the segmentation performance at text edges, in this paper, we
propose Edge-Aware Transformers (EAFormer) for scene text segmentation.
Specifically, EAFormer consists of three main modules: text edge extractor,
edge-guided encoder and text segmentation decoder. The text edge extractor
takes the scene images as input and predicts the text areas and edges. In this
module, we adopt a light-weighted backbone to detect text areas and use the
traditional edge detection algorithm Canny to obtain the edges of whole
images. To alleviate the interference of edges in non-text areas, the masks of
predicted text areas are used to filter out the edges of non-text areas. For
the edge-guided encoder, we adopt the framework of SegFormer [43], which is
composed of four stages. At the first stage of this encoder, we additionally
design a symmetric cross-attention sub-module, which aims at utilizing the
filtered text edges to guide the encoder to focus more on text edges at the
first stage. Finally, the outputs of the edge-guided encoder are fused and
input into the text segmentation decoder to predict text masks.

To validate the effectiveness of EAFormer, we have conducted extensive
experiments on six text segmentation benchmarks. The results demonstrate that
EAFormer can indeed improve the segmentation performance of the baseline
model. However, as aforementioned, the annotations of COCO_TS and MLT_S are
not accurate enough, which may make the experimental results on these two
datasets unconvincing. In order to solve this problem, we have re-annotated
the training, validation, and test sets of COCO_TS and MLT_S. Through
experiments, we observe that the proposed EAFormer can still achieve better
performance than previous methods with more accurate pixel-level annotations.

In summary, the contributions of this paper are as follows:

  * •

For better segmentation performance in text-edge areas, we propose Edge-Aware
Transformers (EAFormer) to explicitly predict text edges and use them to guide
the following encoder.

  * •

Considering the low-quality annotations of COCO_TS and MLT_S, we have re-
annotated them for experiments to make the experimental results of EAFormer on
these two datasets more convincing.

  * •

Extensive experiments on six scene text segmentation benchmarks demonstrate
that the proposed EAFormer can achieve state-of-the-art performance and
perform better in text-edge areas.

##  2 Related Work

###  2.1 Scene Text Detection

Existing scene text detection methods can be divided into two categories:
regression-based methods and segmentation-based methods. Regression-based
methods [27, 53, 17, 18] regard text detection as a distinct object detection
task, where the goal is to locate text regions by predicting the offsets from
anchors or pixels. However, texts exhibit significant variations in scale and
orientation compared to general objects. To handle oriented texts, EAST [53]
directly regresses offsets from boundaries in an anchor-free manner. While
regression-based methods perform well for quadrilateral texts, they struggle
to adapt to texts with arbitrary shapes. Segmentation-based methods [20, 34,
37, 25] consider text detection as a dense binary prediction task. DBNet [20]
introduces differentiable binarization within a segmentation network, allowing
for adaptive threshold prediction. Although various text detection methods
have been proposed, we only employ a light-weighed backbone to detect text
areas in our method. Although some methods [6, 46] in general segmentation
field have proposed to introduce the edge information to improve the
performance, they are not perfectly suitable for the text segmentation task,
which may result from two reasons: 1) To detect edges accurately, most of them
need the annotations of edges, which is time-consuming and labor-intensive. 2)
Directly employing them to solve text segmentation may introduce some edges of
non-text areas, leading to subpar performance.

###  2.2 Semantic Segmentation

Semantic segmentation is a fundamental task in computer vision, which involves
classifying each pixel in input images. Fully convolutional networks (FCN),
which can learn dense prediction efficiently, were previously mainstream for
the semantic segmentation task. To capture contextual information at multiple
scales, several methods [51, 7] introduce dilated convolutions or spatial
pyramid pooling to enlarge the receptive field. Subsequently, attention
mechanisms were introduced to better capture long-range dependencies [14, 47,
52]. Recently, a Transformer-based semantic segmentation method SegFormer [43]
proposes to combine hierarchical Transformer encoders with a lightweight MLP
decoder. Due to its outstanding performance, we adopt it as the baseline model
of our method.

###  2.3 Scene Text Segmentation

Scene text segmentation aims at predicting fine-grained masks for texts in
scene images. In the past, text segmentation methods often rely on
thresholding [30, 36, 33, 29] or low-level features [2, 21, 42, 39] to
binarize scene text images. However, these approaches often struggle with text
images that have complex colors and textures, leading to poor performance.
Recently, deep learning-based text segmentation methods have emerged. For
instance, SMANet [3] adopts the encoder-decoder structure and introduces a new
multi-scale attention module for scene text segmentation. TextFormer [41]
introduces a text decoder into the hierarchical segmentation framework to
enhance its ability to perceive text details. Due to the low labeling quality
of previous datasets, TexRNet [44] proposes the TextSeg dataset with fine-
grained annotations, which contain word- and character-level bounding
polygons, masks, and transcriptions. Considering the lack of Chinese texts in
text segmentation, a bilingual text segmentation dataset called BTS [45] has
been proposed. The authors of BTS also developed PGTSNet, which employs a pre-
trained text detection model to constrain text segmentation on detected text
areas.

##  3 Methodology

In this section, we introduce the proposed EAFormer in detail. First, we
introduce the motivation of proposing EAFormer. Then, we detail each module of
EAFormer, including text edge extractor, edge-guided encoder and text
segmentation decoder. Finally, we introduce the loss function of our method.

Figure 2: Feature clustering results of PGTSNet and EAFormer. The
visualization indicates that PGTSNet can hardly well perceive text edges
compared with EAFormer.

###  3.1 Motivation

It is indisputable that text edges are crucial for the scene text segmentation
task, especially for its downstream tasks like text erasing. Accurately
segmenting text edges can provide more background information for the text
erasing model to fill text areas. As shown in Figure 1, we utilize a pre-
trained inpainting model, taking different types of text masks as input, to
erase texts in scene images. Through experiments, we observe that the text
bounding-box mask is too coarse to provide more backgrounds for the inpainting
model. In addition, the text mask with inaccurate edge segmentation makes the
inpainting model mistakenly regard the pixels belonging to texts as
backgrounds, resulting in poor erasing results. Only when the text mask with
accurate edge segmentation is provided, the inpainting model can generate
satisfactory text erasing results.

Although PGTSNet [45] has realized the significance of text edges and employed
a binary cross-entropy loss for detecting pixels at the text edges, it fails
to explicitly introduce the easily available text edge information as one of
the input information. In order to verify its ability to perceive text edges,
we perform $K$-Means clustering on the features output by the backbone, where
$K$ is set to 3, representing the background, text edge, and text center,
respectively. Through the visualization results shown in Figure 2, we observe
that this method still has certain deficiencies in perceiving text edges.

In addition, we find that the traditional edge detection algorithm can obtain
accurate text edges, which may benefit the scene text segmentation task.
However, since the traditional edge detection method cannot distinguish text
areas and non-text areas, most edges are detected in non-text areas. If the
edge detection results are directly utilized as input to assist text
segmentation, it could potentially confuse the text segmentation model and
adversely impact its performance. More discussions are in Section 5. In the
following subsection, we will introduce how our method leverages the results
of traditional edge detection algorithms to achieve better performance in the
text segmentation task.

Figure 3: Overall structure of EAFormer. EAFormer consists of three modules:
text edge extractor, edge-guided encoder and text segmentation decoder. ‘SA’,
‘CA’, and ‘FFN’ represent self-attention, cross-attention, and feed-forward
network, respectively.

###  3.2 Edge-Aware Transformers (EAFormer)

As shown in Figure 3, the proposed EAFormer consists of three modules: text
edge extractor, edge-guided encoder, and text segmentation decoder. Given the
input scene text image $\mathbf{X}\in\mathbb{R}^{3\times H\times W}$, the text
edge extractor is used to obtain the edges of text areas $\mathbf{E}_{t}$.
Then, the text image $\mathbf{X}$ and detected text edges $\mathbf{E}_{t}$ are
input into the edge-guided encoder to extract edge-aware features. Finally,
the text segmentation decoder takes the features generated by the encoder as
input to produce the corresponding text mask $\mathbf{M}_{t}$.

Text Edge Extractor. Since text edges are crucial for the scene text
segmentation task, we propose a text edge extractor to obtain the edges of
text areas. At first, we use the traditional edge detection algorithm Canny
[5] to acquire the edges of the whole input image $\mathbf{E}_{w}$. As
aforementioned, the edges of non-text areas in $\mathbf{E}_{w}$ may have
negative impacts on text segmentation. Therefore, we introduce a lightweight
text detection model in the text edge extractor to perform edge filtering.
Specifically, we first use ResNet-like [16] backbone to extract multi-level
visual features
$\mathcal{F}^{d}=\\{\mathbf{F}^{d}_{1},\mathbf{F}^{d}_{2},\mathbf{F}^{d}_{3},\mathbf{F}^{d}_{4}\\}$,
where $\mathbf{F}^{d}_{i}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}$
represents the features at the $i$-th layer of the ResNet-like backbone (more
details about the backbone of text detection are introduced in the
supplementary material). Then, a text detection head is employed to predict
the mask of text areas $\mathbf{M}_{a}$, which can be formulated as:

$\mathbf{M}_{a}=\text{Conv}_{1\times
1}(\text{Concat}(\\{\mathbf{F}^{d}_{1},\mathbf{F}^{d}_{2},\mathbf{F}^{d}_{3},\mathbf{F}^{d}_{4}\\}))$
(1)

where $\text{Conv}_{1\times 1}$($\cdot$) and Concat($\cdot$) represent the
$1\times 1$ convolution layer and the concatenation operation, respectively.
With the help of the mask of text areas $\mathbf{M}_{a}$, we can filter out
the edges of non-text areas through pixel-wise multiplication between the mask
of text areas $\mathbf{M}_{a}$ and the detected edges $\mathbf{E}_{w}$.
Therefore, the edges of text areas $\mathbf{E}_{t}$ can be obtained by:

$\mathbf{E}_{t}=\mathbf{M}_{a}\odot\text{SoftArgmax}(\mathbf{E}_{w})$ (2)

It is worth mentioning that we exert the soft argmax operation on
$\mathbf{E}_{w}$ before multiplication since joint optimizing the text
detection and segmentation branch can achieve better text detection
performance. Then, the filtered text edges $\mathbf{E}_{t}$ are subsequently
input into the following edge-guided encoder to enhance its ability to
distinguish pixels around text edges.

Edge-Guided Encoder. Since SegFormer [43] exhibits outstanding abilities in
semantic segmentation, we adopt it as the basic framework of the edge-guided
encoder. As shown in Figure 3, the edge-guided encoder consists of four
stages, and the filtered text edges are merged at the first stage. Each
encoding stage contains three submodules: overlap patch embedding, efficient
self-attention, and feed-forward network. The overlap patch embedding is used
to extract local features around each patch. Then, the features are input into
the self-attention layer to excavate the correlations between pixels. The
vanilla self-attention layer is formulated as follows:

$\text{SA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}(\frac{\mathbf{QK}^{\top}}{\sqrt{d_{\text{head}}}})\mathbf{V},$
(3)

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are obtained by applying
different embedding layers to the same features. To reduce the computational
cost, we follow [43] to introduce the spatial reduction operation for
$\mathbf{K}$ and $\mathbf{V}$. More details of spatial reduction are shown in
the supplementary material. Finally, for the $i$-th stage, a feed-forward
network is employed to generate the output features $\mathbf{F}^{s}_{i}$.
Differently, we additionally introduce a symmetric cross-attention layer after
the feed-forward network of the first stage to merge the extracted edge
guidance $\mathbf{E}_{t}$. Specifically, the symmetric cross-attention layer
includes two cross-attention operations between the features
$\mathbf{F}^{s}_{1}$ from the first stage and the edge guidance
$\mathbf{E}_{t}$. On one hand, $\mathbf{E}_{t}$ is regarded as Query to
extract the edge-aware visual information $\mathbf{F}^{ev}$, where
$\mathbf{F}^{s}_{1}$ is viewed as Key and Value; on the other hand,
$\mathbf{F}^{s}_{1}$ is used as Query to further excavate the useful text edge
information $\mathbf{F}^{te}$, where $\mathbf{E}_{t}$ is seen as Key and
Value. Therefore, the final output of the first stage
$\hat{\mathbf{F}}^{s}_{1}$ can be expressed as:

$\displaystyle\hat{\mathbf{F}}^{s}_{1}$
$\displaystyle=\mathbf{F}^{ev}\oplus\mathbf{F}^{te}\oplus\mathbf{F}^{s}_{1}$
(4) $\displaystyle\mathbf{F}^{ev}$
$\displaystyle=\text{SA}(\mathbf{E}_{t},\mathbf{F}^{s}_{1},\mathbf{F}^{s}_{1})$
$\displaystyle\mathbf{F}^{te}$
$\displaystyle=\text{SA}(\mathbf{F}^{s}_{1},\mathbf{E}_{t},\mathbf{E}_{t})$

where SA($\cdot$) represents the aforementioned self-attention operation,
$\oplus$ denotes the pixel-wise addition. Subsequently,
$\hat{\mathbf{F}}^{s}_{1}$ and outputs of other stages are input into the text
segmentation decoder.

Text Segmentation Decoder. Similar to the previous method [41], we employ
several MLP layers to fuse features and predict the final text masks
$\mathbf{M}_{t}$. First, we unify the channel dimension of the outputs of four
stages through corresponding MLP layers. Then, they are up-sampled into the
same resolution and further fused by an MLP layer. Finally, the fused features
are used to predict the text masks. Assuming that the resolution of features
from the $i$-th stage is $H_{i}\times W_{i}\times C_{i}$, the decoding process
can be formulated as:

$\displaystyle\tilde{\mathbf{F}^{s}_{i}}$
$\displaystyle=\text{MLP}(C_{i},C_{1})(\mathbf{F}),\
\mathbf{F}\in\\{\hat{\mathbf{F}}^{s}_{1},\mathbf{F}^{s}_{2},\mathbf{F}^{s}_{3},\mathbf{F}^{s}_{4}\\}$
(5) $\displaystyle\tilde{\mathbf{F}^{s}_{i}}$
$\displaystyle=\text{UpSample}(H_{1},W_{1})(\tilde{\mathbf{F}^{s}_{i}}),\
i\in\\{1,2,3,4\\}$ $\displaystyle\mathbf{F}^{s}$
$\displaystyle=\text{Fuse}(\mathcal{F}),\
\mathcal{F}=\\{\tilde{\mathbf{F}^{s}_{1}},\tilde{\mathbf{F}^{s}_{2}},\tilde{\mathbf{F}^{s}_{3}},\tilde{\mathbf{F}^{s}_{4}}\\}$
$\displaystyle\mathbf{M}_{t}$
$\displaystyle=\text{MLP}(C_{1},2)(\mathbf{F}^{s})$

where MLP($C_{\text{in}}$, $C_{\text{out}}$)($\cdot$) represents that the
channels of input and output features in MLP are $C_{\text{in}}$ and
$C_{\text{out}}$, respectively. Fuse($\cdot$) denotes that the input features
are first concatenated and then reduced in the channel dimension through an
MLP layer.

###  3.3 Loss Function

Previous text segmentation methods [45, 44] tend to introduce various losses
to improve the performance, which may bring difficulties in choosing
appropriate hyper-parameters. In the proposed EAFormer, only two cross-entropy
losses, text detection loss $\mathcal{L}_{det}$ and text segmentation loss
$\mathcal{L}_{seg}$, are used for optimization, which can be expressed by:

$\mathcal{L}=\underbrace{\text{CE}(\mathbf{M}_{t},\hat{\mathbf{M}_{t}})}_{\mathcal{L}_{seg}}+\
\lambda\
\underbrace{\text{CE}(\mathbf{M}_{a},\hat{\mathbf{M}_{a}})}_{\mathcal{L}_{det}}$
(6)

where $\lambda$ is the hyper-parameter to balance $\mathcal{L}_{det}$ and
$\mathcal{L}_{seg}$; $\hat{\mathbf{M}}_{a}$ and $\hat{\mathbf{M}}_{t}$ are the
ground truth of $\mathbf{M}_{a}$ and $\mathbf{M}_{t}$, respectively. Please
note that the utilized bounding box-level supervision for $\mathbf{M}_{a}$ can
be obtained from the semantic-level annotations, which means that the proposed
method only requires the semantic-level annotations as the same as previous
methods.

##  4 Experiments

Dataset | Images | Train/Test/Valid | Words | Classes | Languages
---|---|---|---|---|---
ICDAR13 FST [19] | 462 | 229/233/- | 1,944 | 36 | English
COCO_TS [3] | 14,690 | 11,882/2,808/- | 139,034 | 36 | English
MLT_S [4] | 6,896 | 5,540/1,356/- | 30,691 | 36 | English
Total-Text [8] | 1,555 | 1,255/300/- | 9,330 | 36 | English
TextSeg [44] | 4,024 | 2,646/1,038/340 | 15,691 | 36 | English
BTS [45] | 14,287 | 10,191/1,366/2,730 | 44,385 | 3,988 | Chinese,English

Table 1: Statistical details about the adopted six text segmentation datasets.

###  4.1 Datasets

In this paper, we have conducted extensive experiments on six text
segmentation benchmarks, including five English text segmentation datasets
(ICDAR13 FST [19], COCO_TS [3], MLT_S [4], Total-Text [8], and TextSeg [44])
and one bi-lingual text segmentation dataset BTS [45]. Some statistical
details about each dataset are shown in Table 1. The examples of each dataset
are displayed in the supplementary material.

Figure 4: Comparison between original and modified annotations. The original
datasets have the problems of missing and inaccurate annotations. Using re-
annotated datasets to train the proposed method makes the experimental results
more convincing.

As shown in Figure 4, the original annotations of COCO_TS and MLT_S are too
coarse to train a text segmentation model with satisfactory performance. Even
if the proposed method achieves better performance on these datasets, it is
not sufficient to demonstrate the effectiveness of our method. To make the
experimental results more convincing, we have re-annotated all samples of
these two datasets and use the newly annotated datasets to conduct
experiments. Figure 4 shows the comparison between original and modified
annotations.

###  4.2 Implementation Details

The proposed method is implemented with PyTorch, and we conduct all
experiments on 8 NVIDIA RTX 4090 GPUs. The AdamW [23] optimizer is adopted
with an initial learning rate 6$\times 10^{-5}$ in all experiments, and the
weight decay is set to 0.01. The batch size is set to 4. Following the
previous methods [41, 32, 45], we also adopt some data augmentation
operations, such as random cropping and flipping, in the training phase.
Different from existing methods employing pre-trained models to detect text
areas [45] or recognize characters [44], all modules in the proposed EAFormer
are jointly trained. In other words, no additional datasets are used when
training EAFormer. Two thresholds of Canny are set to 100 and 200,
respectively. To evaluate the performance of the proposed method, we utilize
both foreground Intersection-over-Union (fgIoU) and F-score on foreground
pixels. The metrics of fgIoU and F-score follow the percentage format and
decimal format, respectively.

Method | ICDAR13 FST | COCO_TS | MLT_S | Total-Text | TextSeg
---|---|---|---|---|---
fgIoU | F-score | fgIoU | F-score | fgIoU | F-score | fgIoU | F-score | fgIoU | F-score
PSPNet [51, 3] | - | 0.797 | - | - | - | - | - | 0.740 | - | -
SMANet [4] | - | 0.785 | - | - | - | - | - | 0.770 | - | -
DeeplabV3+ [7] | 69.27 | 0.802 | 72.07 | 0.641 | 84.63 | 0.837 | 74.44 | 0.824 | 84.07 | 0.914
HRNetV2-W48 [40] | 70.98 | 0.822 | 68.93 | 0.629 | 83.26 | 0.836 | 75.29 | 0.825 | 85.03 | 0.914
HRNet-OCR [49] | 72.45 | 0.830 | 69.54 | 0.627 | 83.49 | 0.838 | 76.23 | 0.832 | 85.98 | 0.918
TexRNet [44] | 73.38 | 0.850 | 72.39 | 0.720 | 86.09 | 0.865 | 78.47 | 0.848 | 86.84 | 0.924
ARM-Net [32] | - | 0.851 | - | - | - | - | - | 0.854 | - | 0.927
TFT [48] | 72.71 | 0.845 | 73.40 | 0.847 | 87.80 | 0.935 | 82.10 | 0.902 | 87.11 | 0.931
TextFormer [41] | 72.27 | 0.838 | 73.20 | 0.745 | 86.66 | 0.908 | 81.56 | 0.887 | 87.42 | 0.933
SegFormer (Base) [43] | 60.44 | 0.753 | 63.17 | 0.774 | 78.77 | 0.863 | 73.31 | 0.846 | 84.59 | 0.916
EAFormer (Ours) | 72.63 | 0.840 | 81.03 | 0.895 | 89.02 | 0.942 | 82.73 | 0.906 | 88.06 | 0.939

Table 2: Performance comparison on five English text segmentation datasets.
The bold and underlined numbers represent the best and second-best results,
respectively.

Method | COCO_TS | MLT_S
---|---|---
fgIoU | F-score | fgIoU | F-score
TextFormer [41] | 52.73 | 0.688 | 74.83 | 0.861
EAFormer (Ours) | 64.82 | 0.786 | 81.92 | 0.900

Table 3: The experimental results on re-annotated COCO_TS and MLT_S. The
results of TextFormer are obtained by re-implementing the method.

###  4.3 Experimental Results

Quantitative Comparison. To comprehensively evaluate EAFormer, we have
conducted experiments on both English and bi-lingual text segmentation
datasets. The experimental results on five English text segmentation datasets
are shown in Table 2. Compared with previous methods, EAFormer can achieve a
clear improvement in fgIoU and F-score on most benchmarks. For example, on
TextSeg, EAFormer outperforms the previous SOTA method TextFormer [41] by
0.64% and 0.6% in fgIoU and F-score, respectively. Although the original
COCO_TS and MLT_S datasets have coarse annotations, the proposed EAFormer can
still exhibit better performance, such as achieving a 7.63% improvement in
fgIoU on the COCO_TS dataset compared with TFT [48]. Considering that the
experimental results based on inaccurate annotations are not convincing
enough, we have re-annotated both the training dataset and the test dataset of
COCO_TS and MLT_S. The experimental results based on re-annotated datasets are
displayed in Table 3. Through experiments, we observe that the proposed
EAFormer can still achieve a considerable performance improvement when the
datasets with more accurate annotations are used for training and test.
Compared with the results on original datasets, the performance on re-
annotated datasets seems to drop a lot. The following two reasons may explain
this phenomenon: 1) There are many blurred texts in the datasets, which indeed
bring certain challenges to the model to deal with text edges; 2) The re-
annotated test datasets are more accurate and there are no ignored areas in
evaluation. In addition, we also conduct experiments on the bi-lingual text
segmentation dataset BTS [45], and the results are shown in Table 4. Although
PGTSNet unfairly introduces a pre-trained text detector, EAFormer can still
achieve an improvement of 1.6%/2.8% in fgIoU/F-score, which validates the
effectiveness of the proposed method.

Since we introduce a light-weighted text detection head, it is inevitable to
introduce more parameters. We have evaluated the number of parameters and
inference speed. Compared with the previous SOTA method TextFormer (85M
parameters and 0.42s per image), the proposed model has 92M parameters and
costs 0.47s per image in average. With slight increasing in the number of
parameters, our method can achieve significant performance improvements.

Method | BTS
---|---
fgIoU | F-score
DeeplabV3+ [7] | 71.15 | 0.796
HRNetV2-W48 [40] | 81.84 | 0.861
HRNetV2-W48+OCR [49] | 82.76 | 0.866
TexRNet (DeeplabV3+, w/o Cls) [44] | 83.68 | 0.883
TexRNet (DeeplabV3+, w/ Cls) [44] | 83.81 | 0.892
PGTSNet [45] | 86.48 | 0.909
TFT [48] | 87.84 | 0.935
TextFormer [41] | 86.97 | 0.930
SegFormer (Baseline) [43] | 84.99 | 0.908
EAFormer (Ours) | 88.08 | 0.937

Table 4: Performance comparison on BTS. The bold and underlined numbers
represent the best and second-best results, respectively.

Qualitative Comparison. We also compare EAFormer with previous methods in term
of segmentation quality through visualizations. As shown in Figure 5, the
proposed EAFormer can perform better than previous methods at text edges,
which benefits from the introduced edge information. In addition, for COCO_TS
and MLT_S, we compare the segmentation results based on both the original and
modified annotations. Although Table 3 indicates that the performance of our
method has declined when using the re-annotated datasets for training and
test, the visualizations in Figure 5 demonstrate that our model is able to
achieve better segmentation results based on the re-annotated datasets. More
visualizations are shown in the supplementary material.

$\lambda$ | TextSeg | BTS
---|---|---
fgIoU | F-score | fgIoU | F-score
0.1 | 84.03 | 0.910 | 86.45 | 0.913
0.5 | 87.33 | 0.926 | 87.03 | 0.931
1.0 | 88.06 | 0.939 | 88.08 | 0.937
5.0 | 87.67 | 0.934 | 87.62 | 0.935
10.0 | 87.94 | 0.937 | 87.58 | 0.933

Table 5: The experimental results of choosing $\lambda$. When $\lambda$ is set
to 1.0, the proposed method can achieve the best performance. Figure 5:
Visualizations of qualitative comparison between different methods or training
with different annotations. ‘OA’ and ‘RA’ indicate training EAFormer with
original annotations and re-annotations, respectively.

###  4.4 Ablation Study

Hyper-parameter $\lambda$. During training EAFormer, two losses are used for
optimization. The hyper-parameter $\lambda$ is to balance the weights of
$\mathcal{L}_{det}$ and $\mathcal{L}_{seg}$, and an appropriate $\lambda$ may
result in better performance. Thus, we conduct some experiments to select
$\lambda$ ranging from $\\{0.1,0.5,1.0,5.0,10.0\\}$, and the experimental
results are shown in Table 5. When $\lambda$ is set to 1.0, EAFormer reaches
the best and it boosts fgIoU/F-score by 3.47%/2.3% compared with the baseline
model on TextSeg. The results shown in Table 5 indicate that $\lambda$ has
little impact on performance when it ranges from {0.5, 1.0, 5.0, 10.0}.
However, if $\lambda$ is set to 0.1, the performance of EAFormer is
unsatisfactory, which may result from that too small $\lambda$ makes it
difficult for the text detection module to converge and further affects the
performance of text segmentation. Therefore, we set $\lambda$ to 1.0 for all
experiments in this paper.

EF | EG | TextSeg | BTS
---|---|---|---
fgIoU | F-score | fgIoU | F-score
|  | 84.59 | 0.916 | 84.99 | 0.908
✓ |  | 86.85 | 0.927 | 87.35 | 0.922
| ✓ | 81.03 | 0.832 | 80.35 | 0.828
✓ | ✓ | 88.06 | 0.939 | 88.08 | 0.937

Table 6: The experimental results of ablation studies. If only the edge
guidance is introduced, the edges of non-text areas have negative impacts on
the proposed method.

Edge Filtering and Edge Guidance. In the proposed EAFormer, the edge filtering
in the text edge extractor and the edge guidance in the edge-guided encoder
are two key components. To evaluate the performance gain of these two
strategies, we conduct ablation experiments on them and the results are shown
in Table 6. Please note that when only edge filtering is used, the extracted
edge information is concatenated with the input image and fed into the
SegFormer-based encoder. As shown Table 6, introducing edge filtering can lead
to a clear performance improvement. However, if only edge guidance is
introduced, the performance of our method is subpar. A possible reason is that
the edges of non-text areas introduce more interference information, resulting
in that the model cannot effectively use the extracted edges to assist text
segmentation. Therefore, both edge filtering and edge guidance are necessary
for our method, and when both of them are adopted, EAFormer can achieve the
SOTA performance.

##  5 Discussions

Filtering out the edges of non-text areas. In the text edge extractor module,
we propose to filter out edge information in non-text areas to avoid their
adverse impact on model performance. In the section of the ablation
experiment, we can know that filtering the edge information of non-text areas
can clearly improve the performance. Through visualizations (see the
supplementary material), we observe that when all edge information is used to
assist segmentation, the model will mistakenly believe that areas with edge
information should be classified as foreground. Therefore, in order to give
the model explicit edge guidance, the proposed method only retains the edge
information of the text area as input.

Introducing text edges at different layers. In the edge-guided encoder, we
extract edge-enhanced feature information only in the first stage through
symmetric cross-attention. It is well known that lower-level features are more
sensitive to text edge information. We have visualized the clustering results
of features at different stages in Figure 6, and the visualizations indicate
that only the features of the first stage focus on the edge information.
Therefore, it is reasonable and effective to introduce detected edges at an
earlier stage. We also tried to introduce the edge guidance at other stages to
conduct experiments (detailed results are shown in the supplementary
material). The experimental results indicate that the higher the stage at
which the detected edges are introduced, the smaller the performance
improvement of EAFormer. In particular, when the detected edges are introduced
at the third or fourth stage, the performance of EAFormer is even lower than
the baseline.

Utilizing an off-the-shelf text detector. In the text edge extractor, we
employ a lightweight text detector consisting of a ResNet-based backbone and
an MLP decoder. In fact, we can utilize an off-the-shelf text detector that
has been pre-trained on text detection datasets, which can help EAFormer
achieve better performance in practical applications. Since this may be unfair
to the previous method, we only explore the performance upper limit of
EAFormer. In experiments, using pre-trained DBNet [20] to replace the
lightweight text detector module, the performance of EAFormer on TextSeg can
reach a new SOTA performance (90.16%/95.2% in fgIoU/F-score).

Differences from previous edge-guided methods. Actually, the incorporation of
edge information for segmentation is a well-explored strategy [26, 9, 22].
However, our method still has some differences from previous works. First,
BCANet [26] and BSNet [9] need edge supervision while the proposed method
directly employs Canny to extract edges. Although EGCAN [22] also uses Canny,
our method additionally introduce edge filtering to reserve useful edge
information, which is specifically designed for text segmentation. In
addition, EGCAN fuses the edge information in all encoder layers while our
method fuses the edge information only in the first layer through our designed
symmetric cross-attention.

Limitations. To filter the edges of non-text areas, we introduce a light-
weighted text detector, which may slightly increase the number of learnable
parameters. In addition, we only utilize the off-the-shelf edge detection
algorithm Canny to extract text edges rather than employing a better deep-
learning-based edge detection method. Introducing a SOTA edge detection method
may further improve the performance of our method.

Figure 6: The clustering results of features from different stages. It is
reasonable to introduce the text-edge guidance at the first stage since only
the features of the first stage focus on edge information.

##  6 Conclusion

In this paper, we propose Edge-Aware Transformers, called EAFormer, to solve
inaccurate text segmentation at text edges. Specifically, the traditional edge
detection algorithm Canny is employed to extract edges. To avoid involving the
edges of non-text areas, a light-weighted text detection module is adopted to
filter out the useless edges for segmenting texts. In addition, based on
SegFormer, we propose an edge-guided encoder to enhance its ability to
perceive text edges. Considering that the low-quality annotations of several
datasets may affect the credibility of experimental results, we have re-
annotated these datasets. Extensive experiments are conducted on publicly
available benchmarks, and the SOTA results validate the effectiveness of
EAFormer in the text segmentation task.

## Acknowledgements

This work was supported in part by the National Natural Science Foundation of
China (No.62176060), STCSM project (No.22511105000), Shanghai Municipal
Science and Technology Major Project (No.2021SHZDZX0103), and the Program for
Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of
Higher Learning.

## References

  * [1] Andreini, P., Ciano, G., Bonechi, S., Graziani, C., Lachi, V., Mecocci, A., Sodi, A., Scarselli, F., Bianchini, M.: A two-stage gan for high-resolution retinal image generation and segmentation. Electronics 11(1), 60 (2021)
  * [2] Bai, B., Yin, F., Liu, C.L.: A seed-based segmentation method for scene text extraction. In: 2014 11th IAPR International Workshop on Document Analysis Systems. pp. 262–266. IEEE (2014)
  * [3] Bonechi, S., Andreini, P., Bianchini, M., Scarselli, F.: Coco_ts dataset: pixel–level annotations based on weak supervision for scene text segmentation. In: International Conference on Artificial Neural Networks. pp. 238–250. Springer (2019)
  * [4] Bonechi, S., Bianchini, M., Scarselli, F., Andreini, P.: Weak supervision for generating pixel–level annotations in scene text segmentation. Pattern Recognition Letters 138, 1–7 (2020)
  * [5] Canny, J.: A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence pp. 679–698 (1986)
  * [6] Chen, J., Li, J., Pan, D., Zhu, Q., Mao, Z.: Edge-guided multiscale segmentation of satellite multispectral imagery. IEEE Transactions on Geoscience and Remote Sensing 50(11), 4513–4520 (2012)
  * [7] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 801–818 (2018)
  * [8] Ch’ng, C.K., Chan, C.S.: Total-text: A comprehensive dataset for scene text detection and recognition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 935–942. IEEE (2017)
  * [9] Cong, R., Zhang, Y., Yang, N., Li, H., Zhang, X., Li, R., Chen, Z., Zhao, Y., Kwong, S.: Boundary guided semantic learning for real-time covid-19 lung infection segmentation system. IEEE Transactions on Consumer Electronics 68(4), 376–386 (2022)
  * [10] Conrad, B., Chen, P.I.: Two-stage seamless text erasing on real-world scene images. In: 2021 IEEE International Conference on Image Processing (ICIP). pp. 1309–1313. IEEE (2021)
  * [11] Dai, Y., Huang, Z., Gao, Y., Xu, Y., Chen, K., Guo, J., Qiu, W.: Fused text segmentation networks for multi-oriented scene text detection. In: 2018 24th international conference on pattern recognition (ICPR). pp. 3604–3609. IEEE (2018)
  * [12] Du, X., Zhou, Z., Zheng, Y., Ma, T., Wu, X., Jin, C.: Modeling stroke mask for end-to-end text erasing. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6151–6159 (2023)
  * [13] Ess, A., Müller, T., Grabner, H., Van Gool, L.: Segmentation-based urban traffic scene understanding. In: BMVC. vol. 1, p. 2. Citeseer (2009)
  * [14] Fu, J., Liu, J., Jiang, J., Li, Y., Bao, Y., Lu, H.: Scene segmentation with dual relation-aware attention network. IEEE Transactions on Neural Networks and Learning Systems 32(6), 2547–2560 (2020)
  * [15] Fujisawa, H., Nakano, Y., Kurino, K.: Segmentation methods for character recognition: from segmentation to document structure analysis. Proceedings of the IEEE 80(7), 1079–1092 (1992)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] He, W., Zhang, X.Y., Yin, F., Liu, C.L.: Deep direct regression for multi-oriented scene text detection. In: Proceedings of the IEEE international conference on computer vision. pp. 745–753 (2017)
  * [18] He, W., Zhang, X.Y., Yin, F., Liu, C.L.: Multi-oriented and multi-lingual scene text detection with direct regression. IEEE Transactions on Image Processing 27(11), 5406–5419 (2018)
  * [19] Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras, L.P.: Icdar 2013 robust reading competition. In: 2013 12th international conference on document analysis and recognition. pp. 1484–1493. IEEE (2013)
  * [20] Liao, M., Wan, Z., Yao, C., Chen, K., Bai, X.: Real-time scene text detection with differentiable binarization. In: Proceedings of the AAAI conference on artificial intelligence. pp. 11474–11481 (2020)
  * [21] Liu, X., Samarabandu, J.: Multiscale edge-based text extraction from complex images. In: 2006 IEEE International Conference on Multimedia and Expo. pp. 1721–1724. IEEE (2006)
  * [22] Liu, Z., Li, J., Song, R., Wu, C., Liu, W., Li, Z., Li, Y.: Edge guided context aggregation network for semantic segmentation of remote sensing imagery. Remote Sensing 14(6), 1353 (2022)
  * [23] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018)
  * [24] Lyu, G., Liu, K., Zhu, A., Uchida, S., Iwana, B.K.: Fetnet: Feature erasing and transferring network for scene text removal. Pattern Recognition 140, 109531 (2023)
  * [25] Lyu, P., Yao, C., Wu, W., Yan, S., Bai, X.: Multi-oriented scene text detection via corner localization and region segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7553–7563 (2018)
  * [26] Ma, H., Yang, H., Huang, D.: Boundary guided context aggregation for semantic segmentation. arXiv preprint arXiv:2110.14587 (2021)
  * [27] Ma, J., Shao, W., Ye, H., Wang, L., Wang, H., Zheng, Y., Xue, X.: Arbitrary-oriented scene text detection via rotation proposals. IEEE transactions on multimedia 20(11), 3111–3122 (2018)
  * [28] Ma, J., Jin, L., Zhang, J., Jiang, J., Xue, Y., He, M.: Textsrnet: Scene text super-resolution based on contour prior and atrous convolution. In: 2022 26th International Conference on Pattern Recognition (ICPR). pp. 3252–3258. IEEE (2022)
  * [29] Mustafa, W.A., Kader, M.M.M.A.: Binarization of document image using optimum threshold modification. In: Journal of Physics: Conference Series. p. 012022. IOP Publishing (2018)
  * [30] Otsu, N.: A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics 9(1), 62–66 (1979)
  * [31] Pack, C., Soh, L.K., Lorang, E.: Perceptual cue-guided adaptive image downscaling for enhanced semantic segmentation on large document images. International Journal on Document Analysis and Recognition (IJDAR) pp. 1–17 (2023)
  * [32] Ren, Y., Zhang, J., Chen, B., Zhang, X., Jin, L.: Looking from a higher-level perspective: Attention and recognition enhanced multi-scale scene text segmentation. In: Proceedings of the Asian Conference on Computer Vision. pp. 3138–3154 (2022)
  * [33] Sauvola, J., Pietikäinen, M.: Adaptive document image binarization. Pattern recognition 33(2), 225–236 (2000)
  * [34] Shi, B., Bai, X., Belongie, S.: Detecting oriented text in natural images by linking segments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2550–2558 (2017)
  * [35] Shu, R., Zhao, C., Feng, S., Zhu, L., Miao, D.: Text-enhanced scene image super-resolution via stroke mask and orthogonal attention. IEEE Transactions on Circuits and Systems for Video Technology (2023)
  * [36] Su, B., Lu, S., Tan, C.L.: Binarization of historical document images using the local maximum and minimum. In: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems. pp. 159–166 (2010)
  * [37] Tang, J., Yang, Z., Wang, Y., Zheng, Q., Xu, Y., Bai, X.: Seglink++: Detecting dense and arbitrary-shaped scene text by instance-aware component grouping. Pattern recognition 96, 106954 (2019)
  * [38] Tang, Y., Wu, X.: Scene text detection and segmentation based on cascaded convolution neural networks. IEEE transactions on Image Processing 26(3), 1509–1520 (2017)
  * [39] Vo, Q.N., Kim, S.H., Yang, H.J., Lee, G.: Binarization of degraded document images based on hierarchical deep supervised network. Pattern Recognition 74, 568–586 (2018)
  * [40] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence 43(10), 3349–3364 (2020)
  * [41] Wang, X., Wu, C., Yu, H., Li, B., Xue, X.: Textformer: Component-aware text segmentation with transformer. In: 2023 IEEE International Conference on Multimedia and Expo (ICME). pp. 1877–1882. IEEE (2023)
  * [42] Wu, Y., Natarajan, P., Rawls, S., AbdAlmageed, W.: Learning document image binarization from data. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 3763–3767. IEEE (2016)
  * [43] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34, 12077–12090 (2021)
  * [44] Xu, X., Zhang, Z., Wang, Z., Price, B., Wang, Z., Shi, H.: Rethinking text segmentation: A novel dataset and a text-specific refinement approach. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12045–12055 (2021)
  * [45] Xu, X., Qi, Z., Ma, J., Zhang, H., Shan, Y., Qie, X.: Bts: a bi-lingual benchmark for text segmentation in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19152–19162 (2022)
  * [46] Yin, X., Li, X., Ni, P., Xu, Q., Kong, D.: A novel real-time edge-guided lidar semantic segmentation network for unstructured environments. Remote Sensing 15(4), 1093 (2023)
  * [47] Yu, C., Wang, J., Gao, C., Yu, G., Shen, C., Sang, N.: Context prior for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12416–12425 (2020)
  * [48] Yu, H., Wang, X., Niu, K., Li, B., Xue, X.: Scene text segmentation with text-focused transformers. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 2898–2907 (2023)
  * [49] Yuan, Y., Chen, X., Wang, J.: Object-contextual representations for semantic segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. pp. 173–190. Springer (2020)
  * [50] Zdenek, J., Nakayama, H.: Erasing scene text with weak supervision. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2238–2246 (2020)
  * [51] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2881–2890 (2017)
  * [52] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6881–6890 (2021)
  * [53] Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., Liang, J.: East: an efficient and accurate scene text detector. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 5551–5560 (2017)
  * [54] Zhou, Y., Feild, J., Learned-Miller, E., Wang, R.: Scene text segmentation via inverse rendering. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 457–461. IEEE (2013)
  * [55] Zu, X., Yu, H., Li, B., Xue, X.: Weakly-supervised text instance segmentation. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 1915–1923 (2023)


# Majorana vortex modes in spin-singlet chiral superconductors with
noncollinear spin ordering: Local density of states study

A. O. Zlotnikov<EMAIL_ADDRESS>Kirensky Institute of Physics, Federal
Research Center KSC SB RAS, 660036 Krasnoyarsk, Russia

###### Abstract

In the present study topologically nontrivial edge and vortex bound states are
described in the coexistence phase of chiral spin-singlet superconductivity
and noncollinear spin ordering on a triangular lattice. Since chiral
superconductivity is nodeless in a wide range of parameters, the obtained
bound states are separated from trivial bulk states. It is proved that
Majorana modes localized at vortex cores are caused by noncollinear long-range
magnetic ordering. Even though nearby excitation energies of subgap states
including the edge-localized and vortex-localized states are very close to
each other, the energy difference between different vortex bound states is an
order of magnitude higher. It opens the way to experimentally detect the zero
energy Majorana modes localized at vortex cores in the considered structures
by using methods measuring local density of states such as tunneling
experiments. For a few pairs of vortices, vortex bound states with near zero
energy can appear. Nevertheless, they are localized on different vortices in
comparison with the localization of Majorana vortex modes. In this case zero
modes are still separated in energy and space from other states in local
density of states results. The difference between density of states near the
vortex and near the antivortex hosting Majorana modes is also demonstrated.

††preprint: APS/123-QED

##  I Introduction

Majorana modes that are localized at the ends of quasi-1D quantum wires have
been actively studied in recent years both theoretically and experimentally
Lutchyn, Sau, and Das Sarma (2010); Oreg, Refael, and von Oppen (2010); Shen
_et al._ (2021); Val’kov  _et al._ (2022). It is believed that such spatially
separated modes can be used for the realization of topologically protected
quantum computations Nayak  _et al._ (2008). The similar modes with zero
excitation energies propagate along the edges in 2D and quasi-2D topological
superconductors, thus preventing their use in braiding. On the other hand,
well separated Majorana modes in 2D topological superconductors can be
localized on inhomogeneities or topological defects, such as Abrikosov
vortices Read and Green (2000); Elliott and Franz (2015), magnetic skyrmions
Yang  _et al._ (2016); Zlotnikov, Shustin, and Fedoseev (2021), corners in
higher-order topological superconductors Volovik (2010); Zhu (2018);
Zlotnikov, Shustin, and Fedoseev (2021), and so on.

Initially the Majorana vortex modes were predicted in $p_{x}+ip_{y}$-wave
superconductors Kraus  _et al._ (2009); Silaev (2013); Akzyanov  _et al._
(2016) in which the presence of a topologically nontrivial phase is caused by
the triplet superconductivity. For superconductors with spin-singlet Cooper
pairings an additional interaction, mixing electrons with different spins,
should exist to induce topological superconductivity with Majorana modes.
Therefore, Majorana modes were predicted in the vortex state in spin-singlet
superconductors with pronounced spin-orbit interaction Iskin (2012); Björnson
and Black-Schaffer (2013), topological insulator / superconductor
heterostructures Chiu, Gilbert, and Hughes (2011); Akzyanov  _et al._ (2014),
iron chalcogenide FeTe0.55Se0.45 Pathak, Plugge, and Franz (2021),
superconducting Dirac and Weyl semimetals Yan, Wu, and Huang (2020).

It should be noted, that the subgap states including Majorana modes in 2D
topological superconductors requires the absence of bulk gapless excitations
in order to be realized. For example, in 2D nodal superconductors, such as
$d$-wave superconductors and extended $s$-wave superconductors with zeros of
the order parameter on nodal lines in the Brillouin zone, the trivial modes
appear near zero energy. Such modes are caused by the nodal bulk excitations
and prevent the formation of Majorana modes. Therefore a mechanism for opening
an energy gap in the bulk spectrum of nodal superconductors should be
suggested. Recently, it was shown that the gap is open for coupled structures
of the nodal superconductor and a magnetic texture crystal Steffensen,
Andersen, and Kotetes (2021). Another way to open the gap is the formation of
a chiral superconducting state with the complex order parameter, such as, for
example, the $d_{x^{2}-y^{2}}+id_{xy}$ state (in the following this symmetry
is marked as $d_{1}+id_{2}$). It should be mentioned that these statements are
correct for 2D systems with open boundary conditions in two directions of the
lattice. If the strip geometry is considered, when open boundaries exist in
one direction of the lattice and the periodic boundary conditions are applied
in the other direction, it becomes possible to study Majorana end modes in
nodal superconductors Sato and Fujimoto (2010).

The chiral d-wave superconductivity is characterized by spin-singlet pairings
and its order parameter vanishes at some points of the Brillouin zone (nodal
points). Therefore, the chiral superconductor has the gapped excitation
spectrum under wide conditions in the homogeneous case with the periodic
boundary conditions. The energy gap is closed only for certain parameters when
the Fermi contour intersects the set of nodal points Zhou and Wang (2008);
Val’kov, Val’kova, and Mitskan (2017). It is supposed that the
$d_{1}+id_{2}$-wave superconductivity can be formed in triangular and
hexagonal lattice systems Baskaran (2003); Black-Schaffer, Wu, and Le Hur
(2014); Kagan, Mitskan, and Korovushkin (2015).

It was proved over 20 years ago that the $d_{1}+id_{2}$ superconductor is the
topological one Volovik (1997) with zero energy edge states which are not
Majoranas due to the presence of the only spin-singlet component of the order
parameter. On the other hand, the vortex bound states in the
$d_{1}+id_{2}$-wave superconductor exist only at finite energy (in contrast to
the $d_{xz}+id_{yz}$-wave pairing symmetry) Lee and Schnyder (2016).

It is known that a helical long-range magnetic ordering in superconducting
structures with the spin-singlet component of pairing can also lead to
topological superconductivity and, consequently, to the formation of Majorana
modes. Among such structures are spin chains on the surface of a
superconductor Choy  _et al._ (2011); Nadj-Perge  _et al._ (2013) and 2D
magnetic superconductors Martin and Morpurgo (2012); Lu and Wang (2013);
Val’kov, Zlotnikov, and Shustin (2018); Bedow  _et al._ (2020); Rex, Gornyi,
and Mirlin (2020); Zlotnikov, Shustin, and Fedoseev (2021). Helical magnet-
superconductor heterostructures were realized experimentally, specifically, a
chain of Fe atoms with 120-degree ordering on a superconducting Re substrate
Kim  _et al._ (2018), as well as a Fe or Mn monolayer on the Re surface
Palacio-Morales  _et al._ (2019); Spethmann  _et al._ (2020). It should be
noted that in the vast majority of the theoretical studies of topological
superconductivity in magnetic superconductors the emphasis has been on the
description of Majorana end modes rather than Majorana vortex modes. The
formation of Majorana modes on the vortex-antivortex pair was briefly
described in Lu and Wang (2013) for the coexistence state of chiral
$d_{1}+id_{2}$ superconductivity and stripe magnetic ordering on a triangular
lattice with periodic boundary conditions.

Majorana bound states in vortices can be detected by scanning tunneling
spectroscopy (for example, Akzyanov  _et al._ (2014); Wang  _et al._ (2018);
Pathak, Plugge, and Franz (2021)). It is known that in the simplest
approximation the tunneling conductance is proportional to the local density
of states. In this article, we focus on the local density of states of the
chiral $d_{1}+id_{2}$ superconductor with 120-degree spin ordering on a
triangular lattice in the presence of Majorana modes localized at different
vortex structures: single vortex or antivortex, pair of vortices and vortex-
antivortex pair, four vortices. The chiral superconductivity provides the
gapped energy spectrum in the model, while 120-degree spin ordering supports
Majorana modes. It is shown that the excitation energy difference between
Majorana vortex modes and other subgap vortex bound states significantly
exceeds the one between edge states. Therefore, the Majorana vortex modes can
be detected experimentally by scanning tunneling microscopy. The conditions
for zero modes formation are found. The difference in the local density of
states is demonstrated for Majorana modes localized at the vortex and the
antivortex cores.

The article has been organized in five sections. In Sec. II the Hamiltonian of
the spin-singlet chiral superconductor with noncollinear spin ordering in the
presence of vortices is introduced and the methods are described. The relation
between the results of previous studies Lu and Wang (2013) and Lee and
Schnyder (2016) is discussed in Sec. III. Namely, appearance of zero energy
vortex bound states in $d_{1}+id_{2}$-wave superconducting state in the
presence of noncollinear spin ordering is noted. In Sec. IV the numerical
results for the excitation energies, and local density of states around the
vortex (or antivortex) cores and near the edges are presented. Conclusions are
given in Sec. V.

##  II Model and methods

In the framework of tight-binding approach and mean-field approximation the
coexistence of superconductivity and in-plane noncollinear spin ordering can
be described by the Hamiltonian

$\displaystyle\mathscr{H}$ $\displaystyle=$
$\displaystyle\sum_{f\sigma}(-\mu-\eta_{\sigma}h_{z})c_{f\sigma}^{{\dagger}}c_{f\sigma}+\sum_{ff_{1}\sigma}t_{ff_{1}}c_{f\sigma}^{{\dagger}}c_{f_{1}\sigma}+$
(1) $\displaystyle+$ $\displaystyle h\sum_{f}\left(\exp({i\bf
QR}_{f})c_{f\uparrow}^{{\dagger}}c_{f\downarrow}+\exp(-i{\bf
QR}_{f})c_{f\downarrow}^{{\dagger}}c_{f\uparrow}\right)+$ $\displaystyle+$
$\displaystyle\sum_{ff_{1}}\left(\Delta_{ff_{1}}c_{f\uparrow}^{{\dagger}}c_{f_{1}\downarrow}^{{\dagger}}+\Delta_{ff_{1}}^{*}c_{f_{1}\downarrow}c_{f\uparrow}\right),$

where $\mu$ is the chemical potential, $h_{z}$ is the Zeeman splitting term
due to external magnetic field along the $z$ axis, $t_{ff_{1}}$ is the hopping
parameter, $h=JM/2$ is the exchange field. The exchange field is caused by
$s-d(f)$ exchange interaction with the parameter $J$ between itinerant
electrons and frozen localized spins having a noncollinear magnetic structure
for which the vector spin operator is defined as

$\left\langle{\bf S}_{f}\right\rangle=M\left(\cos({\bf QR}_{f}),-\sin({\bf
QR}_{f}),0\right).$ (2)

Here, the spin structure vector is ${\bf Q}$, and $M$ is magnetic order
parameter. We suppose that in-plane magnetic anisotropy exist, and ordered
magnetic moments are not canted in low magnetic field along $z$-axis. The
parameter $\Delta_{ff_{1}}$ defines the amplitude of superconducting pairings.

In the Bogoliubov—de Gennes representation the Hamiltonian is expressed as

$\displaystyle\mathscr{H}=\frac{1}{2}\Psi^{{\dagger}}H\Psi,\,\,\,\Psi^{{\dagger}}=[c_{\uparrow}^{{\dagger}}\,c_{\downarrow}^{{\dagger}}\,c_{\downarrow}\,c_{\uparrow}],$
(3)

and

$\displaystyle H$ $\displaystyle=$
$\displaystyle\left({\begin{array}[]{*{20}{c}}{{{\xi}_{\uparrow}}}&{h}&{{D}}&0\\\
{{{h}^{*}}}&{{{\xi}_{\downarrow}}}&0&{-D^{T}}\\\
{D^{{\dagger}}}&0&{-\xi_{\downarrow}^{T}}&{-{h}}\\\
0&-D^{*}&{-h^{*}}&{-\xi_{\uparrow}^{T}}\end{array}}\right).$ (8)

The triangular lattice with the edges along the basic translation vectors
${\bf a}_{1}=a(\sqrt{3}/2,-1/2)$ and ${\bf a}_{2}=a(0,1)$ is considered. The
number of sites on the edges of the lattice is the same and denoted as
$N_{s}$. For this case the each element in the matrix $H$ is the matrix of the
size $N_{s}^{2}$. In the following, different bases defining the radius-vector
in the real space are used: ${\bf R}_{f}=x_{f}{\bf e}_{x}+y_{f}{\bf
e}_{y}=(n-1){\bf a}_{1}+(m-1){\bf a}_{2}$.

The Hamiltonian (1) belongs to the D-symmetry class which is described by the
integer topological invariant $\mathbb{Z}$. In the present study the
topological invariant $\tilde{N}_{3}$ Ishikawa and Matsuyama (1987); Volovik
(2009) expressed in terms of Green functions is used

$\displaystyle\tilde{N}_{3}=\frac{\varepsilon_{\mu\nu\lambda}}{48\pi^{2}}\sum_{\sigma}$
$\displaystyle\int\limits_{-\infty}^{\infty}d\omega\iint\limits_{-\pi}\limits^{\pi}dk_{1}dk_{2}\text{Tr}\left(\widehat{G}_{\sigma}\partial_{\mu}\widehat{G}^{-1}_{\sigma}\widehat{G}_{\sigma}\partial_{\nu}\widehat{G}^{-1}_{\sigma}\widehat{G}_{\sigma}\partial_{\lambda}\widehat{G}^{-1}_{\sigma}\right).$

Here, the repeated indices $\mu$, $\nu$, $\lambda=1,\,2,\,3$ imply summation,
$\varepsilon_{\mu\nu\lambda}$ is the Levi-Civita symbol,
$\partial_{1(2)}\equiv\partial/\partial k_{1(2)}$,
$\partial_{3}\equiv\partial/\partial\omega$, and
$\widehat{G}_{\sigma}(i\omega,{\bf k})$ is the matrix Green function under
periodic boundary conditions whose poles determine the bulk fermion spectrum.

In general the parameter $\Delta_{ff_{1}}$ can define the amplitude of
superconducting pairings between the fermions on the same site ($f=f_{1}$)
leading to s-wave superconductivity. The on-site excange field (the third sum
in (1)) in this case has a strong pair-breaking effect. Nevertheless,
noncollinear spin ordering and superconductivity can coexist if the pairing
interaction is non-local Val’kov and Zlotnikov (2019). Therefore in the
following the pairings between electrons on nearest sites are considered. The
presence of vortices is parameterized through the dependence on coordinates of
the superconducting order parameter:

$\displaystyle\Delta_{<ff_{1}>}=\Delta\exp\left[il\arg(z({\bf R}_{f})-z({\bf
R}_{f_{1}}))\right]e^{i\pi/3}\times$ (10)
$\displaystyle\prod_{i}\tanh\left(\frac{|{\bf R}_{ff_{1}}-{\bf
R}_{v_{i}}|}{\xi}\right)\exp\left[il_{v_{i}}\arg(z({\bf R}_{ff_{1}})-z({\bf
R}_{v_{i}}))\right],$

where the first exponent determines the symmetry of the superconducting order
parameter (for the chiral $d_{1}+id_{2}$ symmetry on the triangular lattice
$l=2$), $z({\bf R}_{f})=x_{f}+iy_{f}$ (in the basis of ${\bf e}_{x}$, ${\bf
e}_{y}$), the additional phase $\exp(i\pi/3)$ is used for convenience. The
last product describes different structures of noninteracting vortices
($l_{v_{i}}=+1$ for a vortex, and $l_{v_{i}}=-1$ for an antivortex), where
${\bf R}_{v_{i}}$ is a coordinate of the vortex (or antivortex) center and
$\xi$ is the characteristic length. Here, ${\bf R}_{ff_{1}}=({\bf R}_{f}+{\bf
R}_{f_{1}})/2$.

The energy spectrum and probability densities of subgap states are found using
the Bogoliubov transformation with Bogoliubov quasi-particle operators
$\alpha_{j}$ ($j=1,\dots,2N_{s}^{2}$):

$\displaystyle\alpha_{j}=\sum_{nm\sigma}\left(u_{jnm\sigma}c_{nm\sigma}+v_{jnm\sigma}c_{nm\sigma}^{{\dagger}}\right).$
(11)

Here we explicitly use the coordinates $n$ and $m$ in the basis of ${\bf
a}_{1}$, ${\bf a}_{2}$ instead of the site index $f$.

From the formula for site-dependent electron concentration $\left\langle
c_{nm\sigma}^{{\dagger}}c_{nm\sigma}\right\rangle$ the spatially and
frequency-resolved density of states (DOS) is written as

$\displaystyle\rho\left({\bf R}_{f},\omega\right)$ $\displaystyle=$
$\displaystyle\sum_{\begin{subarray}{c}\sigma,j\\\
(\varepsilon_{j}>0)\end{subarray}}\left[\left|u_{jnm\sigma}\right|^{2}\delta\left(\omega-\varepsilon_{j}\right)+\right.$
(12) $\displaystyle+$
$\displaystyle\left.\left|v_{jnm\sigma}\right|^{2}\delta\left(\omega+\varepsilon_{j}\right)\right],$

where the positive excitation energies $\varepsilon_{j}$ are chosen. According
to Pathak, Plugge, and Franz (2021) it is more informative to use the local
DOS integrated over a region around a vortex core, edges, etc.:

$\displaystyle\text{LDOS}_{v}\left(\omega\right)=\frac{1}{N_{W}}\sum_{|n-n_{v}|<W}\sum_{|m-m_{v}|<W}\rho\left({\bf
R}_{f},\omega\right).$

For $\omega=\varepsilon_{j}=0$ and an unbroadened zero-energy level DOS
$\rho\left({\bf R}_{f},\omega\right)$ corresponds to the widely used
probability density of the $j$-excitation

$\displaystyle\text{PrD}_{j}\left({\bf
R}_{f}\right)=\sum_{\sigma}\left(\left|u_{jnm\sigma}\right|^{2}+\left|v_{jnm\sigma}\right|^{2}\right).$
(14)

To describe spatially separated Majorana modes in the case of a doubly
degenerate ground state two Majorana quasiparticle operators $b^{\prime}$ and
$b^{\prime\prime}$ (following Kitaev Kitaev (2001)) are introduced. They are
expressed through the initial Majorana operators set
$\gamma_{Anm\sigma}=c_{nm\sigma}+c_{nm\sigma}^{{\dagger}}$ and
$\gamma_{Bnm\sigma}=i\left(c_{nm\sigma}^{{\dagger}}-c_{nm\sigma}\right)$:

$\displaystyle b^{\prime}=\alpha_{1}+\alpha_{1}^{{\dagger}}=$ $\displaystyle=$
$\displaystyle\sum_{nm\sigma}\left(\text{Re}(w_{1nm\sigma})\gamma_{Anm\sigma}-\text{Im}(z_{1nm\sigma})\gamma_{Bnm\sigma}\right),$
$\displaystyle
b^{\prime\prime}=i\left(\alpha_{1}^{{\dagger}}-\alpha_{1}\right)=$
$\displaystyle=$
$\displaystyle\sum_{nm\sigma}\left(\text{Im}(w_{1nm\sigma})\gamma_{Anm\sigma}+\text{Re}(z_{1nm\sigma})\gamma_{Bnm\sigma}\right),$

where $w_{jnm\sigma}=u_{jnm\sigma}+v_{jnm\sigma}$ and
$z_{jnm\sigma}=u_{jnm\sigma}-v_{jnm\sigma}$.

##  III  Appearance of zero energy vortex bound states due to 120-degree
magnetic order

Figure 1:  The chemical-potential dependence of the first excitation energy in
the presence of the vortex-antivortex pair on the triangular lattice with
periodic boundary conditions in two cases: without long-range magnetic order
($M=0$, in agreement with the results of Lee and Schnyder (2016)), and for
coexistence of superconductivity and 120∘ spin ordering (SC+NCO, $M=0.5$, in
agreement with Lu and Wang (2013)). Parameters: $J=4$, $\Delta=0.2$, $h_{z}=0$
(in units of the hopping parameter $|t|$).

To study vortex bound states, it is generally accepted to separate them from
edge states. Therefore, in order to suppress edge-localized states periodic
boundary conditions can be applied. However, the periodicity can be realized
only for special vortex structures providing continuity of the phase of the
order parameter on the opposite edges. For example, on a square lattice this
can be done for the vortex-antivortex pair Lee and Schnyder (2016) which is
described by the expression similar to (10) with the additional factor
providing periodic boundary conditions. As mentioned above, in this case zero
energy excitations are absent, and vortex bound state has a finite excitation
energy. It should be noted that this result is obtained in the continuum
limit.

On the other hand, the authors of Lu and Wang (2013) showed that in the
presence of the vortex-antivortex pair on the triangular lattice with periodic
boundary conditions Majorana vortex modes with zero energy arise. To provide
continuity of the phase of the order parameter on the opposite lattice edges
the vortex-antivortex pair is parameterized through the Jacobi theta-functions
(see Supplemental materials for Lu and Wang (2013)). Although it was not
explicitly emphasized in Refs. Lu and Wang (2013); Lee and Schnyder (2016),
but it can be concluded that namely noncollinear spin ordering leads to
appearance of Majorana vortex modes among the subgap vortex bound states with
finite energy. It is connected with the fact that symmetry class C of the
$d_{1}+id_{2}$-wave state changes to class D in the presence of magnetic
order.

Figure 2:  The chiral d-wave superconducting order parameter (10) in the
presence of the vortex which center is denoted by the large black dot. The
arrows on the nearest-neighbor bonds indicate the magnitude and the phase of
the order parameter.

In the present section this result is repeated in the tight-binding approach
for the triangular lattice of the identical size along ${\bf a}_{1}$ and ${\bf
a}_{2}$ and with periodic boundary conditions. We consider the vortex-
antivortex pair with ${\bf
R}_{v_{1}}=\left(\frac{N_{s}}{3},\frac{N_{s}+1}{2}\right)$ (antivortex) and
${\bf R}_{v_{2}}=\left(\frac{2N_{s}}{3}+1,\frac{N_{s}+1}{2}\right)$ (vortex)
on the triangular lattice ($N_{s}=63$) in two cases: (i) in the presence of
120∘ (${\bf Q}=\left(2\pi/3,2\pi/3\right)$) long-range magnetic order; (ii)
without magnetic ordering. The slightly modified parametrization of the
vortex-antivortex pair is used in comparison with the one in Lu and Wang
(2013). The chemical-potential dependencies of the first excitation energies
in both cases are displayed in Fig. 1 for the parameters $J=4$, $\Delta=0.2$,
$h_{z}=0$ (here and henceforth all energy variables are in units of the
hopping parameter $|t|$). It is seen that the modes with zero excitation
energy appear in the coexistence phase of superconductivity and noncollinear
spin ordering (SC+NCO), while in the absence of magnetic ordering only the
finite-energy vortex bound states are realized. The obtained zero modes
correspond to Majorana vortex modes $b^{\prime}$ and $b^{\prime\prime}$
localized at different cores of the vortex-antivortex pair.

It should be noted that taking into account Zeeman splitting ($h_{z}\neq 0$)
can lead to formation of zero modes even at $M=0$. Although, such modes are
not spatially separated and, consequently, they are not Majorana modes.

Figure 3:  The chemical-potential dependence of the first two excitation
energies in the presence of the vortex with the center at
$((N_{s}+1)/2,(N_{s}+1)/2)$ on the triangular lattice ($N_{s}=63$) with open
boundary conditions in SC+NCO phase. Parameters: $h=1$, $\Delta=0.2$,
$h_{z}=0.1$. Inset: The chemical-potential dependence of the bulk gap in the
excitation spectrum of the homogeneous system under periodic boundary
conditions. The vertical dashed line divide topological phases with different
values of the topological invariant.

In real systems periodic boundary conditions may not be applicable. Therefore
in the following, the simultaneous presence of edge and vortex bound states in
the SC+NCO phase with open boundary conditions, as well as features of local
density of states will be described. Note that topological edge states in the
chiral d-wave state can have zero excitation energy even in the absence of
magnetic order, in contrast to the vortex bound states.

##  IV  Local density of states study

###  IV.1  Single vortex on the lattice

The structure of the order parameter (10) in the presence of the single vortex
with $\xi=4a$ and ${\bf R}_{v}$ at the center of the triangular lattice is
presented in Fig. 2 (the region of the lattice in the vicinity of the vortex
core is only shown). The vortex center position is marked by the large black
dot. It is seen that the pairing amplitudes of the bonds nearest to the vortex
center are suppressed. As usual, the phase of the order parameter on
equivalent bonds denoted by the arrows with the same color changes by $2\pi$
going around the vortex center.

Figure 4:  Majorana modes in the presence of the single vortex on the
triangular lattice (the vortex center coincides with the lattice center) at
$\mu\approx-0.4205$. The distributions of $b^{\prime}$ (a) and
$b^{\prime\prime}$ (b) modes on the lattice are shown. Black color corresponds
to near zero values, the brightest color is for maximum values reaching
$0.24$. The parameters are the same as in Fig. 3.

In this section the vortex with the center located at the point
$(n_{v},m_{v})=((N_{s}+1)/2,(N_{s}+1)/2)$ (in the basis of ${\bf a}_{1}$,
${\bf a}_{2}$) on the triangular lattice with $N_{s}=63$ and open boundaries
is considered. The chosen lattice size when $N_{s}$ is multiple of 3 provides
zero in-plane total magnetic moment. In Fig. 3 the chemical-potential
dependence of two lowest-excitation energies is presented for the parameters
$h=1$, $\Delta=0.2$, $h_{z}=0.1$. The chemical potential values corresponding
to zero modes formation are denoted by dots on the $x$ axis. Such values are
calculated from the system of equations on the coefficients of the Bogoliubov
transformation (the algorithm is described in, for example, Nijholt and
Akhmerov (2016)).

Figure 5:  The excitation energies at $\mu\approx-0.4205$ (this value is
marked by the star in Fig. 3). Square markers denote implementation of
Majorana modes, large points — ingap vortex bound states, small points are for
ingap edge bound states. The bulk gap is denoted by the dashed lines.

The presented on Fig. 3 range of the chemical potential corresponds to the
case when on-site electron concentration, far from inhomogeneities such as a
vorte core and edges, changes from approximately 0.7 to near half filling. For
periodic boundary conditions in the homogeneous case different topological
phases with topological phase transition between them are realized on this
range. To show it the dependence of the size of the bulk gap on chemical
potential is calculated (see inset of Fig. 3). It is seen that gapless
excitations occur at the critical value of the chemical potential separating
topological phases with different topological invariant $\tilde{N}_{3}$.

The phase with $\tilde{N}_{3}=4$ is the topologically non-trivial phase which
is caused by chiral d-wave superconductivity. Such a phase exist also without
long-range magnetic order ($h=0$) Volovik (1997); Zhou and Wang (2008). It is
known that the $\tilde{N}_{3}=4$ phase does not support Majorana modes due to
spin-singlet character of $d_{1}+id_{2}$ superconductivity. On the other hand,
edge and vortex states with zero excitation energy can appear in this phase in
the presence of 120∘ spin ordering. Nevertheless, the modes $b^{\prime}$ and
$b^{\prime\prime}$ (see (II) and (II)) corresponding to such zero-energy
states are not spatially separated.

The topologically non-trivial phase with $\tilde{N}_{3}=1$ appears only in the
presence of noncollinear spin ordering ($h\neq 0$). In this phase the spatial
distributions of $b^{\prime}$ and $b^{\prime\prime}$ modes with zero energy at
$\mu\approx-0.4205$ (the zero-energy solution at this value is obtained in
Fig. 3 and is marked by the star) are shown in Fig. 4. It is seen that the
first Majorana mode is localized in the vortex core, while the second mode is
located near the edges. Such behavior confirms the formation of Majorana modes
at certain conditions in the $\tilde{N}_{3}=1$ phase. The obtained
distribution of Majorana modes in the presence of a single vortex is known for
other topological superconducting systems Björnson and Black-Schaffer (2013).

Figure 6:  (a) Local density of states (LDOS) in the vicinity of the single
vortex (solid line on the lower graph), near the edges (solid line on the
middle graph), and far from inhomogeneities (solid line on the upper graph) at
$\mu\approx-0.4205$ with existing Majorana modes. The broadening parameter for
LDOS is $\delta=\Delta/20$. The dashed line is DOS calculated for comparison
in the homogeneous case with periodic boundary conditions. (b) The same DOS as
shown by the dashed line in (a) but for all energies. (c) The integration
areas on the lattice for calculation of $\text{LDOS}(\omega)$ are highlighted
near the vortex core, near the edges, and far from inhomogeneities,
respectively.

The excitation energies at $\mu\approx-0.4205$ are demonstrated in Fig. 5. The
square markers are zero energy Majorana modes. The spatial separation on the
edge and vortex modes, as it is presented in Fig. 4, is obtained only for zero
energy. The large points denote excitation energies of ingap bound states
which are localized near the vortex core. At finite energy such states are
analogues of Caroli—de Gennes—Matricon bound states for the chiral d-wave
superconductor with triangular lattice. The small points are for energies of
edge states. The bulk gap size is marked by the horizontal dashed lines. The
energy values higher than the gap correspond to bulk excitations (not shown in
Fig. 5). It is seen that the energy difference between vortex bound states,
including the Majorana vortex mode, is an order of magnitude higher than the
difference between the adjacent energies corresponding to the edge states.
Therefore the Majorana mode located in the vortex core can be observed by the
methods probing LDOS.

Dependencies of LDOS on energy near the single vortex core, edges, and far
from inhomogeneities are shown in Fig. 6(a). For comparison the energy
dependence of DOS without vortices and under periodic boundary conditions is
presented in Fig. 6(b). A well-defined bulk superconducting gap is seen. For
the limited lattice in the Shubnikov phase the edge and vortex bound states
appear. Notably, the Majorana vortex mode is well separated by energy from
finite-energy vortex states. On the other hand, a band of overlapping edge
states appears in the superconducting gap for LDOS near the edges with the
chosen broadening parameter $\delta$. The integration areas for
$\text{LDOS}(\omega)$ are shown in Fig 6(c). In calculations of
$\text{LDOS}(\omega)$ the broadening parameter $\delta=\Delta/20$ for the
energy levels is used (to compare with the experimental STM resolution in
FeTe0.55Se0.45 see Pathak, Plugge, and Franz (2021)). It should be also noted
that in the presence of Zeeman splitting $h_{z}=0.1$ the average z-component
of spin operator $\left\langle S_{f}^{z}\right\rangle$ is non-zero
(approximately $0.025$) near the vortex core and edges, while it is close to
zero in the bulk of the lattice.

###  IV.2  Single antivortex on the lattice

It is known that there is difference between the wave functions of vortex
bound states and bound states in the case of an antivortex in chiral
superconductors ($p_{x}+ip_{y}$-wave and $d_{1}+id_{2}$-wave) Kraus  _et al._
(2009); Lee and Schnyder (2016). The reason is that the phase winding of the
order parameter is not equivalent in the case of the vortex and antivortex in
chiral superconductors, while for s-wave superconductivity this effect is
absent. We show that such difference is also manifested for Majorana bound
states localized at the vortex or the antivortex cores in the case of the
coexistence of chiral $d_{1}+id_{2}$ superconductivity and 120∘ spin ordering
on the triangular lattice.

Figure 7:  (a) The spatially and energetically resolved density of states
$\rho\left({\bf R}_{f}=(n,m),\omega\right)$ near the single vortex center
$\left(n_{v},m_{v}\right)$ and zero energy at $\mu\approx-0.4205$ supporting
Majorana vortex mode. The coordinate $n=n_{v}$ is fixed, while $m$ is changed
from $m_{v}-d$ to $m_{v}+d$ with $d=5$. (b) The same map of $\rho\left({\bf
R}_{f},\omega\right)$ for the single antivortex (the antivortex center is in
the center of the triangular lattice) at slightly different
$\mu\approx-0.3962$ (the other parameters do not changed). Tuning $\mu$ is
done to obtain Majorana modes for the antivortex case.

For the antivortex (with the same parameters ${\bf R}_{v}$ in the center of
the triangular lattice, $\xi=4a$, $N_{s}=63$) a number of zero modes appears
at slightly different values of chemical potential comparing to the case of
the vortex in Sec. IV.1. However, the Majorana modes are also localized near
the antivortex core and near the lattice edges in the topological phase with
$\tilde{N}_{3}=1$, and the energy spectrum with subgap states is qualitatively
the same. As it is mentioned above, $\rho\left({\bf R}_{f},\omega\right)$ in
the vicinity of the antivortex center should be distinguished from the one in
the case of the vortex. To show it the dependencies of $\rho\left({\bf
R}_{f},\omega\right)$ on coordinates and energy are presented in Fig. 7(a) for
the vortex at $\mu\approx-0.4205$ and in Fig. 7(b) for the antivortex at
$\mu\approx-0.3962$ (the other parameters remain unchanged). The chosen
conditions support Majorana modes formation for both cases. The highlighted
line on the graphs corresponds to the energy dependence of $\rho\left({\bf
R}_{f},\omega\right)$ exactly at the center of the vortex/antivortex ${\bf
R}_{f}={\bf R}_{v}=\left(n_{v},m_{v}\right)$ (in the basis of ${\bf a}_{1}$
and ${\bf a}_{2}$). To obtain the rest energy dependencies the coordinate $m$
of ${\bf R}_{f}$ is changed from $m_{v}-d$ to $m_{v}+d$ ($d=5$), while $n$
remains fixed $n=n_{v}$. It is seen that for the Majorana vortex mode
$\rho\left({\bf R}_{f},\omega\right)$ has a maximum exactly at the center of
the vortex and at zero energy. On the other hand, the maximum of
$\rho\left({\bf R}_{f},\omega\right)$ for the antivortex is shifted from zero
energy and the antivortex center. It should be noted that LDOS for the
antivortex case qualitatively does not changed from LDOS shown in Fig. 6.
Therefore, to distinguish the obtained features of $\rho\left({\bf
R}_{f},\omega\right)$ for the vortex and the antivortex fairly precise
positioning of a tip in STM experiments for larger vortices should be carried
out.

###  IV.3  Pair of vortices or vortex-antivortex pair on the lattice

Figure 8:  The excitations energies in the presence of the pair of vortices
with the vortex centers at $\left(\frac{N_{s}}{3},\frac{N_{s}+1}{2}\right)$
and $\left(\frac{2N_{s}}{3}+1,\frac{N_{s}+1}{2}\right)$ (the lattice size is
$N_{s}=99$) at $\mu\approx-0.389$. Highlighting markers at zero energy denote
Majorana modes, large points — ingap vortex bound states, small points are for
ingap edge bound states. The bulk gap is denoted by the dashed lines.

As it was discussed in Sec. III, Majorana modes have to be localized at the
cores of the vortices considering the vortex-vortex or vortex-antivortex
pairs. Usually the corresponding eigenstate problem is solved with periodic
boundary conditions. Existence of edges in real systems can influence on zero
modes: if the vortex is located close enough to the edge then each zero energy
mode is represented as a superposition of vortex bound states and edge bound
states. In the model (1) this is the case for the pair of vortices with
coordinates of the centers $\left(\frac{N_{s}}{3},\frac{N_{s}+1}{2}\right)$
and $\left(\frac{2N_{s}}{3}+1,\frac{N_{s}+1}{2}\right)$ on the triangular
lattice with $N_{s}=63$ (the same as in Sec. III where the case with periodic
boundary conditions is described). To exclude a mixing of vortex and edge
bound states in the lattice with open boundary conditions, the lattice size
should be increased. Therefore in this section the triangular lattice with
$N_{s}=99$ is considered, and the vortices are parameterized by using (10).

Considering the pair of vortices, Majorana vortex modes and zero-energy edge
modes can be separated from each other. Therefore, in the topological phase
with $\tilde{N}_{3}=1$, a part of zero modes at certain values of chemical
potential contain two Majorana modes localized at the vortex cores, while for
the other values of chemical potential the zero modes are near the edges. This
is in contrary with the case of the single vortex, when all the zero energy
solutions in the $\tilde{N}_{3}=1$ phase contain simultaneously one Majorana
mode inhabiting at the vortex core and the other Majorana mode localized in
the vicinity of the edges.

Next, we are focused on the Majorana vortex modes instead of the edge modes
and demonstrate their experimentally accessible energy resolution from the
other bound states. The result for the excitation spectrum in the presence of
the pair of vortices at $\mu\approx-0.389$ is presented in Fig. 8. As before,
the markers at zero energy are for Majorana modes, the large points are vortex
bound states, the small points correspond to energies of edge states.

As it is known, an increase of the lattice size increases a number of subgap
states. Through comparing Fig. 8 and Fig. 5, we can see this result. It is
also seen that the difference between second and first excitation energies is
decreased comparing to the case of Fig. 5. However, the second excitation
energy corresponds to the edge bound states, while the first excitation with
zero energy matches to the formation of spatially separated Majorana modes
localized on the different vortex cores. The distributions of modes
$b^{\prime}$ and $b^{\prime\prime}$ are demonstrated in Fig. 9. The
probability density near the edges is negligible. For the pair of vortices the
energy difference between Majorana vortex modes and the other vortex bound
states as well as LDOS behavior remain qualitatively the same as in the case
of the single vortex shown in Fig. 6(a).

Figure 9:  Majorana modes $b^{\prime}$ (a) and $b^{\prime\prime}$ (b)
localized on the pair of vortices. Black color corresponds to near zero
values, the brightest color is for maximum values $\approx 0.125$. The
parameters are the same as in Fig. 8.

When the zero modes are localized only near the edges at certain parameters,
the second excitation energy corresponds to the formation of vortex bound
states. Further increase in the lattice size causes the second excitation
energy to shift to near zero energy.

For the vortex-antivortex pair the Majorana modes localized at the cores also
exist and the excitation energy spectrum with LDOS are qualitatively the same
as for the pair of vortices. Although comparing with the case of the vortex
center, the tiny features of $\rho\left({\bf R}_{f},\omega\right)$ near the
antivortex center described in Sec. IV.2 are preserved for the vortex-
antivortex pair.

Figure 10:  PrD1 for the first excitation with zero energy (a) and PrD2 for
the second excitation energy ($\varepsilon_{2}\approx 2.7\,10^{-4}$). (b) in
the presence of four vortices. Majorana modes $b^{\prime}$ and
$b^{\prime\prime}$ in (a) are localized on different vortex cores.

###  IV.4  Four vortices on the lattice

In this section four vortices are described with ${\bf R}_{v_{i=1\dots
4}}=\left(\frac{N_{s}}{3},\frac{N_{s}}{3}\right)$,
$\left(\frac{2N_{s}}{3}+1,\frac{2N_{s}}{3}+1\right)$,
$\left(\frac{N_{s}}{3},\frac{2N_{s}}{3}+1\right)$,
$\left(\frac{2N_{s}}{3}+1,\frac{N_{s}}{3}\right)$ on the triangular lattice
with $N_{s}=99$. The zero energy solutions exist on the same interval of
chemical potential as in the previous sections. The main difference from the
previous results is that vortex bound states appear also for the second
excitation nearby with zero-energy Majorana vortex modes. However, a pair of
vortices are distinguished to form Majorana modes. This result is demonstrated
in Fig. 10a where PrD1 (14) of the first excitation for $\mu=-0.5902$ is
shown.

Majorana modes $b^{\prime}$ and $b^{\prime\prime}$ with zero energy are
localized on different cores of the pair of vortices in Fig. 10a. Further,
vortex bound states corresponding to the second excitation energy (as before
for the pair of vortices, $\varepsilon_{2}\sim 10^{-4}$) are formed on the
another pair of vortices (PrD2 for the second excitation is presented in Fig.
10b). The energy difference between such low-energy vortex bound states and
higher-energy vortex bound states is still $\approx 0.05$ as in the case of
the single vortex or the pair of vortices, although a number of nearby high-
energy excitations corresponding to vortex bound states grows. This result
confirms that Majorana vortex modes in the presence of several pairs of
vortices on the lattice locally are well separated on energy scale to be
experimentally detected by methods dealing with LDOS.

Note that there are also zero energy solutions corresponding to modes
localized along the edges in the four-vortices case. In this case the second
and third excitations involve vortex bound states. These bound states are
distributed over all four vortex cores or over three vortex cores in contrast
to the localization on two cores presented in Fig. 10.

##  V  Summary

In the present article it is shown that there appear many subgap states in
Shubnikov phase of 2D topological superconducting materials with coexistence
of superconductivity and noncollinear spin ordering. The majority of them are
edge bound states, while the rest are vortex bound states (like Caroli-de
Gennes-Matricon states). The formation of Majorana vortex modes in the
proposed system is also proved. In general, finite-energy vortex bound states
can have nearby excitation energies forming a cluster, and the number of such
energies corresponds to the number of vortices. In turn, the energy difference
between different clusters of vortex bound states in the excitation spectrum
is an order of magnitude higher then the difference between the adjacent
energies. Therefore, zero energy Majorana modes localized on the single vortex
core or the cores of the pair of vortices can be distinguished from other
higher energy vortex bound states in experiments. This can be done by tools
measuring local density of states near the vortex core, such as scanning
tunneling microscopy. For a number of vortices more than two on the considered
limited triangular lattice, vortex bound states appear not only with zero
energy, but also with the excitation energy nearby to zero. Nevertheless, the
conditions can be found when the localization of Majorana modes and
localization of nearby vortex bound states occur on different vortex cores.
Therefore they still can be separated from each other. The use of such modes
in braiding operations remains the open question. It is supposed that the
presented results remain applicable for larger vortex cores when the scale of
the core and the lattice size is preserved.

###### Acknowledgements.

I am grateful for fruitful discussions with participants of the seminar of
Laboratory of Theoretical Physics in Kirensky Institute of Physics. The
reported study was funded by the Theoretical Physics and Mathematics
Advancement Foundation “BASIS” and partly by Russian Foundation for Basic
Research, Government of Krasnoyarsk Territory, and Krasnoyarsk Regional Fund
of Science (project No. 20-42-243001).

## References

  * Lutchyn, Sau, and Das Sarma (2010) R. M. Lutchyn, J. D. Sau, and S. Das Sarma, Phys. Rev. Lett. 105, 077001 (2010).
  * Oreg, Refael, and von Oppen (2010) Y. Oreg, G. Refael, and F. von Oppen, Phys. Rev. Lett. 105, 177002 (2010).
  * Shen  _et al._ (2021) J. Shen, G. W. Winkler, F. Borsoi, S. Heedt, V. Levajac, J.-Y. Wang, D. van Driel, D. Bouman, S. Gazibegovic, R. L. M. Op Het Veld, D. Car, J. A. Logan, M. Pendharkar, C. J. Palmstrøm, E. P. A. M. Bakkers, L. P. Kouwenhoven, and B. van Heck, Phys. Rev. B 104, 045422 (2021).
  * Val’kov  _et al._ (2022) V. V. Val’kov, M. S. Shustin, S. V. Aksenov, A. O. Zlotnikov, A. D. Fedoseev, V. A. Mitskan, and M. Y. Kagan, Physics-Uspekhi 65, 2 (2022).
  * Nayak  _et al._ (2008) C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma, Reviews of Modern Physics 80, 1083 (2008).
  * Read and Green (2000) N. Read and D. Green, Phys. Rev. B 61, 10267 (2000).
  * Elliott and Franz (2015) S. R. Elliott and M. Franz, Rev. Mod. Phys. 87, 137 (2015).
  * Yang  _et al._ (2016) G. Yang, P. Stano, J. Klinovaja, and D. Loss, Phys. Rev. B 93, 224505 (2016).
  * Zlotnikov, Shustin, and Fedoseev (2021) A. O. Zlotnikov, M. S. Shustin, and A. D. Fedoseev, J Supercond Nov Magn 34, 3053 (2021).
  * Volovik (2010) G. E. Volovik, JETP Lett. 91, 201 (2010).
  * Zhu (2018) X. Zhu, Phys. Rev. B 97, 205134 (2018).
  * Kraus  _et al._ (2009) Y. E. Kraus, A. Auerbach, H. A. Fertig, and S. H. Simon, Phys. Rev. B 79, 134515 (2009).
  * Silaev (2013) M. A. Silaev, Phys. Rev. B 88, 064514 (2013).
  * Akzyanov  _et al._ (2016) R. S. Akzyanov, A. L. Rakhmanov, A. V. Rozhkov, and F. Nori, Phys. Rev. B 94, 125428 (2016).
  * Iskin (2012) M. Iskin, Phys. Rev. A 85, 013622 (2012).
  * Björnson and Black-Schaffer (2013) K. Björnson and A. M. Black-Schaffer, Phys. Rev. B 88, 024501 (2013).
  * Chiu, Gilbert, and Hughes (2011) C.-K. Chiu, M. J. Gilbert, and T. L. Hughes, Phys. Rev. B 84, 144507 (2011).
  * Akzyanov  _et al._ (2014) R. S. Akzyanov, A. V. Rozhkov, A. L. Rakhmanov, and F. Nori, Phys. Rev. B 89, 085409 (2014).
  * Pathak, Plugge, and Franz (2021) V. Pathak, S. Plugge, and M. Franz, Annals of Physics 435, 168431 (2021).
  * Yan, Wu, and Huang (2020) Z. Yan, Z. Wu, and W. Huang, Phys. Rev. Lett. 124, 257001 (2020).
  * Steffensen, Andersen, and Kotetes (2021) D. Steffensen, B. M. Andersen, and P. Kotetes, Phys. Rev. B 104, 174502 (2021).
  * Sato and Fujimoto (2010) M. Sato and S. Fujimoto, Phys. Rev. Lett. 105, 217001 (2010).
  * Zhou and Wang (2008) S. Zhou and Z. Wang, Phys. Rev. Lett. 100, 217002 (2008).
  * Val’kov, Val’kova, and Mitskan (2017) V. V. Val’kov, T. A. Val’kova, and V. A. Mitskan, Journal of Magnetism and Magnetic Materials 440, 129 (2017).
  * Baskaran (2003) G. Baskaran, Phys. Rev. Lett. 91, 097003 (2003).
  * Black-Schaffer, Wu, and Le Hur (2014) A. M. Black-Schaffer, W. Wu, and K. Le Hur, Phys. Rev. B 90, 054521 (2014).
  * Kagan, Mitskan, and Korovushkin (2015) M. Y. Kagan, V. A. Mitskan, and M. M. Korovushkin, Phys.-Usp. 58, 733 (2015).
  * Volovik (1997) G. E. Volovik, JETP Lett. 66, 522 (1997).
  * Lee and Schnyder (2016) D. Lee and A. P. Schnyder, Phys. Rev. B 93, 064522 (2016).
  * Choy  _et al._ (2011) T.-P. Choy, J. M. Edge, A. R. Akhmerov, and C. W. J. Beenakker, Phys. Rev. B 84, 195442 (2011).
  * Nadj-Perge  _et al._ (2013) S. Nadj-Perge, I. K. Drozdov, B. A. Bernevig, and A. Yazdani, Phys. Rev. B 88, 020407 (2013).
  * Martin and Morpurgo (2012) I. Martin and A. F. Morpurgo, Phys. Rev. B 85, 144505 (2012).
  * Lu and Wang (2013) Y.-M. Lu and Z. Wang, Phys. Rev. Lett. 110, 096403 (2013).
  * Val’kov, Zlotnikov, and Shustin (2018) V. V. Val’kov, A. O. Zlotnikov, and M. S. Shustin, Journal of Magnetism and Magnetic Materials 459, 112 (2018).
  * Bedow  _et al._ (2020) J. Bedow, E. Mascot, T. Posske, G. S. Uhrig, R. Wiesendanger, S. Rachel, and D. K. Morr, Phys. Rev. B 102, 180504 (2020).
  * Rex, Gornyi, and Mirlin (2020) S. Rex, I. V. Gornyi, and A. D. Mirlin, Phys. Rev. B 102, 224501 (2020).
  * Kim  _et al._ (2018) H. Kim, A. Palacio-Morales, T. Posske, L. Rózsa, K. Palotás, L. Szunyogh, M. Thorwart, and R. Wiesendanger, Sci. Adv. 4, eaar5251 (2018).
  * Palacio-Morales  _et al._ (2019) A. Palacio-Morales, E. Mascot, S. Cocklin, H. Kim, S. Rachel, D. K. Morr, and R. Wiesendanger, Sci. Adv. 5, eaav6600 (2019).
  * Spethmann  _et al._ (2020) J. Spethmann, S. Meyer, K. von Bergmann, R. Wiesendanger, S. Heinze, and A. Kubetzka, Phys. Rev. Lett. 124, 227203 (2020).
  * Wang  _et al._ (2018) D. Wang, L. Kong, P. Fan, H. Chen, S. Zhu, W. Liu, L. Cao, Y. Sun, S. Du, J. Schneeloch, R. Zhong, G. Gu, L. Fu, H. Ding, and H.-J. Gao, Science 362, 333 (2018).
  * Ishikawa and Matsuyama (1987) K. Ishikawa and T. Matsuyama, Nuclear Physics B 280, 523 (1987).
  * Volovik (2009) G. E. Volovik,  _The Universe in a Helium Droplet_ (Oxford University Press, 2009).
  * Val’kov and Zlotnikov (2019) V. V. Val’kov and A. O. Zlotnikov, JETP Lett. 109, 736 (2019).
  * Kitaev (2001) A. Y. Kitaev, Phys.-Usp. 44, 131 (2001).
  * Nijholt and Akhmerov (2016) B. Nijholt and A. R. Akhmerov, Phys. Rev. B 93, 235434 (2016).