# Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream
NLP Tasks

Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon  Samsung SDS, Seoul,
Korea

###### Abstract

Contextualized representations from a pre-trained language model are central
to achieve a high performance on downstream NLP task. The pre-trained BERT and
A Lite BERT (ALBERT) models can be fine-tuned to give state-of-the-art results
in sentence-pair regressions such as semantic textual similarity (STS) and
natural language inference (NLI). Although BERT-based models yield the [CLS]
token vector as a reasonable sentence embedding, the search for an optimal
sentence embedding scheme remains an active research area in computational
linguistics. This paper explores on sentence embedding models for BERT and
ALBERT. In particular, we take a modified BERT network with siamese and
triplet network structures called Sentence-BERT (SBERT) and replace BERT with
ALBERT to create Sentence-ALBERT (SALBERT). We also experiment with an outer
CNN sentence-embedding network for SBERT and SALBERT. We evaluate performances
of all sentence-embedding models considered using the STS and NLI datasets.
The empirical results indicate that our CNN architecture improves ALBERT
models substantially more than BERT models for STS benchmark. Despite
significantly fewer model parameters, ALBERT sentence embedding is highly
competitive to BERT in downstream NLP evaluations.

##  I Introduction

Pre-trained language models have impacted the way modern natural language
processing (NLP) applications and systems are built. An important paradigm is
to train a language model on large corpora to serve as a platform upon which
an NLP application can be built and optimized. Such platform is shareable and
can be distributed. Self-supervised learning with large corpora provides an
appropriate starting point for extra task-specific layers being optimized from
scratch while reusing the pre-trained model parameters.

Transformer [1], a sequence transduction model based on attention mechanism,
has revolutionized the design of a neural encoder for natural language
sequences. By skipping any recurrent or convolutional structures, the
transformer architecture enables the learning of sequential information in an
input solely via attention, thanks to multihead self-attention layers in an
encoder block. Devlin  _et al._ [2] have proposed Bidirectional Encoder
Representations from Transformers (BERT) to improve on predominantly
unidirectional training of language models.

By jointly conditioning on both left and right context in all layers, BERT
uses the masked language modeling (MLM) loss to make the training of deep
bidirectional language encoding possible. BERT uses an additional loss for
pre-training known as next-sentence prediction (NSP). NSP is designed to learn
high-level linguistic coherence by predicting whether or not given two text
segments should appear consecutively as in the original text. NSP is expected
to improve downstream NLP task performances such as semantic textual
similarity (STS) and natural language inference (NLI) that need to infer
reasoning about inter-sentence relations.

A Lite BERT [3] is proposed to scale up the language representation learning
via parameter reduction techniques. In ALBERT, cross-layer parameter sharing
and factorization of embedding parameters can be thought as a regularization
that helps stabilize its training. Furthermore, ALBERT uses an updated self-
supervised loss known as sentence-order prediction (SOP) that enhances the
ineffectiveness of NSP confused between topic and coherence predictions. SOP
has been shown to consistently help downstream tasks with multi-sentence
inputs.

The pre-training tasks are intrinsic compared to downstream tasks. A key
disadvantage of BERT is that no independent sentence embeddings are computed.
As a higher means of abstraction, sentence embeddings can play a central role
to achieve good downstream performances like machine reading comprehension
(MRC).

The specifics of NLP applications are well-abstracted by downstream tasks. For
this reason, downstream performance is a good indicator for a language model.
When pre-trained language models are used for downstream task evaluations,
pre-trained models can generate additional feature representations in addition
to being provided as a platform for fine-tuning.

In this paper, we are interested in learning sentence representation using
out-of-the-box BERT and ALBERT token embeddings. Sentence embedding models are
essential for clustering and semantic search where a sentence input is mapped
in a high-dimensional semantic vector space such that sentence vectors with
similar meanings are close in distance. NLP researchers have started to input
an individual sentence into BERT to derive a fixed-size embedding. A commonly
accepted sentence embedding for BERT-based models is the [CLS] token used for
sentence-order prediction (_i.e._ , NSP or SOP) during the pre-training.

Averaging the representations obtained from the BERT or ALBERT output layer
(_i.e._ , token embeddings) gives an alternative. Using the [CLS] token, which
is optimized by an intrinsic task of the pre-training, is considered
suboptimal while the average pooling of token embeddings has a limitation of
its own. Nonetheless, it can be time consuming to perform multi-sentence tasks
associated with semantic search, summarization, and paraphrase.

Computing sentence embeddings from contextualized language models is an
active, ongoing research problem. In our exploration for more elaborate
sentence embedding models, we first consider Sentence-BERT (SBERT) [4], a
modified BERT network with siamese and triplet network structures to derive
semantically meaningful sentence embeddings. SBERT is computationally
efficient and can compare sentences using only cosine-similarity at run-time.
We then take the SBERT architecture and simply replace BERT with ALBERT to
form Sentence-ALBERT (SALBERT). We also apply a convolutional neural net (CNN)
instead of average pooling that takes in the BERT or ALBERT token embedding
outputs.

We have evaluated the empirical performance of all sentence embedding models
by using the STS and NLI datasets. We find that our CNN architecture improves
ALBERT models up to 8 points in Spearman’s rank correlation for STS benchmark,
which is substantially larger than the case for BERT models with an
improvement of only 1 point. Despite significantly fewer model parameters,
ALBERT sentence embedding is highly competitive to BERT in downstream NLP
evaluations.

This paper is structured in the following manner. Section II presents related
work. Section III describes all sentence embedding models of our
consideration. In Section IV, we empirically evaluate the sentence embedding
models using the STS and NLI datasets. The paper concludes in Section V.

##  II Related Work

Language models provide core building blocks for downstream NLP tasks. Task-
specific fine-tuning of a pre-trained language model is a contemporary
approach to implement an NLP system. BERT [2] is a pre-trained transformer
encoder network [1] fine-tuned to give state-of-the-art results in question
answering, sentence classification, and sentence-pair regression. A Lite BERT
(ALBERT) [3] incorporates parameter reduction techniques to scale better than
BERT. ALBERT is known to improve on inter-sentence coherence by a self-
supervised loss from sentence-order prediction (SOP) compared to the next
sentence prediction (NSP) loss in the original BERT.

The BERT network structure contains a special classification token [CLS] as an
aggregate sequence representation for NSP. (Similarly for ALBERT, [CLS] is
used for SOP.) The [CLS] token therefore can serve a sentence embedding.
Because there are no other independently computed sentence embeddings for BERT
and ALBERT, one can average-pool the token embedding outputs to form a fixed-
length sentence vector.

Previously, sentence embedding research looked over convolutional and
recurrent structures as building blocks. Kim [5] proposed a CNN with max
pooling for sentence classification. In Conneau _et al._ [6], bidirectional
LSTM (BiLSTM) was used as sentence embedding for natural language inference
tasks. More complex neural nets such as Socher  _et al._ [7] introduced
recursive neural tensor network (RNTN) over parse trees to compute sentence
embedding for sentiment analysis. Zhu  _et al._ [8] and Tai  _et al._ [9]
proposed tree-LSTM while Yu & Munkhdalai [10] suggested neural semantic
encoder (NSE) based on memory augmented neural net.

Recently, sentence embedding research is exploring attention mechanisms.
Vaswani  _et al._ [1] have proposed Transformer, a self-attention network for
the neural sequence-to-sequence task. A self-attention network uses multi-head
scaled dot-product attention to represent each word as a weighted sum of all
words in the sentence. The idea of self-attention pooling has existed before
self-attention network as in Liu  _et al._ [11] that have utilized inner-
attention within a sentence to apply pooling for sentence embedding. Choi  _et
al._ [12] have developed a fine-grained attention mechanism for neural machine
translation, extending scalar attention to vectors.

Complex contextualized sentence encoders are usually pre-trained like language
models, but they can be improved by supervised transfer tasks such as natural
language inference (NLI). InferSent by Conneau  _et al._ [6] has consistently
outperformed unsupervised methods like SkipThought. Universal Sentence Encoder
[13] trains a transformer network and augments unsupervised learning with
training on the Stanford NLI (SNLI) dataset. Hill  _et al._ [14] show that the
task on which sentence embeddings are trained significantly impacts their
quality. According to Conneau  _et al._ [6] and Cer  _et al._ [13], the SNLI
datasets are suitable for training sentence embeddings. Yang  _et al._ [15]
present a method to train siamese deep averaging network (DAN) and
transformer, using conversations from Reddit to yield good results on the STS
benchmark.

In Sentence-BERT (SBERT) [4], a comprehensive evaluation on the pre-trained
BERT combined with siamese and triplet network structures is presented. To
alleviate the run-time overhead, SBERT’s more elaborate fine-tuning mechanisms
such as softmax on augmented sentence representations and triplet loss are
replaced by the cosine similarity at inference. The simplistic SBERT inference
helps reduce the effort for finding the most similar pair from 65 hours with
BERT to about 5 seconds, while hardly impacting the accuracy.

##  III Models

The output of BERT or ALBERT constitutes token embeddings for a given text
input. With a large output size (_e.g._ , up to 512 token vectors of 768
dimensions each), the contextualized word embeddings can be fine-tuned for any
downstream task. To do sentence-level regressions such as semantic textual
similarity (STS), fixed-size sentence embeddings would be necessary. In this
section, we describe sentence embedding models for BERT and ALBERT.

###  III-A The [CLS] token embedding

The most straightforward sentence embedding model is the [CLS] vector used to
predict sentence-level context (_i.e._ , BERT NSP, ALBERT SOP) during the pre-
training. The [CLS] token summarizes the information from other tokens via a
self-attention mechanism that facilitates the intrinsic tasks of the pre-
training. A similar reasoning applies such that the [CLS] token can be further
optimized while fine-tuning the downstream task. After the fine-tuning, the
[CLS] token is expected to capture more semantically-relevant sentence-level
context specific to the downstream task.

###  III-B Pooled token embeddings

Averaging the token embedding output gives our next model. The model works
like a pooling layer in a convolutional neural net. Average pooling turns the
token embeddings into a fixed-length sentence vector. An alternative would use
max pooling instead, although max pooling tends to select the most important
features rather than taking representative summary. In this paper, we choose
to go with the average-pooling model.

###  III-C Sentence-BERT (SBERT)

Reimers & Gurevych [4] propose SBERT that modifies a pre-trained BERT with
siamese and triplet network structures to derive semantically meaningful
sentence embeddings comparable using only cosine similarity. The siamese
architecture is computationally efficient. Note that using a single copy of
pre-trained BERT would require to run all possible combinations of sentence
pairs from a dataset to form a representation for sentence pairs. SBERT first
average-pools a pair of the BERT embeddings to fixed-size sentence embeddings.
Using the two sentence embeddings and an element-wise difference between them,
SBERT can run a softmax layer configured for classification and regression
tasks.

###  III-D Sentence-ALBERT (SALBERT)

Based on ALBERT, SALBERT has the same siamese and triplet networks as SBERT.
The siamese network structure in SBERT and SALBERT is illustrated in Fig. 1.

Figure 1: Siamese network structure used in SBERT and SALBERT

###  III-E CNN-SBERT

In SBERT, average pooling is used to make the BERT embeddings into fixed-
length sentence vectors. CNN-SBERT instead employs a CNN architecture that
takes in the token embeddings and computes a fixed-size sentence embedding
through convolutional layers with the hyperbolic tangent activation function
interlaced with pooling layers. In CNN-SBERT, all the pooling layers use max
pooling except the final average pooling. The CNN architecture used in CNN-
SBERT is described in Fig. 2.

###  III-F CNN-SALBERT

Similarly, CNN-SALBERT uses the same CNN architecture used in CNN-SBERT.

Figure 2: CNN architecture used in CNN-SBERT and CNN-SALBERT. B, T, and H
means mini-batch size, number of tokens, and transformer hidden size.

##  IV Experiments

We evaluate the performance of the sentence embedding models on Semantic
Textual Similarity (STS) and Natural Language Inference (NLI) benchmarks.
Following the methodology by Reimers & Gurevych [4], we use cosine-similarity
as a main metric to evaluate the similarity between two sentence embeddings.
We compute both Pearson and Spearman’s rank coefficients to indicate how our
cosine-similarity estimate and a ground-truth label provided by the datasets
are correlated. We use pre-trained BERT and ALBERT models from Hugging Face
[16]111https://github.com/huggingface.

###  IV-A Datasets and tasks

We fine-tune the BERT and ALBERT sentence embedding models on the Semantic
Textual Similarity benchmark (STSb) [17], the Multi-Genre Natural Language
Inference (MultiNLI) [18], and the Stanford Natural Language Inference (SNLI)
[19] datasets.

####  IV-A1 Semantic Textual Similarity benchmark

STSb gives a set of English data used for STS tasks organized in International
Workshop on Semantic Evaluation (SemEval) [20] between 2012 and 2017. The
dataset includes 8,628 sentence pairs from image captions, news headlines, and
user forums that are partitioned in train (5,749), dev (1,500) and test
(1,379) sets. They are annotated with a score from 0 to 5 indicating how
similar a pair of sentences are in terms of semantic relatedness.

####  IV-A2 Multi-genre Natural Language Inference

The MultiNLI corpus [18] is a crowd-sourced collection of 433k sentence pairs
annotated with textual entailment information. The dataset is used to evaluate
entailment classification task. MultiNLI is modeled on the SNLI corpus,
differing in its coverage of genres of spoken and written text. MultiNLI
supports a distinctive cross-genre generalization evaluation. Each sentence
pair in MultiNLI has a label that distinguishes whether the two sentences are
contradiction, entailment, or neutral.

####  IV-A3 Stanford Natural Language Inference

The SNLI corpus [19] contains 570k human-written English sentence pairs
manually labeled for balanced classification with the labels entailment,
contradiction, and neutral for natural language inference (NLI), also known as
recognizing textual entailment (RTE). The General Language Understanding
Evaluation (GLUE) benchmark [21] recommends the SNLI dataset used as an
auxiliary training data for MultiNLI task. Conneau  _et al._ [6] and Cer  _et
al._ [13] find SNLI suitable for training sentence embeddings for asserting
reasoning about the semantic relationship within sentences.

###  IV-B Training

In our evaluation, we consider only BERT and ALBERT base models (_i.e._ ,
multi-head attention over 12 layers) in the transformer package downloaded
from Hugging Face [16]. We use GLUE benchmark to fine-tune the [CLS] token
embedding and average-pooled token embedding models with a learning rate of
$3\times 10^{-5}$. We train all of our models using the Adam optimizer with a
linear learning rate warm-up for 10% of the training data. We use a learning
rate of $2\times 10^{-5}$ for SBERT and SALBERT as suggested by the original
SBERT architecture and $1\times 10^{-5}$ for CNN-SBERT and CNN-SALBERT. Using
the MultiNLI and SNLI data, we optimize SBERT and SALBERT on the 3-way softmax
loss.

####  IV-B1 STSb

To train STS benchmark task, we use siamese network as shown in Fig 1. We run
10 training epochs with a batch size of 32.

####  IV-B2 NLI (MultiNLI + SNLI)

To train NLI tasks, we adopt the siamese architecture in Fig 1. We use a
softmax classifier instead of cosine similarity in training NLI tasks with a
cross-entropy loss. We train 1 epoch because the NLI train set is much bigger
than STSb. We use a batch size of 16.

####  IV-B3 NLI + STSb

After fine-tuning on the NLI dataset, we train on the STS benchmark with a
batch size of 32.

TABLE I: Evaluation on the STSb by fine-tuning sentence embeddings on STS, NLI, and both Model | Spearman (Pearson) |
---|---|---
Not fine-tuned |
BERT [CLS]-token embedding | 6.43 (1.70) |
BERT Avg. pooled token embedding | 47.29 (47.91) |
ALBERT [CLS]-token embedding | 0.86 (4.57) |
ALBERT Avg. pooled token embedding | 47.84 (46.57) |
Fine-tuned on STSb |
BERT [CLS]-token embedding | 12.96 (7.49) |
BERT Avg. pooled token embedding | 55.76 (54.90) |
SBERT | 84.66 (84.86) |
CNN-SBERT | 85.72 (86.15) |
ALBERT [CLS]-token embedding | 37.98 (27.89) |
ALBERT Avg. pooled token embedding | 61.06 (60.41) |
SALBERT | 74.33 (75.26) |
CNN-SALBERT | 82.30 (83.08) |
Fine-tuned on NLI (MultiNLI + SNLI) |
BERT [CLS]-token embedding | 32.72 (26.88) |
BERT Avg. pooled token embedding | 69.57 (68.49) |
SBERT | 77.22 (74.53) |
CNN-SBERT | 76.77 (75.31) |
ALBERT [CLS]-token embedding | 24.87 (4.11) |
ALBERT Avg. pooled token embedding | 54.21 (53.58) |
SALBERT | 74.05 (70.78) |
CNN-SALBERT | 73.70 (72.24) |
Fine-tuned on NLI (MultiNLI + SNLI) and STSb |
BERT [CLS]-token embedding | 44.77 (38.74) |
BERT Avg. pooled token embedding | 67.61 (65.30) |
SBERT | 85.32 (84.51) |
CNN-SBERT | 85.91 (85.63) |
ALBERT [CLS]-token embedding | 40.35 (33.46) |
ALBERT Avg. pooled token embedding | 60.24 (59.98) |
SALBERT | 77.59 (77.82) |
CNN-SALBERT | 83.49 (83.87) |
TABLE II: Evaluation on the GLUE STSb task. Model | Spearman (Pearson)
---|---
BERT | 88.58 (88.89)
ALBERT | 90.13 (90.46)
TABLE III: Evaluation on various STS tasks. Numbers represent Spearman (Pearson). Model | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | Avg.
---|---|---|---|---|---|---|---
SBERT | 72.37 (77.61) | 87.49 (88.01) | 89.57 (89.87) | 89.76 (89.54) | 82.41 (80.59) | 85.32 (84.51) | 84.49 (85.02)
CNN-SBERT | 69.80 (75.04) | 88.92 (89.86) | 89.23 (90.53) | 89.35 (89.37) | 82.81 (82.03) | 85.91 (85.63) | 84.34 (85.41)
SALBERT | 63.87 (68.15) | 84.04 (84.59) | 84.89 (86.09) | 86.41 (86.31) | 75.26 (74.32) | 77.59 (77.82) | 78.68 (79.55)
CNN-SALBERT | 65.41 (68.58) | 86.76 (87.29) | 86.17 (87.66) | 87.84 (88.12) | 81.58 (81.48) | 83.49 (83.87) | 81.76 (82.70)

###  IV-C Results

####  IV-C1 Effect of fine-tuning

Table I presents the STS benchmark results. Note that performance we report is
$\rho\times 100$, where $\rho$ is Spearman’s rank or Pearson correlation
coefficient. In general, fine-tuning results in a better performance than no
fine-tuning. Without fine-tuning, the [CLS] token as a sentence embedding
gives poor downstream task performance. Quality of sentence embedding
reflected on the STSb performance seems to be affected by how related train
sets used for fine-tuning are to the task. We consider STSb train set, which
is directly related to the task of STSb. We also consider NLI (_i.e._ ,
MultiNLI and SNLI) train sets that are not directly related to STSb. We have
experimented with the following: i) fine-tuning with only STSb train set, ii)
with only NLI train sets, and iii) with both NLI and STSb train sets. Fine-
tuning with only STSb train set gives a reasonably good performance whereas
fine-tuning with irrelevant NLI train sets only have yielded a suboptimal
performance as expected. Our best STSb results are obtained by fine-tuning
with both STSb and NLI train sets.

####  IV-C2 Model comparison

We expect a more elaborate sentence embedding model to give a better
performance in STSb. We have found that pooling token embeddings will form a
better sentence representation than [CLS]. We have also found that siamese
structure further helps sentence embeddings. Generally, our CNN-based sentence
embedding models give the best performance among all sentence embedding
models.

####  IV-C3 Performance of ALBERT

ALBERT-based sentence embedding models generally achieve lower performance
than the BERT counterparts in STSb evaluations. Before fine-tuning, there is
no significant difference between ALBERT and BERT. The gap, however, increases
after fine-tuning. Only for [CLS] token embedding and average-pooled
embeddings, ALBERT has a better performance than BERT when fine-tuned on STSb.
SALBERT has much lower performance than SBERT even though they both have the
same siamese architecture. This is surprising because ALBERT has a higher
score than BERT when evaluated on STSb using GLUE as shown in Table II. The
performance of SALBERT catches up with SBERT when the CNN architecture
applies, but CNN-SALBERT is still slightly inferior to CNN-SBERT.

####  IV-C4 Effect of CNN

In Table I, we find that the best score is from CNN-based models trained on
NLI and STSb. According to these scores, the CNN architecture seems to have a
positive impact on sentence embedding performances. The CNN architecture,
however, improves the ALBERT-based sentence embedding models more than the
BERT-based models. We have found that the improvement by CNN to ALBERT models
can be as high as 8 points, which is compared to 1 point for the case of BERT
models. We have empirically observed that ALBERT exposes more instability (due
to parameter sharing) compared to BERT. Such instability can be alleviated by
CNN, and this is a possible explanation for more improvement on ALBERT by
adding CNN than BERT.

####  IV-C5 Evaluation of STS12–STS16 Tasks

In Table III, we present a comprehensive evaluation on the STS tasks from 2012
to 2016 [22, 23, 24, 25, 26] after fine-tuning with both the NLI and STSb
train sets. We show the results of our best two models (_i.e._ , SBERT/SALBERT
and CNN-SBERT/CNN-SALBERT). The STSb result is also presented for comparison.
The purpose of the evaluation is to verify the improvement by CNN beyond STSb.
In general, we see a similar trend that CNN architecture improves ALBERT-based
sentence embedding models substantially more than BERT-based. On the average,
SBERT embeddings achieve a Spearman’s rank correlation point of 84.49 while
the average for CNN-SBERT is 84.34. The CNN architecture seems almost no
effect on BERT-based sentence embedding models. On the other hand, the average
correlation score of CNN-SALBERT is improved by 2 points.

##  V Conclusion and Future Work

In this paper, we have presented an evaluation of BERT and ALBERT sentence
embedding models on Semantic Textual Similarity (STS). Knowing limitations of
the [CLS] sentence vector, we facilitate the STS sentence-pair regression task
with the siamese and triplet network architecture by Reimers & Gurevych for
BERT and ALBERT. We have additionally developed a CNN architecture that takes
in the token embeddings to compute a fixed-size sentence vector. Our CNN
architecture improves ALBERT models up to 0.08 (8 points in percentile) in
Spearman’s rank correlation for STS benchmark, which is substantially larger
than the case for BERT models with an improvement of only 0.01 (1 point).
Despite significantly fewer model parameters, ALBERT sentence embedding is
highly competitive to BERT in downstream NLP evaluations.

For our future work, we plan to evaluate sentence embedding with larger ALBERT
models— _i.e._ , ALBERT-large and ALBERT-xlarge. (Note that the total number
of parameters in ALBERT-xlarge is still fewer than that of BERT-base.) The
ALBERT results in this paper are obtained with the number of groups for the
hidden layers (num_hidden_groups) set to 1. We also plan to optimize the
num_hidden_groups hyperparameter for better performance.

## References

  * [1] Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia, “Attention is All you Need,” in _Advances in Neural Information Processing Systems 30_ , 2017.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A Lite BERT for Self-supervised Learning of Language Representations,” _arXiv preprint arXiv:1909.11942_ , 2019.
  * [4] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019.
  * [5] Y. Kim, “Convolutional neural networks for sentence classification,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * [6] A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “Supervised learning of universal sentence representations from natural language inference data,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017.
  * [7] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , 2013.
  * [8] X. Zhu, P. Sobhani, and H. Guo, “Long Short-Term Memory over Recursive Structures,” in _Proceedings of the 32nd International Conference on International Conference on Machine Learning_ , 2015.
  * [9] K. S. Tai, R. Socher, and C. D. Manning, “Improved semantic representations from tree-structured long short-term memory networks,” in _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2015.
  * [10] T. Munkhdalai and H. Yu, “Neural semantic encoders,” in _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_ , 2017.
  * [11] Y. Liu, C. Sun, L. Lin, and X. Wang, “Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention,” _CoRR_ , vol. abs/1605.09090, 2016.
  * [12] H. Choi, K. Cho, and Y. Bengio, “Fine-Grained Attention Mechanism for Neural Machine Translation,” _CoRR_ , vol. abs/1803.11407, 2018.
  * [13] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil, “Universal sentence encoder for English,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 2018.
  * [14] F. Hill, K. Cho, and A. Korhonen, “Learning distributed representations of sentences from unlabelled data,” in _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2016.
  * [15] Y. Yang, S. Yuan, D. Cer, S.-y. Kong, N. Constant, P. Pilar, H. Ge, Y.-H. Sung, B. Strope, and R. Kurzweil, “Learning semantic textual similarity from conversations,” in _Proceedings of The Third Workshop on Representation Learning for NLP_ , 2018.
  * [16] Hugging Face, “Open Source NLP,” https://huggingface.co.
  * [17] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation,” in _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , 2017.
  * [18] A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018.
  * [19] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015.
  * [20] Special Interest Group on the Lexicon of the Association for Computational Linguistics, “SemEval: International Workshop on Semantic Evaluation,” https://semeval.github.io/.
  * [21] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , 2018.
  * [22] E. Agirre, D. Cer, M. Diab, and A. Gonzalez-Agirre, “SemEval-2012 task 6: A pilot on semantic textual similarity,” in _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , 2012.
  * [23] E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo, “*SEM 2013 shared task: Semantic textual similarity,” in _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_ , 2013.
  * [24] E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe, “SemEval-2014 task 10: Multilingual semantic textual similarity,” in _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , 2014.
  * [25] E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea, G. Rigau, L. Uria, and J. Wiebe, “SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability,” in _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , 2015.
  * [26] E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe, “SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation,” in _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , 2016.


# Oscillations of Majorana neutrinos in supernova and CP violation

###### Abstract

Leptonic CP violation is one of the most important topics in neutrino physics.
CP violation in the neutrino sector is strongly related to the nature of
neutrinos: whether they are Dirac or Majorana particles. In [1] we have shown
that for Majorana neutrinos appearance of nonzero Majorana CP-violating phases
combined with strong magnetic field during supernova core-collapse can induce
new resonances in neutrino oscillations, for example in
$\nu_{e}\to\bar{\nu}_{\tau}$ channel. In this paper we further study new
resonances in Majorana neutrino oscillations, in particular energy dependence
of amplitudes of resonant oscillations. Our findings suggest a potential
astrophysical setup for studying the nature of neutrino masses and leptonic CP
violation and may be important for future neutrino experiments, such as JUNO,
Hyper-Kamiokande and DUNE.

a Department of Theoretical Physics,
Moscow State University, 119991 Moscow, Russia

PACS: 13.15.$+$g; 14.60.Pq; 14.60.St

## Introduction

Neutrino electromagnetic properties are closely tied with the nature of
neutrino mass. In particular, the matrix of neutrino magnetic moments has a
different form for the Dirac and Majorana cases [2], namely

$\mu^{D}=\begin{pmatrix}\mu_{11}&\mu_{12}&\mu_{13}\\\
\mu_{12}&\mu_{22}&\mu_{23}\\\
\mu_{13}&\mu_{23}&\mu_{33}\end{pmatrix},\;\;\;\mu^{M}=\begin{pmatrix}0&i\mu_{12}&i\mu_{13}\\\
-i\mu_{12}&0&i\mu_{23}\\\ -i\mu_{13}&-i\mu_{23}&0\end{pmatrix},$ (1)

where $\mu_{ik}$ are real numbers. Thus, studying evolution of neutrinos in a
magnetic field can be a potential way to distinguish between Dirac and
Majorana cases. Astrophysical objects possessing strong magnetic fields, such
as supernovae, are among most promising setups to probe neutrino
electromagnetic properties.

In this paper we study oscillations of Majorana neutrinos in supernova
magnetic field and matter. In particular, we focus on CP-violating effects.
Previously, CP-violation in oscillations of supernova neutrino was studied in
[3, 4]. However, in these papers effects of neutrino interaction with a
magnetic field were neglected.

##  1 Majorana neutrino oscillations in supernova media

It is well known that neutrino mixing matrix for the case of Majorana
neutrinos can be written as follows:

$U=U_{PMNS}(\theta_{12},\theta_{13},\theta_{23},\delta)\cdot\text{diag}(e^{i\alpha_{1}},e^{i\alpha_{2}},1),$
(2)

where $U_{PMNS}$ is the Pontecorvo–Maki–Nakagawa–Sakata matrix, $\theta_{ik}$
are mixing angles, $\delta$ is Dirac CP-violating phase, and $\alpha_{1}$ and
$\alpha_{2}$ are Majorana CP-violating phases that only present in case if
neutrinos are Majorana particles. In our paper [1] we have studied the process
of Majorana neutrino oscillations in environment peculiar for supernova
explosion, in particular strong magnetic field ($10^{12}$ Gauss and stronger)
and dense matter. It was shown that presence of nonzero Majorana CP-violating
phases can modify patterns of neutrino-antineutrino oscillations induced by
the supernova magnetic field.

Majorana neutrino interaction with a magnetic field is described by the
following Lagrangian:

$\mathcal{L}_{mag}^{M}=-\sum_{i,k}\mu_{ik}^{M}\left[\overline{(\nu_{i}^{L})^{c}}\bm{\Sigma}\bm{B}\nu_{k}^{L}+\overline{\nu_{i}^{L}}\bm{\Sigma}\bm{B}(\nu_{k}^{L})^{c}\right],$
(3)

where $\bm{B}$ is the magnetic field vector and

$\Sigma_{i}=\begin{pmatrix}\sigma_{i}&0\\\ 0&\sigma_{i}\end{pmatrix},$ (4)

where $\sigma_{i}$ are the Pauli matrices.

Majorana neutrino interaction with electrically neutral supernova matter is
described by the Lagrangian

$\mathcal{L}_{mat}^{M}=-\sum_{\alpha}V^{(f)}_{\alpha\alpha}\left[\overline{\nu^{L}_{\alpha}}\gamma_{0}\nu_{\alpha}^{L}-\overline{(\nu^{L}_{\alpha})^{c}}\gamma_{0}(\nu_{\alpha}^{L})^{c}\right],$
(5)

where

$V^{(f)}=\operatorname{diag}\left(\frac{G_{F}n_{e}}{\sqrt{2}}-\frac{G_{F}n_{n}}{2\sqrt{2}},-\frac{G_{F}n_{n}}{2\sqrt{2}},-\frac{G_{F}n_{n}}{2\sqrt{2}}\right)$
(6)

is the well-known Wolfenstein potential. Here $n_{e}$ and $n_{n}$ are electron
and neutron number densities of supernova media respectively. Note that due to
matter neutrality, proton number density $n_{p}$ equals to electron number
density $n_{e}$.

To study neutrino oscillations in supernova environment we numerically solve
the following equation

$(i\gamma^{\mu}\partial_{\mu}-m_{i}-V^{(m)}_{ii}\gamma^{0}\gamma_{5})\nu_{i}(x)-\sum_{k\neq
i}\big{(}\mu_{ik}^{M}\bm{\Sigma}\bm{B}+V^{(m)}_{ik}\gamma^{0}\gamma_{5}\big{)}\nu_{k}(x)=0,$
(7)

where $V^{(m)}=U^{{\dagger}}V^{(f)}U$ is the matter potential in the massive
neutrinos basis, and $n_{e}$, $n_{n}$ are electron and neutron number
densities respectively. The magnetic moments matrix of Majorana neutrinos
$\mu^{M}$ from Eq. (7) is antisymmetric and imaginary, and is given by Eq.
(1).

The values of neutrino magnetic moments are presently unknown. Currently, the
best upper bounds for transition magnetic moments of Majorana neutrinos are
given by [5]

$\displaystyle|\mu_{12}^{M}|$ $\displaystyle\leq$ $\displaystyle 0.64\times
10^{-11}\mu_{B},$ $\displaystyle|\mu_{13}^{M}|$ $\displaystyle\leq$
$\displaystyle 0.75\times 10^{-11}\mu_{B},\;\;\;\;\text{(90\% C.L.)}$ (8)
$\displaystyle|\mu_{23}^{M}|$ $\displaystyle\leq$ $\displaystyle 1.1\times
10^{-11}\mu_{B},$

where $\mu_{B}$ is the Bohr magneton.

Using Eq. (7), we numerically study the amplitudes of neutrino-antineutrino
oscillations. Here we take realistic values for supernova magnetic field
$B=10^{12}$ Gauss, matter density $n_{n}=10^{30}$ cm-3 and neutrino energy
$E_{\nu}=12$ MeV. For simplicity we also assume
$\mu_{12}=\mu_{13}=\mu_{23}=10^{-12}\mu_{B}$, which does not contradict the
current experimental results. We study the dependence of the neutrino
oscillations probabilities on various parameters, including the electron
fraction $Y_{e}=n_{e}/(n_{n}+n_{e})$, since it varies significantly along the
neutrino path, and the neutrino energy $E_{\nu}$. Figure 1a shows the maximal
probability of $\nu_{e}\to\bar{\nu}_{\mu}$ and $\nu_{e}\to\bar{\nu}_{\tau}$
oscillations as functions of electron fraction $Y_{e}$ in supernova media for
the case $\alpha_{1}=0$ and $\alpha_{2}=0$. The amplitude of
$\nu_{e}\to\bar{\nu}_{\tau}$ transition is close to zero, while
$\nu_{e}\to\bar{\nu}_{\mu}$ oscillations undergo resonant enhancement at
$Y_{e}=0.5$. This well known phenomenon of resonant spin-flavour conversion
was proposed for the first time in [6, 7]. In Figure 1b we show the amplitudes
of neutrino-antineutrino oscillations for the case of nonzero Majorana CP-
violating phases, namely $\alpha_{1}=\pi$ and $\alpha_{2}=0$. In this case
resonant enhancement in $\nu_{e}\to\bar{\nu}_{\tau}$ oscillations appears
instead of $\nu_{e}\to\bar{\nu}_{\mu}$ oscillations at $Y_{e}=0.5$. In [1] we
have shown that this new resonance can result in potentially observable
phenomena during supernova core-collapse, in particular to modification of
$\nu_{e}$ and $\bar{\nu}_{e}$ fluxes ratio.

Figure 1: Amplitudes of Majorana neutrino oscillations in supernova media for
$B=10^{12}$ Gauss as functions of electron fraction $Y_{e}$. (a)
$\alpha_{1}=0$, $\alpha_{2}=0$; (b) $\alpha_{1}=\pi$, $\alpha_{2}=0$.

Figure 2 shows amplitudes of resonant conversion $\nu_{e}\to\bar{\nu}_{\mu}$
(left) and $\nu_{e}\to\bar{\nu}_{\tau}$ (right) at $Y_{e}=0.5$ as functions of
both Majorana CP-violating phases $\alpha_{1}$ and $\alpha_{2}$. We find that
$\nu_{e}\to\bar{\nu}_{\mu}$ oscillations are present for CP-conserving values,
i.e. $\alpha_{1},\alpha_{2}=0,2\pi$, while $\nu_{e}\to\bar{\nu}_{\tau}$ arise
when $\alpha_{1}$ is close to $\pi$.

Figure 2: Amplitudes of resonant conversion $\nu_{e}\to\bar{\nu}_{\mu}$ (left)
and $\nu_{e}\to\bar{\nu}_{\tau}$ (right) at $Y_{e}=0.5$ as functions of
Majorana CP-violating phases $\alpha_{1}$ and $\alpha_{2}$.

Throughout [1] we assumed that neutrino energy is constant $E=10$ MeV, which
is considered to be a typical value for supernova neutrinos. It is also
important to study behaviour of new resonance at different neutrino energies.
Figure 3a and Figure 3b show the amplitudes of resonant
$\nu_{e}\to\bar{\nu}_{\mu}$ and $\nu_{e}\to\bar{\nu}_{\tau}$ conversions as
functions of neutrino energy $E$. We notice that in both cases resonant
enhancement is indeed observed for supernova neutrinos with energy of 1 MeV
and higher. However, for energies lower that 1 MeV resonant enhancement may
disappear.

Figure 3: Amplitudes of resonant conversion $\nu_{e}\to\bar{\nu}_{\tau}$ at
$Y_{e}=0.5$ for $B=10^{12}$ Gauss as functions of neutrino energy. (a)
$\alpha_{1}=0$, $\alpha_{2}=0$; (b) $\alpha_{1}=\pi$, $\alpha_{2}=0$.

##  2 Conclusion

In this paper we have studied the process of Majorana neutrino oscillations in
supernova media, in particular in a strong magnetic field. We have shown that
for certain nonzero values of Majorana CP-violating phases $\alpha_{1}$ and
$\alpha_{2}$, resonant enhancement of $\nu_{e}\to\bar{\nu}_{\tau}$
oscillations can occur in region of supernova media where electron fraction
$Y_{e}=0.5$. Note that $Y_{e}=0.5$ is expected to be realized during supernova
core-collapse (see [8]). We show that this resonant amplification is present
for realistic supernova neutrino energies, but it can disappear for energies
lower that $\sim$ 100 keV.

The appearance of the considered resonant amplification of neutrino
oscillations can lead to a modification of supernova neutrino burst flavour
composition. This will lead to a modification of expected event rates from the
Galactic supernova explosion in future neutrino experiments, such as JUNO [9],
Hyper-Kamiokande [10] and DUNE [11], in particular
$N(\bar{\nu}_{e})/N(\nu_{e})$ ratio. Thus, we expect that supernova neutrino
data can be used to probe neutrino magnetic moments, as well as to test the
hypothesis of Majorana nature of neutrino mass.

Note that accurate prediction of supernova event rates requires modelling of
neutrino fluxes using a specific model of supernova medium, in particular
density and magnetic field profiles, and is outside the scope of this paper.
In addition, since supernova neutrinos are emitted from the spherical surface
called the neutrinosphere, the total neutrino flux is averaged over the
emission angle (see for example [12]). Averaging over the emission angle will
lead to a smearing of the considered effects.

## Acknowledgements

The work is supported by the Russian Science Foundation under grant
No.22-22-00384. The work of A.P. has been supported by the Foundation for the
Advancement of Theoretical Physics and Mathematics “BASIS” under Grant No.
21-2-2-26-1 and by the National Center for Physics and Mathematics (Project
“Study of coherent elastic neutrino-atom and -nucleus scattering and neutrino
electromagnetic properties using a high-intensity tritium neutrino source”).

## References

  * [1] _Popov A., Studenikin A._ Manifestations of nonzero Majorana CP-violating phases in oscillations of supernova neutrinos // Phys. Rev. D —- 2021\. —- V. 103, no. 11. —- 115027 —- arXiv:2102.07991 [hep-ph].
  * [2] _Giunti C., Studenikin A._ Neutrino electromagnetic interactions: a window to new physics. Rev.Mod.Phys —- 2015\. —- V. 87, no. 531. —- arXiv:1403.6344 [hep-ph].
  * [3] _Balantekin B., Gava J., Volpe C._ Possible CP-Violation effects in core-collapse Supernovae // Phys.Lett.B —- 2008\. —- V. 662, no. 5. —- P.396-404 —- 0710.3112 [astro-ph].
  * [4] _Gava J., Volpe C._ Collective neutrinos oscillation in matter and CP-violation // Phys.Rev.D —- 2008\. —- V. 78, no. 8. —- 083007 —- 0807.3418 [astro-ph].
  * [5] _de Gouvêa A., Jusino Sánchez G., Machado P., Tabrizi Z._ Majorana versus Dirac Constraints on the Neutrino Dipole Moments //  arXiv:2209.03373 [hep-ph].
  * [6] _Akhmedov E.K._ Resonant Amplification of Neutrino Spin Rotation in Matter and the Solar Neutrino Problem // Phys.Lett.B —- 1988\. —- V. 213, no. 1. —- P. 64-68.
  * [7] _Lim C.-S., Marciano W.J._ Resonant Spin - Flavor Precession of Solar and Supernova Neutrinos // Phys. Rev. D —- 1988\. —- V. 37, no. 1368. —- P. 1368-1373.
  * [8] _Buras R., Rampp M., Janka H.-Th., Kifonidis K._ Two-dimensional hydrodynamic core-collapse supernova simulations with spectral neutrino transport. 1. Numerical method and results for a 15 solar mass star // Astron.Astrophys. —- 2006\. —- V. 447, no. 3. —- P. 1049 - 1092 —- astro-ph/0507135 [astro-ph].
  * [9] _Fengpeng A. et al. [JUNO Collaboration]_ Neutrino Physics with JUNO // J.Phys.G —- 2016\. —- V. 43, no. 3. —- 030401 —- 1507.05613 [physics.ins-det].
  * [10] _Abe K. et al. [Hyper-Kamiokande]_ Supernova Model Discrimination with Hyper-Kamiokande // Astrophys.J. —- 2021\. —- V. 916, no. 1. —- 15 —- 2101.05269 [astro-ph.IM].
  * [11] _Abi B. et al. [DUNE Collaboration]_ Supernova neutrino burst detection with the Deep Underground Neutrino Experiment // Eur.Phys.J.C —- 2021\. —- V. 81, no. 5. —- 423 —- 2008.06647 [hep-ex].
  * [12] _Cherry J.F., Carlson J., Friedland A., Fuller G.M., Vlasenko A._ Halo Modification of a Supernova Neutronization Neutrino Burst // Phys.Rev.D —- 2013\. —- V. 87, no. 8. —- 085037 —- 1302.1159 [astro-ph.HE].