# Multi-granular Legal Topic Classification on Greek Legislation

Christos Papaloukas${}^{\;\;\dagger}$ Ilias Chalkidis‡
Konstantinos Athinaios${}^{\;\dagger}$ Despina-Athanasia
Pantazi${}^{\;\dagger}$ Manolis Koubarakis${}^{\;\dagger}$
${}^{\dagger\;}$National and Kapodistrian University of Athens, Greece
[christospap, dpantazi<EMAIL_ADDRESS>
${}^{\ddagger\;}$University of Copenhagen, Denmark
<EMAIL_ADDRESS>

###### Abstract

In this work, we study the task of classifying legal texts written in the
Greek language. We introduce and make publicly available a novel dataset based
on Greek legislation, consisting of more than 47 thousand official,
categorized Greek legislation resources. We experiment with this dataset and
evaluate a battery of advanced methods and classifiers, ranging from
traditional machine learning and RNN-based methods to state-of-the-art
Transformer-based methods. We show that recurrent architectures with domain-
specific word embeddings offer improved overall performance while being
competitive even to transformer-based models. Finally, we show that cutting-
edge multilingual and monolingual transformer-based models brawl on the top of
the classifiers’ ranking, making us question the necessity of training
monolingual transfer learning models as a rule of thumb. To the best of our
knowledge, this is the first time the task of Greek legal text classification
is considered in an open research project, while also Greek is a language with
very limited NLP resources in general.

##  1 Introduction

In recent years, there has been intensified activity in the adaptation of
Artificial Intelligence technologies to the legal domain Chalkidis and Kampas
(2018); Zhong et al. (2020); Chalkidis et al. (2021), in which legal
practitioners are required to analyze and review an overwhelming amount of
legal data, mostly being plain text documents. This process requires
dedication and an extraordinary level of resources, both concerning human
resources along with the use of automated techniques to sift rationally
through data. However, more sophisticated automated techniques are able to
assist legal experts in making obsolete many labour-intensive manual tasks.
These techniques are mostly contained in the area of machine learning and
natural language processing (NLP).

With legal text processing being an emerging subarea of NLP, many relevant
applications have been derived such as legal topic classification Nallapati
and Manning (2008); Chalkidis et al. (2020a) legal information extraction
O’Neill et al. (2017); Chalkidis et al. (2018), legal entity recognition
Chalkidis et al. (2017a); Angelidis et al. (2018); Leitner et al. (2019),
court opinion generation and analysis Wang et al. (2012); Ye et al. (2018),
legal judgement prediction Aletras et al. (2016); Xiao et al. (2018);
Chalkidis et al. (2019a) and many more. However, current legal NLP studies are
mainly focused in English and Chinese, with very limited resources being
available in other languages.

Our work focuses on the task of _multi-class legal topic classification_ ,
where the goal is to identify the relevant thematic topic that represents a
document. In our case, thematic topics (categories) are available in a multi-
level hierarchy from broader to more specialized ones. The main contributions
of our work are listed below:

  * •

We introduce Greek Legal Code (GLC), a dataset consisting of approx. 47k legal
resources from Greek legislation. The origin of GLC is “Permanent Greek
Legislation Code - Raptarchis”, a collection of Greek legislative documents
classified into multi-level (from broader to more specialized) categories.

  * •

We study the task of multi-class legal topic classification for Greek
legislation by examining a battery of advanced methods, ranging from
traditional machine learning techniques and RNN-based methods, to state-of-
the-art Transformer-based methods. We discuss the results and lay the
groundwork for further research.

Considering that Greek is a language with few NLP resources, we anticipate
that our study will be a significant contribution for the Greek NLP community.
To enhance the available NLP resources and foster reproducible results, we
make both our code and dataset publicly available.

##  2 Related Work

Mencia and Fürnkranzand (2007) introduced a legal topic classification task
using a dataset obtained from the EUR-LEX111See https://eur-lex.europa.eu/.
database, which includes EU laws that have been tagged with EUROVOC concepts.
They used multiple binary Perceptrons, one for each label, and multi-label
pairwise Perceptrons on top of Bag-of-Words (BoW) representations. While the
methods seem primal and inefficient by today’s standards, the EURLEX dataset
is widely adopted as a notable benchmark in Large-scale Multi-label Text
Classification (LMTC) literature.

Nallapati and Manning (2008) were also among the first who investigated the
task of text classification in the legal domain where machine learning
classifiers such as SVMs were insufficient. They experimented with a dataset
of 5.5k US docket entries of court cases. The authors stress the importance of
feature selection in such specialized domains and expose the limitations of
classifiers relying on BoW featues to capture the intricacies of natural
language, widespread in specialized domains such as the legal one.

Undavia et al. (2018) applied neural networks on legal document classification
in a similar task, classification of legal court opinions. They used a dataset
of 8k US Supreme Court (SCOTUS) opinions, where they targeted two sub-tasks,
depending on the total output categories: 15 broad and 279 finer-grained
categories. They experimented with shallow neural networks using different
word embeddings, where their best model (word2vec + CNN) scored 72.4% accuracy
in the 15-classes task and 31.9% accuracy in the 279-classes task. They
concluded claiming that an RNN-based network together with domain-specific
word embeddings could possibly tackle the task with higher accuracy.

Chalkidis et al. (2019c) experimented with several classifiers on a novel
dataset of 57k legislative documents (EURLEX57k) from EUR-LEX in English. They
demonstrated that that BiGRUs with self attention outperform CNN-based methods
that employ the label-wise attention mechanism. Using domain-specific word
embeddings and context-sensitive ELMO Peters et al. (2018) embeddings improves
the overall performance. Furthermore, the authors experimented with BERT
Devlin et al. (2019) obtaining the best results

In a more recent and extended version of this study, Chalkidis et al. (2020b)
evaluated a battery of LMTC methods ranging from RNN-based Label-Wise
Attention Networks (LWANs) to Probabilistic Label Trees (PLTs) Prabhu et al.
(2018); Khandagale et al. (2019); You et al. (2019) and Transformer-based
models (BERT, ROBERTA) Devlin et al. (2019); Liu et al. (2019) on three
English datasets: EURLEX57k Chalkidis et al. (2019c), MIMIC-III Johnson et al.
(2016) and AMAZON13k Lewis et al. (2004). The experimental results show that
PLT-based methods outperform LWANs, while Transformer-based approaches surpass
state-of-the-art in two out of three datasets. Furthermore, a new state-of-
the-art method is introduced which combines BERT and LWAN, giving the best
results overall. Furthermore, the case of few and zero-shot learning is
studied with new models that leverage the label hierarchy and yield better
results.

Following literature, we examine traditional machine learning methods and the
RNN-based methods used in Chalkidis et al. (2019c) and Chalkidis et al.
(2020b) and investigate whether these methods can perform equally well in
multi-class text classification as they do in the multi-label setting.
Furthermore, we examine several BERT-based methods, including multi-lingual
models Devlin et al. (2019); Conneau et al. (2020) that have not been studied
to date in the context of legal NLP.

Most of the preceding efforts focus on the English language and limited
studies and resources exist considering languages other than English and
Chinese. While recently there is a new wave of studies on NLP tasks focused on
the Greek language Athanasiou and Maragoudakis (2017); Papantoniou and
Tzitzikas (2020); Pitenis et al. (2020); Koutsikakis et al. (2020), there is
limited work for legal NLP in Greek Angelidis et al. (2018). To the best of
our knowledge, this is the first study on the task of Greek legal text
classification, where experiments range from traditional machine learning to
transfer learning models.

##  3 GLC Dataset

###  3.1 Original Data

The “Permanent Greek Legislation Code - Raptarchis222Mr. P. Raptarchis is the
original curator of this collection.” is a thorough catalogue of Greek
legislation since the creation of the Greek state in 1834 until 2015. It
includes Laws, Royal and Presidential Decrees, Regulations and Decisions,
retrieved from the Official Government Gazette, where Greek legislation is
published. This collection is one of the official, publicly available sources
of classified Greek legislation suitable for our classification
task.333Another official source is that of European Legislation written in
modern Greek, available at: https://eur-
lex.europa.eu/browse/directories/legislation.html.

Currently, the original catalogue is publicly offered in MS Word (.doc) format
through the portal e-Themis444The portal is hosted at
https://www.secdigital.gov.gr/e-themis/, where you can find the thematic index
(hierarchy)., the legal database and management service of it, under the
administration of the Ministry of the Interior
(Affairs)555https://www.ypes.gr. E-Themis is primarily focused on providing
legislation on a multitude of predefined thematic categories, as described in
the catalogue. The main goal is to help users find legislation of interest
using the thematic index.

The original collection follows a bibliographic structure (Figure 1).666In
fact, it originated through a proper printed thesaurus. It consists of 47
legislative volumes and each volume corresponds to a main thematic topic.
Inside each volume, the main thematic topic is divided into thematic
subcategories which are called chapters and subsequently, each chapter breaks
down to subjects which contain the legal resources. The total number of
chapters is 389 while the total number of subjects is 2285, creating an
interlinked thematic hierarchy. Thus the task is defined as a _multi-level
text classification task_ , where the goal is to predict the thematic category
at each level (volume, chapter, subject).

An example of this thematic hierarchy is the volume of “Criminal Law”, which
is divided into 9 chapters. “International Criminal Law” and “Military
Criminal Law” are two out of the nine chapters. Subsequently, “International
Criminal Law” is subdivided into 8 subjects (e.g., “Genocide”,
“Counterfeiting” etc.) and “Military Criminal Law” is subdivided into 4
subjects (e.g., “Military Criminal Code”, “Legal Remedies” etc.). Another
example is the volume of “Labour Law”, divided into 17 chapters. Two of those
are “Collective Employment Contracts” and “Hygiene and Safety of Employees”.
“Collective Employment Contracts” is subdivided into 7 subjects (e.g., “Limits
of Salaries”, “Holiday Allowance” etc.) and “Hygiene and Safety of Employees”
is subdivided into 3 subjects (e.g., “Hygiene And Safety Of Workplaces And
Employees”, “Work Health Books” etc.).

Figure 1: Original GLC thematic hierarchy

For the study presented in this paper, we curated and publicly release a new
dataset named Greek Legal Code (GLC), containing all the available legislation
from the original catalogue in JSON format. Each JSON file contains a legal
resource (law) along with its metadata, i.e., thematic topics, publication
year, document type, etc., as they were extracted from the original
documents.777In Appendix A, we present in great detail the pre-processing
procedure.

###  3.2 Dataset Statistics

In this section, we present a detailed quantitative analysis of the final
dataset, consisting of 47,563 documents (i.e., categorized legal resources).

#### Data Split

GLC is split into three subsets: training (60%), development (20%) and test
(20%), as shown in Table 1. The documents are distributed uniformly for all
levels of the class hierarchy in order to achieve the same level of
partitioning from bottom to top (i.e., from each subject to the whole
dataset).

Subset | Docs | Mean # of tokens / doc | (<100 tokens)
---|---|---|---
Train (60%) | 28536 | 600 | 15412 (54%)
Dev. (20%) | 9511 | 574 | 5175 (54.4%)
Test. (20%) | 9516 | 595 | 5075 (53.3%)
Total: | 47563 | 594 | 25662 (54%)

Table 1: Dataset split and statistics. Figure 2: Tokens’ distribution over
docs in GLC

#### Document Size

In Figure 2, we observe that more than half of the documents in our dataset
have less than 100 tokens. We see that most of the documents are not that
long, with their total number of tokens being below the mean number of tokens
across all documents. Many records (documents) in GLC, especially the older
ones have only the descriptive title of the legal resource or a small part of
it and not the full text.

| Total | Frequent | Few-shot (<10) | Zero-shot
---|---|---|---|---
Volume | 47 | 47 (100%) | 0 | 0
Chapter | 389 | 333 (85.6%) | 53 (13.6%) | 3 (00.7%)
Subject | 2285 | 712 (31.2 %) | 1431 (62.6%) | 142 (06.2%)

Table 2: Number of classes per thematic level and their distribution to
frequent-few-zero categories

#### Label Frequency

GLC classes are divided into three categories for each thematic level:
frequent classes, which occur in more than 10 training documents and can be
found in all three subsets (training, development and test); few-shot
classes888The terms few-shot and zero-shot are used to express the under-
representation of classes in GLC rather than transfer learning approaches as
in other deep learning literature. which appear in 1 to 10 training documents
and also appear in the documents of the development and test sets, and zero-
shot classes which appear in the development and/or test, but not in the
training documents. As demonstrated in Table 2, many classes are under-
represented, especially in the thematic level of subjects, causing the
appearance of few- and zero-shot categories. The appearance of
underrepresented classes increases as we move towards more specific thematic
levels. Table 3 shows the total number of documents per category and thematic
level.

| Total | Frequent | Few-shot (<10) | Zero-shot
---|---|---|---|---
Volume | 47563 | 47563 (100%) | 0 | 0
Chapter | 47563 | 47108 (99.0%) | 445 (00.9%) | 10 (<00.1%)
Subject | 47563 | 38475 (80.9%) | 8870 (18.6%) | 218 (00.5%)

Table 3: Number of documents as labeled per thematic level and their
distribution to frequent-few-zero categories

In the volume level, all the classes belong to the frequent category and are
sufficiently represented, as more than 10 documents per class exist in the
training data. In the chapter level, few-shot classes appear and are rather
underrepresented as most documents are classified among frequent classes,
leaving less than 1% of the total documents to be associated with ~14% of the
total classes. In the subject level, data are even more unequally distributed
over classes. The majority of documents are classified into frequent classes,
leaving more than half of the total classes (~63%) to be associated with less
than 20% of the total documents, along with 142 classes having zero
representation at the training subset.

Figure 3: Illustration of (i) BIGRU-MAX, (ii) BIGRU-ATT, (iii) BIGRU-LWAN and
(iv) BERT

##  4 Task and Methods

###  4.1 Multi-class Legal Topic Classification for Greek Legislation

We study multi-class legal topic classification for Greek legislation using
the GLC dataset999Available at:
https://huggingface.co/datasets/greek_legal_code. GLC offers 3 hierarchical
levels of thematic categorization, forming a tree structure that follows the
original GLC organization as explained in Section 3.1. We divide the
classification task for GLC into three separate classification tasks. Each
task deals with a different level of the thematic hierarchy (i.e., volume,
chapter or subject) and the classifier’s objective is to predict the correct
class out of all the classes in this specific thematic level.

###  4.2 Methods

In our study, we consider an arsenal of nine methods ranging from traditional
machine learning techniques (2) and RNN-based methods (3), to state-of-the-art
Transformer-based methods (4).

SVM-BOW: Support Vector Machines (SVMs) is a strong baseline for the multi-
class classification problem as it is one of the highest performing
traditional ML methods. We represent the legal documents using Bag-of-Words
(BoW) features, getting the most frequent n-grams across all training data,
weighted by TF-IDF.

XGBOOST-BOW: XGBoost Chen and Guestrin (2016) is a scalable, fast and robust
open-source implementation101010Available at: https://github.com/dmlc/xgboost/
of the Gradient Boosting decision tree algorithm. In XGBoost, the model is
fitted on the gradient of loss generated from the previous step and the
gradient boosting algorithm is modified so that it works with any
differentiable loss function. In our case, the documents are represented using
BoW features weighted by TF-IDF (similarly to SVM-BOW).

BIGRU-MAX: The first RNN-based method that we examine is BiGRU with max-
pooling, employing pre-trained, domain-specific word embeddings Angelidis et
al. (2018).111111Angelidis et al. (2018) released a Word2Vec model trained on
Greek legal corpora, including Greek national and EU legislation. The stacked
BiGRU encoder converts the pre-trained word embeddings wi into context-aware
embeddings hi. Context-aware token embeddings pass through a max-pooling layer
to produce the final document representation d by reducing the initial matrix
(token-wise vectors) into a single vector but hopefully, keep the most salient
information. Finally, a dense layer with L output units and the softmax()
activation is deployed, to transform the document representation d into a
probability distribution over L classes. L is defined according to the
examined task, i.e., L=47 for the volume-level task, L=389 for the chapter-
level task and L=2285 for the subject-level task.

BIGRU-ATT: The second RNN-based method is a BiGRU network with self-attention
Xu et al. (2015); Chalkidis et al. (2019c). This methods uses the very same
word embedding and BiGRU encoder, similarly to the previously described method
(BIGRU-MAX). Instead of max-pooling, it uses the self-attention mechanism to
produce a final document representation d. This representation is computed as
the weighted sum of the BiGRU context-aware embeddings hi, weighted by the
self-attention scores ai produced as:

$a_{i}=\frac{\exp\left(\mathbf{h}_{i}^{\top}\mathbf{u}\right)}{\sum_{j}\exp\left(\mathbf{h}_{j}^{\top}\mathbf{u}\right)}$
(1)

$\mathbf{d}=\frac{1}{T}\sum_{i=1}^{T}a_{i}\mathbf{h}_{i}$ (2)

T represents the document’s length in words while u is a trainable vector used
to compute the attention scores ai over hi. Similarly to BIGRU-MAX, a final
dense layer with L output units and softmax() activation is deployed to
predict the correct output class using a probability distribution over all the
classes.

BIGRU-LWAN: The third RNN-based method replaces the self-attention mechanism
of BIGRU-ATT with the label-wise attention mechanism. The original Label-Wise
Attention Network (LWAN) was introduced in Mullenbach et al. (2018) using a
CNN-based encoder. Later on, Chalkidis et al. (2019c, b) replaced it with a
BiGRU encoder. In contrast with BIGRU-ATT, this label-wise attention technique
uses L independent attention heads, one per class, generating L label-wise
document representations dl from the sequence of hi vectors produced by the
BiGRU encoder. The intuition is that each label-wise document embedding is
dedicated in predicting the corresponding class, focusing on possibly
different aspects of each representation hi. In effect, different parts of the
representation may be more relevant for different classes.

$a_{li}=\frac{\exp\left(\mathbf{h}_{i}^{\top}\mathbf{u}_{l}\right)}{\sum_{i^{\prime}}\exp\left(\mathbf{h}_{i^{\prime}}^{\top}\mathbf{u}_{l}\right)}$
(3)

$\mathbf{d}_{l}=\frac{1}{T}\sum_{i=1}^{T}a_{li}\mathbf{h}_{i}$ (4)

Again, T represents the document’s length in words while ul ($l=1,2,...,L$) is
a trainable vector used to compute the attention scores ali of the l-th
attention head. Then, each attention head dl goes through an independent dense
layer for each class and similarly to the previous methods, softmax()
activation is deployed to transform the document representation d into a
probability distribution over L classes.

M-BERT: BERT Devlin et al. (2019) is a Transformer-based Vaswani et al. (2017)
language model initially developed by Google. In BERT, deep bidirectional
representations are pre-trained from unlabeled text by jointly conditioning on
both left and right context. As a result, for any new task, the pre-trained
BERT model can be fine-tuned with just one additional task-specific output
layer trained with task-specific data. We employ the multilingual version of
the 12-layer BERT-BASE-CASED model, which supports modern Greek, alongside 99
languages, out-of-the-box. We add linear layer on top of M-BERT encoder
followed by a softmax() activation. This extra dense layer is fed with the so-
called “classification token” ([cls]) of the BERT encoder as described in
Devlin et al. (2019), serving as the final document representation.

XLM-ROBERTA: The next BERT-based model we examine is XLM-RoBERTa Conneau et
al. (2020), a multilingual adaptation of RoBERTa Liu et al. (2019). RoBERTa is
built following BERT’s architecture, while removed the next-sentence pre-
training objective and trained with much larger batches and higher learning
rate. Based on this study, Conneau et al. (2020) proposed the XLM-RoBERTa
model, which supports 100 different languages (Greek included) and is trained
on 2.5TB of filtered Common Crawl data.

GREEK-BERT: We also experiment with GREEK-BERT Koutsikakis et al. (2020), a
native monolingual version of BERT, trained solely on modern Greek, achieving
state-of-the-art results in several NLP tasks. GREEK-BERT was pre-trained on
29GB of text from a corpus consisting of the Greek part of Wikipedia, the
Greek part of the European Parliament Proceedings Parallel Corpus (Europarl)
Koehn (2005) and OSCAR Ortiz Suárez et al. (2019), a clean version of Common
Crawl. Even though multilingual models like the previous two offer exceptional
performance, monolingual models usually surpass them in most downstream tasks.
Again, GREEK-BERT follows the previous BERT models configuration.

GREEK-LEGAL-BERT: Finally, we experiment with GREEK-LEGAL-BERT Athinaios
(2020), another BERT-flavored model for modern Greek. Its pre-training corpus
is based entirely on Greek legal documents and consists of around 5GB of Greek
and EU legislation documents retrieved by
Nomothesia121212http://legislation.di.uoa.gr. Chalkidis et al. (2017b), a
Greek legislative Knowledge Base. It also follows the previously described
configuration.

Method | Hyperparameters
---|---
SVM-BOW |  | Kernel: [linear, rbf]
---
C: [0.25, 0.50, 0.75, 1]
N-Grams: [(1,3), (1,5)]
Max-Features: [200k, 400k]
XGBOOST-BOW |  | Booster: gbtree with softmax
---
N-Estimators: 800
Max-Depth: [4, 5, 7, 10]
Min-Child-Weight: [2, 5, 10]
BIGRU-* |  | HYPEROPT131313See: https://hyperopt.github.io/hyperopt/on:
---
Layers: [1, 2]
Units: [200, 300, 400]
Dropout: [0.1, 0.2, 0.3, 0.4]
Word Dropout: [0, 0.01, 0.02]
Batch Size: [8, 16]
*BERT* |  | Dropout: 0.1
---
Learning Rate: [1e-5, 2e-5, 3e-5, 5e-5]
Batch Size: 8

Table 4: Hyper parameters search space for all models

MODEL | _Volume (47 classes)_ | _Chapter (389 classes)_
---|---|---
| ALL LABELS | FREQUENT | ALL LABELS | FREQUENT | FEW-SHOT
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
SVM-BOW | 85.3 | 85.3 | 85.3 | 85.3 | 85.3 | 85.3 | 77.9 | 77.9 | 77.9 | 77.9 | 78.6 | 78.2 | 90.0 | 09.3 | 16.8
XGBOOST-BOW | 77.2 | 77.2 | 77.2 | 77.2 | 77.2 | 77.2 | 67.5 | 67.5 | 67.5 | 67.8 | 68.1 | 67.9 | 19.2 | 10.3 | 13.4
BIGRU-MAX | 84.3 | 84.3 | 84.3 | 84.3 | 84.3 | 84.3 | 77.5 | 77.5 | 77.5 | 77.9 | 77.9 | 77.9 | 44.9 | 45.4 | 45.1
BIGRU-ATT | 86.4 | 86.4 | 86.4 | 86.4 | 86.4 | 86.4 | 81.1 | 81.1 | 81.1 | 81.1 | 81.6 | 81.3 | 86.7 | 40.2 | 54.9
BIGRU-LWAN | 84.1 | 84.1 | 84.1 | 84.1 | 84.1 | 84.1 | 76.8 | 76.8 | 76.8 | 76.9 | 77.3 | 77.1 | 63.8 | 30.9 | 41.7
M-BERT | 85.8 | 85.8 | 85.8 | 85.8 | 85.8 | 85.8 | 80.2 | 80.2 | 80.2 | 80.1 | 80.6 | 80.4 | 81.0 | 43.9 | 56.6
XLM-ROBERTA | 85.0 | 85.0 | 85.0 | 85.0 | 85.0 | 85.0 | 80.6 | 80.6 | 80.6 | 80.6 | 81.0 | 80.8 | 80.3 | 37.9 | 51.3
GREEK-BERT | 87.5 | 87.5 | 87.5 | 87.5 | 87.5 | 87.5 | 82.4 | 82.4 | 82.4 | 82.4 | 82.8 | 82.6 | 81.0 | 45.2 | 57.8
GREEK-LEGAL-BERT | 89.4 | 89.4 | 89.4 | 89.4 | 89.4 | 89.4 | 84.3 | 84.3 | 84.3 | 84.4 | 84.8 | 84.6 | 79.0 | 45.8 | 57.8

Table 5: Experimental results for _Volume_ and _Chapter_ levels.

##  5 Experiments

###  5.1 Experimental SetUp

We tune all methods by grid-searching over the core hyper-parameters (per
method) presented in Table 4, except the RNN-based methods where we used
HyperOpt due to the large search space. We select the models with the best
performance on the development set. We use early stopping on the development
loss using the Adam Kingma and Ba (2015) optimizer. All BERT models follow the
base configuration, i.e., 12 layers with 786 units and 12 attention heads
each.141414All models are available at https://huggingface.co/models. We
report micro-averaged Precision (P), Recall (R), and F1-score (F1) on the test
set. For each method, we perform 5 runs with different seeds and report the
average. We report results for the frequent and few-shot classes (when
available), omitting results for zero-shot classes since our methods are
incapable of zero-shot learning. We release our code for
reproducibility:151515See: https://github.com/christospi/glc-nllp-21.

###  5.2 Volume-level Classification Evaluation

In volume-level classification, all the 47 possible classes are sufficiently
represented and belong to the frequent category, while the mean number of
documents per class is approximately 1k. However, we acknowledge class
imbalance (label skewness) as a challenge in our dataset, though not so
evident here. Table 5 (first zone) demonstrates the results for this task.
SVM-BOW proves to be a strong competitor in the first classification task
(85.3 in F1). Interestingly, it outperforms two of our neural methods with
domain-specific word embeddings, namely BIGRU-MAX and BIGRU-LWAN with 84.3 and
84.1 F1 scores, respectively. As for XGBoost, although it seemed quite
promising and very fast at training, its inadequate F1 score of 77.2 places it
at the bottom of the list for this task.

Among the RNN-based neural methods, BIGRU-ATT achieves an F1 score of 86.4
outperforming the rest of the RNN-based methods. Its results indicate the
significance of two of its fundamental features: (i) the domain-specific
word2vec embeddings and (ii) the cumulative self-attention head that provides
an advantageous final document representation. Compared to BIGRU-MAX, we
believe that its max-pooling layer hinders some of the document’s
particularities and thus, it yields a lower score. Likewise, the BIGRU-LWAN
method with L different attention heads seems to be more tailor-made for
multi-label classification tasks, as it does not offer any performance
improvement compared to BIGRU-ATT.

GREEK-LEGAL-BERT proves to be the best method we experimented with, achieving
a score of 89.4 in F1, followed by the generic GREEK-BERT with a score of 87.5
in F1. The two multilingual models (M-BERT and XLM-ROBERTA) also achieve
satisfying F1 scores (85.8 and 85.0), confirming their claim to offer top-
notch results in most downstream NLP tasks. The results demonstrate that
monolingual models are able to surpass other advanced multilingual
transformer-based models. We hypothesize that the fact that GREEK-LEGAL-BERT
is pre-trained entirely on Greek legal corpora is the main reason of its
superiority, while the specialization in the Greek language, alongside GREEK-
BERT, seems even more critical.

###  5.3 Chapter-level Classification Evaluation

In Table 5 (second zone) we observe again that XGBOOST-BOW has the lowest
performance with 67.5 in overall F1 score with SVM-BOW being the second worst
overall (77.9) alongside BIGRU-MAX. The ranking of BERT-based models is
similar to the previous task (Volume-level classification), i.e., monolingual
models outperform their multilingual counterparts, while GREEK-LEGAL-BERT is
better by 2% compared to the generic GREEK-BERT model. These results further
supporting our intuition on the importance of native language support and
domain knowledge.

Inspecting the few-shot label results, we observe that BIGRU-ATT and BERT
models outperform BIGRU-MAX by 10%, while the traditional machine learning
approaches have terrible performance (approx. 15 F1). These results highlight
the importance of the attention mechanism and its capability to focus in
specific parts of the text that are more prominent in relation to the
downstream task and the specific labels.

###  5.4 Subject-level Classification Evaluation

In the last sub-task, where there are many more classes (2285 in total), we
observe a similar ranking in the methods (Table 6). However, the absolute
difference in few-shot labels’ scores is higher this time, when comparing
either the attention-based method with BIGRU-MAX (approx. +20% F1) or
monolingual with multilingual BERT models (approx. +9% F1).

MODEL | _Subject (2285 classes)_
---|---
| ALL LABELS | FREQUENT | FEW-SHOT
| P | R | F1 | P | R | F1 | P | R | F1
SVM-BOW | 37.9 | 37.9 | 37.9 | 37.9 | 47.8 | 42.3 | 00.0 | 00.0 | 00.0
XGBOOST-BOW | 55.3 | 55.3 | 55.3 | 56.1 | 64.8 | 60.1 | 46.9 | 19.1 | 27.2
BIGRU-MAX | 62.9 | 62.9 | 62.9 | 66.0 | 70.5 | 68.1 | 47.1 | 37.8 | 42.0
BIGRU-ATT | 74.8 | 74.8 | 74.8 | 75.3 | 79.6 | 77.4 | 72.6 | 61.1 | 66.3
BIGRU-LWAN | 65.2 | 65.2 | 65.2 | 68.1 | 72.8 | 70.4 | 50.7 | 40.4 | 45.0
M-BERT | 76.8 | 76.8 | 76.8 | 79.8 | 82.5 | 81.1 | 64.4 | 59.7 | 62.0
XLM-ROBERTA | 78.0 | 78.0 | 78.0 | 80.3 | 83.4 | 81.8 | 68.2 | 62.1 | 65.0
GREEK-BERT | 79.4 | 79.4 | 79.4 | 80.5 | 83.9 | 82.2 | 74.6 | 67.0 | 70.6
GREEK-LEGAL-BERT | 81.2 | 81.2 | 81.2 | 83.0 | 85.5 | 84.2 | 73.6 | 69.4 | 71.4

Table 6: Experimental results for _Subject_ level.

###  5.5 General Observations

There are two main observations:

  * •

Pre-trained Transformer-based models perform exceptionally and the performance
gains compared to traditional machine learning methods and RNN-based methods
are increasing in relation to the number of labels. We can only speculate the
following: (a) sub-word units used by Transformer-based models potentially
play an important role in morphological rich languages like Greek, compared to
full words processed by the rest of the examined models; and (b) the multi-
head attention mechanism can better distill important detailed (specialized)
information to distinguish categories (labels).

  * •

The Label-wise Attention Network (LWAN) under-performs in the newly introduced
multi-class text classification task. Again, we can only speculate that the
produced label-wise document representations that improve results over the
standard universal attention mechanism in the multi-label setting Chalkidis et
al. (2020a) lead to a “greedy” over-scoring (produced logits) followed by an
aggressive label competition (softmax over logits) in the examined multi-task
setting leading to negative (poor) results.

We aim to further review and validate these observations, while also study the
reasons behind those thoroughly in the future, possibly using more datasets
and ablation studies (e.g., train RNN-based methods with pretrained sub-word
embeddings).

##  6 Conclusions and Future Work

We introduced Greek Legal Code (GLC), a new publicly available dataset
consisting of 47k Greek legislation resources. Relying on this dataset, we
experimented with several classifiers, ranging from traditional machine
learning and recurrent models to state-of-the-art transfer learning models.
Through their performance evaluation, we realized that although traditional
machine learning classifiers (e.g., SVM-BOW) set strong baselines for some of
the considered tasks, they fall short against more sophisticated methods. In
contrast, RNN-based methods relying on BiGRUs provide improved overall
performance and were competitive to multilingual transformer-based
architectures (M-BERT, XLM-ROBERTA). Beyond doubt, monolingual transformer-
based models (GREEK-BERT and GREEK-LEGAL-BERT) lead to state-of-the-art
results, especially when they are pre-trained on in-domain corpora.

Nonetheless, more emphasis should be given to the qualitative and quantitative
characteristics of the examined datasets. Intricacies like class imbalance,
data scarcity and diversity apparently need special handling. Regarding our
study, we noticed that few-shot and especially zero-shot classes need to be
properly handled with appropriate methods, as standard classifiers are
insufficient. As for the recent trend to develop novel monolingual BERT-based
models, results show that already established multilingual models are
incredibly powerful even in monolingual tasks. While research is on-going and
these models are continuously being improved, also taking into consideration
the computational costs, it is quite challenging to motivate researchers into
making an effort to train monolingual models for medium or small-sized
languages; especially when multilingual models can perform equally well or
occasionally, even better.

In future work, we plan to investigate specialized methods with improved few-
shot and zero-shot performance Hu et al. (2018); Rios and Kavuluru (2018);
Chalkidis et al. (2019c) that leverage various data properties (e.g., label
descriptors and label hierarchy). Also, we intend to apply deep learning
techniques that take into account the hierarchy of classes in datasets like
GLC, where there is an underlying taxonomy Kowsari et al. (2017); Chalkidis et
al. (2020b); Manginas et al. (2020). Furthermore, experimenting with similar
datasets like that of EU Legislation written in Greek will allow us to confirm
our current conclusions. For example in the future, we could consider Cypriot
legislation161616http://www.cylaw.org/nomoi/ to evaluate the out-of-domain
generalization of models trained on Greek legislation. Finally, our long term
goal is to support and encourage further research in NLP for the Greek
language by publishing novel datasets, introducing and experimenting with
state-of-the-art methods.

## Acknowledgements

This work has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 825258. This work is also
partly funded by the Innovation Fund Denmark
(IFD)171717https://innovationsfonden.dk/en under File No. 0175-00011A.

## References

  * Aletras et al. (2016) N. Aletras, D. Tsarapatsanis, D. Preotiuc-Pietro, and V. Lampos. 2016.  Predicting judicial decisions of the european court of human rights: a natural language processing perspective.  _PeerJ Comput. Sci._ , 2.
  * Angelidis et al. (2018) Iosif Angelidis, Ilias Chalkidis, and Manolis Koubarakis. 2018.  Named entity recognition, linking and generation for greek legislation.  In _JURIX_.
  * Athanasiou and Maragoudakis (2017) V. Athanasiou and M. Maragoudakis. 2017.  A novel, gradient boosting framework for sentiment analysis in languages where nlp resources are not plentiful: A case study for modern greek.  _Algorithms_ , 10:34.
  * Athinaios (2020) Konstantinos Athinaios. 2020.  Named Entity Recognition using a Novel Linguistic Model for Greek Legal Corpora based on BERT model.  _BS Thesis, School of Science, Department of Informatics and Telecommunications_.
  * Chalkidis et al. (2019a) Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019a.  Neural legal judgment prediction in English.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4317–4323, Florence, Italy. Association for Computational Linguistics.
  * Chalkidis et al. (2017a) Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2017a.  Extracting contract elements.  In _Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law_ , ICAIL ’17, page 19–28, New York, NY, USA. Association for Computing Machinery.
  * Chalkidis et al. (2018) Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2018.  Obligation and prohibition extraction using hierarchical RNNs.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 254–259, Melbourne, Australia. Association for Computational Linguistics.
  * Chalkidis et al. (2019b) Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2019b.  Extreme multi-label legal text classification: A case study in EU legislation.  In _Proceedings of the Natural Legal Language Processing Workshop 2019_ , pages 78–87, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Chalkidis et al. (2019c) Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019c.  Large-scale multi-label text classification on EU legislation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6314–6322, Florence, Italy. Association for Computational Linguistics.
  * Chalkidis et al. (2020a) Ilias Chalkidis, Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020a.  An empirical study on large-scale multi-label text classification including few and zero-shot labels.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7503–7515, Online. Association for Computational Linguistics.
  * Chalkidis et al. (2020b) Ilias Chalkidis, Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020b.  An empirical study on large-scale multi-label text classification including few and zero-shot labels.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7503–7515, Online. Association for Computational Linguistics.
  * Chalkidis et al. (2021) Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2021.  LexGLUE: A Benchmark Dataset for Legal Language Understanding in English.
  * Chalkidis and Kampas (2018) Ilias Chalkidis and Dimitrios Kampas. 2018.  Deep learning in law: early adaptation and legal word embeddings trained on large corpora.  _Artificial Intelligence and Law_ , 27:171–198.
  * Chalkidis et al. (2017b) Ilias Chalkidis, Charalampos Nikolaou, Panagiotis Soursos, and Manolis Koubarakis. 2017b.  Modeling and querying greek legislation using semantic web technologies.  In _The Semantic Web_ , pages 591–606, Cham. Springer International Publishing.
  * Chen and Guestrin (2016) T. Chen and C. Guestrin. 2016.  Xgboost: A scalable tree boosting system.  _ACM-SIGKDD_.
  * Conneau et al. (2020) A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  _ArXiv_ , abs/1911.02116.
  * Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Hu et al. (2018) Z. Hu, X. Li, C. Tu, Z. Liu, and M. Sun. 2018.  Few-shot charge prediction with discriminative legal attributes.  In _COLING_.
  * Johnson et al. (2016) A. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Celi, and R. G. Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * Khandagale et al. (2019) Sujay Khandagale, Han Xiao, and Rohit Babbar. 2019.  Bonsai - Diverse and Shallow Trees for Extreme Multi-label Classification.  _CoRR_ , abs/1904.08249.
  * Kingma and Ba (2015) D. P. Kingma and J. Ba. 2015.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _MTSUMMIT_.
  * Koutsikakis et al. (2020) John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020.  Greek-bert: The greeks visiting sesame street.  In _11th Hellenic Conference on Artificial Intelligence_ , SETN 2020, page 110–117, New York, NY, USA. Association for Computing Machinery.
  * Kowsari et al. (2017) K. Kowsari, D. Brown, M. Heidarysafa, K. Meimandi, M. Gerber, and L. Barnes. 2017\.  Hdltex: Hierarchical deep learning for text classification.  _ICMLA_.
  * Leitner et al. (2019) Elena Leitner, Georg Rehm, and Julian Moreno-Schneider. 2019.  Fine-grained Named Entity Recognition in Legal Documents.  In _Semantic Systems. The Power of AI and Knowledge Graphs. Proceedings of the 15th International Conference (SEMANTiCS 2019)_ , number 11702 in Lecture Notes in Computer Science, pages 272–287, Karlsruhe, Germany. Springer.  10/11 September 2019.
  * Lewis et al. (2004) D. D. Lewis, T. G. Rose Y. Yang, and F. Li. 2004.  Rcv1: A new benchmark collection for text categorization research.  _J. Mach. Learn. Res._ , 5:361–397.
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Manginas et al. (2020) Nikolaos Manginas, Ilias Chalkidis, and Prodromos Malakasiotis. 2020.  Layer-wise guided training for BERT: Learning incrementally refined document representations.  In _Proceedings of the Fourth Workshop on Structured Prediction for NLP_ , pages 53–61, Online. Association for Computational Linguistics.
  * Mencia and Fürnkranzand (2007) Eneldo Loza Mencia and Johannes Fürnkranzand. 2007.  An Evaluation of Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain.  In _Proceedings of the 1st Linguistic Annotation Workshop_ , pages 126–132, Halle, Germany.
  * Mullenbach et al. (2018) J. Mullenbach, S. Wiegreffe, J. Duke, J. Sun, and J. Eisenstein. 2018.  Explainable prediction of medical codes from clinical text.  _ArXiv_ , abs/1802.05695.
  * Nallapati and Manning (2008) R. Nallapati and C. D. Manning. 2008.  Legal docket classification: Where machine learning stumbles.  _EMNLP_.
  * O’Neill et al. (2017) J. O’Neill, P. Buitelaar, C. Robin, and L. O’ Brien. 2017.  Classifying Sentential Modality in Legal Language: A Use Case in Financial Regulations, Acts and Directives.  In _Proceedings of the 16th International Conference on Artificial Intelligence and Law (ICAIL)_ , pages 159–168, London, UK.
  * Ortiz Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut für Deutsche Sprache.
  * Papantoniou and Tzitzikas (2020) K. Papantoniou and Y. Tzitzikas. 2020.  Nlp for the greek language: A brief survey.  _11th Hellenic Conference on Artificial Intelligence_.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Pitenis et al. (2020) Z. Pitenis, M. Zampieri, and T. Ranasinghe. 2020.  Offensive language identification in greek.  In _LREC_.
  * Prabhu et al. (2018) Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018\.  Parabel: Partitioned Label Trees for Extreme Classification with Application to Dynamic Search Advertising.  In _Proceedings of the 2018 World Wide Web Conference_ , WWW ’18, pages 993–1002, Republic and Canton of Geneva, Switzerland.
  * Rios and Kavuluru (2018) A. Rios and R. Kavuluru. 2018.  Few-shot and zero-shot multi-label learning for structured label spaces.  _EMNLP_.
  * Undavia et al. (2018) S. Undavia, A. Meyers, and J. Ortega. 2018.  A comparative study of classifying legal documents with neural networks.  _FedCSIS_.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017.  Attention is all you need.  In _NIPS_.
  * Wang et al. (2012) W. Yang Wang, E. Mayfield, S. Naidu, and J. Dittmar. 2012.  Historical analysis of legal opinions with a sparse mixed-effects latent variable model.  In _ACL_.
  * Xiao et al. (2018) Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018.  CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction.  _arXiv:1807.02478 [cs]_.  ArXiv: 1807.02478.
  * Xu et al. (2015) K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _ICML_.
  * Ye et al. (2018) H. Ye, X. Jiang, Z. Luo, and W. Chao. 2018.  Interpretable charge predictions for criminal cases: Learning to generate court views from fact descriptions.  _ArXiv_ , abs/1802.08504.
  * You et al. (2019) Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019.  AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification.  In _Advances in Neural Information Processing Systems 32_ , pages 5812–5822. Curran Associates, Inc.
  * Zhong et al. (2020) Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020.  How does NLP benefit legal system: A summary of legal artificial intelligence.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5218–5230, Online. Association for Computational Linguistics.

##  Appendix A Dataset Curation

We describe in-detail the processing method we followed to generate GLC and
describe the data structure of the final JSON documents.

###  A.1 Parsing the Original Documents

The original legislative volumes are encoded in MS Word (.doc) format. While
most of them follow the double-column format, there are cases where they also
include text in single-column format or even include scanned documents or
images as legal resources, making the initial data quite noisy. Considering
our objective, these abnormalities should be revised, and all the additional
metadata contained in the .doc file (e.g., font style, size, page margins)
should be removed. Thus, converting these documents into plain text files was
of significant importance. To achieve that, we used docx2txt181818Available
at: https://github.com/ankushshah89/python-docx2txt/., a python utility that
detects and extracts text from .doc files.

Examining the output text files, we encountered problematic samples that
needed special handling. For example, we found significant keywords (e.g.,
“$\Theta$EMA” which means subject) missing from text or even having typos,
subject IDs found inline with subject titles etc., mostly due to minor
inaccuracies in the text conversion. Furthermore, multiple white spaces,
multiple new lines and special or corrupted characters occurred in the text.
To overcome these complications, we performed data cleaning using heuristics
and regular expressions to produce neat text files that follow the same
normalized structural pattern.

Next, we built a rule-based parser in Python which receives these text files
as input and produces JSON files that will be the inputs to our classifiers.
The goal here is to separate legal resources into single documents, along with
all their related metadata such as ID, publication year, title etc. and their
classification hierarchy (i.e., in which volume, chapter and subject they
belong to). Each final JSON file represents a unique legal resource, ready to
be fed into the machine-learning models we built. The parser builds in memory
a tree of depth 4 that represents the whole GLC hierarchy. The first level
consists of the thematic volumes while the second level contains all the
thematic chapters for each volume. The third level includes the thematic
subjects of the individual chapters, and finally, the leaf nodes represent the
legal resources. An overview of the tree can be found in Figure 4.

Figure 4: Tree representation of GLC thematic hierarchy

With the use of regular expressions, the parser organizes the content to its
thematic hierarchy. Each legal resource may contain the whole original
legislative document, some articles of it or even a short sentence (usually
its original title or a short description). Hence, the parser attempts to
identify and separate any existing articles. However, if this is not feasible,
it just keeps the whole body as a text chunk. No deeper parsing is performed
(i.e., in paragraphs, sentences) as this is out of scope. The final step is to
populate the leaf nodes (i.e., the legal resources) with the appropriate
metadata and enhance the available text samples. To accomplish this, the
parser uses the metadata fragment of each legal resource to extract the
necessary information. Specifically, the words of interest are shown in Figure
5, depicting an example of a metadata fragment.

Figure 5: Legal resource’s metadata (translated)

Again, with proper regular expressions, the parser manages to retrieve the
requested information. Also, having available the type, the year of
publication and the ID of each legal resource, the parser is able to uniquely
identify each one of them by using the following pattern: _{
type}/{year}/{id}_. Exploiting that, it searches for duplicate legal resources
that may exist in the dataset. For example, one law may be present in more
than one subject due to the thematic variety of its articles. To avoid any
complexities and because our task is multi-class and not multi-label
classification, the parser removes these resources entirely from the dataset.

Moreover, the parser manages to enhance the content of some legal resources
(depending on their type191919See the supported legislation types at:
http://legislation.di.uoa.gr/search/.) by utilizing the platform
Nomothesia202020See: http://legislation.di.uoa.gr/. Chalkidis et al. (2017b).
Nomothesia makes Greek legislation available as open linked data using
semantic web technologies. Through its RESTful API and by adopting the
following URI template: _http://www.legislation.di.uoa.gr/eli/
{type}/{year}/{id}/data/json_ the parser manages to retrieve the text of any
legal resource in JSON format, as offered through Nomothesia. Then, it
compares the number of tokens of the original and the fetched text fragments
and eventually keeps the more extensive. In that way, the parser succeeds in
enhancing the size and quality of the dataset.

For each final document, the text content consists of the header along with
the body. In case of successful parsing the body consists of multiple
articles. Otherwise, it only contains a single text passage. Evaluating the
final data, we noticed that many legal resources have limited tokens count (as
shown in charts of Section 3.2). However, we consider this not to be a crucial
problem since meaningful information (e.g., highly representative words) is
quite dense in most of these samples as shown in Figure 6.

Figure 6: Small-sized sample of GLC indicating highly representative words
(translated)

Finally, the complete dataset, consisting of JSON files following the format
of Figure 7 is distributed to train, development and test subsets as described
in Section 3.2.

Figure 7: Final legal resource as JSON. The legal resource has been parsed and
enhanced with two articles, as fetched from Nomothesia web platform


###### Abstract

Motivated by the significance of modular symmetry in describing neutrino
masses and flavor structure, we apply the $A_{4}$ modular symmetry in 3+1
scheme of active-sterile neutrino mixings. Neutrino oscillation observables in
3$\sigma$ range are successfully reproduced through the vev of the modulus
$\tau$ in normal hierarchy(NH) as well as inverted hierarchy(IH). We have also
studied other phenomenologies regarding effective masses $m_{\beta}$ and
$m_{\beta\beta}$ in neutrinoless double beta decay. Mixings between active and
sterile neutrinos are analysed in detail and the mixing elements $|V_{i4}|$
are found to satisfy the experimental bounds. The best-fit values of the
neutrino mixing angles and mass squared differences ratio are determined using
minimum $\chi^{2}$ analysis. This model predicts best-fit values of the
neutrino oscillation observables as $\sin^{2}\theta_{23}=0.572$,
$\sin^{2}\theta_{12}=0.313$, $\sin^{2}\theta_{13}=0.022$ and $r=\sqrt{\Delta
m_{21}^{2}/\Delta m_{31}^{2}}=0.172$ for NH whereas
$\sin^{2}\theta_{23}=0.602,$ $\sin^{2}\theta_{12}=0.288,$
$\sin^{2}\theta_{13}=0.022$ and $r=\sqrt{\Delta m_{21}^{2}/|\Delta
m_{32}^{2}|}=0.172$ for IH. Our analysis is also consistent with the latest
Planck cosmological upper bound on the sum of neutrino masses $\sum
m_{i}<0.12$ eV.

##  I Introduction

Origin of neutrino masses and flavour structure is one of the most important
problems of Standard Model(SM) of particle physics. Since the discovery of
neutrino oscillation in various experiments such as SNO, SuperKamiokande,
etc., various extensions of SM have been studied. Models with non-Abelian
discrete symmetries such as $A_{4}$[1, 2, 3], $S_{4}$[4, 5, 6], $S_{3}$[7, 8,
9, 10], etc. found their distinct places in describing some of the
experimental results as well as new predictions regarding unresolved problems
of SM. Among these, the absolute mass scale of neutrino, Dirac CP violating
phase, Baryogenesis, Dark matter, etc. are some of the main questions in high
energy physics.

Motivated by the observations of LSND [11] and MiniBooNE [12, 13], many
authors proposed the existence of a fourth state of neutrino called sterile
neutrino. Sterile neutrinos are incorporated with the three neutrino theory in
various literatures in a 3+1 scheme, 3+1+1, 3+2 etc. One of the simplest
extension is the 3+1 scheme where a singlet sterile neutrino is added to the
three active neutrinos and sterile neutrino gets mass through minimal extended
seesaw mechanism(MES)[14]. Many authors have used discrete symmetries to
describe the neutrino masses and its flavor structures. The main drawbacks of
such approaches are the presence many hypothetical scalar fields as well as
additional symmetry groups. Recently, an interesting method has been proposed
in which modular symmetry is used along with the discrete symmetries as its
subgroup. An important feature of modular symmetry framework is that the
Yukawa couplings can transform non-trivially as modular forms under the
modular group. These modular forms are written as a function of a single
parameter $\tau$ which is called the modulus. A minimal or no scalar flavons
are required to break the symmetry as the lepton masses are generated from the
symmetry breaking by the vev of the modulus $\tau$. Modular forms of level 3
denoted by $\Gamma(3)$ is isomorphic to $A_{4}$. Detailed analysis on $A_{4}$
modular groups and its application in neutrino model building are studied in
Refs. [15, 16, 17]. There are other works based on numerous modular groups
$A_{4}$ [18, 19, 20, 21, 22, 23, 24, 25, 26, 27], $A_{5}$ [28, 29, 30, 31, 32,
33], $S_{4}$[34, 35, 36, 37, 38, 39], etc. to study neutrino phenomenologies,
dark matter, leptogenesis, etc. For instance, in Ref.[24], modular $A_{4}$ is
used to study lepton and quark masses and mixing based on inverse seesaw
mechanism, Ref.[26] studies lepton mixing and leptogenesis in linear seesaw
while scotogenic dark matter scenario is studied in Ref.[25, 19].

The special feature of this work is that we have performed a detailed study of
the effects of an eV-scale sterile neutrino on neutrino phenomenology using
modular $A_{4}$ symmetry. We have successfully reproduced neutrino oscillation
data within the 3$\sigma$ bounds in normal hierarchy(NH) as well as Inverted
hierarchy(IH). The particle contents and model predictions are different from
other works. Dirac CP-violating phase and the two Majorana phases are also
determined from the active-sterile mixing matrix. Finally, the best-fit values
of the model parameters and neutrino observables are determined using the
$\chi^{2}$ analysis. Significant results are obtained for the active-sterile
mixing parameters $|V_{i4}|$, (where $i=1,2,3$) within the experimental
bounds. There are literatures which study 3+1 active-sterile mixing using the
general $A_{4}$ discrete symmetry group [40, 41, 42, 43, 44]. However, to the
best of our knowledge, modular $A_{4}$ group has not been used in MES
mechanism for 3+1 scheme. The structure of this manuscript is organised as
follows. We present a detailed description of the model in section II,
followed by the numerical analysis of the model in section III. Results of the
analysis is presented in section IV. We conclude with a brief summary and
discussion in section V.

Parameter | Normal Hierarchy (best-fit$\pm 1\sigma$) | Inverted Hierarchy (best-fit$\pm 1\sigma$)
---|---|---
$|\Delta m^{2}_{21}|:[10^{-5}eV^{2}]$ | 6.82 – 8.03 $(7.41^{+0.21}_{-0.20})$ | 6.82 – 8.03 $(7.41^{+0.21}_{-0.20})$
$|\Delta m^{2}_{31}|:[10^{-3}eV^{2}]$ | 2.428 – 2.597 $(2.511^{+0.028}_{-0.027})$ | 2.408 – 2.581 $(2.498^{+0.032}_{-0.025})$
$\sin^{2}\theta_{12}$ | 0.270 – 0.341 $(0.303^{+0.012}_{-0.011})$ | 0.270 – 0.341 $(0.303^{+0.012}_{-0.011})$
$\sin^{2}\theta_{23}$ | 0.406 – 0.620 $(0.572^{+0.018}_{-0.023})$ | 0.412 – 0.623 $(0.578^{+0.016}_{-0.021})$
$\sin^{2}\theta_{13}/10^{-2}$ | 2.029 – 2.391 $(2.203^{+0.056}_{-0.059})$ | 2.047 – 2.396 $(2.219^{+0.060}_{-0.057})$
$\delta_{\rm CP}/^{o}$ | 108 - 404 $(197^{+42}_{-0.25})$ | 192 - 360 $(286^{+27}_{-32})$
$r=\sqrt{\frac{\Delta m_{21}^{2}}{|\Delta m_{3l}^{2}|}}$ | 0.1675 - 0.1759 (0.1718) | 0.1683 - 0.1765 (0.1722)
$|U_{14}|^{2}$ | 0.012 - 0.047 | 0.012 - 0.047
$|U_{24}|^{2}$ | 0.005 - 0.03 | 0.005 - 0.03
$|U_{34}|^{2}$ | 0 - 0.16 | 0 - 0.16
Table 1: Updated global-fit data for three neutrino oscillation, $Nufit$ 2022
[45]. For 3+1 mixing, data taken from [46, 43, 47].

##  II Description of the model

In this model, we use $A_{4}$ modular symmetry for studying lepton masses and
mixings. We consider three right-handed neutrinos $N_{i}$ as triplet under
$A_{4}$ and a sterile neutrino field $S$ as singlet. The SM lepton doublet $L$
is also transformed as triplet of $A_{4}$ while the Higgs $H_{u}$ and $H_{d}$
having hypercharges $+1/2$ and $-1/2$ respectively are represented at
singlets. The right-handed charged leptons $e_{r},\mu_{r}$ and $\tau_{r}$ are
assigned $1,1^{\prime\prime},1^{\prime}$ of $A_{4}$ respectively. As a result,
there are three independent coupling constants
$\alpha^{\prime},\beta^{\prime}$ and $\gamma^{\prime}$ in the superpotential
of the charged lepton sector. We have considered one scalar field $\zeta$ as
an $A_{4}$ singlet in order to generate sterile neutrino mass matrix $M_{s}$.
Finally, the Yukawa coupling is transformed as a modular function
$Y_{i}(\tau)$ which is triplet under $A_{4}.$ The role of triplet scalar field
$\phi$ with zero modular weight is to simplify our analysis by making the
charged lepton mass matrix diagonal. It does not have any significance in the
neutrino sector. The complete particle contents with their corresponding group
charges and modular weights are given in Table 2. The modular forms
$Y_{i}(\tau)$ of weight 2 which transform as triplet under $A_{4}$ can be
expressed in terms of Dedekind eta-function $\eta(\tau)$ as [15]

$\displaystyle y_{1}=$
$\displaystyle\frac{i}{2\pi}\left[\frac{\eta^{\prime}(\frac{\tau}{3})}{\eta(\frac{\tau}{3})}+\frac{\eta^{\prime}(\frac{\tau+1}{3})}{\eta(\frac{\tau+1}{3})}+\frac{\eta^{\prime}(\frac{\tau+2}{3})}{\eta(\frac{\tau+2}{3})}-\frac{27\eta^{\prime}(3\tau)}{\eta(3\tau)}\right]$
$\displaystyle y_{2}=$
$\displaystyle\frac{-i}{\pi}\left[\frac{\eta^{\prime}(\frac{\tau}{3})}{\eta(\frac{\tau}{3})}+\omega^{2}\frac{\eta^{\prime}(\frac{\tau+1}{3})}{\eta(\frac{\tau+1}{3})}+\omega\frac{\eta^{\prime}(\frac{\tau+2}{3})}{\eta(\frac{\tau+2}{3})}\right]$
(1) $\displaystyle y_{3}=$
$\displaystyle\frac{-i}{\pi}\left[\frac{\eta^{\prime}(\frac{\tau}{3})}{\eta(\frac{\tau}{3})}+\omega\frac{\eta^{\prime}(\frac{\tau+1}{3})}{\eta(\frac{\tau+1}{3})}+\omega^{2}\frac{\eta^{\prime}(\frac{\tau+2}{3})}{\eta(\frac{\tau+2}{3})}\right]$

where $\omega=e^{2\pi i/3}$ and $\eta(\tau)$ is defined as

$\eta(\tau)=q^{1/24}\prod_{n=1}^{\infty}(1-q^{n}),\ \ \ \ q\equiv
e^{i2\pi\tau}.$ (2)

The overall coefficient in eq.(1) is one possible choice and it cannot be
determined.

$\frac{Fields}{Charges}$ |  $L$ |  $e_{r},\mu_{r},\tau_{r}$ |  $H_{u,d}$ |  $N_{i}$ |  $S$ |  $\phi$ |  $\zeta$ |  $Y$
---|---|---|---|---|---|---|---|---
$SU(2)_{L}$ |  2 |  1 |  2 |  1 |  1 |  1 |  1 |  1
$U(1)_{Y}$ |  -1/2 |  1 |  $\pm$ 1/2 |  0 |  -1 |  -1 |  +1 |  0
$A_{4}$ |  3 |  1,1′′,1′ |  1 |  3 |  1 |  3 |  1 |  3
$k_{i}$ |  $k_{L}$ |  $k_{e}$ |  $k_{H}$ |  $k_{N}$ |  $k_{S}$ |  $k_{\phi}$ |  $k_{\zeta}$ |  $k_{Y}$
Table 2: Particle contents of the model and their group charges.

The modular invariant Lagrangian of the charged lepton is given below.

$\displaystyle\mathcal{L}_{c}=\
\alpha^{\prime}e_{r}(H_{d}\bar{L}\phi)_{1}+\beta^{\prime}\mu_{r}(H_{d}\bar{L}\phi)_{1^{\prime\prime}}+\gamma^{\prime}\tau_{r}(H_{d}\bar{L}\phi)_{1^{\prime}}$
(3)

The charged lepton masses can be reproduced by adjusting the values the
coefficients $\alpha^{\prime},\beta^{\prime}$ and $\gamma^{\prime}$. Choosing
the vev of $\phi$ along $(1,0,0)$, eq.(3) gives a diagonal charged lepton mass
matrix

$M_{L}=diag(\alpha^{\prime},\beta^{\prime},\gamma^{\prime})\ v_{\phi}v_{d}$
(4)

For the neutrino sector, the invariant Lagrangian is

$\displaystyle\mathcal{L}_{D}=$ $\displaystyle\ gH_{u}\bar{L}YN_{i}$ (5)
$\displaystyle=$ $\displaystyle\
g_{1}H_{u}(\bar{L}Y)_{3S}N_{i}+g_{2}H_{u}(\bar{L}Y)_{3A}N_{i}$
$\displaystyle\mathcal{L}_{R}=$ $\displaystyle\ \lambda YN_{i}^{c}N_{i}$ (6)
$\displaystyle\mathcal{L}_{s}=$ $\displaystyle\ \delta\zeta YS^{c}N_{i}$ (7)

The weights $k_{i}$ should satisfy the following conditions,

$\displaystyle k_{e}+k_{H}+k_{L}+k_{\phi}=0,\ \ k_{H}+k_{L}+k_{N}+k_{Y}=0,$
$\displaystyle k_{N}+k_{N}+k_{Y}=0,\ \ \ \ \ k_{\zeta}+k_{S}+k_{N}+k_{Y}=0$
(8)

For $-k_{Y}=2$, we get

$k_{N}=k_{L}=1,\ k_{\phi}=k_{H}=0,k_{S}=2,\ k_{\zeta}=-1,\ \mbox{and}\
k_{e}=-1$ (9)

The neutrino mass matrices along with the v.e.v $\langle H_{u}\rangle=v_{u}$
and $\langle\zeta\rangle=v_{\zeta}$ are

$\displaystyle M_{D}=$ $\displaystyle\
v_{u}\left(\begin{array}[]{ccc}2g_{1}y_{1}&y_{3}(g_{2}-g_{1})&y_{2}(-g_{1}-g_{2})\\\
y_{3}(-g_{1}-g_{2})&2g_{1}y_{2}&y_{1}(g_{2}-g_{1})\\\
y_{2}(g_{2}-g_{1})&y_{1}(-g_{1}-g_{2})&2g_{1}y_{3}\\\ \end{array}\right);$
$\displaystyle M_{R}=$ $\displaystyle\
\lambda\left(\begin{array}[]{ccc}2y_{1}&-y_{3}&-y_{2}\\\
-y_{3}&2y_{2}&-y_{1}\\\ -y_{2}&-y_{1}&2y_{3}\\\ \end{array}\right);$
$\displaystyle M_{s}=$ $\displaystyle\ \delta
v_{\zeta}\left(\begin{array}[]{ccc}y_{1}&y_{2}&y_{3}\\\ \end{array}\right).$

In the 3+1 MES mechanism, the active neutrino mass and the sterile neutrino
mass are calculated using the following relations [14]

$\displaystyle m_{\nu}\simeq$ $\displaystyle\
M_{D}M_{R}^{-1}M_{S}^{T}\left(M_{S}M_{R}^{-1}M_{S}^{T}\right)^{-1}M_{S}\left(M_{R}^{-1}\right)^{T}M_{D}^{T}-M_{D}M_{R}^{-1}M_{D}^{T}\
;$ (10) $\displaystyle m_{s}\simeq$ $\displaystyle-M_{S}M_{R}^{-1}M_{S}^{T}.$
(11)

The $(4\times 4)$ active-sterile mass matrix is diagonalised by a unitary
$(4\times 4)$ mixing matrix given by [48]

$V\simeq\left(\begin{matrix}(1-\frac{1}{2}RR^{\dagger})U&R\\\
-R^{\dagger}U&1-\frac{1}{2}R^{\dagger}R\end{matrix}\right),$ (12)

where $R$ represents the strength of active-sterile mixing given by

$\displaystyle R=$ $\displaystyle
M_{D}M_{R}^{-1}M_{S}^{T}(M_{S}M_{R}^{-1}M_{S}^{T})^{-1}$ (13)

Deviation of the active neutrino mixing matrix $U$ from unitarity due to the
presence of sterile neutrino is determined by $\frac{1}{2}RR^{\dagger}$. The
$4\times 4$ neutrino mixing matrix can be parameterized by six mixing angles
$(\theta_{12},\theta_{13},\theta_{23},\theta_{14},\theta_{24},\theta_{34})$,
three Dirac phases $(\delta_{13},\delta_{14},\delta_{24})$ and three Majorana
phases $(\alpha,\beta,\gamma)$ [49] as,

$V^{4\times
4}=\left(\begin{matrix}c_{12}c_{13}c_{14}&c_{13}c_{14}s_{12}e^{i\frac{\alpha}{2}}&c_{14}s_{13}e^{i\frac{\beta}{2}}&s_{14}e^{-i\frac{\gamma}{2}}\\\
U_{\mu 1}&U_{\mu 2}&U_{\mu
3}&c_{14}s_{24}e^{-i\left(\frac{\gamma}{2}-\delta_{14}+\delta_{24}\right)}\\\
U_{\tau 1}&U_{\tau 2}&U_{\tau
3}&c_{14}c_{24}s_{34}e^{-i\left(\frac{\gamma}{2}-\delta_{14}\right)}\\\
U_{s1}&U_{s2}&U_{s3}&c_{14}c_{24}c_{34}e^{-i\left(\frac{\gamma}{2}-\delta_{14}\right)}\end{matrix}\right).$
(14)

The six neutrino mixing angles can be determined from the mixing elements of
$V$ using the following relations [50],

$\displaystyle\sin^{2}\theta_{14}\ $ $\displaystyle=\ |V_{e4}|^{2},\ \
\sin^{2}\theta_{24}\ =\ \frac{|V_{\mu 4}|^{2}}{1-|V_{e4}|^{2}},\ \
\sin^{2}\theta_{34}\ =\ \frac{|V_{\tau 4}|^{2}}{1-|V_{e4}|^{2}-|V_{\mu
4}|^{2}}$ $\displaystyle\sin^{2}\theta_{12}\ $ $\displaystyle=\
\frac{|V_{e2}|^{2}}{1-|V_{e4}|^{2}-|V_{e3}|^{2}},\ \ \sin^{2}\theta_{13}\ =\
\frac{|V_{e3}|^{2}}{1-|V_{e4}|^{2}},$ (15) $\displaystyle\sin^{2}\theta_{23}\
=$ $\displaystyle\ \frac{|V_{e3}|^{2}(1-|V_{e4}|^{2})-|V_{e4}|^{2}|V_{\mu
4}|^{2}}{1-|V_{e4}|^{2}-|V_{\mu 4}|^{2}}+\frac{|V_{e1}V_{\mu 1}+V_{e2}V_{\mu
2}|^{2}(1-|V_{e4}|^{2})}{(1-|V_{e4}|^{2}-|V_{e3}|^{2})(1-|V_{e4}|^{2}-|V_{\mu
4}|^{2})}.$

where $V_{ij}$ are the elements of mixing matrix in eq.(12). We also try do
determine the Jarlskog invariant $J$ for the active sterile mixing. According
to the parameterisation in eq.(14), the Jarlskog invariant
$J_{3+1}=Im[V_{e1}V_{\mu 2}V^{*}_{e2}V^{*}_{\mu 1}]$ takes the form [51]

$J_{3+1}=J_{3}^{cp}c_{14}^{2}c_{24}^{2}+s_{24}s_{14}c_{24}c_{23}c^{2}_{14}c^{3}_{13}c_{12}s_{12}\sin(\delta_{14}-\delta_{24}),$
(16)

where $J_{3}^{cp}=s_{23}c_{23}s_{12}c_{12}s_{13}c_{13}^{2}\sin\delta_{13}$ is
the Jarlskog invariant for the three neutrino framework and
$s_{ij}=\sin\theta_{ij},c_{ij}=\cos\theta_{ij}$ are the mixing angles. The two
physical Majorana phases $\alpha$ and $\beta$ are determined from $V^{4\times
4}$ using the invariants $I_{1}$ and $I_{2}$ defined as follows

$\displaystyle I_{1}=Im[V_{e1}^{*}V_{e2}]\ =\
c_{12}c_{13}^{2}c_{14}^{2}s_{12}\sin\left(\frac{\alpha}{2}\right),$ (17)
$\displaystyle I_{2}=Im[V_{e1}^{*}V_{e3}]\ =\
c_{12}c_{13}c_{14}^{2}s_{13}\sin\left(\frac{\beta}{2}-\delta_{13}\right).$
(18)

Another important parameter in neutrino physics is the effective neutrino mass
$m_{\beta\beta}$ and effective electron mass $m_{\beta}$. A combined analysis
from KAMLand-Zen[52] and GERDA provided an upper bound on $m_{\beta\beta}$ in
the range $m_{\beta\beta}<(0.071-0.161$) eV [53, 54]. Recent result from
latest KATRIN [55] experiment constrains the effective electron neutrino mass
$m_{\beta}$ to be less than 1.1 eV. These parameters are determined from the
neutrinoless double beta decay and beta decay respectively using the relations
[56]

$m_{\beta\beta}=|\sum_{j=1}^{4}|V_{ej}|^{2}m_{j}|,$ (19)

$m_{\beta}=\left(\sum_{i=1}^{4}|V_{ei}|^{2}m_{i}^{2}\right)^{1/2}.$ (20)

Figure 1: Variation plot for NH showing the dependence of Yukawa
($y_{1},y_{2},y_{3}$) among themselves in (a). Relation of $y_{3}$ with
Re[$\tau$] is shown in (b) while (c) shows the variation of $y_{1},y_{2}$ with
Re[$\tau$]. The dependence of Im[$\tau$] with ($y_{1},y_{2},y_{3}$) is shown
in (d).

##  III Numerical analysis

In this section, we carry out the detailed numerical analysis to determine the
allowed regions of the free parameters in the model, which satisfy the current
neutrino oscillation data. The lepton mass matrices in eq.(3) and eq.(10)
depend on the effective parameters
$\alpha^{\prime},\beta^{\prime},\gamma^{\prime},g_{1},g_{2}$ and $\tau.$
Parameters $\alpha^{\prime},\beta^{\prime},\gamma^{\prime}$ in the charged
lepton sector can be taken as real and they are determined by the ratios of
the charged lepton masses $m_{e}/m_{\tau}$ and $m_{\mu}/m_{\tau}$. On fixing
$\tau$, the modular symmetry is broken and the neutrino mass eigenvalue,
mixing angles and Dirac as well as Majorana phases are completely determined.
We define the ratio of neutrino mass squared differences $r=\sqrt{\Delta
m_{21}^{2}/\Delta m_{31}^{2}}=m_{2}/m_{3}$ for NH $(m_{1}\approx
0<<m_{2}<m_{3}<m_{4})$ and $r=\sqrt{\Delta m_{21}^{2}/|\Delta
m_{32}^{2}|}=\sqrt{1-\frac{m_{1}^{2}}{m_{2}^{2}}}$ for IH $(m_{3}\approx
0<<m_{1}<m_{2}<m_{4})$. The absolute scale of active neutrino masses can be
fixed by adjusting the overall factor $v_{u}^{2}g_{1}/\lambda.$ The neutrino
mixing angles and $r$ depend on only two free parameters $g_{2}/g_{1}$ and the
modulus $\tau.$ Since these parameters are complex, we consider

$\tau=Re[\tau]+i\ Im[\tau],\ \ \frac{g_{2}}{g_{1}}=ge^{i\phi_{g}}$ (21)

The fundamental domain of $\tau$ is given in Ref.[15]. We randomly scan the
parameters in the range Re$[\tau]=[-1.5,1.5]$ for both ordering. Further, we
take $\ g=[1,2]$ and Im$[\tau]=[1,1.5]$ for NH while $g=[1.5,2]$,
Im$[\tau]=[0.6,1]$ for IH . We also consider $\phi_{g}=[-\pi,\pi]$. The active
neutrino mass matrix is numerically diagonalised using the relation
$m_{\nu}=Um_{\nu}^{d}U^{\dagger}$, where
$m_{\nu}^{d}=diag(m_{1},m_{2},m_{3})$, and $m_{1},m_{2},m_{3}$ are the
neutrino mass eigenvalues. The mixing angles can be calculated using the
general formula given in eq.(II). We filter the allowed values of the model
parameters using the 3$\sigma$ bounds of the three mixing angles and ratio $r$
given in Table 1. For the sterile sector, the coefficient $k\equiv\delta
v_{\zeta}$ is solved by constraining the sterile neutrino mass $m_{s}$ in the
range $[0.8,10]$eV. Finally, the best-fit values of the neutrino observables
and the corresponding best-fit values of model parameters $\tau$ and $g$ are
evaluated using the $\chi^{2}$ analysis. We use the $\chi^{2}$ function given
by

$\chi^{2}(x_{i})=\sum_{j}\left(\frac{y_{j}(x_{i})-y_{j}^{bf}}{\sigma_{j}}\right)^{2}$
(22)

where $x_{i}$ are the free parameters in the model and $j$ is summed over the
observables
$\\{\sin^{2}\theta_{12},\sin^{2}\theta_{13},\sin^{2}\theta_{23},r\\}$. Here,
$y_{j}(x_{i})$ denotes the model predictions for the observables and
$y_{j}^{bf}$ are their best-fit values obtained from the global analysis.
$\sigma_{j}$ denotes the corresponding uncertainties obtained by symmetrizing
$1\sigma$ range of the neutrino observables given in Table 1. By minimizing
the overall $\chi^{2}$ function, we can calculate the best-fit values of our
model parameters and predict the values of neutrino observables.

Figure 2: (a) and (b) show the dependence of Yukawa ($y_{1},y_{2},y_{3}$) with
Im[$\tau$] and Re[$\tau$] respectively for IH.

##  IV Results

The range of Yukawa couplings which satisfy all the neutrino observables are
shown in Fig.(1)-(2) as variation plot with Re[$\tau$] and Im[$\tau$]
respectively for NH and IH respectively. Values of $|y_{1}|$ and $|y_{2}|$ lie
very close to each other in IH at around $0.98$ while all the couplings are
well separated in case of NH. It is observed that the couplings lie in the
ranges $0.984\leq|y_{1}(\tau)|\leq 0.989$, $0.593\leq|y_{2}(\tau)|\leq 0.662$,
$0.178\leq|y_{3}(\tau)|\leq 0.222$ for NH. The allowed values of Im$(\tau)$
continuously vary in the range $(1.048-1.102)$ while the values of Re$(\tau)$
are concentrated at specific regions around $\pm 0.5$ and $\pm 1.5$ in the
fundamental domain of $\tau$ as evident from Fig.(1): (b),(c). Predicted
values of neutrino mixing angles $\sin^{2}\theta_{13}$ are plotted with
$\sin^{2}\theta_{12}$ and $\sin^{2}\theta_{23}$ for both orderings in Fig.(3).
From these plots, the values of $\sin^{2}\theta_{23}$ are concentrated at
regions greater than $0.55$ implying the higher octant of $\theta_{23}$. Plots
between individual neutrino masses $m_{2},m_{3}$ for NH ($m_{1}=0$) and
$m_{1},m_{2}$ for IH ($m_{3}=0$) with sum of active neutrino masses $\sum
m_{i}$ are shown respectively in Fig.(4). Choosing the factor
$v_{u}^{2}g_{1}/\lambda\sim\mathcal{O}(10^{-2})$, we observe that the sum of
active neutrino masses $\sum m_{i}$ satisfies the Planck upper bound $\sum
m_{i}<0.12$eV. For NH, sum of active neutrino masses is observed in a very
narrow range $\sum m_{i}\sim(0.0145-0.045)$eV while it is $\sum
m_{i}\sim(0.083-0.121)$eV for IH. The individual neutrino masses are found in
the range $m_{2}\sim(0.00214-0.00658)$eV, $m_{3}\sim(0.0124-0.0382)$eV for NH
and $m_{1}\sim(0.041-0.06)$eV, $m_{2}\sim(0.042-0.061)$eV for IH. For the
mixings between active and sterile neutrino, we have plotted the active-
sterile mixing elements $|V_{14}|,|V_{24}|$ and $|V_{34}|$ in Fig.(7) as a
function of sterile neutrino mass $m_{s}$. It is evident from these plots that
the active-sterile mixing elements satisfy the observed bounds given in
Table.2. Variation of active-sterile mixing angles $\sin^{2}\theta_{i4}$
($i=1,2,3$) with parameter $k$ is shown in Fig.(6) for both ordering. Fig.(5)
shows the relation between the mixing angle $\sin^{2}\theta_{14}$ with
$\sin^{2}\theta_{24}$ and $\sin^{2}\theta_{34}$ for NH. The non-unitarity
effect due to the presence of active-sterile mixing is determined by
$\frac{1}{2}RR^{\dagger}$. For our analysis it is observed to be less than
$\mathcal{O}(10^{-3}).$ It is also important to note from the plots shown
above that the number of data points in case of NH is much larger compared to
IH. Thus, we can infer that our model favours NH.

Figure 3:  Plot between mixing angle $\sin^{2}\theta_{13}$ with
$\sin^{2}\theta_{23}$ and $\sin^{2}\theta_{12}$ for NH in (a) and IH in (b).

Figure 4:  Variation between neutrino masses $m_{2}$ and $m_{3}$ ($m_{1}=0$)
with sum of active neutrino masses $\sum m_{i}$ for NH is shown in (a). For
IH, (b) shows plot between $m_{1}$ and $m_{2}$ ($m_{3}=0$) with $\sum m_{i}$.
The vertical dotted line is the cosmological Planck upper bound $\sum
m_{i}<0.12$eV.

Proceeding further, we have calculated the model prediction of the Jarlskog
invariant in the 3+1 scenario. Fig.(8)(a) shows the variation of mixing angles
with $J_{3+1}$ for NH. Prediction of Dirac CP-violating phase $\delta_{13}$ is
shown in Fig.(9)(a). Majorana phases $\alpha$ and $\beta$ are also evaluated
using eq.(17) - (18) and the predictions from our model are shown in
Fig.(9)(b). Unknown phases for the active-sterile sector $\delta_{14}$ and
$\delta_{24}$ are randomly chosen in the range $[-\pi,\pi]$. For the case of
IH, the variation of active neutrino mixing angles with Jarlskog invariant
$J_{3+1}$ is shown in Fig.(8)(b). The value of $J_{3+1}$ is observed in the
vicinity of $\pm 0.06$ which is out of the bounds for $J_{max}^{CP}=0.0332\pm
0.0008$ at 1$\sigma$ for both orderings [45].

Other phenomenological studies on neutrinoless double beta decay experiments
are also carried out. The effective mass parameter $m_{\beta\beta}$ and
effective electron mass $m_{\beta}$ are determined using eq.(19)-(20).
Fig.(10) shows the variation of $m_{\beta\beta}$ and $m_{\beta}$ as a function
of $\sum m_{i}$ for NH. It is observed that these effective mass parameters
lie in the range $0.03$ eV$\leq m_{\beta}\leq 0.18$ eV and $0.001$ eV $\leq
m_{\beta\beta}\leq 0.006$ eV. In case of IH shown in Fig.(11), the effective
parameters are in range $0.179$ eV$\leq m_{\beta}\leq 0.413$ eV and $0.005$ eV
$\leq m_{\beta\beta}\leq 0.017$ eV. Values of $m_{\beta\beta}$ and $m_{\beta}$
are observed to be comparatively larger in case of IH.

Figure 5:  Plot between active-sterile mixing angle $\sin^{2}\theta_{14}$ with
$\sin^{2}\theta_{24}$ and $\sin^{2}\theta_{34}$ for NH.

Figure 6:  Variation between active-sterile mixing angle $\sin^{2}\theta_{i4}$
where $i=1,2,3$ with sterile neutrino mass coefficient $k=\delta v_{\zeta}$
for NH in (a) and IH in (b).

Figure 7:  Dependence of active-sterile mixing elements $V_{i4}$ where
$i=1,2,3$ on sterile neutrino mass $m_{s}$ for NH in (a) and IH in (b).

Figure 8:  (a). Plot between Jarlskog invariant $J_{3+1}$ in the active-
sterile sector with active neutrino mixing angles $\sin^{2}\theta_{12}$ and
$\sin^{2}\theta_{23}$ for NH. (b). Variation of $J_{3+1}$ with
$\sin^{2}\theta_{13},\sin^{2}\theta_{12}$ and $\sin^{2}\theta_{23}$ for IH.

Figure 9: (a). Variation of CP-violating Dirac phase $\delta_{13}$ with
$\sin^{2}\theta_{23}$ for NH. (b). Plot between the two Majorana phases
$\alpha$ and $\beta$ for NH.

Finally, the best-fit values of the model parameters and predictions of
neutrino observables are evaluated using $\chi^{2}$ analysis as shown in
eq.(22). For NH at $\chi^{2}_{min}=1.20$, the best-fit of the parameter $\tau$
is obtained at Re$[\tau]=-0.45$, Im$[\tau]=1.07$ and $g=1.49$. The
corresponding values of Yukawa are found to be
$|y_{1}|=0.987,|y_{2}|=0.644,|y_{3}|=0.197$. The best-fit values of neutrino
observables are obtained at $\sin^{2}\theta_{23}=0.572$,
$\sin^{2}\theta_{12}=0.313$, $\sin^{2}\theta_{13}=0.022$ and $r=0.172$.
Similarly, for IH at $\chi^{2}_{min}=3.54$, the values of the model parameters
are found as Re$[\tau]=0.35$, Im$[\tau]=0.87$ and $g=1.89$, while the Yukawa
are found to be $|y_{1}|=0.969,|y_{2}|=0.951,|y_{3}|=0.466$. The corresponding
best-fit values of the neutrino observables are $\sin^{2}\theta_{23}=0.602,$
$\sin^{2}\theta_{12}=0.288,$ $\sin^{2}\theta_{13}=0.022$ and $r=0.172$.

For the charged lepton sector, by comparing eq.(3) with the experimental
values for masses of the charged leptons given in Ref.[57], $m_{e}=0.51099$
MeV, $m_{\mu}=105.65837$ MeV and $m_{\tau}=1776.86$ MeV, the parameters
$\alpha^{\prime},\beta^{\prime},\gamma^{\prime}$ are determined to be

$\frac{\alpha^{\prime}}{\gamma^{\prime}}=0.287\times 10^{-2},\ \ \ \ \
\frac{\beta^{\prime}}{\gamma^{\prime}}=0.059.$ (23)

Figure 10:  Dependence of (a). Effective electron mass $m_{\beta}$ with $\sum
m_{i}$ and (b). Effective neutrino mass $m_{\beta\beta}$ from $\nu
0\beta\beta$ with $\sum m_{i}$ for NH.

Figure 11:  Dependence of (a). Effective electron mass $m_{\beta}$ with $\sum
m_{i}$ and (b). Effective neutrino mass $m_{\beta\beta}$ from $\nu
0\beta\beta$ with $\sum m_{i}$ for IH.

##  V Summary and Discussion

We have successfully constructed a new neutrino model based on modular $A_{4}$
by extending the SM with an $A_{4}$ triplet right-handed neutrino $N$ and a
singlet sterile neutrino in a 3+1 scheme. MES mechanism is imposed to generate
the active and eV-scale sterile neutrino masses in both NH and IH. The main
motivation is to avoid hypothetical scalar flavons using modular symmetry and
at the same time, reproduce all the neutrino observables through the vev of a
single parameter $\tau.$ However, to simplify our analysis, we have used a
triplet scalar $\phi$ whose vev makes the charged lepton mass matrix diagonal.
Two free model parameters $\tau$ and $g$ are scanned randomly in a particular
domain and active neutrino mass matrix is numerically diagonalised. We have
conducted the numerical analysis using the 3$\sigma$ bounds of neutrino
observables in such a way that all the neutrino observables evaluated from the
model simultaneously satisfy these bounds. Our analysis of neutrino mass is
also consistent with the cosmological upper bound on the sum of neutrino
masses $\sum m_{i}<0.12$ eV. Effects of the eV-scale sterile neutrino on
neutrino mixing angles, effective mass parameters $m_{\beta},m_{\beta\beta}$
and unitarity of the active neutrino mixing matrix are studied in detail.
Neutrino mixing angles are evaluated from the $(4\times 4)$ active-sterile
mixing matrix $V$ without avoiding the non-unitarity effects of sterile
neutrino. Active-sterile mixing angles $\sin^{2}\theta_{14}$,
$\sin^{2}\theta_{24}$ and $\sin^{2}\theta_{34}$ are observed to be
$0.000144\leq\sin^{2}\theta_{14}\leq 0.00236$,
$0.000133\leq\sin^{2}\theta_{24}\leq 0.00394$ and
$0.00023\leq\sin^{2}\theta_{34}\leq 0.00796$ for NH. In case of IH, these
mixing angles are observed in the range $0.001032\leq\sin^{2}\theta_{14}\leq
0.002937$, $0.00172\leq\sin^{2}\theta_{24}\leq 0.00359$ and
$0.00248\leq\sin^{2}\theta_{34}\leq 0.00606$. Jarlskog invariant in the 3+1
sector is also determined. CP-violating Dirac phase is successfully predicted
for NH in the ranges $\delta_{13}\sim(180.12^{o}-230.70^{o})$ and
$\delta_{13}\sim(305.15^{o}-358.83^{o})$. Finally, we have used the minimum
$\chi^{2}$ analysis to predict the best-fit values of the model parameters as
well as the neutrino oscillation data. From our analysis, we can conclude that
NH is more favoured in our model compared to IH. The MES mechanism has an
advantage in generating keV-MeV scale sterile neutrino also along with the
active neutrino mass in the eV scale. There are other works which study the
possibility of keV-MeV sterile neutrino to be a dark matter candidate which we
shall leave for future study. To conclude, we emphasize that modular $A_{4}$
is very successful in reproducing neutrino phenomenology and other beyond
Standard Model problems without the need for extra flavons as in conventional
discrete symmetry models.

## Acknowledgements

One of the authors(MKS) would like to thank DST-INSPIRE, govt. of India for
providing the fellowship for the research under INSPIRE fellowship (ID
IF180349).

## References

  * [1] Guido Altarelli and Ferruccio Feruglio.  Discrete flavor symmetries and models of neutrino mixing.  Reviews of Modern Physics, 82(3):2701, 2010.
  * [2] Stephen F King and Michal Malinskỳ.  A4 family symmetry and quark–lepton unification.  Physics Letters B, 645(4):351–357, 2007.
  * [3] Kaladi S Babu, Ernest Ma, and JWF Valle.  Underlying a4 symmetry for the neutrino mass matrix and the quark mixing matrix.  Physics Letters B, 552(3-4):207–213, 2003.
  * [4] Ernest Ma.  Neutrino mass matrix from s4 symmetry.  Physics Letters B, 632(2-3):352–356, 2006.
  * [5] Guido Altarelli, Ferruccio Feruglio, and Luca Merlo.  Revisiting bimaximal neutrino mixing in a model with s4 discrete symmetry.  Journal of High Energy Physics, 2009(05):020, 2009.
  * [6] Federica Bazzocchi and Luca Merlo.  Neutrino mixings and the s4 discrete flavour symmetry.  Fortschritte der Physik, 61(4-5):571–596, 2013.
  * [7] Rabindra N Mohapatra, S Nasri, and Hai-Bo Yu.  S3 symmetry and tri-bimaximal mixing.  Physics Letters B, 639(3-4):318–321, 2006.
  * [8] Stefano Morisi and Marco Picariello.  The flavor physics in unified gauge theory from an s 3$\times$ p discrete symmetry.  International Journal of Theoretical Physics, 45:1267–1277, 2006\.
  * [9] Walter Grimus and Luis Lavoura.  S3$\times z2$ model for neutrino mass matrices.  Journal of High Energy Physics, 2005(08):013, 2005.
  * [10] Yoshio Koide.  S 3 symmetry and neutrino masses and mixings.  The European Physical Journal C, 50:809–816, 2007.
  * [11] A Aguilar-Arevalo, LSND collaboration, et al.  Evidence for neutrino oscillations from the observation of anti-neutrino (electron) appearance in a anti-neutrino (muon) beam.  Phys. Rev. D, 64:112007, 2001.
  * [12] AA Aguilar-Arevalo, BC Brown, L Bugel, G Cheng, JM Conrad, RL Cooper, R Dharmapalan, A Diaz, Z Djurcic, DA Finley, et al.  Significant excess of electronlike events in the miniboone short-baseline neutrino experiment.  Physical review letters, 121(22):221801, 2018.
  * [13] CA Argüelles, I Esteban, M Hostert, Kevin J Kelly, J Kopp, PAN Machado, I Martinez-Soler, and YF Perez-Gonzalez.  Microboone and the $\nu$ e interpretation of the miniboone low-energy excess.  Physical Review Letters, 128(24):241802, 2022.
  * [14] He Zhang.  Light Sterile Neutrino in the Minimal Extended Seesaw.  Phys. Lett. B, 714:262–266, 2012.
  * [15] Ferruccio Feruglio.  Are neutrino masses modular forms?  In From My Vast Repertoire… Guido Altarelli’s Legacy, pages 227–266. World Scientific, 2019.
  * [16] Tatsuo Kobayashi, Kentaro Tanaka, and Takuya H Tatsuishi.  Neutrino mixing from finite modular groups.  Physical Review D, 98(1):016004, 2018.
  * [17] Tatsuo Kobayashi, Naoya Omoto, Yusuke Shimizu, Kenta Takagi, Morimitsu Tanimoto, and Takuya H Tatsuishi.  Modular a4 invariance and neutrino mixing.  Journal of High Energy Physics, 2018(11), 2018.
  * [18] Di Zhang.  A modular a4 symmetry realization of two-zero textures of the majorana neutrino mass matrix.  Nuclear Physics B, 952:114935, 2020.
  * [19] Mitesh Kumar Behera, Shivaramakrishna Singirala, Subhasmita Mishra, and Rukmani Mohanta.  A modular A 4 symmetric scotogenic model for neutrino mass and dark matter.  J. Phys. G, 49(3):035002, 2022.
  * [20] Priya Mishra, Mitesh Kumar Behera, Papia Panda, and Rukmani Mohanta.  Type iii seesaw under $a\\_4$ modular symmetry with leptogenesis and muon $g-2$.  arXiv preprint arXiv:2204.08338, 2022.
  * [21] Tatsuo Kobayashi, Takaaki Nomura, and Takashi Shimomura.  Type ii seesaw models with modular a 4 symmetry.  Physical Review D, 102(3):035019, 2020.
  * [22] Hiroshi Okada and Yutaro Shoji.  A radiative seesaw model with three higgs doublets in modular a4 symmetry.  Nuclear Physics B, 961:115216, 2020.
  * [23] Takaaki Nomura and Hiroshi Okada.  A modular a4 symmetric model of dark matter and neutrino.  Physics Letters B, 797:134799, 2019.
  * [24] Mohammed Abbas.  Fermion masses and flavor mixing in modular a 4 symmetry.  Physical Review D, 103(5):056016, 2021.
  * [25] Takaaki Nomura, Hiroshi Okada, and Oleg Popov.  A modular a4 symmetric scotogenic model.  Physics Letters B, 803:135294, 2020.
  * [26] Mitesh Kumar Behera, Subhasmita Mishra, Shivaramakrishna Singirala, and Rukmani Mohanta.  Implications of a4 modular symmetry on neutrino mass, mixing and leptogenesis with linear seesaw.  Physics of the Dark Universe, 36:101027, 2022.
  * [27] Mitesh Kumar Behera and Rukmani Mohanta.  Inverse seesaw in modular symmetry.  Journal of Physics G: Nuclear and Particle Physics, 49(4):045001, 2022.
  * [28] Gui-Jun Ding, Stephen F King, and Xiang-Gan Liu.  Neutrino mass and mixing with a 5 modular symmetry.  Physical Review D, 100(11):115005, 2019.
  * [29] PP Novichkov, JT Penedo, ST Petcov, and AV Titov.  Modular a5 symmetry for flavour model building.  Journal of High Energy Physics, 2019(4):1–34, 2019.
  * [30] Chang-Yuan Yao, Xiang-Gan Liu, and Gui-Jun Ding.  Fermion masses and mixing from the double cover and metaplectic cover of the ${A}_{5}$ modular group.  Phys. Rev. D, 103:095013, May 2021.
  * [31] Xin Wang, Bingrong Yu, and Shun Zhou.  Double covering of the modular ${A}_{5}$ group and lepton flavor mixing in the minimal seesaw model.  Phys. Rev. D, 103:076005, Apr 2021.
  * [32] Xin Wang, Bingrong Yu, and Shun Zhou.  Double covering of the modular a 5 group and lepton flavor mixing in the minimal seesaw model.  Physical Review D, 103(7):076005, 2021.
  * [33] Mitesh Kumar Behera and Rukmani Mohanta.  Inverse seesaw in $A_{5}^{\prime}$ modular symmetry.  J. Phys. G, 49(4):045001, 2022.
  * [34] Xin Wang and Shun Zhou.  The minimal seesaw model with a modular s4 symmetry.  Journal of High Energy Physics, 2020(5):1–38, 2020.
  * [35] JT Penedo and ST Petcov.  Lepton masses and mixing from modular s4 symmetry.  Nuclear Physics B, 939:292–307, 2019.
  * [36] Xin Wang.  Dirac neutrino mass models with a modular s4 symmetry.  Nuclear Physics B, 962:115247, 2021.
  * [37] Gui-Jun Ding, Stephen F King, Xiang-Gan Liu, and Jun-Nan Lu.  Modular s4 and a4 symmetries and their fixed points: new predictive examples of lepton mixing.  Journal of High Energy Physics, 2019(12):1–35, 2019.
  * [38] Tatsuo Kobayashi, Yusuke Shimizu, Kenta Takagi, Morimitsu Tanimoto, and Takuya H. Tatsuishi.  ${A}_{4}$ lepton flavor model and modulus stabilization from ${S}_{4}$ modular symmetry.  Phys. Rev. D, 100:115045, Dec 2019.
  * [39] Xiang-Gan Liu, Chang-Yuan Yao, and Gui-Jun Ding.  Modular invariant quark and lepton models in double covering of ${S}_{4}$ modular group.  Phys. Rev. D, 103:056013, Mar 2021.
  * [40] Pritam Das, Ananya Mukherjee, and Mrinal Kumar Das.  Active and sterile neutrino phenomenology with a4 based minimal extended seesaw.  Nuclear Physics B, 941:755–779, 2019.
  * [41] Mayengbam Kishan Singh, S. Robertson Singh, and N. Nimai Singh.  Active-Sterile Neutrino Masses and Mixings in A4 Minimal Extended Seesaw Mechanism.  Int. J. Theor. Phys., 61(9):228, 2022.
  * [42] VV Vien.  3+ 1 active–sterile neutrino mixing in b-l model for normal neutrino mass ordering.  The European Physical Journal C, 81(5):1–11, 2021.
  * [43] VV Vien.  B- l model with a 4$\times$ z 3$\times$ z 4 symmetry for 3+ 1 active-sterile neutrino mixing.  Journal of Physics G: Nuclear and Particle Physics, 49(8):085001, 2022.
  * [44] Pritam Das, Mrinal Kumar Das, and Najimuddin Khan.  Phenomenological study of neutrino mass, dark matter and baryogenesis within the framework of minimal extended seesaw.  JHEP, 03:018, 2020.
  * [45] Ivan Esteban, Maria Conceptión González-García, Michele Maltoni, Thomas Schwetz, and Albert Zhou.  The fate of hints: updated global analysis of three-flavor neutrino oscillations.  Journal of High Energy Physics, 2020(9):1–22, 2020.
  * [46] James Barry, Werner Rodejohann, and He Zhang.  Light sterile neutrinos: models and phenomenology.  Journal of High Energy Physics, 2011(7):1–28, 2011.
  * [47] S Gariazzo, C Giunti, M Laveder, Y F Li, and E M Zavanin.  Light sterile neutrinos*.  Journal of Physics G: Nuclear and Particle Physics, 43(3):033001, jan 2016.
  * [48] J Schechter and José WF Valle.  Neutrino decay and spontaneous violation of lepton number.  Physical Review D, 25(3):774, 1982.
  * [49] Stefano Gariazzo, C Giunti, M Laveder, YF Li, and EM Zavanin.  Light sterile neutrinos.  Journal of Physics G: Nuclear and Particle Physics, 43(3):033001, 2016.
  * [50] S. Dev, Desh Raj, Radha Raman Gautam, and Lal Singh.  New mixing schemes for (3+1) neutrinos.  Nuclear Physics B, 941:401–424, 2019.
  * [51] Priyanka Kumar and Mahadev Patgiri.  Minimal extended seesaw and group symmetry realization of two-zero textures of neutrino mass matrices.  Nuclear Physics B, 957:115082, 2020.
  * [52] S. Abe et al.  Precision measurement of neutrino oscillation parameters with kamland.  Phys. Rev. Lett., 100:221803, Jun 2008.
  * [53] M Agostini, AM Bakalyarov, M Balata, I Barabanov, L Baudis, C Bauer, E Bellotti, S Belogurov, Alice Bettini, L Bezrukov, et al.  Improved limit on neutrinoless double-$\beta$ decay of ge 76 from gerda phase ii.  Physical review letters, 120(13):132503, 2018.
  * [54] Srubabati Goswami, Ananya Mukherjee, Nimmala Narendra, et al.  Leptogenesis and ev scale sterile neutrino.  Physical Review D, 105(9):095040, 2022.
  * [55] Max Aker, Konrad Altenmüller, Marius Arenz, Woo-Jeong Baek, John Barrett, Armen Beglarian, Jan Behrens, Anatoly Berlev, Uwe Besserer, Klaus Blaum, et al.  First operation of the katrin experiment with tritium.  The European Physical Journal C, 80(3):1–18, 2020.
  * [56] Steffen Hagstotz, Pablo F. de Salas, Stefano Gariazzo, Sergio Pastor, Martina Gerbino, Massimiliano Lattanzi, Sunny Vagnozzi, and Katherine Freese.  Bounds on light sterile neutrino mass and mixing from cosmology and laboratory searches.  Physical Review D, 104(12), 12 2021.
  * [57] R. L. Workman and Others.  Review of Particle Physics.  PTEP, 2022:083C01, 2022.