Exploring Graph Neural Networks for Indian Legal Judgment Prediction
The burdensome impact of a skewed judges-to-cases ratio on the judicial
system manifests in an overwhelming backlog of pending cases alongside an
ongoing influx of new ones. To tackle this issue and expedite the judicial
process, the proposition of an automated system capable of suggesting case
outcomes based on factual evidence and precedent from past cases gains
significance. This research paper centres on developing a graph neural
network-based model to address the Legal Judgment Prediction (LJP) problem,
recognizing the intrinsic graph structure of judicial cases and making it a
binary node classification problem. We explored various embeddings as model
features, while nodes such as time nodes and judicial acts were added and
pruned to evaluate the model's performance. The study is done while considering
the ethical dimension of fairness in these predictions, considering gender and
name biases. A link prediction task is also conducted to assess the model's
proficiency in anticipating connections between two specified nodes. By
harnessing the capabilities of graph neural networks and incorporating fairness
analyses, this research aims to contribute insights towards streamlining the
adjudication process, enhancing judicial efficiency, and fostering a more
equitable legal landscape, ultimately alleviating the strain imposed by
mounting case backlogs. Our best-performing model with XLNet pre-trained
embeddings as its features gives the macro F1 score of 75% for the LJP task.
For link prediction, the same set of features is the best performing giving ROC
of more than 80%
Mann Khatri, Mirza Yusuf, Yaman Kumar, Rajiv Ratn Shah, Ponnurangam Kumaraguru
# Exploring Graph Neural Networks for Indian Legal Judgment Prediction

###### Abstract

The burdensome impact of a skewed judges-to-cases ratio on the judicial system manifests in an overwhelming backlog of pending cases alongside an ongoing influx of new ones. To tackle this issue and expedite the judicial process, the proposition of an automated system capable of suggesting case outcomes based on factual evidence and precedent from past cases gains significance. This research paper centres on developing a graph neural network-based model to address the Legal Judgment Prediction (LJP) problem, recognizing the intrinsic graph structure of judicial cases and making it a binary node classification problem. We explored various embeddings as model features, while nodes such as time nodes and judicial acts were added and pruned to evaluate the model's performance. The study is done while considering the ethical dimension of fairness in these predictions, considering gender and name biases. A link prediction task is also conducted to assess the model's proficiency in anticipating connections between two specified nodes. By harnessing the capabilities of graph neural networks and incorporating fairness analyses, this research aims to contribute insights towards streamlining the adjudication process, enhancing judicial efficiency, and fostering a more equitable legal landscape, ultimately alleviating the strain imposed by mounting case backlogs. Our best-performing model with XLNet pretrained embeddings as its features gives the macro F1 score of 75% for the LJP task. For link prediction, the same set of features is the best performing giving ROC of more than 80%.

Keywords:Legal NLP Judgment Prediction Graph Neural Networks

## 1 Introduction

Many cases are pending in the Indian judiciary, and the courts face a meager judge-to-case ratio1, which requires a fair, reliable, and automated system to
predict the verdict of a case. One such task developed recently is the Legal Judgement Prediction (LJP). It aims to predict and suggest the judgement decisions of a court case based on facts and aid in how judgements have passed in previous years. With the access and development of large legal datasets, conducting studies on these tasks becomes imperative. In our work, we model the Indian judiciary using graphs due to its inherent inter-connected structure of cases and laws.

We also aim to analyse the effect of time and acts on the outcome of the cases. To ensure a verdict is fair, we check our model for bias to discover how fair the decisions are. Extending the task to link prediction, we check how well the model understands the relationship between two cases by predicting whether an edge exists between their corresponding graph nodes. In the paper, we answer the following four research questions:

**RQ1:** How does a real-world setting like graph neural networks perform on the task of LJP?

**RQ2:** How does the model behave when we prune or add time and act nodes?

**RQ3:** How does the model perform when trained temporally, i.e. trained till a particular period and how well can the model predict an edge between two given nodes?

**RQ4:** How fair are the decisions in the task of LJP?

Following is the summary of the research contributions done in this paper:

1. We employ a graph neural network (GNN) under different embedding settings by adding and removing two different node characteristics, i.e. time and acts.
2. A link prediction task to observe how well the model can predict an edge between two nodes representing a particular case citing another case.
3. A set of temporal experiments to see the effect of time on the training of the model.
4. For fairness, we check how biased the model is while making predictions.

## 2 Literature Review

There has been a great deal of research on the text in the legal domain and various tasks have been suggested, such as prior case retrieval [13], crime classification [23], and judgment prediction [28].

For the LJP challenge, various strategies and corpora have been introduced. In order to get around BERT's input token count restriction for the LJP problem, [2] presented a hierarchical variant of BERT [4]. Using datasets from the Chinese AI and Law Challenge (CAIL2018), [25] deployed a Multi-Perspective Bi-Feedback Network to forecast the corresponding legal accusations, offences, and periods of punishment. On three Chinese datasets (CJO, PKU, and CAIL), [27] used topological multi-task learning on a directed acyclic network to predict charges, including theft, traffic violation, and deliberate homicide.
To predict the charges on a dataset of Criminal Law of the People's Republic of China, [18] suggested an attention-based model given the case's facts and the relevant articles. Similarly, in a few-shot configuration, [11] implemented an attribute-attentive model based on the case's facts. Using a legal reading comprehension technique on a Chinese dataset, [17] predicted the case's outcome. Given the facts and charges on a dataset created from documents of the Supreme People's Court of China, [3] used a deep gating network to predict prison terms. [1] employed a linear support vector machine (SVM) to predict violations based on the facts of cases from the European Court of Human Rights. [22] implemented SVM in the LJP task on cases from the French Supreme Court. [15] proposed a random forest model to forecast the judges "Reverse", "Affirm", and "Other" judgments in the US Supreme Court.

In their research, [6] proposed a method for representing legal knowledge using logic rules in a co-attention network, which improves interpretability and logical reasoning. They demonstrate the effectiveness of their approach through comprehensive experiments conducted on a civil loan scenario. Similarly, [20] utilizes a real courtroom dataset to predict legal judgments. Using multi-task learning, they extensively analyze multi-role dialogues, including plaintiff's claims and court debate data, to understand facts and discriminate claims for final judgments. The works of [26] introduce NeurJudge, a framework for predicting legal judgments that consider crime circumstances. They leverage intermediate sub-task results to identify and utilize different circumstances for predicting other subtasks.

Another approach by [19] employs LSTM [9] to predict legal judgments by comprehensively understanding case inputs, court debates, and multi-role dialogues. They also utilize multi-task learning to discriminate claims and reach final judgments. [12] propose a unified text-to-text Transformer for LJP, where the auto-regressive decoder's dependencies among sub-tasks can be naturally established. They highlight the advantage of establishing dependencies among sub-tasks.

Furthermore, [24] uses a graph neural network to differentiate confusing charges. They leverage a novel attention mechanism to automatically learn subtle differences between law articles and extract effective discriminative features from fact descriptions. [5] employ a graph neural network (GNN) to address the LJP problem as a node classification task on a global consistency graph derived from the training set. They utilize a masked transformer network for case-aware node representations and leverage relational learning for local consistency through neighbours' label distribution. Variational expectation minimization optimizes both the node encoder and classifier.

[16] introduce MANN, a multichannel attentive neural network model for the integrated LJP task. MANN learns from previous judgment documents and utilizes attention-based neural networks to capture latent feature representations focused on case facts, defendant persona, and relevant law articles. A two-tier structure empowers attentive sequence encoders to hierarchically model semantic interactions at word and sentence levels in the case description.
In their work, [14] introduce the Hindi Legal Documents Corpus (HLDC) consisting of over 900K legal documents in Hindi. They also propose a Multi-Task Learning (MTL) based model incorporating summarization as an auxiliary task alongside the primary task of bail prediction. [21] present ILDC, a vast corpus containing 35k Indian Supreme Court cases annotated with original court decisions. They explore various baseline models for case predictions and propose a hierarchical occlusion-based model to enhance explainability. [7] suggest a moco-based supervised contrastive learning approach to acquire distinguishable representations and determine optimal positive example pairs for all three LJP subtasks. They also enhance fact description representation by incorporating pre-trained numeracy models to utilize crime amounts for predicting penalty terms.

## 3 Experiments

For our experiments, we use graph neural network architecture (GraphSAGE [8]) for node classification where nodes are cases, acts and time nodes. Text embeddings were used as node features and time & act nodes are referred to as characteristic nodes as they are hypothesised to have an impact on the graphical model's performance. Further, to analyse the impact of time on verdicts, we divide the dataset into train and test based on the year up to which we want to train the model. For our experiments, we use the ILDC dataset curated by [21], in which they provide preprocessed cases for the task with their corresponding binary labels. In each case brought before the Supreme Court of India (SCI), the judge or panel determines whether the assertions made by the appellant/petitioner against the respondent should be deemed as "accepted" or "rejected" and accordingly, labels are assigned in the dataset. The dataset already has split examples into train, test and development sets(5082/1517/994). We use the train and development splits as the train split and the test split as itself.

As we employed the graph neural network for our experiments, we had to increase the size of the dataset because the dataset [21] does not provide the cases that were cited by the cases. Using the ikanoon API1, we extracted 24,907 additional cases and added them as nodes to the graph network to complete the citation network. This achieved the semi-supervised setting, enabling message passing between nodes and learning from other cases.

Footnote 1: [https://api.indiankanoon.org/](https://api.indiankanoon.org/)

### Different embeddings

* **Random** We initialize random embeddings as node features to train the model.
* **XLNet** We initialize XLNet embeddings as node features to train the model.
* **XLNet Pretrained** We take the previously trained XLNet model on the task of judgment prediction from [21] and extract embeddings for the nodes to train the model.
* **hierarchical** For the train split in the dataset, we initialize embeddings from XLNet pre-trained on the judgment prediction, and for the test split, we initialize embeddings from XLNet (not pre-trained).

### Edge Types

Another set of experiments included the edge type between two nodes in the graph.

* **Directed:** Given a case, an edge is directed from it to all the cases it cites, enabling message-passing from that case node to the cited case node.
* **Rev-Directed:** Given a case, an edge is directed to it from all the cases it cites, enabling message-passing to the case node from the cited case node. It is the most practical use case as the message has to be passed between various nodes to make the network aware of the legal knowledge and decisions.
* **Undirected:** Given a case, an undirected edge is present between it and the cases it cites, enabling message-passing from that case node to the cited case node and vice versa.

### Time

The dataset contains cases from the year 1956 to the year 2021. To study the impact of time on these verdicts, we added new nodes, called time nodes, in the

\begin{table}
\begin{tabular}{|c|c|c|} \hline
**Year range** & \begin{tabular}{c} **Number of** \\ **Training** \\ **Examples** \\ \end{tabular} & 
\begin{tabular}{c} **Cumulative** \\ **Frequency** \\ \end{tabular} \\ \hline
1956-1960 & 447 & 447 \\ \hline
1961-1965 & 955 & 1402 \\ \hline
1966-1970 & 714 & 2116 \\ \hline
1971-1975 & 728 & 2844 \\ \hline
1976-1980 & 622 & 3466 \\ \hline
1981-1985 & 515 & 3981 \\ \hline
1986-1990 & 547 & 4528 \\ \hline
1991-1995 & 366 & 4894 \\ \hline
1996-2000 & 24 & 4918 \\ \hline
2001-2005 & 10 & 4928 \\ \hline
2006-2010 & 5 & 4933 \\ \hline
2011-2015 & 1 & 4934 \\ \hline
2016-2020 & 1 & 4935 \\ \hline \end{tabular}
\end{table}
Table 1: Number of training examples distributed year-wise with their cumulative frequency.
graph with their connection to the corresponding case in that particular year. The node features of these time nodes are randomly initialised embeddings.

### Acts

Cases come with supporting arguments referencing acts from the Indian constitution to make an argument better. We experiment with the retention and removal of those act nodes in the graph model to observe how the model prediction changes based on underlying act nodes with their connection to the case.

### Simple Training

A graph network is trained on the dataset as specified by the train and test splits [21]. Experimentation involved training with and without characteristic nodes. We recorded the macro precision, recall and F1 scores.

### Temporal Training

To study the temporal aspect of how a verdict is made based on cases and acts present in that particular split and how the law has changed over time. The model is trained on cases for a particular number of years and tested on the rest.

#### 3.6.1 Training direction

* **Forward** We train the model for a particular number of years and then predict future case verdicts. For example, we train the model till 2001 and then predict the verdicts from 2002 to 2021 to check how verdicts present till a particular year affect future cases. We have a range from 1956 to 2021. We tend to predict cases in 5 years from the year of training till the range, i.e. if the model is trained till 1956, predictions are made on cases from 1957-1961, 1962-1966, 1967-1971 and so on till 2021.

\begin{table}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|} \hline \multirow{2}{*}{Embeddings} & \multicolumn{3}{l|}{no change} & \multicolumn{3}{l|}{undirected} & \multicolumn{3}{l|}{rev edges} \\ \cline{2-13}  & \multicolumn{1}{l|}{mn acts} & \multicolumn{1}{l|}{keep acts} & \multicolumn{1}{l|}{mn acts} & \multicolumn{1}{l|}{keep acts} & \multicolumn{1}{l|}{mn acts} & \multicolumn{1}{l|}{keep acts} & \multicolumn{1}{l|}{mn acts} & \multicolumn{1}{l|}{keep acts} \\ \cline{2-13}  & yes time & no time & yes time & no time & yes time & no time & yes time & no time & yes time & no time & yes time & no time \\ \hline vanilla & 58.53 & 57.84 & 59.69 & 57.45 & 59.12 & 58.49 & 58.64 & 58.97 & 58.3 & 58.37 & 59.28 & 59.62 \\ \hline pretrained & **74.81** & **75.14** & **75.0** & **74.98** & **74.85** & **75.35** & **73.86** & **74.46** & **75.35** & **75.3** & **74.49** & **74.22** \\ \hline random & 53.80 & 52.75 & 54.63 & 53.21 & 54.58 & 52.83 & 55.72 & 53.52 & 52.10 & 53.24 & 53.88 & 53.05 \\ \hline hierar & 54.88 & 55.53 & 55.80 & 55.80 & 55.42 & 55.45 & 55.10 & 54.9 & 55.52 & 55.18 & 54.53 & 54.49 \\ \hline \end{tabular}
\end{table}
Table 2: Results from simple training. The table shows the different embeddings used with time and act settings. Pretrained embeddings were the best performing of all the embeddings used in the experiments. Acts and Time nodes were less significant in the model.
* **Reverse** We train the model for a particular number of years and then predict past case verdicts. For example, we train the model from 2001 till 2021 and then predict the verdicts from 1956 to 2000 to check how new laws can comprehend past cases and make good predictions. We tend to predict cases in 5 years from the year of training till the range, i.e. if the model is trained till 2001, predictions are made on years 1996-2000, 1991-1995, 1976-1990 and so on till 1956.

Figure 1: Judgment Prediction Task: Results of temporal training on different embeddings. The thickest line represents an undirected edge, the less thick line represents the rev_edges setting, and the thinnest line represents the no_change setting. The broken line represents that training is done in a reversed manner.
### Link Prediction

Link prediction is predicting if any edge exists between two nodes. We experimented with the task of link prediction based on the temporal aspect of the cases. We used the complete dataset of 24,907 cases from the early 1800s to 2021, divided nodes into train and test, and ran the above experiments to observe how well the link prediction works in the given settings. We plotted the ROC curve Figure 2 of the result.

### Redaction

We redacted gender-biased terms, in our case pronouns (he, him, his, her, she) and name tokens with _[gender]_ and _[REDACTED]_ tokens and fine-tuned the

Figure 2: Link Prediction Task: Results of temporal training on different embeddings. The thickest line represents an undirected edge, the less thick line represents the rev_edges setting, and the thinnest line represents the no_change setting. The broken line represents that training is done in a reversed manner.
XLNet and graph models again on the above-given settings for link prediction. We used Indian legal NER5 to extract name entities.

Footnote 5: [https://huggingface.co/opennyaiorg/en_legal_ner_trf](https://huggingface.co/opennyaiorg/en_legal_ner_trf)

## 4 Results and Discussion

### Simple training

In Table 2, we can observe that the difference in the observations is mainly due to the different features i.e. the text embeddings used. Pretrained embeddings were the top features for the model, followed by vanilla XLNet embeddings,

Figure 3: ROC curves for link prediction task using pre-trained embeddings. Line thickness increases from the earliest to the latest year in the dataset.
hierarchical and random. The type of edges between nodes has minimal effect. Moreover, adding and removing the time nodes and acts did not significantly change the model's performance. We can only observe a minute change in the model's performance when we add time nodes.

### Temporal Training

The trend for all the graphs except for the model with pre-trained embeddings is the same. Figure 1 shows that in the forward training direction, as we keep training the graph on consecutive years, the F1 score of the model keeps increasing as the data increases, resulting in a non-negative slope from a negative slope. Training in the reverse direction gives us a negative slope over the years, indicating that trained on future predictions, the model performs poorly on past verdicts when only the future data is used to train the model. The model with pre-trained features gets roughly the same F1 score for both classes on the test data; see Figure 4. Comparison is also made based on the intersection of test samples in the simple and the created temporal datasets. However, when we only take the test samples from the temporal dataset, we can see that the model has an F1 score in the range of around 80%-85% across all years, and we can see a drop after 2011. This observation also includes samples used to pretrain XLNet to get text embeddings. To confirm that the model is not learning irrelevant patterns, as some instances are used to train XLNet, as mentioned, and that the F1 score is not based on pre-trained embeddings, we shuffled the labels by 50% [10] and trained the model. In every case, we got an F1 score lower than 50%, confirming that the close F1 scores by the model in temporal and simple settings are not random, and the model is learning the patterns. For the vanilla embeddings, we can see in Figure 5 that it performs better than simple training in some scenarios, like when time nodes are not added to the model. After 1981, we can see the kink changing its direction in the graphs from and after 2001-2006; this is observed in the models with features except for pre-trained embeddings.

### Link Prediction

For the link prediction task, in forward training, we can see in Figure 2 when the edges are undirected, the model can predict it more efficiently, followed by rev_edges and no_change. The embeddings have significantly less effect in the forward training direction.

Figure 3 shows ROC curves broken down to years for each pre-trained embedding and each setting. More are present in the appendix. The area under the curve (AUC) for pre-trained is the most, followed by hierarchical, vanilla and random.

The addition of time and act nodes decreases the AUC of the curve, which means the model is not very efficient in predicting an edge between a case and time and\(\backslash\)or act nodes.
Reverse training also exhibited the same observations except for the part where the AUC was lower than forward training, specifically for the reverse training setting.

Figure 4: F1 scores of binary labels of the GNN model with pre-trained features calculated on different numbers of samples distributed according to the year as present in Table 1. The dotted line represents label 0 and s_0 represents the performance of label 0 training in the simple setting and continuous line represents label 1 and s_1 represents the performance of label 1 while training in simple setting. Line t_0 represents the performance of label 0 while training temporally. Line t_1 represents the performance of label 1 while training temporally. The X-axis represents the number of training samples presented according to Table 1 and the Y-axis represents the F1 score.
### Redaction

Training our model with redacted tokens gives almost the same output for all the models compared to the unredacted dataset, hinting at little to no bias in the judgments.

## 5 Conclusion

By this study, we conclude that the time nodes have a negative impact on the performance of the model, as we can observe in Figure 4, 5, 6 and 7 where the difference F1 score of both the classes is more significant with the time nodes. The opposite is observed in the case of acts; when we have the act nodes, the difference between F1 scores of both the classes reduces, which implies the best setting is with no time nodes and keeping act nodes in the model. Secondly, embeddings have the least significant impact on the task of link prediction, and it mainly predicts the presence of an edge between two nodes. Predicting the edge's direction is challenging for the model as undirected edges are predicted better. Lastly, we found no significant change in the model's performance and classes with the redacted tokens.

## 6 Limitations

As per the limitations, the explainability of judgments was out of the scope of the paper. We limit ourselves to existing graph models for our study. We mainly focus on a more profound analysis of the LJP task using graph neural networks, so we are on par with the best-performing model proposed in [21].

## 7 Ethics

Our study aims to advance research and automation in the legal domain, focusing on the Indian legal system. We are committed to making the extended dataset and resources we use publicly available, ensuring accessibility for all. Given the substantial number of pending cases in lower courts, our efforts are directed towards enhancing the legal system, which stands to benefit millions of people. Our work aligns with previous efforts in legal NLP, such as creating legal corpora and predicting legal judgments.

Nevertheless, we acknowledge the potential risks associated with developing AI systems based on legal corpora, which could adversely affect individuals and society. To address this concern, we took proactive measures to identify and mitigate biases in the corpus. One of these measures includes anonymizing entities, such as names and gender, in the dataset. This is particularly important as previous research [21] and [14] have shown biases in legal datasets.

Additionally, we explored the task of Link Prediction in the Indian legal domain, which holds significance for developing recommendation systems in the
legal field. This area is relatively novel in NLP research and remains nascent in India. Consequently, further research and investigations are essential, especially concerning potential biases and societal impacts.

## 8 Acknowledgements

We would like to acknowledge iHub Anubhuti IIIT Delhi for funding our research which enabled us to get the required resources.
