11institutetext:  Salvador University, Salvador, Bahia, Brazil
11email<EMAIL_ADDRESS>22institutetext: Salvador University,
Salvador, Bahia, Brazil
22email<EMAIL_ADDRESS>
# A Blockchain-based Educational Record Repository

Emanuel E. Bessa Bessa, E. is with UNIFACS IPQoS research group11 Joberto S.
B. Martins Prof. Dr. Martins, J. is with UNIFACS IPQoS and NUPERC research
groups22

###### Abstract

The Blockchain technology was initially adopted to implement various
cryptocurrencies. Currently, Blockchain is foreseen as a general purpose
technology with a huge potential in many areas. Blockchain-based applications
have inherent characteristics like authenticity, immutability and consensus.
Beyond that, records stored on Blockchain ledger can be accessed any time and
from any location. Blockchain has a great potential for managing and
maintaining educational records. This paper presents a Blockchain-based
Educational Record Repository (BcER2) that manages and distributes educational
assets for academic and industry professionals. The BcER2 system allows
educational records like e-diplomas and e-certificates to be securely and
seamless transferred, shared and distributed by parties.

##  0.1 Introduction

Blockchain is a technology considered by many to be something as relevant as
the rise of the Internet. There have been experiments with blockchains since
the early 1990’s, but it was only in 2008, with the release of a white paper
by an individual or group of individuals under the pseudonym of Satoshi
Nakamoto, that blockchains gained wide adoption [5].

Educational records are used worldwide and, from the user point of view, is an
important asset for individuals pledging for scholarships, jobs and
professional and academic visibility in general. Currently, our educational
records management systems are mostly physically localized, require specific
and non-trivial procedures to access information, are in many cases unreliable
and, finally, do not follow or have any educational standards.

With the blockchain capabilities and citizen’s global visibility perspective
in mind this paper presents a Blockchain-based Educational Records Repository
(BcER2). BcER2 is intended to allow any individual to be able to store
educational records and access multiple type of educational records with
authenticity on a worldwide basis. It is a system to ensure educational
records distributed management and access with inherent more security like
authenticity and privacy.

In the next part of this article, section 2 summarizes the fundamental aspects
of Blockchain technology. Section 3 indicates the relevant work being done
related to Blockchain and BcER2 system. Section 4 describes the architecture,
entities, components and implementation of the BcER2 system. Section 5
presents a proof-of-concept of the BcER2 implementation. Finally, section 6
presents the final considerations and future work.

##  0.2 Fundamental aspects of Blockchain Technology

A blockchain is essentially a distributed database of records that keeps
potentially all kind of data, like transactions, contracts and events. All
information handling takes place across a peer-to-peer network and is
maintained chronologically in digital blocks. These basic features and
capabilities make Blockchain transparent, secure, decentralized and with
almost unlimited storage capacity[3].

Blockchain uses the concept of hashing. The ”hash” is a block signature and
considers all data and transactions involved. In summary, a cryptography hash
function takes a input string and turns it into a unique n-digit string [5].

###  0.2.1 Key Blockchain Characteristics

With traditional methods for recording transactions and tracking assets,
participants on a network keep their own ledgers and other records. This
traditional method can be expensive, partially because it involves
intermediaries that charge fees for their services. It’s clearly inefficient
due to delays in executing agreements and the duplication of effort required
to maintain numerous ledgers. It’s also vulnerable because if a central system
(for example, a bank) is compromised due to fraud, cyberattack, or a simple
mistake, the entire business network is affected. To solve or improve
traditional method Blockchain has a set of key characteristics: consensus,
provenance, immutability and finality [5] [2].

All relevant participants make decisions by consensus, in this process most
participants must agree that a transaction is valid. This goal is achieved
through the implementation of consensus algorithms. Each network enforces the
conditions under which transactions are performed or the exchange of assets
may occur. Provenance guarantees that participants know where the asset came
from and how its ownership has changed over time. With immutability, no
participant can tamper with a transaction after it has been recorded to the
ledger. If a transaction is in error, a new transaction must be used to
reverse the error, and both transactions are then visible. With finality, a
single shared ledger provides one place to go to determine the ownership of an
asset or the completion of a transaction.

###  0.2.2 Why ”Blockchain” to Manage and Improve Visibility for Educational
Records

An “education record” in the context of this paper is a record containing
files, documents, and other materials which [5]: i) Contains information
directly related to the academic historical of a student or a professional;
and ii) From a local perspective are typically maintained by an educational
institution or by other entity acting for such institution.

There are significant advantages and benefits in using a Blockchain-based
educational repository [8] [10]: i) Educational records (e-diplomas,
e-certificates, other) uploaded and managed on the Blockchain ledger are more
secure and resistant to “physical wear and tear” than paper documents [8]; ii)
Educational records are seamless and efficiently transferred and shared among
parties (universities, schools and employers) fostering worldwide visibility;
and iii) Educational records stored on the blockchain can be accessed any
time, from any location.

In summary, educational records managed by Blockchain technology stimulate the
knowledge/reward principle, makes credentials more trustworthy and keeps
educational records safe and easy to access [8].

##  0.3 Related Work

In recent years, blockchain technology has been widely used as the basic
construct for crypto-coins such as Bitcoin [7].

MIT has a system for building Blockchain-based applications that issues and
verifies official records called ”Blockcerts Wallet”. It allows, for instance,
the creation of a certificate wallet for students to receive virtual diplomas
via their smart devices [6]. Different from Blockchain-based Educational
Records Repository (BcER2), MIT Blockcerts Wallet system is a building
application platform that has a similar target in terms of allowing
educational records creation and dissemination using Blockchain.

New promisingly Blockchain-based solutions include ’intelligent contracts’.
Ethereum, discussed in [9], allows the creation of contracts that are self-
managed. Contracts are triggered by an event such as passing an expiration
date or achieving a specific price goal. In response, the smart contract
manages itself by making adjustments as needed and without the input of
external entities [1].

##  0.4 The Blockchain-based Educational Records Repository (BcER2) -
Architecture and Components

In order to meet the different alternatives of use, blockchain-based
applications can be implemented using 3 types of general structures as
follows: i) Public Blockchain; ii) Private Blockchain; and iii) Consortium
Blockchain.

The BcER2 repository adopted the consortium system since only authorized
persons are able to create certificates records on the network. On the other
hand, anyone can verify their authenticity. Thus, when registering an
educational record, for example, the responsible for creating the record
writes in the registry or in the database using its own private key. Users who
want to check the veracity of the record must have a corresponding identifier
number to be inserted into the system.

The Blockchain consortium is a semi-private and partially decentralized chain
system, in this scenario the nodes are responsible for the validation of the
transactions and how this happens depends on the implementation of the
consensus methods. The form of access and consultation of the records can be
public or private and the owner of the network is responsible for its
configuration.

The BcER2 effective structure uses the basic steps and operation flow of a
blockchain-based application: i) A transaction is requested by someone who has
prior authorization and needs to create an educational record; ii) The request
record transaction is sent to the nodes belonging to the BcER2 system; iii)
The educational record transaction is verified by the ledger; and iv) A new
block of data corresponding to the educational record transaction is accessed
or created and annexed to the ledger becoming permanent and immutable
completing the transaction.

###  0.4.1 Blockchain-based Educational Records Repository (BcER2) Entities

The entities belonging to the BcER2 educational repository are the following:
i) Assets; ii) Registers; iii) Transactions; and iv) Participants.

An ”asset” can be anything of value that will be kept securely by the
educational repository. Educational records like certificates, diplomas,
educational records an similar documents are BcER2 assets.

”Participants” are the educational organization representatives, students and
people in general that are somehow interested in either distributing or
accessing educational records. Participants are defined in the ”business
network model” adopted by the blockchain application process. For BcER2,
coordinators, students, and anyone else interested in having access to the
educational records are the participants belonging to the network. Each one
has their specifically assigned functions, responsibilities and access
restrictions.

”Transactions” are submitted by participants to create or access the assets
held in the blockchain-based asset registries on the blockchain ledger.
Transactions, in general, do belong to a business network and, as such, do
require a ”business network model”. The business network model, from the
blockchain system perspective, define the operation involved with the assets.

”Registers” can be defined as the set of data that involve the assets,
transactions and the participants. this set is what will be included in the
block after the validation. Registers are new information that is added to the
blockchain ledger.

In BcER2 educational records repository the addition of a ”register” is
carried out through the execution of steps. In these steps, the business model
adopts the following ”business” premises: i) The course coordinator or the
educational institution representative are the authority to create new assets;
and ii) Students and general public are participants that access the validated
and secure educational assets maintained by the system.

Figure 1: ”Certificate Register” creation in BcER2

The creation of a register is executed as illustrated in Figure 1: i) A
coordinator proceeds to write a record to the Blockchain account, which means
create the certificate with an identifier. In this process the coordinator
selects the certificate and through its identifier number it is possible to
link it to a student; ii) The record is saved and time-stamped in a block
using arithmetic operations; iii) The block is subsequently validated by
network pre-selected nodes through cryptography techniques; and iv) The block
is dated and added to the block chain, so that all users can have access to
the same chain since each node builds its own exemplary independently.

Once these steps have been executed, we can access educational records with
authenticity and integrity by simply using a credential (ID Card) through a
web browser.

###  0.4.2 Blockchain-based Educational Records Repository (BcER2) Business
Network

The business network is a fundamental definition for the BcER2 educational
registry repository deployment.

In summary, it models the BcER2 ”educational model”, defining the existing
assets, transactions and participants related to them. The business network
defines the transactions that interact with assets. The model also includes
the definition of participants who interact with assets and associates a
unique identity, across multiple business networks. As described before, BcER2
is composed of assets, participants and transactions, with each of these
entities modeled in relation to the educational operation.

###  0.4.3 Blockchain-based Educational Records Repository (BcER2) Components

The basic components belonging to the the BcER2 educational records repository
are illustrated in Figure 2 and basically reflect the business network adopted
which is suitable for an educational records repository that registers,
manages and provide access to them.

Figure 2: BcER2 basic components

The ”asset information” component contains information related to the
educational record being managed by BcER2. This component is responsible for
asset’s definition and consistency.

The ”Business Model Information” component contains information related to the
process involved in the asset management. It defines basically the
participants, name space and transactions involved in the process.

The ”Transaction Process Function” component contains the information
concerning the specif functions invoke in the business model to manage the
asset.

The ”Access Rules” component contains, as the name suggests, the access rules
including all priorities among participants involved in the business model
adopted.

BcER2 managers and general users access the repository through a web browser
using identification cards (ID Cards) which includes connection profiles and
credentials.

Assets are effectively deployed in the BcER2 blockchain ledger in this specif
system by the Hyperledger Composer framework described in section 4.4.

###  0.4.4 Blockchain-based Educational Records Repository (BcER2)
Implementation

The Hyperledger Composer [4] was used to implement the BcER2 educational
repository. Hyperledger Composer is an open source development tool set and
framework aiming to support the development of blockchain applications. It
allows the modeling of the business network and integrates existing systems
components and data deploying as such the blockchain application.

The use case adopted by the actual implementation is initially intended to
verify the authenticity of student’s certificates generated by Salvador
University (UNIFACS) under-graduation courses.

BcER2 is composed of assets, participants and transactions, with each of these
entities being represented within Hyperledger framework as configuration
files.

The .CTO hyperledger component is responsible for implementing the assets,
participants, and transactions, including all relevant information. A
Hyperledger Composer CTO file is composed of the following elements: i) A
name-space with resources declaration; ii) Resources definition including
assets, transactions, participants and events; and iii) Optional resource
import declarations from other name-spaces.

The .ACL hyperledger component provides declarative access control for the
elements in the domain model. By defining access and control (ACL) rules you
can determine which users/roles are permitted to create, read, update or
delete elements in a business network’s domain model.

The ’Business Network’ definition, from the Hyperledger Composer perspective,
is composed by a set of model files defining assets, participants and
transactions. The ”.JS” script file is responsible for maintain a set of
scripts. The scripts contain transaction process functions that implement the
transactions defined in the ’Business Model’. Transaction processing functions
are automatically invoked at run-time when transactions are submitted and
their structure are composed by a JavaScript function.

##  0.5 BcER2 Repository - Proof-of-Concept

The main objective of the BcER2 repository proof-of-concept is to validate the
deployment of the business network and verify the operation steps of the
repository by: i) Creating assets; and ii) Accessing them and verifying the
effectiveness of credentials and other distribution and security aspects.

The proof-of-concept of the BcER2 repository operation was implemented by
emulating participants as follows: i) ’Users’ are the general public accessing
educational records; and ii) The ’Coordinator’ (Register Authority) is a
UNIFACS authority creating educational record entries.

The experiment was executed using the following infrastructure: i) The BcER2
system runs on Notebook Core i7, 2.0 Ghz, 8 GB RAM, Ubuntu Server Operating
System 16.04 x64; and ii) The coordinator and users access the BcER2 system
using any browser.

The software components installed to run the BcER2 system are i) Node Version
8.12; ii) NPM Version 6.4.1; iii) Visual Studio Code Version Version 1.28; iv)
Docker Engine Version 18.06; and v) Docker Composer Version 1.23.

The proof-of-concept method and parameters used to validate the operation of
BcER2 was the creation of a set of educational records followed by
authenticity verification and access by the system administrator and
distributed users. The experimental setup included the creation of 10
different educational records with course certificates and various nodes
(N$\textgreater$10) simulating different users acting on the validation
process (transactions) and verifying their authenticity. The results allowed
the distributed access to educational records enabling verification through
blockchain technology. Security access was also validated by trying to access
educational records without the adequate credential. The scalability of the
solution was not evaluated and will be addressed by future work.

##  0.6 Final Considerations and Future Work

We propose in this work to use Blockchain technology as a tool to provide a
secure and efficient way to access certificate with authenticity. We argue
that the proposed Blockchain BcER2 repository has the potential to support the
education sector by providing better support for certificate management and
distribution.

In the current scenario, the proposed application covers only the UNIFACS
network for the purpose of managing diplomas and certificates issued by the
university. The BcER2 application has the potential to cover additional areas
in which digital certificates provide interesting opportunities such as: i)
Corporate Training - Many large companies offer a multitude of training
opportunities to their employees, but lack the systems to track and store
results reliably. Current human resources systems often do not interact with
corporate databases and there are no consistent standards for comparing skills
and accomplishments; and ii) Workforce Development - There are millions of
records and learning certificates, but there are no systems to manage them.

In terms of future work, it is intended to evaluate the scalability issues and
impacts associated with the deployment of a huge repository. Another aspect to
be considered is to bring together stakeholders such as employers, students,
teachers and contractors in a way that they interact with each other enabling
wide-spread use of trustable e-certificates. A final target will be to adopt a
fully standardized asset representation as an additional step towards a secure
and decentralized way of conferring a wide-spread use of the system.

##  0.7 Acknowledgements

Authors thanks FAPESB (Fundação de Apoio à Pesquisa do Estado da Bahia) by the
scientific initiation (IC) scholarship support.

## References

  * [1] Habib Azam.  What are the different generations of blockchains?, January 2018.
  * [2] Sanjaya Baru.  Blockchain: The next innovation to make our cities smarter.  page 48, January 2018.
  * [3] Michael Crosby.  BlockChain Technology: Beyond Bitcoin.  (2):16, 2016.
  * [4] Linux Foundation.  Introduction | Hyperledger Composer.
  * [5] Alexander Grech and Antony F. Camilleri.  Blockchain in Education.  Technical Report EUR 28778 EN, European Union, Luxembourg, 2017.
  * [6] MIT Media Lab.  What we learned from designing an academic certificates system on the blockchain, June 2016.
  * [7] Satoshi Nakamoto.  Bitcoin: A Peer-to-Peer Electronic Cash System.  page 9, 2008.
  * [8] Institute of Blockchain™.  Blockchain and Education, June 2018.
  * [9] Nathan Reiff.  Blockchain Technology’s Three Generations, July 2018.
  * [10] Raza Sheeraz.  Blockchain Can Help People Keep Their Educational Records Intact, May 2018.

] Emanuel E. Bessa - Computer science student at Salvador University
(UNIFACS). Research interest include Blockchain technology and Smart Cities.
Emanuel is a member of IPQoS Research Group and RePAF Project (Resource
Allocation Framework for MPLS, Elastic Optical Networks (EON), Network
Function Virtualization (NFV) and IoT) at UNIFACS.

] Prof. Dr. Joberto S. B. Martins - PhD in Computer Science at Université
Pierre et Marie Curie – UPMC, Paris (1986), PosDoc at ICSI/ Berkeley
University (1995) and PosDoc Senior Researcher at Paris Saclay University –
France (2016). Invited Professor at HTW – Hochschule für Techknik und
Wirtschaft des Saarlandes (Germany) (since 2004) and Université d’Evry
(France). Full Professor at Salvador University on Computer Science, Head of
NUPERC and IPQoS research groups with research interests on Cognitive
Management, Artificial Intelligence, Resource Allocation, SDN/ OpenFlow,
Internet of Things, Smart Grid and Smart Cities. Senior member of IEEE Smart
Grid and IEEE Smart City Research Committees and former Chair of the IEEE CGAA
(Committee on Global Accreditation Activities).


# Challenges and Considerations in Annotating Legal Data: A Comprehensive
Overview

Harshil Darji0000-0002-8055-1376 Corresponding Author: Harshil Darji,
<EMAIL_ADDRESS>Jelena Mitrovic0000-0003-3220-8749  Michael
Granitzer0000-0003-3566-5507  Chair of Data Science, University of Passau,
Innstraße 41, 94032 Passau, Germany  The Institute for Artificial Intelligence
Research and Development of Serbia, Serbia

###### Abstract

The process of annotating data within the legal sector is filled with distinct
challenges that differ from other fields, primarily due to the inherent
complexities of legal language and documentation. The initial task usually
involves selecting an appropriate raw dataset that captures the intricate
aspects of legal texts. Following this, extracting text becomes a complicated
task, as legal documents often have complex structures, footnotes, references,
and unique terminology. The importance of data cleaning is magnified in this
context, ensuring that redundant information is eliminated while maintaining
crucial legal details and context. Creating comprehensive yet straightforward
annotation guidelines is imperative, as these guidelines serve as the road map
for maintaining uniformity and addressing the subtle nuances of legal
terminology. Another critical aspect is the involvement of legal professionals
in the annotation process. Their expertise is valuable in ensuring that the
data not only remains contextually accurate but also adheres to prevailing
legal standards and interpretations. This paper provides an expanded view of
these challenges and aims to offer a foundational understanding and guidance
for researchers and professionals engaged in legal data annotation projects.
In addition, we provide links to our created and fine-tuned datasets and
language models. These resources are outcomes of our discussed projects and
solutions to challenges faced while working on them.

###### keywords:

AnnotationLegal DataInformation ExtractionChallengesNER

, and

##  1 Introduction

Legal data annotation is a critical step in developing machine learning models
and natural language processing tools tailored for the legal domain. The
significance of this process stems from the unique nature of legal texts,
characterized by their complicated structures, specialized terminology, and
the importance of context. While data annotation is a common practice across
various fields, the legal sector presents challenges that necessitate a
specialized approach.

The initial phase of this process involves selecting a raw data set. Given the
multifaceted nature of legal documents, this step is crucial in ensuring that
the data captures the complexities inherent to legal language. Once a data set
is chosen, the task of text extraction begins. This is a complex endeavor, as
legal documents often contain footnotes, references, and other elements that
can complicate the extraction process. Moreover, redundant information
necessitates rigorous data cleaning to retain essential legal details while
eliminating irrelevant content.

Annotation guidelines play a pivotal role in this process. They provide a
framework that ensures consistency in annotation, considering the nuances of
legal language. Furthermore, the involvement of legal professionals in the
annotation process cannot be overstated. Their expertise ensures that the
annotated data is not only contextually accurate but also aligns with current
legal standards and interpretations.

In addition to these challenges, practical considerations also come into play.
For instance, many annotators, especially those from a legal background, may
prefer using familiar tools like Microsoft Word for annotation. This
preference introduces another layer of complexity, as extracting annotations
from resulting formats and converting them into a structured format like CoNLL
can be a daunting task. Addressing these practical challenges is essential to
ensure the smooth progression of the annotation project and to maintain the
quality and integrity of the annotated data. Through this paper, we aim to
shed light on these challenges and offer a comprehensive overview to assist
researchers and professionals in their legal data annotation projects.

##  2 Related work

Legal data annotation, particularly in the domain of discourse phenomena, has
been the subject of extensive research. The intricacies of extracting
substantive legal knowledge from legal sources have been highlighted by
Santosuosso et al. [1], emphasizing the challenges this presents in legal
analytics. Their exploration resonates with the complexities faced in the
legal data annotation process, emphasizing the need for robust solutions.

Wyner et al. [2] have explored the challenges associated with the manual
extraction of legal rules from legal texts, emphasizing the importance of
clear annotation guidelines and the involvement of legal professionals in the
annotation process. Their work underscores the complexities of legal language
and the challenges of extracting meaningful information from dense legal
texts. The challenges of semantic role labeling in legal texts have been
explored by Ceci et al. [3], highlighting the unique syntactic and semantic
structures present in legal documents. Their work emphasizes the need for
specialized tools and approaches tailored to the legal domain.

Rättzén et al. [4] highlight the challenges in legal data annotation for
automated contract review, emphasizing the complexity of translating contract
language into machine-readable formats. They note the potential for machine
learning to revolutionize contract scrutiny despite the need for meticulous
manual annotation and the inherent risks of confidentiality breaches.

The complexities of annotating multiparty discourse in legal texts have been
highlighted by Wacholder et al. [5], emphasizing the challenges of achieving a
reliable Inter-Annotator Agreement (IAA). Their observations resonate with the
challenges faced in legal data annotation, where identifying unit boundaries
can be remarkably intricate.

Walker et al. [6] address the need for annotated legal corpora and effective
human annotation protocols in legal document analysis. They emphasize the lag
in applying advanced analytics to legal texts compared to other fields,
attributing this to the absence of practical theories of legal reasoning for
software development and the complexity of legal argumentation for AI.

The study by Correia et al. [7] addresses the challenge of annotating legal
documents for Named Entity Recognition (NER) in the context of the Brazilian
Supreme Court. It presents a fine-grained legal entity annotation case study,
utilizing law students as annotators to create a corpus of 594 decisions.

Urchs et al. [8] presented a corpus of 32,748 German legal decisions and a
subset of 200 randomly chosen judgments annotated for the German legal writing
style Urteilsstil. As per the authors, annotating legal texts is a complicated
task for machine learning, as it involves defining clear legal terms and
including facts under legal definitions. However, legal language is nuanced
and context-dependent, making it challenging to annotate legal datasets in
detail.

##  3 Challenges of Legal Annotations

Legal data annotation, especially within German legal texts, is filled with
complexities that arise from the nature of legal language, the architecture of
legal documents, and the subtleties inherent in legal references. In this
section, we discuss the similar challenges we faced during our previous works.

###  3.1 Dataset structuring and availability

One of the primary hurdles in the legal NLP domain is the need for well-
structured datasets. Existing datasets in the German legal domain often
encompass entire legal cases or specific named entities within those cases.
However, there is a pressing need for datasets that provide comprehensive
texts for each legal reference cited in a legal case, along with specific
paragraph texts. This challenge directly impacts the annotation process, as
the quality and structure of the dataset determine the accuracy and efficiency
of legal data annotation.

One of our previous works explored semantic similarities between legal texts
and corresponding referred laws [9]. This task requires a large number of
legal cases as well as legal laws with their definitions. We fulfilled the
necessary condition of gathering legal cases by relying on Open Legal
Data111https://de.openlegaldata.io/ introduced by Ostendorff et al. [10]

However, all the legal cases in the downloaded raw dataset were in HTML
format. Because of this, important information, such as tenor, tatbestand,
gründe, and entscheidungsgründe, was difficult to extract due to the
inconsistency of HTML structure throughout the dataset. We dealt with these
inconsistencies by making some assumptions, one of which was to assume that
all the important titles, such as tenor and gründe, are within the h2 tag. We
also made sure the content within h2 tag is all alphabetic and is within a
certain length. The reason is that there were some instances where titles were
written as 1\. tenor or t e n o r. Considering we were only focusing on a few
important titles, it was an excellent choice to make assumptions related to
this to avoid any false positives.

After cleaning the raw dataset using such assumptions, we now have a dataset
of approximately 1.1 GBs, with 43337 rows and 12 features (see table 1). This
dataset is currently publicly available on
Zenodo222https://zenodo.org/records/6631931.

Feature | Total | Example content
---|---|---
id | 43337 | 127981
slug | 43337 | ag-volklingen-2002-07-10-5c-c-24102
ecli | 10831 | NaN
date | 43337 | 2002-07-10
court | 43337 | Amtsgericht Völklingen
jurisdiction | 43337 | Ordentliche Gerichtsbarkeit
level_of_appeal | 43337 | Amtsgericht
type | 43337 | Urteil
tenor | 36282 | 1\. Die Beklagten werden als Gesamtschuldner…
tatbestand | 24243 | Auf die Darstellung des Tatbestandes…
gründe | 27144 | Die Klage ist zulässig und begründet. Die…
entscheidungsgründe | 24038 | Die Klage ist zulässig und begründet. Di…
Table 1: Here, slug can be used to view the web version of any case by
appending it to https://de.openlegaldata.io/case/[slug]. ECLI, which stands
for European Case Law Identifier, simplifies the accurate citing of rulings
from European and national courts.

###  3.2 Information extraction

As mentioned in the previous section, calculating semantic similarities
between legal text and corresponding law also requires access to the
definition of the corresponding law. For example, if the following is part of
text from a legal case,

Die Klägerin hat ein rechtliches Interesse an der begehrten Feststellung (§
256 Abs. 1 ZPO), da sie von den …

Then, the corresponding law definition will be the law text for the law § 256
Abs. 1 ZPO, which can be found on Gesetze im Internet333https://www.gesetze-
im-internet.de/ as,

Auf Feststellung des Bestehens oder Nichtbestehens eines Rechtsverhältnisses,
auf Anerkennung einer Urkunde oder auf Feststellung ihrer Unechtheit kann
Klage …

As seen in this example, this requires the extraction of similar laws from the
available dataset of legal cases. However, the dataset we worked with did not
have a section that collected all the laws referred to in every legal case
available. This issue can be resolved using regex, which filters and extracts
specific data patterns. However, using regex has problems if the given dataset
uses different ways for a similar string (e.g., §§ 47 Abs. 1, § 154 Abs. 2, or
Art. 6), as shown in the following examples:

1. Let’s begin with a simple regex to identify basic references to German legal statutes:

        \s*\d+\s*(Abs\.\s*\d+)?\s*(Satz\s*\d+)?

It captures patterns that denote a section (§) followed by a section number
and, optionally, subsections and sentences within that section. However, while
using this regex on a given set of examples, we ran into the issue of it
constantly missing the main section numbers (like § 433, § 434, etc.).
2. The following regex example is intended to identify specific references to articles in major German legal codes such as the GG, BGB, StGB, and HGB:

        Art\.\s*\d+\s*(Abs\.\s*\d+)?\s*(Satz\s*\d+)?
        \s*(GG|BGB|StGB|HGB)

However, as with the first regex example, this also failed to identify the
main article number in law references. Another issue with this regex is its
inability to identify unabbreviated forms of similar German legal codes.
3. The syntax of these law references is very inconsistent throughout the dataset, e.g., it often starts with “§” or “§§” etc. [11] An example of regex that can be used to resolve this issue is:

        (?|Art\.)\s*(\d+\w*(?:-\d+\w*)?)

However, this regex is restricted to “§”, “§§”, and “Art.” and cannot be
generalized. To make it suitable for similar inconsistencies, it needs to be
modified each time one is encountered.

These are just a few examples where regex fails to extract the information due
to differences in representation and inconsistent formatting of law
references. Regex is precise for well-defined and consistent patterns, but
legal documents are not always consistent.

To tackle this problem, we fine-tuned the bert-base-
cased444https://huggingface.co/bert-base-cased language model using the Legal-
Entity-Recognition555https://github.com/elenanereiss/Legal-Entity-Recognition
dataset. This fine-tuned BERT model is capable of identifying laws in a given
legal case with an F1 score of 99.29 [12]. Another advantage of fine-tuning a
language model over regex is its ability to identify and extract more than
just law references. Our fine-tuned BERT model can recognize 19 different
named entities with high precision. We have made it publicly available on
HuggingFace666https://huggingface.co/PaDaS-Lab/gbert-legal-ner, which
currently has over 3000 downloads.

The mentioned 19 named entities are: Person (PER), Judge (RR), Lawyer (AN),
Country (LD), City (ST), Street (STR), Landscape (LDS), Organization (ORG),
Company (UN), Institution (INN), Court (GRT), Brand (MRK), Law (GS), Ordinance
(VO), European legal norm (EUN), Regulation (VS), Contract (VT), Court
decision (RS), Legal literature (LIT).

###  3.3 Manual annotation and expertise

Building on the previous section, the next part of the process is to create
annotations for each extracted law. This includes identifying various parts of
a law, such as Artikel, Satz, Absatz, etc. Consider the following example:

For a given law § 14 Abs. 3 Satz 2 SchVG, • Artikel = 14 • Absatz = 3 • Satz =
2

As shown in the above example, some information is easily accessible from a
given law. However, other information, such as Gesetzbuch, law_title,
absatz_text can only be found by visiting the corresponding online link for
each law. Such annotations can not be done programmatically and require manual
efforts, which is not only labor-intensive but also mandates deep expertise.
We tackled this problem with the help of three legal experts who have already
passed their first state examination in law. These experts manually annotated
extracted law references with following 21 distinct labels, creating a dataset
of 2944 unique law references:

Gesetzbuch/Norm, Buch , Teil, Abschnitt, Titel, Untertitel, Kapitel, Artikel,
Absatz, Buchstabe, Unterabsatz, Satz, Nummer, Buchstabe , Online Link
Gesetzbuch, Online Link Exakt, Alternative Schreibeweise 1, Alternative
Schreibeweise 2, law_title, full_text, absatz_text

The inter-annotator agreement between these three annotators, calculated using
Fleiss’ Kappas score, is 0.87 [13]. Considering that this dataset, alongside
the Open Legal Data, can be used by researchers for various purposes, such as
law reference prediction or link prediction, it is now publicly available on
HuggingFace777https://huggingface.co/datasets/PaDaS-Lab/legal-reference-
annotations.

While on the topic of manual annotation, another challenge we faced during our
most recent work was due to the annotation tool being used. Our most recent
work focuses on creating a dataset of privacy policies annotated using GDPR-
compliant named entities, such as Data Controller, Data Processor, Authority,
etc. For this process, we retained two legal experts who annotated law
references.

To begin with, we collected privacy policies in English from 45 different
online platforms such as 9gag, Amazon, Airbnb, etc. Since the legal experts
are familiar with Microsoft Word, these privacy policies were stored in the
same format. Both annotators used the comment feature of Microsoft Word to
perform annotations using 33 distinct labels. However, the challenge was
extracting the commented parts and comments and storing them in a format
suitable for further analysis. Popular libraries such as
Aspose888https://docs.aspose.com/words/python-net/ and python-
docx999https://python-docx.readthedocs.io/en/latest/ failed, considering they
only managed to extract less than 35% of total comments.

Finally, we tackled this problem by using XPath, an expression language
designed for selecting nodes from an XML document, supporting both queries and
transformations. Using Python, when a “.docx” file is treated as a “.zip”
file, its output includes “comments.xml” and “document.xml”. After extracting
comments from “comments.xml” and using it with the following XPath expression
on “document.xml”, we managed to extract commented parts and corresponding
commented labels with utmost precision.

    # XPath expression to find text associated with the comment
    xpath_expr = f’//w:commentRangeStart[@w:id="{k}"]/following:’ \
                 f’:w:t[following::w:commentRangeEnd[@w:id="{k}"]’ \
                 f’ and not(preceding::w:commentRangeStart’ \
                 f’[@w:id="{int(k)+1}"])]’

This XPath expression can be integrated into a Python
script101010https://gist.github.com/harshildarji/8480e6f083021c8bd0fce0d297a9855c
as it is and can be interpreted as follows:

$\mathbf{//w:commentRangeStart}$
Selects all $<w:commentRangeStart>$ elements in the XML document, regardless
of their location.

$\mathbf{[@w:id=``{k}"]}$
Filters the selected $<w:commentRangeStart>$ elements by the $w:id$ attribute
value equal to ${k}$. This attribute value will likely be a variable or
placeholder being dynamically replaced with an actual value when the XPath
expression is used.

$\mathbf{/following::w:t}$
Selects all $<w:t>$ elements that appear after the filtered
$<w:commentRangeStart>$ element(s).

$\mathbf{[following::w:commentRangeEnd[@w:id=``{k}"]}$
Filters the selected $<w:t>$ elements further by checking if there is a
$<w:commentRangeEnd>$ element with a matching $w:id$ attribute value of ${k}$
that appears after the current $<w:t>$ element.

$\mathbf{and\ not(preceding::w:commentRangeStart[@w:id=``{int(k)+1}"])]}$
Additionally filters the selected $<w:t>$ elements by checking if there is no
$<w:commentRangeStart>$ element with a $w:id$ attribute value equal to
${int(k)+1}$ that appears before the current $<w:t>$ element.

AAno Considering the fact that this dataset can help improve current privacy
policy initiatives, we also plan to make this dataset public in CoNLL format
[14].

##  4 Conclusion

Efforts to structure and annotate legal datasets, especially in the German
legal domain, come with various challenges. The complex nature of legal
language and the diverse yet inconsistent structure of legal domains require a
careful and detailed approach to dataset creation and annotation. In this
paper, we discussed our experiences highlighting the need for a well-
structured and consistent dataset. Despite facing challenges with HTML
inconsistencies and needing tedious manual annotations, we have created
datasets that provide access to clean Open Legal Data and corresponding
information for individual laws referred to in legal cases. We also fine-tuned
a language model to tackle the problem of extracting information from the
available dataset, which allowed for a fine-tuned BERT model capable of
recognizing 19 distinct named entities in German legal cases.

We also discussed problems that emerged due to the use of Microsoft Word for
annotation purposes and the use of its comments feature to annotate privacy
policies with GDPR-compliant named entities. Although using Word as an
annotation tool caused concern while extracting the commented parts and
corresponding comments/labels, we found a way to do so using XPath expression
properly. While investigating the low inter-annotator agreement score, we also
inferred that one of the main reasons behind this is the differences in
comment boundaries rather than the usual disagreement between annotators.

In this paper, we reintroduced several related datasets and language models we
created and fine-tuned, respectively. After solving various challenges during
our work on the mentioned projects, we have concluded that it is crucial to
understand these challenges as the field of legal data annotation continues to
expand. Addressing these challenges ensures the success of the annotation
projects and maintains the quality of the annotated data in the legal domain.

##  5 Acknowledgements

The project on which this report is based was funded by the German Federal
Ministry of Education and Research (BMBF) under the funding code 01—S20049.
The author is responsible for the content of this publication.

## References

  * [1] Santosuosso A, Pinotti G. Bottleneck or Crossroad? Problems of Legal Sources Annotation and Some Theoretical Thoughts. Stats. 2020 Sep 9;3(3):376-95.
  * [2] Wyner AZ, Peters W, Katz D. A Case Study on Legal Case Annotation. InJURIX 2013 Jan (pp. 165-174).
  * [3] Ceci M, Lesmo L, Mazzei A, Palmirani M, Radicioni DP. Semantic annotation of legal texts through a framenet-based approach. InAI Approaches to the Complexity of Legal Systems. Models and Ethical Challenges for Legal Systems, Legal Language and Legal Ontologies, Argumentation and Software Agents: International Workshop AICOL-III, Held as Part of the 25th IVR Congress, Frankfurt am Main, Germany, August 15-16, 2011. Revised Selected Papers 3 2012 (pp. 245-255). Springer Berlin Heidelberg.
  * [4] Rättzén M. Automated Contract Review: Challenges and Outcomes of a Data Annotation Framework.
  * [5] Wacholder N, Muresan S, Ghosh D, Aakhus M. Annotating multiparty discourse: Challenges for agreement metrics. In Proceedings of LAW VIII-The 8th Linguistic Annotation Workshop 2014 Aug (pp. 120-128).
  * [6] Walker VR. The need for annotated corpora from legal documents, and for (Human) protocols for creating them: the attribution problem.
  * [7] Correia FA, Almeida AA, Nunes JL, Santos KG, Hartmann IA, Silva FA, Lopes H. Fine-grained legal entity annotation: A case study on the Brazilian Supreme Court. Information Processing & Management. 2022 Jan 1;59(1):102794.
  * [8] Urchs S, Mitrovic J, Granitzer M. Design and Implementation of German Legal Decision Corpora. InICAART (2) 2021 Jul (pp. 515-521).
  * [9] Darji H, Mitrović J, Granitzer M. Exploring Semantic Similarity Between German Legal Texts and Referred Laws. InInternational Joint Conference on Knowledge Discovery, Knowledge Engineering, and Knowledge Management 2021 Oct 25 (pp. 37-50). Cham: Springer International Publishing.
  * [10] Ostendorff M, Blume T, Ostendorff S. Towards an open platform for legal information. InProceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 2020 Aug 1 (pp. 385-388).
  * [11] Milz T, Granitzer M, Mitrovic J. Analysis of a German Legal Citation Network. InKDIR 2021 (pp. 147-154).
  * [12] Darji H, Mitrović J, Granitzer M. German BERT model for legal named entity recognition. arXiv preprint arXiv:2303.05388. 2023 Mar 7.
  * [13] Darji H, Mitrović J, Granitzer M. A Dataset of German Legal Reference Annotations. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (ICAIL ’23). Association for Computing Machinery, New York, NY, USA, 392–396. Available from: https://huggingface.co/datasets/PaDaS-Lab/legal-reference-annotations
  * [14] Sang EF, De Meulder F. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. 2003 Jun 12.