# Bridging Law and Data: Augmenting Reasoning via a Semi-Structured Dataset
with IRAC methodology

Xiaoxi Kang1, Lizhen Qu2∗, Lay-Ki Soon1,Zhuang li2,Adnan Trakic 2
School of Information Technology, Monash University Malaysia1,
Monash University2
1{xiaoxi.kang,soon.layKi } @monash.edu
2{lizhen.qu,zhuang.li,adnan.trakic} @monash.edu

###### Abstract

The effectiveness of Large Language Models (LLMs) in legal reasoning is often
limited due to the unique legal terminologies and the necessity for highly
specialized knowledge. These limitations highlight the need for high-quality
data tailored for complex legal reasoning tasks. This paper introduces
LegalSemi, a benchmark specifically curated for legal scenario analysis.
LegalSemi comprises 54 legal scenarios, each rigorously annotated by legal
experts, based on the comprehensive IRAC (Issue, Rule, Application,
Conclusion) framework. In addition, LegalSemi is accompanied by a structured
knowledge graph (SKG). A series of experiments were conducted to assess the
usefulness of LegalSemi for IRAC analysis. The experimental results
demonstrate the effectiveness of incorporating the SKG for issue
identification, rule retrieval, application and conclusion generation using
four different LLMs. LegalSemi will be publicly available upon acceptance of
this paper.

##  1 Introduction

Access to justice is a universal social challenge. Two-thirds of people in the
United States experienced at least one legal issue in the past four years,
with less than half of those problems completely resolved
111https://iaals.du.edu/publications/justice-needs-and-satisfaction-united-
states-america. In India, more than 10,490 legal cases in the Supreme Court of
India have been pending for more than a decade (Madhana and Subhashree, 2022).
These backlogs are often caused by the complexity in legal practice, as well
as the scarcity of legal professionals. IRAC framework Metzler (2002), stands
for issue, rule, application, and conclusion, is the problem solving framework
widely used by legal professionals to determine the underlying legal issues,
followed by extracting and transforming facts in a legal scenario for legal
reasoning, which eventually leads to a legal conclusion.

Figure 1: An example of a legal scenario pertinent to Malaysian Contract Law
with annotations for IRAC analysis. The new types of annotations are legal
concepts, court cases, and links to SKGs.

AI models, in particular Large Language Models (LLMs), demonstrate great
potentials to improve access to justice Krasadakis et al. (2024). However, it
remains a challenge for LLMs to perform IRAC analysis on legal scenarios
accurately. A recent study Kang et al. (2023) identifies two key problems in
analysing legal scenarios. First, ChatGPT draws wrong conclusions on
approximately 50% of the legal scenarios on average. Even if the conclusions
are correct, there are mistakes in the intermediate reasoning steps. Secondly,
ChatGPT is not able to cite correct legal rules when analysing majority of the
legal scenarios. In real world, it is crucial for legal professionals to
understand every single reasoning step that leads to the final conclusion. In
addition, our empirical study finds that LLMs struggle to cope with the
language gaps between legalese and everyday language. We conjecture that LLMs
still cannot fully comprehend the underlying legal knowledge and perform
complex legal reasoning accurately.

Recent advances show that it is possible to mitigate the hallucination problem
of LLMs by leveraging structured knowledge graphs (SKGs) Pan et al. (2024a).
SKGs can enhance LLMs in terms of interpretability and faithfulness by
providing external knowledge Kim et al. (2024). If legal knowledge is stored
in SKGs, it is also easy to keep it up-to-date, in accordance with the
revisions of legislation. Unfortunately, existing IRAC datasets do not contain
any SKGs for legal knowledge.

To address the problems above, we curate LegalSemi, a dataset comprising legal
scenarios pertaining to Malaysian Contract Law, accompanied by rich structured
IRAC analysis carried out by top law students, as illustrated in Fig. 1.
Compared to (Kang et al., 2023), we do not only extend their dataset by
doubling the legal scenarios in Malaysian Contract Law but also introduce new
annotation types to all 54 scenarios, including legal concepts and court
cases.

To support reasoning with structured legal knowledge, we extract semantic
information from a law textbook and a legislation automatically to build the
SKG. In the SKG, a node represents either a legal concept, a court case, a
legal rule, the interpretation of a legal rule or a concept in lay language,
or relevant meta information, while an edge between two nodes denotes their
relation. The rigorous layout in the textbook and the legislation facilitates
rule-based extraction of semantic relations between legal concepts as well as
their relations to legal rules and interpretations. We demonstrate the
usefulness of the SKG for LLMs through extensive experiments and obtain the
following key findings:

  * •

Following Kang et al. (2023), we apply an LLM to decompose a legal question
into a set of issues, followed by retrieving rules and performing legal
analysis to address each issue. Incorporating legal concepts from the SKG, the
quality of the issue generation is improved by over 21.4% across all evaluated
LLMs. The annotated issues subsequently lead to significant improvement on
application generation.

  * •

By enhancing an LLM with the structured legal knowledge in the SKG, we achieve
a 60% increase in recall and a 12% improvement in the F1 score at top-5
results of rule retrieval. We find out that legal concepts are significant in
bridging the semantic gaps between facts in scenarios and rules in the
legislation. The interpretations in lay language further reduce language gaps
between scenarios in lay language and statutes in legalese.

  * •

Our findings indicate that while LLMs are adept at identifying high-level
legal concepts, there is still a need for improvement in recognizing the
details of these concepts.

##  2 Dataset

IRAC provides a comprehensive problem-solving framework for legal
professionals. It takes four stages to transform facts acquired from a legal
scenario into legal conclusions: (i) identifying legal issues, (ii)
determining the legal rules and precedents pertinent to the issues, (iii)
performing analysis by applying the law to the facts and the issues, which
requires strong legal reasoning skills, and (iv) drawing conclusions based on
the analysis. Legal reasoning is defeasible such that there are often more
than one reasoning traces leading to the same conclusion or different
plausible reasoning traces lead to different reasonable conclusions (Billi et
al., 2021). Given a legal scenario, legal professionals’ concern is not just
about the final conclusion, but also why the conclusion is drawn. Therefore,
it is essential to build automatic IRAC analysis tools that produce outcomes
for each stage and help them identify any missing reasoning steps, and suggest
alternative analysis, when necessary.

While LLMs demonstrate a great potential to automate IRAC analysis in the
absence of supervised training data, they suffer from three key limitations:
i) wrong references to statutes and precedents, ii) weak legal reasoning
capability, and iii) difficulties in filling the gaps between legalese and
everyday language. In contrast, prior studies demonstrate the effectiveness of
utilizing Retrieval-Augmented Generation (RAG) and neuro-symbolic approaches
with knowledge bases to enhance the factuality and reasoning capability of
LLMs (Gao et al., 2023). Therefore, LegalSemi builds the first SKG as a legal
knowledge base to facilitate research on neuro-symbolic approaches for legal
reasoning and provides an annotated corpus to evaluate system outcomes for
each stage of IRAC.

###  2.1 Structured Knowledge Graphs

Neuro-symbolic systems have garnered increasing interest due to their ability
in enhancing the reasoning capabilities of deep neural networks by
incorporating symbolic reasoning, such as logic. Recent advances indicate that
it is possible to mitigate the hallucination problem of LLMs and enhance the
factual accuracy of their responses by incorporating knowledge graphs (KGs)
(Pan et al., 2024b). These approaches are considered neuro-symbolic because
KGs essentially implement the principles of description logic Baader et al.
(2017).

We consider Malaysian Contract Law as the target area of law due to the
importance of contracts in everyday life. The corresponding SKG is
automatically constructed from the textbook “Law for Business” (Trakic et al.,
2022), the Contracts Act 1950 (the primary legislation governing contracts in
Malaysia), and 76 court cases pertinent to contracts downloaded from Malaysia
e-judgement 222e-Judgement:
https://cms2.kehakiman.gov.my/CommonWeb/ejudgment/SearchPage.aspx?JurisdictionType=ALL.
It is easy to implement rules to extract legal knowledge from legal documents
because The layout of a legal document often resembles the structure of legal
knowledge, as evident by the screenshots of the textbook in Appendix E.

Legal concepts serve as the building blocks of legal doctrine, often act as
bridges that connect related knowledge from diverse sources. For example,
under the Contract Act 1950, Section 2(a) states: "when one person signifies
to another his willingness to do or to abstain from doing anything, with a
view to obtaining the assent of that other to the act or abstinence, he is
said to make a proposal;" . This section is linked to paragraph P4-014 in the
text book via the legal concept "offer".

We derive the skeleton of the SKG from the textbook and enrich the skeleton
with statutes from the primary legislation. The index of the book organizes
the key concepts of contract law hierarchically, as illustrated in Appendix E.
We extract those concepts from the index and annotate them as nodes at the
corresponding levels, such as main_concept and subconcept. The children nodes
are linked to their parent nodes using the relation subconcept_of.
Additionally, we represent each chapter title as a node, indicating specific
aspects of the Contract Act 1950, such as Void Agreements.

Furthermore, we extract the titles, section titles etc. from the Contracts Act
1950 and represent each as a node. Then we introduce several relations to
associate the nodes derived from the book with the relevant ones in the
legislation. For example, each chapter is associated with the relevant
sections of the legislation. Figure 2 shows a snippet of the SKG.

Figure 2: A snippet of the SKG.

To bridge the language gap between legalese and plain English, we extract
interpretations from the book that provides layman explanations of the
corresponding statutes in the legislation. Each interpretation is represented
as a node in the SKG, and the mentions relation is used to link an
interpretation to the relevant statute. Overall, the SKG comprises 3,114 nodes
and 1,811 edges, stored in Neo4j for easy data exchange. Further details about
the SKG can be found in Table 4 in Appendix C.

###  2.2 Corpus Construction

From a pool of applicants from Malaysia, we carefully selected six data
annotators to assist with scenario selection and annotating IRAC analysis.
This annotator team comprises four second-year law students from three
distinct Malaysian universities and two junior lawyers. The annotated corpus
comprises 54 legal scenarios covering five chapters in the textbook and 55
subtopics, ensuring extensive coverage of various aspects of Malaysian
Contract Law. Each scenario is designed to reflect real-world legal problems.
The rigorous annotation task is challenging, because it takes approximately
three hours to annotate one scenario with legal concepts and complete IRAC
analysis, though we develop an easy-to-use annotation tool (Appendix D), which
significantly improve the productivity of annotators.

####  2.2.1 Scenario Selection

To ensure diversity of scenarios and coverage of legal concepts pertinent to
formation of contracts, we gather scenarios based on the law textbook “Law for
Business” Trakic et al. (2022) used by law students when studying contract
law. In particular, we choose five main topics: offer and acceptance,
consideration, certainty, capacity, and intention to create legal relations.
The corresponding chapters in the text book are Chapter 4 "Formation of
Contract: Proposal and Acceptance", Chapter 5 "Consideration", Chapter 6
"Promissory Estoppel", and Chapter 7 "Intention to Create Legal Relationships
and Capacity". The section headings of these chapters represent the
corresponding subtopics, such as proposal, acceptance, and minors etc.. There
are 55 unique subtopics extracted from the subheadings of the text book.

First, we asked two annotators to create 24 scenarios which were modified from
tutorial questions, books, and past exam questions. Next, for the remaining
subtopics, we utilized ChatGPT to suggest candidate scenarios with the prompt
: " You are a legal professional, based on the example scenarios, main topic,
and subtopics, create a new scenario around avg_length"̇. The average length
is calculated based on the human-authored scenarios. This parameter is used to
guide ChatGPT to generate scenarios with a length that matches those curated
by humans. As the result, the main topics are evenly distributed among all the
scenarios, and each subtopic is covered by at least one scenario. To ensure
the quality of the scenarios, another two of the six law students evaluated
the quality of the candidate scenarios using the following questions, as shown
in Fig. 3.

Figure 3: A scenario with quality assessment questions.

Based on annotators’ feedback, our experts revised all 54 scenarios to ensure
their quality. Their average length is 800 words.

####  2.2.2 IRAC Annotations

Legal concepts act as a bridge, connecting the facts in a scenario to the
professional legal knowledge, including statutes and precedents. Therefore, we
first annotated legal concepts, followed by performing IRAC analysis. The
detailed annotation guideline is provided in Appendix A.

##### Legal Concepts.

Using the legal concepts in the SKG, annotators were asked to identify and
highlight relevant legal concepts within a given scenario. They were
instructed to look up legal concepts first in the textbook “Law for Business”,
which ensures that the identified legal concepts are part of the SKG. If a
concept, such as “offeror”, is not absent from the textbook, they are allowed
to add new concepts into the SKG.

##### Issues.

A legal issue is a point of dispute that involves the interpretation,
application, or violation of laws and regulations. Six annotators were asked
to identify the issues in given scenarios. Across the scenarios, the main
question is whether there is a valid contract between involved parties. To
answer the question, the target problem is decomposed into issues, which are
articulated in the style of questions. For example, an issue in our example
scenario (Fig. 1) is “Whether there was an acceptance on the part of
Vanessa?”.

##### Rules.

A rule specifies the laws applicable to the issues. The annotators are asked
to locate the appropriate cases and/or statute sections from the Contract Act
1950 pertinent to issues. For statutory law, the annotation tool offers a
drop-down menu to select relevant sections from the 280 sections available in
the SKG. For case law, the tool includes text fields for related court cases
along with the corresponding page numbers in the cases. For instance, Eckhardt
Marine GMBH v Sheriff, High Court of Malaya, Seremban & Ors [2001] 4 MLJ 4
(CA) [3/4]. To enable reuse and reference of those rules, the provided cases
are displayed as buttons in the user interface so that annotators can refer to
those cases by clicking on the buttons (see Fig. 9 in the Appendix D).

##### Application.

In the Application section, annotators applied the rules identified in the
rules section to the specific facts of the issues in a given scenario step by
step. They are encouraged to use the conditional statements in form of
“IF…THEN…” to articulate each reasoning step. Figure 7 in Appendix B
illustrates tje application section of our example. As legal reasoning is
defeasible, annotators can make assumptions due to incomplete information.
Different assumptions may lead to different conclusions, thus it is essential
to discuss and justify these assumptions in the corresponding reasoning step.
The application is the most important and challenging section of an IRAC
because it develops the answer to the issue at hand.

##### Conclusion.

The conclusion section directly answers the questions in the issue section,
without introducing any new rules and analysis. Following the common practice
in law, annotators were asked to write the full sentence of a conclusion, such
as “ There is no contract between Vanessa and Niko.”

####  2.2.3 Data Quality Assurance

Given a scenario, there are many plausible IRAC analysis, because different
assumptions and different interpretations of rules may lead to different
conclusions. As it takes roughly three hours to perform a single IRAC analysis
and it is infeasible to annotate all possible IRAC, we verified the quality of
an IRAC analysis by asking another annotator to act as an evaluator.
Specifically, an evaluator can either agree, disagree, or partially agree with
an IRAC analysis. The ratio of overall agreements across all 54 scenarios
exceeds 0.8, indicating a high level of annotation quality.

##### Annotator Quality.

IRAC analysis is challenging. Hence, as mentioned above, we selected only
annotators who have a strong legal background. The law students were required
to have achieved at least a B grade in related law subjects. All annotators
must pass a specialized pre-test before being recruited. Financial
compensation is MYR30 per hour, above the minimum wage MYR7.21 in Malaysia,
reflecting the complexity and rigour of the annotation tasks.

|  | No.
---
Scenario
| Full
---
IRAC
| Legal
---
Concept
Rules |  | Annotated
---
Application
SKG
SIRAC | 40 | yes | 0 | 58 | Yes | No
LegalSemi | 54 | yes | 297 | 90 | Yes | Yes
SARA_entailment | 277 | no | 38 | 9 | No | No
SARA_numeric | 100 | no | 38 | 9 | No | No
Legal Bench | 59 | no | 0 | 18 | No | No

Table 1: Comparison between LegalSemi and the most relevant datasets.

####  2.2.4 Summary of the Corpus

Our corpus comprises 54 scenarios, 243 issues as decomposed questions, 197
mentions of legal concepts (70 of them are unique), 268 sections of the
Contracts Act 1950 (44 of them are unique), 76 court cases, and 607 reasoning
paths. On average, each application involves 11.25 reasoning steps to draw a
conclusion for the main questions. The most common legal concepts encountered
include "offeror", "offeree", and "proposal", reflecting the frequent focus on
contract formation. Similarly, the law sections most often cited are s2(a),
s2(d), s7(a), s2(e), and s2(b).

##### Dataset Comparison.

We compare our corpus with the publicly available corpora in Table 1. Among
them, SIRAC Kang et al. (2023) is the only one that includes annotations of
full IRAC analysis for legal scenarios. LegalSemi improves upon their work by
i) adding a SKG to support neuro-symbolic approaches, ii) introducing
annotations of legal concepts to facilitate evaluation of neuro-symbolic
approaches, iii) decomposing main legal questions into scenario-based issues,
instead of using fixed issues across scenarios as in SIRAC, and iv) include a
test set with longer scenarios, closer to the real world scenarios, that
require more complex reasoning.

SARA Holzenberger and Van Durme (2021) focuses on legal question answering in
Taxation Law. It annotates structured reasoning paths involving merely seven
rules in total for the QAs but does not include any annotations of IRAC. Legal
Bench Guha et al. (2022) covers diverse legal AI tasks but does not include
full IRAC analysis, particularly the detailed analysis in Application.
Instead, it designs simple QAs or classification tasks for a certain IRAC
stage or reasoning steps, such as asking if a specific rule is relevant to a
simplified scenario or not.

##  3 Experiments and Results

We empirically demonstrate the usefulness of LegalSemi for IRAC analysis and
highlight the open challenges for future research.

###  3.1 Legal Concept Identification

Legal concepts play a key role in neuro-symbolic approaches for legal
reasoning by linking facts in scenarios with legal knowledge. We investigate
how accurate the state-of-the-art (SOTA) LLMs can identify legal concepts in
scenarios, because LLMs demonstrate remarkable performance on zero-shot and
few-shot learning (Brown et al., 2020).

We adopt four LLMs for legal concepts identification: GPT-3.5 turbo, Llama 2,
Mistral, and Gemini. The configurations of those models are detailed in
Appendix G. Our prompt for those models is shown in Fig. 13. It begins with
instructing the LLM to select relevant concepts from a comprehensive list of
concept candidates, followed by providing a scenario and main legal questions.
At the end of the prompt, it specifies the output format to a Python list for
easy post-processing.

Figure 4: Comparison of F1 Score for predicting both high-level and lower-
level concepts.

##### Evaluation Details.

We extract a list of legal concepts from each model output, and compare them
with the ground truth concepts in terms of precision, recall and F1 score. As
the concepts are organized into a hierarchy in the textbook, we report the
results for top-level and lower-level concepts, respectively, in order to
highlight open challenges.

##### Results.

As illustrated in Figure 4, all four LLMs perform significantly better at
predicting top-level concepts compared to the lower-level ones. For the top-
level concepts, GPT-3.5 turbo achieves the highest precision (35%), while
Gemini obtains the highest recall (93%). We conjecture that compared to top-
level concepts, e.g. "invitation to treat", the lower-level concepts associate
with specific details of contracts, such as "audition" and "advertisement".
Hence, they appear less frequently in the pre-training data of LLMs. This
sheds light on the importance of constructing dedicated supervised training
data for future research.

###  3.2 Issue Identification

Prior works Kang et al. (2023); Guha et al. (2022) employ a set of fixed
issues to decompose a main legal questions into simpler questions. Since
issues can vary significantly from case to case in practice, we investigate
the extent to which LLMs can generate scenario-based issues and identify the
helpfulness of legal concepts at this stage.

We apply the same four LLMs as in concept identification for issue generation.
The prompt instructs LLMs to break a main legal question into a list of issues
by leveraging relevant ground-truth legal concepts. In the prompt, we also
instruct an LLM to self-evaluate its outputs by ensuring they are reasonable,
as inspired by (Hao et al., 2023). The prompt is detailed in Appendix H.

##### Evaluation Details.

As issue generation is a language generation task, following (Kang et al.,
2023), we apply GPT-3.5 turbo to compare predicted issues with annotated
reference issues with a list of criteria detailed in the prompt (see Appendix
H). An LLM is expected to select one option from: strongly agree, neutral, or
disagree, which is further mapped to a score of 1, 0, and -1, respectively.

To investigate the quality of this automatic metric, we compare the results of
the automatic evaluation with those of human evaluation. In the human
evaluation, we assess the quality of an IRAC analysis using a rubric that is
widely used in Malaysian contract Law courses Gerhardt (2008); Carter (2006).
The issues of an IRAC analysis receive a Pass when they satisfy the
corresponding criteria detailed in Appendix F.1, otherwise, they are marked as
Fail.

We ask two annotators to independently assess the issues generated by two best
performing models (Gemini and GPT-3.5 turbo) in three different configurations
(e.g. with or without legal concepts) on 10 randomly selected scenarios. In
case of disagreement between their assessments, we ask the most experienced
annotator with a strong legal background to resolve it. Each generated output
marked as Pass receives a score of 2, otherwise, a score of 1. We then rank
all LLM configurations according to their average scores and compute the
Spearmann rank correlation with the counterpart using the automatic metric. A
strong correlation of 0.8 suggests the effectiveness of the automatic metric.
Further details are in Appendix F.

Figure 5: The results of issue identification.

##### Results.

Figure 5 depicts the average scores of the automatic metric across all 54
scenarios. Legal concepts are beneficial for all LLMs especially for GPT-3.5
turbo, which increased by 21.4%. The self-evaluation instruction further
enhances the performance of all LLMs, with the best performance observed when
both legal concepts and self-evaluation are combined.

No initial retrieval |  | index: legalese
---
| index: interpret (text book)
---
| index: GPT_interpret
---
@ top5 | @ top10 | @ top50 | @ top5 | @ top10 | @ top50 | @ top5 | @ top10 | @ top50
Precision | 2.60% | 1.70% | 1.40% | 4.30% | 4.90% | 7.80% | 3.30% | 4.40% | 3.20%
Recall | 2.90% | 3.30% | 12.50% | 0.90% | 1.85% | 15.70% | 2.30% | 9.00% | 29.40%
F1 score | 2.50% | 2.00% | 2.50% | 1.50% | 2.54% | 9.50% | 2.60% | 5.50% | 5.60%
Initial retrieval |  | index: law
---
| index: interpret (text book)
---
| index: GPT_interpret
---
with legal concepts | @ top5 | @ top10 | @ top50 | @ top5 | @ top10 | @ top50 | @ top5 | @ top10 | @ top50
Precision | 9.70% | 7.50% | 3.10% | 11.80% | 13.30% | 11.80% | 10.30% | 9.00% | 4.40%
Recall | 32.20% | 32.60% | 37.20% | 35.30% | 31.20% | 35.30% | 33.20% | 36.50% | 48.50%
F1 score | 13.90% | 11.50% | 5.60% | 16.30% | 17.20% | 16.30% | 14.60% | 13.50% | 7.90%

Table 2: Results for rule retrieval. GPT_interpret denotes using the
interpretations generated by GPT-3.5 turbo.

###  3.3 Rule Retrieval

Given a scenario annotated with legal concepts, we investigate what
information in the SKG is beneficial for retrieving relevant legal rules from
the Contract Act 1950. A key challenge herein is the gaps between the lay
language used in scenarios and the legalese used to express legal rules. Given
a scenario as the query, we apply a TF-IDF based search engine (Pedregosa et
al., 2011) to retrieve rules indexed by three types of representations,
detailed below. Those experimental results are compared with the setting that
only the legal rules associated with the same legal concepts pertinent to the
scenario are considered for retrieval. This is achieved by performing
retrieval in two stages: i) sending the legal concepts of a scenario as the
structured query to the SKG to identify the set of legal rules associated with
those concepts, and ii) using the TF-IDF based search engine to rank the legal
rules in the results of the initial retrieval.

##### Indexing.

We consider three representation types of a legal rule for indexing: i)
original legalese, ii) interpretation extracted from the textbook, and iii)
combination of the interpretations from the textbook and the additional
interpretation generated by GPT-3.5 turbo (detailed in Appendix H), because
the textbook covers only 18.5% of the legal rules.

##### Evaluation Metrics.

We consider precision, recall, and F1 scores at top-$k$ retrieved results,
where $k=$5, 10, and 50, respectively.

##### Results.

Table 2 shows that the retrieval of legal concepts can be used effectively as
the search scope for scenario-based retrieval, where it boosts both precision
and recall of rule retrieval. Using the interpretations from the text book as
index further improves the retrieval quality overall. The highest F1 score at
top-5 reaches 16.3%. In contrast, direct application of those LLMs for rule
generation falls below 3% at top-$5$. Those automatically generated
interpretations are beneficial only in terms of recall. In addition, it can be
observed that the quality of the generated interpretations are often prone to
the hallucination problem of GPT-3.5 turbo.

Figure 6: Results of application generation.

###  3.4 Application Generation

We investigate the effectiveness of utilizing issues and rules for application
generation. We reuse the same four LLMs in the previous stages and apply the
prompt (Figure 16 in the Appendix H) to generate legal analysis.

##### Evaluation Details.

Similar to issue generation, we apply GPT-3.5 turbo to compare the generated
application sections with the annotated reference for each scenario. The
possible outcomes of the evaluation include strongly agree (1), neutral (0),
or disagree (-1). We also conduct a similar human evaluation to assess the
quality of this automatic evaluation metric, and obtain a correlation of 0.86.
The details are covered in Appendix F.

##### Results.

Figure 6 shows the results when the prompts were incrementally added with
issues and rules. Notably, all LLMs greatly benefit from the identified issues
except for Gemini. GPT-3.5 turbo shows the most significant increase in
performance, with an improvement of 18.9% from the self-evaluation prompt to
the self-evaluation prompt with issues and rules. These results highlight the
effectiveness of incorporating rules and issues in the legal reasoning steps.

###  3.5 Conclusion Generation

We apply the same LLMs to generate conclusions, based on given scenarios, main
questions, and reference application sections. The details of the prompt can
be found in Appendix H.

##### Evaluation Details.

Same as the previous IRAC stages, we apply GPT-3.5 turbo to evaluate
conclusions by comparing them with ground truth, as detailed in Appendix F.
The correlation with the human evaluation results is 91%, as presented in
Appendix F.

##### Results.

Figure 21 in Appendix H illustrates the automatic evaluation results, which
indicate improvements across all models. Notably, GPT-3.5 turbo and Gemini
exhibit substantial enhancements. ChatGPT’s score improved by 71.4%. These
significant gains underscore the effectiveness of incorporating the
application component into the prompt, aligned with findings from Kang et al.
(2023).

##  4 Related Work

##### Legal Reasoning

Savelka et al. (2023) analyzed how effectively GPT-4 produces definitions for
legal terms found in legislation. Huang et al. (2023) addressed the challenge
of improving Large Language Models (LLMs), such as Llama 2, for domain-
specific tasks in the legal field. Legal Bench Guha et al. (2022) is created
through an interdisciplinary procedure for legal scenario analysis using the
IRAC methodology. However, their work did not utilize the same legal scenarios
for the completed IRAC tasks. Large Language Models (LLMs) have demonstrated
significant reasoning abilities, especially when chain-of-thought (CoT)
prompting Wei et al. (2022) is employed. Hu et al. (2023) applied LLM to
generate a reasoning chain along with the final answer given the legal
question. Hao et al. (2023) proposed Reasoning via Planning (RAP). RAP
enhances the LLM with a world model and employs principled planning, namely
Monte Carlo Tree Search (MCTS), to generate high-reward reasoning traces
following effective exploration, demonstrating its superiority over several
contemporary CoT-based reasoning approaches. However, these approaches,
including RAP, have yet to be applied in the legal domain as artificial
intelligence (AI) for legal tasks requires highly domain-specific legal
knowledge rather than just common sense knowledge.

##### Structured Knowledge Graph

SKILL Moiseev et al. (2022) demonstrated that the results show improvements
with pre-trained models on the Wikidata KG, beating the T5 baselines on
FreebaseQA, WikiHop, and the Wikidata-answerable subset of TriviaQA and
NaturalQuestions. Knowledge graphs with external knowledge can help the model
improve accuracy and reduce confusion. Leveraging the power of structured
knowledge graphs is able to enhance the performance of the LLMs. For legal
reasoning, we need highly specialized legal knowledge to ensure accurate
answers. Unfortunately, the current approach mainly focuses on common sense
knowledge.

##  5 Conclusion

We introduce LegalSemi, which consists of 54 scenarios annotated with IRAC
analysis in Malaysian Contract Law, and a SKG for legal knowledge extracted
from a law textbook and legislation. The SKG covers legal concepts, legal
rules, interpretations in lay language and their relations. Legal concepts
from the SKG are particularly useful for improving the quality of issue
generation, which in turn significantly enhance legal analysis in Application
across all four evaluated LLMs. Besides, LLMs fall short of identifying
relevant legal rules accurately by having the mean precision at top-$5$ below
3%. By leveraging the SKG, we achieve a significant improvement in rule
retrieval, with an increase of 17.2% in F1 score at top-5.

## Ethical Statement

Our research practices align with the principles of the ACL Code of Ethics.
Our investigation complies with these ethical standards. LegalSemi was created
and evaluated with a keen awareness of ethical considerations, especially
regarding the involvement of human annotators. We recognize that the necessity
for human-annotated data to train conditional independence classifiers in our
method demands significant effort. We have taken careful measures to ensure
that this process is ethically sound, honoring the annotators’ contributions
by respecting their time and providing equitable compensation. Moreover, the
central objective of LegalSemi is to create an IRAC methodology-based
benchmark. It is designed without generating any information that could be
deemed harmful or violate privacy.

## Limitation

In this study, our primary emphasis revolves around examining scenarios that
pertain specifically to the ’Formation of Contract’ as delineated within
Malaysian Contract Law. While our dataset may exhibit limitations in terms of
the breadth of legal scenarios available for analysis, it remains robust in
its coverage of all essential topics related to contract formation. Despite
potential constraints, such as data availability or accessibility, our dataset
is meticulously curated to encompass a comprehensive spectrum of scenarios
relevant to the legal domain, ensuring a thorough investigation into the
intricacies of contract formation under Malaysian law.

Furthermore, an additional limitation inherent in our study lies in the
selection of LLMs employed for our experiments. Our study opts for a more
focused approach by utilizing a limited subset of these models. While this
decision may result in a narrower scope of analysis compared to studies
incorporating a broader array of LLMs, it ensures consistency and reliability
in our experimental methodology. Despite this limitation, our choice of
employing the most widely used and recognized LLM ensures that our findings
are grounded in established practices within the field of natural language
processing and legal analysis.

## References

  * Baader et al. (2017) Franz Baader, Ian Horrocks, Carsten Lutz, and Uli Sattler. 2017.  _Introduction to description logic_.  Cambridge University Press.
  * Billi et al. (2021) Marco Billi, Roberta Calegari, Giuseppe Contissa, Francesca Lagioia, Giuseppe Pisano, Galileo Sartor, and Giovanni Sartor. 2021.  Argumentation and defeasible reasoning in the law.  _J_ , 4(4):897–914.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Carter (2006) John W Carter. 2006.  Carter’s guide to australian contract law.  _(No Title)_.
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_.
  * Gerhardt (2008) Michael J Gerhardt. 2008.  How a judge thinks.  _Minn. L. Rev._ , 93:2185.
  * Guha et al. (2022) Neel Guha, Daniel E Ho, Julian Nyarko, and Christopher Ré. 2022.  Legalbench: Prototyping a collaborative benchmark for legal reasoning.  _arXiv preprint arXiv:2209.06120_.
  * Hao et al. (2023) Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023.  Reasoning with language model is planning with world model.  _arXiv preprint arXiv:2305.14992_.
  * Holzenberger and Van Durme (2021) Nils Holzenberger and Benjamin Van Durme. 2021.  Factoring statutory reasoning as language understanding challenges.  _arXiv preprint arXiv:2105.07903_.
  * Hu et al. (2023) Tongxin Hu, Zhuang Li, Xin Jin, Lizhen Qu, and Xin Zhang. 2023.  Tmid: A comprehensive real-world dataset for trademark infringement detection in e-commerce.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_ , pages 176–184.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_.
  * Kang et al. (2023) Xiaoxi Kang, Lizhen Qu, Lay-Ki Soon, Adnan Trakic, Terry Yue Zhuo, Patrick Charles Emerton, and Genevieve Grant. 2023.  Can chatgpt perform reasoning using the irac method in analyzing legal scenarios like a lawyer?  _arXiv preprint arXiv:2310.14880_.
  * Kim et al. (2024) Kyungha Kim, Sangyun Lee, Kung-Hsiang Huang, Hou Pong Chan, Manling Li, and Heng Ji. 2024.  Can llms produce faithful explanations for fact-checking? towards faithful explainable fact-checking via multi-agent debate.  _arXiv preprint arXiv:2402.07401_.
  * Krasadakis et al. (2024) Panteleimon Krasadakis, Evangelos Sakkopoulos, and Vassilios S Verykios. 2024.  A survey on challenges and advances in natural language processing with a focus on legal informatics and low-resource languages.  _Electronics_ , 13(3):648.
  * Madhana and Subhashree (2022) B Madhana and S Subhashree. 2022.  A study on backlog of cases.  _Issue 5 Int’l JL Mgmt. & Human._, 5:942.
  * Metzler (2002) Jeffrey Metzler. 2002.  The importance of irac and legal writing.  _U. Det. Mercy L. Rev._ , 80:501.
  * Moiseev et al. (2022) Fedor Moiseev, Zhe Dong, Enrique Alfonseca, and Martin Jaggi. 2022.  Skill: structured knowledge infusion for large language models.  _arXiv preprint arXiv:2205.08184_.
  * Pan et al. (2024a) Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024a.  Unifying large language models and knowledge graphs: A roadmap.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Pan et al. (2024b) Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024b.  Unifying large language models and knowledge graphs: A roadmap.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Sajid (2023) Haziqa Sajid. 2023.  Leveraging the power of knowledge graphs: Enhancing large language models with structured knowledge.  Wisecube AI Blog.
  * Savelka et al. (2023) Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023.  Large language models (gpt) struggle to answer multiple-choice questions about code.  _arXiv preprint arXiv:2303.08033_.
  * Statistics (2013) Laerd Statistics. 2013.  Spearman’s rank-order correlation.  _Laerd Statistics_ , page 33.
  * Trakic et al. (2022) Adnan Trakic, Nagiah Ramasamy, Cheah You Sum, Paul Linus Andrews, Sri Bala Murugan, P Vijayganesh, and Kanchana Chandran. 2022.  _Law for Business, Third Edition_.  Sweet & Maxwell Malaysia.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.

##  Appendix A Annotation Guidelines

##### Project Overview

Develop a machine learning system for in-depth analysis of legal scenarios,
specifically focusing on Contract Law utilising the IRAC (Issue, Rule,
Analysis, and Conclusion) methodology.

##### Methodology:

Apply Contract Law principles to annotate data using the IRAC framework.

##### Project Requirements

  * •

Contract Law Expertise: A comprehensive understanding of Contract Law,
particularly in relation to contract formation, is essential. You need to have
B+ and above for the related subject.

  * •

Responsibility and Time Management: Commitment to assigned tasks and timely
completion is crucial.

  * •

Basic IT Knowledge: Familiarity with computer systems and basic IT concepts is
preferred.

  * •

Communication and Teamwork: Strong communication skills and ability to
collaborate effectively within a team are important.

  * •

Pass the pre-test before starting the real annotation work.

##### Data Annotation Outcomes

  * •

Publication: The annotated dataset will be used for benchmarking and may be
published in a journal or presented at a conference.

  * •

Further Research: The annotated data will serve as a resource for subsequent
machine learning research.

##### Benefits

  * •

Research Assistant Experience: Opportunity to work as a Data Annotator on a
research project.

  * •

Flexibility: Remote work with flexible hours.

  * •

Compensation: RM 30 per hour.

##### Annotation Tasks

  * •

Evaluation of Legal Scenarios: Analyse and evaluate legal scenarios as per the
IRAC framework.

  * •

IRAC Analysis for Contract Formation: Apply IRAC methodology to analyse
contract formation in provided scenarios.

  * •

Decomposed Questions and Court Case References: Generate relevant decomposed
questions for each IRAC segment and include related court cases with page
numbers.

##  Appendix B Examples of the Application annotation.

Figure 7 exemplifies our annotation process for legal scenario analysis using
the IRAC methodology. It demonstrates the structured approach we take to break
down and evaluate each aspect of a legal problem. The figure uses logical
steps to progress from identifying an initial legal issue to applying relevant
rules and statutes, analyzing the facts, and drawing a conclusion. Each step
is clearly annotated with references to legal cases and statutes, ensuring
that the reasoning is well-supported and transparent. The annotations also
include conditional statements and assumptions, highlighting how various legal
principles and factual circumstances are considered to reach a final
conclusion.

Figure 7: An example of the application section.

Table 3 lists all the condition types used in the annotation. We have a total
of six different condition types. The most commonly applied condition type is
the IF…THEN… structure.

Conditional Type | Use | Example | Count
---|---|---|---
If… Then | Used to state a condition and its consequence |  "IF {She placed an advertisement on a social media platform selling a limited edition vinyl for the price of $500 [advertisements: invitation to treat]} is an invitation to treat, THEN the call from a customer, Niko, to reserve that vinyl is an offer." | 54
According to… |  Used to refer to legal cases, statutes, or authoritative sources |  "ACCORDING TO {Eckhardt Marine GmbH v Sheriff (2001) 4 MLJ 49[53-54]}, IF {Lowel[offeror]} puts up {advertisement[Advertisements to treat]}, THEN it is not an offer but an invitation to treat." | 39
Since… Then | Used to show a reason and its result |  "SINCE {Lowel responded by saying that 500 is too cheap, and that the lowest price she is willing to sell is RM 700 .[Counter offers of initial proposal by proposee]} is not absolute and unqualified THEN there is no valid acceptance." | 52
However… If… Then |  Used to introduce an exception or a contrasting condition and its consequence |  "{HOWEVER} IF the agreement between Penny and Tina is not supported by considerations THEN it will be void even if it is supported by intention to create legal relation." | 42
Even if… Then |  Used to express a consequence that applies despite a condition |  "EVEN IF {Nina[Capacity contracting with]} has lied about her age, she would not be estopped from pleading incapacity." | 10
Only if… Then |  Used to indicate that a consequence will occur solely under a specific condition |  IF (7) THEN agreement is supported by intention ONLY IF stated condition is fulfilled. {Confetti Records (A Firm) and others v Warner Music UK Ltd and another [2003] EWHC 1274 (Ch)[]} | 12

Table 3: Application conditional types

##  Appendix C Structured Knowledge Graphs

Structured Knowledge Graphs (SKGs) significantly enhance Large Language Models
(LLMs) by providing organized, interconnected data representations. This
methodical arrangement allows LLMs to make coherent and clear interpretations,
aligning seamlessly with their ability to recognize data patterns and
relationships. This is particularly beneficial in domains that demand
precision, such as scientific research, financial analysis, and medical
diagnostics Sajid (2023).

Legal text often resembles structured knowledge. For example, under the
Contract Act 1970, Section 2(a) states: "when one person signifies to another
his willingness to do or to abstain from doing anything, with a view to
obtaining the assent of that other to the act or abstinence, he is said to
make a proposal;" .This section is related to the legal concept "offer" and
corresponds to paragraph P4-014 in the text book.

Given the nature of legal knowledge and the benefits of SKGs for LLMs, we
design an SKG based on the legal knowledge from book paragraphs, legal
concepts, laws, and court cases. The details of the SKG is shown in the Table
4. Figure 2 shows partial of the graph. This graph illustrates the structure
of a Social Knowledge Graph (SKG) in Neo4j, showcasing the relationships
between various sections, chapters, interpretations, and main concepts through
nodes and edges.

Figure 8The diagram illustrates a section of the SKG, demonstrating the
hierarchical and relational structure of legal statutes. The central pink node
represents a Chapter, which connects to various Section nodes (yellow) through
"BELONGS_TO" relationships. Each Section, like Section with a Title (dark
brown) via "HAS_TITLE" relationships and has Interpretations (orange)
connected by "HAS_INTERPRETATION" links. These Interpretations, such as
P4-109, detail specific provisions and connect to main Concepts (green) and
sub-concepts (light brown).

Figure 8: Details example of SKG in Neo4j.

Node name | Details | No of nodes | Example
---|---|---|---
Chapter |  | Each chapter covers a specific aspect
---
of the Contract Act 1950.
24 | Void Agreements
Title |  | Each title focuses on specific legal
---
points within the Contract Act 1950.
210 |  | Misrepresentation,Acceptance
---
must be absolute
Section |  | Sections of the Contract Act 1950,
---
providing detailed legal
provisions and serving as the main
reference for the statute.
304 |  | Section 5.2, An acceptance may be revoked at any
---
time before the communication …
Interpretation |  | Interpretations of the contract law,
---
automatically extracted from the
book content. Each is labeled with
its content and paragraph IDs.
1623 |  | If a statement does not satisfy the elements of proposal
---
as discussed&above, the statement would
more likely be an invitation to treat or ….
Extend content |  | Footnotes or extensions of the
---
interpretations, sharing the same
paragraph ID as the related interpretation.
307 |  | Partridge v Crittenden [1968] 1 WLR Facts:
---
The defendant advertised the …
Main concept |  | Key legal concepts summarized from the
---
interpretations, auto extracted from the
index of the book content.
189 | proposal revocation
Sub Concept | Detailed extensions of the main concepts. | 351 | communication
Sub sub concept | More detailed information on the sub-concepts. | 106 | thrid party
| Total | 3114 |
Edge name | Details | No of eges | Example
Belongs_to | Connects the Section and Chapter. | 304 |  | chapter_title : OF CONTRACTS, VOIDABLE CONTRACTS
---
AND VOID AGREEMENTS
content: the court regards it as immoral,
or opposed to public policy.
section_id : 24e
title;What considerations and objects are lawful, and what not
has_title | Connects the Section and Title. | 304 | Same as above
mentions | Connects the Interpretation and Section. | 193 |  | The definition of “agent” and “principal” is provided
---
in section 135 of;the Contracts Act 1950…
title “Agent” and “principal”
related_to | Connects the Interpretation and Extend Content. | 307 | Hughes v Metropolitan Rly Co (1877) 2 App ……
concept_of | Connects the Main Concept and Interpretation. | 184 | Acceptance concept_of Under section 3, an acceptance can…..
subconcept_of | Connects the Main Concept and Sub Concept. | 364 | communication sub concept of proposal revocation
subSubconcept_of | Connects the Sub Sub Concept and Sub Concept. | 155 | third paty sub sub concept of communication
| Total | 1811 |

Table 4: The table illustrates the structure of the Social Knowledge Graph
(SKG) implemented in Neo4j. It includes detailed information about the nodes
representing entities and edges depicting relationships between these
entities. The nodes can represent various entities such as people,
organizations, and concepts, while the edges capture the interactions and
connections among them. Attributes associated with nodes and edges are also
detailed, providing a comprehensive view of the SKG.

##  Appendix D Annotation Tool

To facilitate this intricate annotation process, we developed an online data
annotation platform, grounded in the principles of IRAC methodology. It is
designed for universal accessibility, requiring only an internet connection.
It features a ’Review’ function, allowing annotators to refine and adjust
their inputs as necessary. Data output is organized into a structured .json
and ./txt format, significantly enhancing efficiency and streamlining the data
processing workflow for subsequent analysis. Figure 9 shows an example of the
annotation.

Figure 9: The web-based annotation tool developed to enable the legal scenario
analysis. Figure 10: The screenshot of the structure of the textbook.

##  Appendix E Textbook details

Figure 10 showcase various sections and subsections that illustrate the
organization of legal knowledge within the textbook. The index of the book’s
structured format, including headings, subheadings, and bullet points, mirrors
the hierarchical nature of legal documents, making it conducive for rule-based
knowledge extraction. At the end of the index is a link to the paragraph on
that legal concept.

Figure 11: Human Evaluation Guidelines.

##  Appendix F Human Evaluation

Three human evaluators participate in the evaluation session. We select 10
scenarios and two models (Gemini and GPT-3.5 turbo) with all the experiment
settings for them to evaluate. We select GPT-3.5 turbo and Gemini since it has
the best performance from the auto evaluation result. They attend a briefing
meeting to discuss and clarify their understanding of the marking rubric.
After the briefing, they independently evaluate the scenarios. The third
evaluator, who is more experienced, serves as the final decision-maker in
cases where the first two annotators disagree, ensuring the reliability and
accuracy of the final results. This method follows the steps for identifying
issues, which involve decomposing questions for this experiment. The remainder
of the evaluation focuses on the application of these guidelines

###  F.1 Human Evaluation Guidelines

These guidelines are based on the marking rubric and evaluation criteria for
contract law Carter (2006). They determine how and why the conclusion is made.
The answer needs to demonstrate the facts, which include the description of
the circumstances, the main questions, and the issues (decomposed questions)
raised by the question. It should also cover the rules and principles related
to the issues, as well as the conclusions drawn. The proper approach is
similar to that of the court’s judgments. The grade in figure 11 shows is a
helpful step-by-step process to evaluate the legal reasoning process.

##### Human Evaluation Results

Human evaluation is conducted for issue identification, application, and
conclusion. These aspects require human judgment based on expertise and
credentials to ensure accuracy and reliability in the evaluation process.

We compared the human evaluation results with the auto-evaluation results by
examining the rankings of the models’ outcomes.

Figure 12 displays the human evaluation results for all experiments. We
calculate the Spearman’s rank correlation coefficient Statistics (2013) for
each experiment to assess the alignment between human and automated
evaluations.

We rank models based on the ’Pass’ rate and compare it to the ’Agree’ rate
from the automated evaluation matrix. For human evaluation, we use a five-
level grading scale: 1 (Fail), 2 (Pass), 3 (Credit), 4 (Distinction), and 5
(High Distinction), and compare these rankings with the ’Agree’ rate from the
automated evaluation. In conclusion, we compare the entailment rate with the
’Agree’ rate from the automated evaluation matrix.

From the result, the average Spearman’s rank correlation coefficient,
indicated by the red dashed line, is approximately 0.89. This average further
emphasizes the overall strong positive correlation between human and automated
evaluations across different criteria.

Figure 12: Human Evaluation Results.

##  Appendix G Models Details

We apply four Large Language Models (LLMs): GPT-3.5 turbo, Llama 2, Mistral,
and Gemini.

In the GPT-3.5 turbo, our settings include a temperature of 0.7, a common and
general setting for GPT models, balancing creativity and coherence in
responses. We also set the maximum token count to 1000, allowing for extensive
and detailed answers.

We choose Llama 2 70B, Mistral 7B, and Gemini as comparative models to analyze
their performance against GPT-3.5 turbo Turbo in handling complex legal
scenarios. This comparison aims to assess the efficacy of each model in terms
of accuracy, coherence, and relevance to the given legal context. Llama 2 is
selected for its extensive parameter count, which may enhance its ability to
understand intricate details. Mistral 7B is known for its efficiency and
speed, making it an interesting contrast to the larger models. Gemini is
included for its promising performance in previous legal text analyses,
providing a benchmark for evaluation. By comparing these models, we aim to
determine which is best suited for tasks requiring precise legal understanding
and reasoning.

##  Appendix H Experiment Details

##### Legal Concepts prediction

Figure 13 displays the structure of the legal concepts prediction. The figure
shows the different components of the prompt. For different experimental
settings, we sometimes remove the legal concepts list from the potential legal
concepts to compare the results.

Figure 13: The prompt for legal concept identification.

##### Legal Concept Evaluation

We use automatic evaluation metrics for this task. The outcome of the legal
concept list is compared with the ground truth. The comparison is separated
into two different levels: top-level and lower levels. The top level refers to
more general concepts, such as "invitation to treat." The lower level includes
more detailed aspects of the concept. For example, under "invitation to
treat," there are specifics like "audition," "advertisement," etc.

To evaluate the accuracy of our predictions, we use precision, recall, and F1
score. Precision measures the proportion of correctly identified legal
concepts out of all identified concepts. Recall measures the proportion of
correctly identified legal concepts out of all relevant concepts in the ground
truth, indicating the model’s completeness. The F1 score provides a mean of
precision and recall, offering a single metric that balances both aspects of
accuracy.

##### Issue Identification

Figure 14 displays the structure of the issue identification. The figure shows
different components of the prompt. For different experimental settings, we
sometimes remove the legal concepts list from the ground truth to compare the
results.

Figure 14: Details of the experiment of the decompose questions

##### Issue Identification Evaluation

Figure 15 displays the structure of the issue identification evaluation
prompt. The evaluation is based on the evaluation guidelines.

Figure 15: Prompts used for the automatic evaluation of the decomposed
questions.

##### Application Generation

Figure 16 displays the structure of the application generation prompt. The
figure shows different components of the prompt. For different experimental
settings, we sometimes remove the issues or self-evaluation prompt to compare
the result.

Figure 16: Application Prompt.

##### Application Generation Evaluation

Figure 17 displays the structure of the application evaluation prompt. The
evaluation is based on the evaluation guidelines.

Figure 17: Application Evaluation Prompt.

##### Rule Retrieval: Details GPT-3.5 turbo interpretation

We generate the interpretation of the Malaysia Contract Act 1950 using GPT-3.5
turbo to interpret the law. The interpretation is then compared with the
interpretation from textbooks.Figure 18 display the structure of the prompt.
Table 5 shows the example of the output of the interpretation.

Figure 18: Rule interpretation Prompt.

section_id | content | interpreation | real life example
---|---|---|---
s_2a |  | when one person signifies to
---
another his willingness to do
or to abstain from doing anything,
with a view;to obtaining the assent
of that other to the act or abstinence,
he is said to make a proposal;
| A bailment is simply the
---
;transfer of possession and
control of personal property
(goods) from one
;person (the bailor) to another person (the bailee)
for a specific purpose,
;with the understanding that the goods will b
e returned to the bailor or
otherwise disposed of according to their
instructions once the purpose is fulfilled.
| John wants to sell his car and
---
approaches his friend
Mark with an offer.
John tells Mark that he
is willing to sell his car for
$10,000. In this scenario,
John is making a proposal by
expressing his willingness
to sell his car at a certain price.
Mark can either accept or reject the proposal.
s_2c |  | the person making the proposal
---
;is called the "promisor" and the
person accepting the proposal is
called the "promisee";
| In a legally enforceable promise, there are
---
two parties involved. The person who
makes the offer is known as the ’promisor’ and
the person who accepts the offer is
referred to \as the ’promisee.’
| "In a real-estate
---
transaction, the seller proposes
to sell their property to the buyer at a
specific price. The buyer accepts this offer,
resulting in a legally binding
contract between them. In this scenario,
the seller is the ’promisor’ because
they made the offer, and the buyer is the
’promisee’ because they acceptedthe offer.

Table 5: Example of the interpretation.

##### Conclusion Generation

Figure 19 displays the structure of the conclusion generation prompt. The
figure shows different components of the prompt. For different experimental
settings, we sometimes remove the application or self-evaluation prompt to
compare the result.

Figure 19: Conclusion Prompt.

##### Conclusion Generation Evaluation

Figure 20 displays the structure of the conclusion evaluation prompt. The
evaluation is based on the evaluation guidelines.

Figure 20: Conclusion Evaluation Prompt.

##### Conclusion Result

Figure 21 display the result of the conclusion. The comparison is between
adding the application and not adding the application in the prompt.

Figure 21: Result of Conclusion.


# An adaptive self-organizing fuzzy logic controller in a serious game for
motor impairment rehabilitation

Shabnam Sadeghi Esfahlani, Silvia Cirstea, George Wilson and Alireza Sanaei
Department of Computing and Technology
Anglia Ruskin University, Cambridge, UK

Email<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Rehabiliation robotics combined with video game technology provides a means of
assisting in the rehabilitation of patients with neuromuscular disorders by
performing various facilitation movements. The current work presents
ReHabGame, a serious game using a fusion of implemented technologies that can
be easily used by patients and therapists to assess and enhance sensorimotor
performance and also increase the activities in the daily lives of patients.
The game allows a player to control avatar movements through a Kinect Xbox,
Myo armband and rudder foot pedal, and involves a series of reach-grasp-
collect tasks whose difficulty levels are learnt by a fuzzy interface. The
orientation, angular velocity, head and spine tilts and other data generated
by the player are monitored and saved, whilst the task completion is
calculated by solving an inverse kinematics algorithm which orientates the
upper limb joints of the avatar. The different values in upper body quantities
of movement provide fuzzy input from which crisp output is determined and used
to generate an appropriate subsequent rehabilitation game level. The system
can thus provide personalised, autonomously-learnt rehabilitation programmes
for patients with neuromuscular disorders with superior predictions to guide
the development of improved clinical protocols compared to traditional
theraputic activities.

##  1 Introduction

Computational intelligence (CI) and artificial intelligence (AI) can offer
unsupervised and hybrid instruction that enables relearning sensorimotor
function without the need of direct supervision [1] and [2]. Video games and
virtual reality (VR) systems when combined with AI and CI algorithms provide
the possibility of designing and developing an environment through which the
well-being and mobility of a person could be maintained. More generally, video
games can be employed for physiotherapy as well as pain management to distract
awareness [3]. Unlike more traditional therapeutic activities, therapy using
VR or video games does not rely on passive movements that could sometimes
accompany painful manipulation of the limbs.
It is well known that a person can manage a complicated process on his/her own
using heuristics whereby the expert knowledge is acquired by experimentation
or trial and error without the aid of any closed control loop [4]. Actions
based on the expert knowledge requires sophisticated methods to justify the
non-linearity and complexity of real-world problems, especially when these
issues could not be addressed through sufficient theories, mathematical or
analytic models [5]. Systems with complex non-linear control problems require
highly sophisticated computer systems on which to build the knowledge [4]. In
general, it is of vital importance to design a framework that addresses more
efficient and effective diagnostic and rehabilitation methods [1]. This
structure can be implemented using a fuzzy logic that utilises a linguistic
system by employing the terms of natural language and makes decisions based on
prior knowledge of the scheme [6]. It is one of the great strengths of fuzzy
control that expert knowledge is incorporated in the absence of mathematical
theories. This is capable of reflecting reality and able to generalise the
experience in uncertain cases. The Mamdani-Assilian linguistic influence
system is arguably the best known method, which is based on the fundamental
work of Zadeh (1971) [5], [7] and [8].
In this study, an integrated system called ’ReHabGame’ is developed for body
posture monitoring and physical therapy rehabilitation. The framework is based
on a Kinect Xbox One and Myo armband skeleton and joint tracking devices and a
rudder foot pedal that enables navigating the virtual world. The adaptive
rehabilitation system is built on robotic rehabilitation techniques, inverse
kinematics (IK), and Mamdani’s fuzzy logic method. The decisions are made by
the system based on the player’s performance and his/her level of physical
ability. The data is extracted and analysed via Mamdani’s fuzzy logic
according to the sequence of postures and orientations of the player and the
virtual robotic instructor in the 3D environment. ReHabGame is a serious game
designed specifically to cater for the issues of a particular therapeutic task
and involves the development of four different game scenarios. Real-time
feedback and other data are provided through the Kinect Xbox, Myo armband, and
Rudder Pedal devices. The system is developed through the device interfaces
and the 3D virtual games via the Unity3D game engine. The recorded and
measured quantities are imported as input into a fuzzy rule system for
fuzzification. Every rule has a weight (between 0 and 1) from which the
implication method is implemented. Defuzzification is then performed to
convert the collection of data into a crisp output. These output data are
presented as ”progress into the next level, repeat the level, make the level
more simple or the activity was harmful” and are fed back into Unity. If the
crisp output suggests progression to a next level, the game shifts to a more
advanced stage. The work presented here shows the effectiveness of machine
learning techniques in refining physiotherapy arrangements in a way to achieve
a desirable outcome.

###  1.1 Background & Related Work

The development and simulation of fuzzy logic based learning mechanisms
related to robotic rehabilitation is well-documented, utilizing various
devices to emulate human motor learning [2], [9], [10], [11], [12], [13], and
[14].
The relationships between movements of upper extremities identifying joint
angles using a fuzzy logic system have been reported [15]; these authors
analysed a range of joint angles and rhythmical movement variables to design a
fuzzy expert system. [9] utilised fuzzy inference to develop an internal model
of a dynamic environment experienced during planar reaching movements with the
upper limb. They considered trajectory movement and velocity errors as input
variables to update parameters and showed that the fuzzy learning strategy is
successful in incrementally updating an internal model of the environment
dynamics. [16] have introduced and assessed an adaptive home based
rehabilitation framework for wrist training. A fuzzy logic adaptation engine
was used and evaluated while coping with different levels of exercise. A
physical therapy rehabilitation serious game was developed by [17] and is a 2D
game based on the Kinect tracking skeleton in which the authors used a fuzzy
logic method to analyse the sequence of player’s postures which could be
compared against a specification of the prescribed exercise. A self-adaptive
home based rehabilitation game has also been developed by [18], who present a
2D game where gameplay is continuously monitored using a fuzzy system. They
showed that computational intelligence could offer a feasible and robust
method of home rehabilitation. [19] have proposed a fuzzy logic risk-predictor
of Parkinson’s disease through a 2D game called Pumpkin Garden. The game takes
a player’s historical in-game finger movement and other behaviour data as
input and calculates the player’s risk of developing the disease accordingly.

##  2 Skeleton joints and their hierarchy

The Kinect skeleton tracking system detects the position and orientation of
individual joints and the speed of joint movement. The upper limb orientation
with the maximum and minimum values are shown in Fig. 1-a. The joint hierarchy
is illustrated in Fig. 1-b. The joints have their origin in the Spine (SP) and
extend downwards to the centre of the hip (CH) and from there to the right hip
(RH) and the left hip (LH). The SP extends upwards to the middle of the
shoulders (CS) and head (H), towards the right and left shoulders (SR) and
(SL), elbow (ER) and (EL), wrist (WR) and (WL), and hand (RH) and (LH). Fig.
1-c shows the terms used for upper limb movements. The upper arm abduction
(moves the limb laterally away from the midline of the body), adduction (the
opposing action that brings the arm toward the body or across the midline),
flexion (decreases the angle between the bones or bending of the joint), and
extension (increases the angle and straightens the joint).

Figure 1: Skeleton and body orientation, joints and their hierarchy.

##  3 Kinect, Myo Armband, and Saitek Pro Flight Rudder Pedal

Microsoft Kinect Xbox One version 2 (Microsoft in 2010) and Myo armband
(Thalmic Labs in 2016) devices offer a portable 3D motion capture capability,
enabling users to control and interact with a computer in real time. Kinect V2
is illustrated in Fig. 2-b and consists of an infrared laser based IR emitter
and a colour (RGB) camera. It has the capability to detect the position and
orientation of $25$ individual joints, the speed of player’s joint and body
movements, as well as to track gestures performed with a standard controller.
The upper limb is modeled as a kinematic chain consisting of: three degrees of
freedom (DoF) for the shoulder spherical joint (i.e. abduction-adduction,
flexion-extension and internal-external rotation of the upper arm), one DoF
for the revolute elbow joint (i.e. forearm flexion-extension, indicated),
three DoF for the wrist spherical common (i.e. pronation-supination, ulnar-
radial deviation and flexion-extension of the hand).

Figure 2: Devices that are interfaced with the rehabilitation game via
Unity3D.

The Myo armband is shown in Fig. 2-a; it streams the accelerometer, gyroscope,
and electromyography (EMG) data and the inertial measurement unit (IMU) data
at $200$ $Hz$ and $50$ $Hz$ frequency, respectively. The three-axis
accelerometer, magnetometer and gyroscope senses and records motion and
collects real-time data with a high accuracy and precision. It is made of
eight medical grade stainless steel EMG sensors that detect the electric
impulses in the muscles. The armband is connected via Bluetooth USB adapter
and interacts with the VR through the movements of the hand and wrist.
Underlying electrical activity in the muscle is detected (electromyography)
and displayed as a waveform (an electromyogram).
The Saitek Pro Flight Rudder Pedal111http://www.saitek.com/uk/prod/pedals.html
with 3-axes is used for navigation inside the virtual 3D environment and to
simulate movements. The movement is controlled by the player while he/she is
sited and pushing the pedals in alternative sequences for forward movement,
and left or right toe push for rotating towards left or right. It is connected
to the game engine and virtual game via a USB connector and has an adjustable
tension dial that is used to select a suitable resistance in order to allow
exertion of an appropriate pressure within a range of forces. The differential
braking provides independent control of the left and right toe brakes that are
used for moving left and right. The footrests adjust to accommodate a range of
sizes and include non-slip materials to help keep the pedals on the floor
(Fig. 2-c).

##  4 Kinematics of leading joints and Inverse Kinematics

Kinematics refer to the mathematical description of motion without considering
the underlying physical forces. The kinematics of the human body is
specifically concerned with formulating and solving for the translation,
rotation, position and velocity of each body segment in real-world motions
[20]. The movement of a kinematic chain, whether it is a robot or an animated
character, is modeled by the kinematic equations of the chain [20], [21], [22]
and [23]. The IK determines the typical parameters that provide a desired
position of the end-effector to achieve the task and is known as action
planning [23]. IK techniques provide direct control over the placement of an
end effector object at the end of a kinematic chain of joints, solving for the
joint rotations which place the object at the desired location. IK offers an
alternative to explicitly rotating individual joints within a skeleton [21],
[22], and [24]. As such, the virtual robotic avatar is animated using IK in
that the position of an end effector is presented to the algorithm so that the
system automatically computes the joint angles needed to collect the object
(Fig 3-a). The IK equation is defined in Eqn. 1.
The data collected from Kinect, Myo armband and Rudder foot pedal devices are
used to monitor a player’s performance. Below is the list of parameters and
data that can be collected from these devices:

$\begin{array}[]{llllll}\theta_{1}\leftarrow\arctan 2(y,x)\\\
\theta_{2}\leftarrow\arctan 2(z-l_{1},\sqrt[]{x^{2}+y^{2}})-\arctan
2(l_{3}s_{3},l_{2}+l_{3}c_{3})\\\ \theta_{3}\leftarrow\arctan
2(s_{3},c_{3})\\\ \\\
where:\left\\{\begin{array}[]{ll}c_{3}\leftarrow\frac{x^{2}+y^{2}+(z-l_{1})^{2}-l_{2}^{2}-l_{3}^{2}}{2l_{2}l_{3}}\\\
\\\ s_{3}\leftarrow\sqrt[]{1-c_{3}^{2}}\end{array}\right.\end{array}$ (1)

Figure 3: Spinal reflex arc (a) and organisation in the brain and its sensory
neurones (b)

Data taken from Kinect XBox One

  * •

The orientation (O) and position (P) of the wrist ($O_{W}$), ($P_{W}$) the
elbow ($O_{E}$), ($P_{E}$), and the shoulder ($O_{S}$), ($P_{S}$) when the
objects are spawned relative to their final location (effector).

  * •

Average angular velocity (AV) of the wrist($AV_{W}$), the elbow ($AV_{E}$),
and the shoulder ($AV_{S}$). It is defined by the Eqn. 2, where $v(t)$ is the
angular velocity, $t_{1}$ is the starting time of the task and $t_{2}$ is the
time the task is finished [16].

$AV=\frac{1}{t_{2}-t_{1}}\int^{t_{2}}_{t_{1}}v(t)dt$ (2)

  * •

The time an object is ”spawned”, ”reached” and ”collected”.

  * •

Head Tilt $(T_{H})$ and Spine Tilt $(T_{S})$ are used to measure the postural
body stability.

Data taken from Rudder Foot Pedal

  * •

Pressure on Foot Pedal: The amount of pressure applied by each foot are
monitored. It ranges between $[-1,1]$ to determine (I) if the steps are taken
in order (otherwise the forward movement would not take place), (II) measure
the amount of pressure applied on each pedal; Left foot pressure $P_{(LF)}$ is
$-1$, neutral $P_{(N)}$ is $0$ and right foot pressure $P_{(RF)}$ is $+1$. The
data is recorded per frame, and the average value is calculated $P_{(Avg)}$.
If the $P_{(Avg)}$ is negative then this shows the left foot is dominant and
if it is positive the right foot is the dominant one.

Data taken from Myo Armband

  * •

Eight ElectroMyoGraphy (EMG) data sets are collected to evaluate the
electrical activity produced by the muscles of the upper or lower arm. These
data are saved individually as $EMG_{1}$, $EMG_{2}$, $EMG_{3}$, $EMG_{4}$,
$EMG_{5}$, $EMG_{6}$, $EMG_{7}$, and $EMG_{8}$.

  * •

The gyroscope measures the rotation around one of the axes (X, Y, and Z). When
it is combined with the accelerometer, it provides a useful, compact
measurement system, known as inertial measurement unit (IMU). The data is used
to measure the arm movements and its activities when playing the game (motion,
speed, times, vertical and horizontal displacements as well as kinematic
parameters). Data is recorded as $IMU_{X}$, $IMU_{Y}$, and $IMU_{Z}$.

##  5 Adaptive Rehabilitation Games

In designing the ReHabGame, a broad spectrum of rehabilitation exercises were
studied with advice obtained through direct collaboration and consultancy
sessions with physiotherapy experts. As such, the involvement of various body
segments is considered in developing the games to reach the desired goals.
Fig. 3 (b) and (c) show the spinal reflex arc, brain stem and the
musculoskeletal mechanics of the body. As shown, the brain has the important
role in transferring the signals from muscle and motor nerve fibre to sensory
neurones and receptors. Accordingly, data is transferred back to the spinal
cord to trigger motor behaviour or any action. Any impairment in this system
could effect the data transfer between various segments and as a result could
cause reduced functionality of the sensitive parts and connectors. Thus, the
adaptive ReHabGame is developed to assist with various sensorimotor functions
considering the following factors:

  * •

spinal reflexes (reflex of the joint during abnormal stress condition) and
brain stem activity (cognitive awareness of voluntary movements).

  * •

Postural stability and balance in standing and walking. It involves the
control of position and motion of the centre of body mass about the base of
support. In upright posture, instability may cause the whole body to slightly
sway while standing still, so there is a constant subtle fluctuation between
stability and mobility to maintain balance.

  * •

Residual cognitive impairment and deficiencies in the upper and lower limb
movements.

The rehabilitation therapy is performed through four game scenarios [25]. Fig
4-I shows the main menu of the game in that the patient or player’s data must
be entered and will be saved in a folder before choosing any of the games.
This folder accumulates the player’s activity during the game session with the
exact times, dates, joints and muscles data. Fig 4-II illustrates the second
menu with different quantities that are used for parametrizing the
constraints. The parameterized quantities are; maximum and minimum range of
shoulder, elbow and hand orientation and position, the right-handed or the
left-handed game, the object generator algorithms, and some iterations, the
basket size, fruit size, and the speed of fruit that are spawned. The four
scenarios of the ReHabGame are listed below:

  * •

”Reach-Grasp-Collect fruits” game to engage the upper limbs and body posture
that is a ”Two Players” game as depicted in Fig 4-a and the ”Single Player”
game is shown in Fig 4-b. The player/s should reach the fruits generated in
the scene to grab and collect them in the designated basket and gain score.

Figure 4: The main menu of the ReHabGame (I) second menu to adjust constraints
(II), ”Two players” game (a), ”Single player” game (b), Button press and hold
game (c), Slingshot/hit game (d) and maneuvering in the scene and fruit
collection in the virtual world game (e) and (f).

  * •

”Button Reach-Press-Hold” game in that buttons are generated in a matrix
format. The player should reach the buttons and hold them steadily for a
particular amount of time or just reach and press it Fig 4-c.

  * •

”Sling Shot and Hit” game for engaging torso and upper limb. The player should
pull the sling stretching it with the aim of hitting the target cubes on
release Fig 4-d.

  * •

”Manoeuvring and Reach-Grasp-Collect fruits” game to stimulate lower and upper
limb activity. It is done by manoeuvring in the 3D VR scene. The game is
interfaced with the foot pedal to simulate the walking of the avatar. The
player must reach the highlighted random areas in the VR using the foot pedal,
attain the fruits generated and collect them in the basket. The player must
reach those areas and collect the rewards.

Figure 5: General schematic of fuzzy regulation.

###  5.1 Fuzzy Logic and Machine Learning Algorithm

The fuzzy interface is adapted first to ”Reach-Grasp-Collect fruit” game and
then is expanded to other games. The fuzzy logic adaptation system makes
decisions based on the player’s performance and accordingly proposes an
appropriate level of rehabilitation game. It is done based on human’s logic
that mimics the same reasoning strategy [4] and [26]. The intensity of the
game and game progression is based on completion of a task rather than by per-
session. In the fuzzy logic, there is a gradual transition from one value to
another rather than abrupt change (granulation). There are overlapping fuzzy
interval changes that mimic the way in which human perceive linguistic
reasoning. Fuzzy logic provides a framework for the description of imprecise
dependencies and commands through the use of fuzzy $if/then$ or $or/and$
rules. The concepts which play central roles in the application of fuzzy set
are linguistic variable and values, membership functions, and the universe of
discourse [7]. A fuzzy set is a collection of real numbers having partial
membership in the set. The total number of members in the set is specified by
a membership value of $1$, absolute exclusion from the game has a member value
of $0$ and partial membership in the set is determined by a membership value
between $0$ and $1$ [4]. A membership function (MF) is a curve that defines
how each point in the input space is mapped to a membership value in $[0,1]$.
The role of the membership functions is to find the standard range for each
input [4], [9], [15], [27] and [28], MF specifies a fuzzy set $A$ as Eqn. 3,
[29].

$\mu:X\rightarrow[0,1]~{}or~{}A:X\rightarrow[0,1].$ (3)

A fuzzy set $A$ in $X$ is directly specified by the function $(x,\mu_{A}(x))$,
mathematically as Eqn. 4:

$A=\\{(x,\mu_{A}(x))\mid x\in X\\}$ (4)

The collection of numbers on which a variable is defined are the universe of
discourse for the variable. Consider the system where a variable within a
universe of discourse is $X$, and $x$ is a real number where ($x\in X$) and
let $A$ denote a fuzzy set defined on $X$. For these conditions a membership
function $\mu_{A}(x)$ associated with $A$ is a function that maps $X$ into
$[0,1]$ and gives the grade of membership of $X$ as $A$. The MF that is used
for the ReHabgame fuzzy logic system is triangular with the only condition
that it must be in the range of $[0,1]$ [4]. The triangular membership
function is defined by Eqn. 5 where $\alpha$, $\beta$, and $\gamma$ are the
parameters. The fuzzy logic operates on the input $x$ to produce the crisp
output $\mu$.

$\mu_{A}(x:\alpha,\beta,\gamma)=\left\\{\begin{array}[]{llll}0\ \ \ \ \ \ \ \
\ x\leqslant\alpha\\\ \frac{x-\alpha}{\beta-\alpha}\ \ \
\alpha<x\leqslant\beta\\\ \frac{\gamma-x}{\gamma-\beta}\ \ \ \beta\leqslant
x<\gamma\\\ 0\ \ \ \ \ \ \ \ \ x\geqslant\gamma\\\ \end{array}\right.$ (5)

In this study a multi-input and single-output (MISO) fuzzy system is developed
[4]. The MISO model is designed with $n$ inputs and every $i^{th}$ fuzzy if-
then rule ($R^{i}$) is specified by a set of fuzzy input sets $A^{i}_{n}$, an
output fuzzy set $B^{i}$ and a set of parameters $\theta$ that can be
represented using Eqn. 6 [29] and [4]:

$R^{i}:\textbf{{IF}}~{}{and_{n=1}^{N}}~{}X_{n}~{}\textbf{{is}}~{}A^{i}_{n}~{}\textbf{{THEN}}~{}\mu~{}\textbf{{is}}~{}B^{i}(\theta,x_{0})$
(6)

$X_{n}$ and $\mu$ denotes linguistic variables (inputs and output,
respectively), and $B^{i}(\theta,x_{0})$ represents a parameterized linguistic
value of the output linguistic variable $\mu$.
Each triangular function is composed by a set of $if-then$ rules as in Eqn. 7.

$\left\\{\begin{array}[]{lll}$If
$((x\leqslant\alpha)\land(x\geqslant\gamma))\Rightarrow\ \ \ f(x)=0\\\ $If
$((x>\alpha)\land(x\leqslant\beta))\Rightarrow\ \ \
f(x)=\frac{x-\alpha}{\alpha-\beta}\\\ $If
$((x\geqslant\beta)\land(x<\gamma))\Rightarrow\ \ \
f(x)=\frac{\gamma-x}{\gamma-\beta}\\\ \end{array}\right.$ (7)

According to the schematic of the fuzzy system illustrated in Fig. 5 in order
to get crisp output from the system the defuzzification interface is required
in that the contribution of each fuzzy set inferred is individually considered
by means of a characteristic value center of gravity and the final crisp value
is obtained by means of a weighted average aggregation operator [30] and [31].

###  5.2 Fuzzy Logic Controller in the ReHabgame

Two main tasks are needed to design an intelligent system for the ReHabgame.
Firstly the fuzzy operators involved in the Inference System need to be
selected, and secondly adequate knowledge about the problem being solved needs
to be derived [31], [32], [33], and [34]. The intelligent fuzzy logic
inference system proposes an exercise that is more appropriate for the player
based on his/her performance with out any further harm. The normality factor
of the training is adopted on input parameters that provides a quantified
value that describes how well a patient performed in a particular task [17]
and [19]. Input quantities are standardised and transferred for fuzzification
through fuzzy sets, a set of predefined rules are applied to fuzzy sets
through the fuzzy inference that moves them through defuzzification and
denormalization step to the crisp output $\mu$.
Below is the list of steps towards designing the fuzzy logic system:
Input and output variables: The input variables that are used depend on the
subjective assessment and the severity of the constraints in the games. A
maximum of thirteen fuzzy variables (inputs) are considered depending on each
specific task and the constraints and potentially generate an output which
directs the player towards Progression”, Repetition”, ”Simplification” or
”Harmfulness”. This input data can be activated or deactivated upon request by
the clinician or the player before attempting the game depending on the
constraints that need to be applied. Four subsets are assigned to the input
variables position, orientation and average angular velocity as follows: ”Very
Good (VG), Good(G), Bad (B), and Harmful (H)”. The timing subsets are ”Very
Good (VG), Good(G), Bad (B)”. Some fuzzy logic input functions are described
below along with the output function.

  * •

$O.E_{W}$, ($P.E_{W}$), $O.E_{E}$, ($P.E_{E}$) and $O.E_{S}$ ($P.E_{S}$): The
Orientation (and Position) deviation (error) of the Wrist, Elbow and Shoulder,
defined as the difference between the player controlled avatar joints
orientation (and position) against virtual robotic avatar. The orientation
subsets errors change within the range of $[0^{\circ},90^{\circ}]$ where
$0^{\circ}$ is ”VG” and $90^{\circ}$ is ”H”. and the positions are within
$[0,90]$. Where $0$ is ”VG” and $90$ is ”H”.

  * •

$AV.E_{W}$, $AV.E_{E}$, and $AV.E_{S}$: The Average Angular Velocity Error of
the Wrist, Elbow and Shoulder. Defined as the difference between the joints
average angular velocities. The element of a universe of discourse is in the
range of $[0,90]$ where $0$ is ”VG” and $90$ is ”H”.

  * •

$T.E_{H}$ and $T.E_{S}$: The Head and spine Tilt Error. These parameters
monitor the player’s upright pose (the body posture) in that the spine tilt
error $T.E_{S}$ is in the range of $[0^{\circ},36^{\circ}]\deg$ in that
$36^{\circ}$ is ”H”. The head inclination is in the range of
$[0^{\circ},32^{\circ}]$ in each direction where $0^{\circ}$ is ”VG” and
$32^{\circ}$ is ”H”.

  * •

$T.E_{C}$ and $T.E_{R}$: The Collection and Release Time Error, defined as the
difference between the collection and release time. These two variables are in
the range of $[0,6]$ where $0$ is ”VG” and $6$ is ”B”.

  * •

The output variable ”GameProgress”: with four subsets ranging in $[0,80]$.
Where $[0-20]$ is ”Progression”, $[20-40]$ is ”Repetition”, $[40-60]$ is
”Simplification” and $[60-80]$ is ”Harmfulness”.

Figure 6: Some examples of fuzzy logic input variables defined in the
ReHabGame.

The triangular fuzzy set diagrams are consistent within the constraints of the
above conditions with small arbitrary error overlaps (Fig. 6). This follows a
standard practice for fuzzy set design of a new system and simplifies the
calculations so reducing the demand on computer resources.
Linguistic Rules and Constrains: The control rules are implemented using fuzzy
conditional statements (if/then, or/and, and not) and the relations between
subsets. $A\times B$ (maximum/or), $A+B$ (minimum/and) and $\Gamma A$
(negation/not), Eqn. 8.

$\begin{array}[]{lllllll}A\times
B=\sum_{i}\sum_{j}min\\{\mu_{A}(u_{i}),\mu_{B}(\nu_{j})\\}\\\
A+B=\sum_{i}\sum_{j}max\\{\mu_{A}(u_{i}),\mu_{B}(\nu_{j})\\}\\\ \Gamma
A=1-\sum_{i}{\mu_{A}(u_{i})}\\\ \Gamma B=1-\sum_{j}{\mu_{B}(\nu_{j})}\\\
~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\small{i=1,2,3,...,m}\\\
~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\small{j=1,2,3,...,n}\end{array}$
(8)

Where $m$ and $n$ are the numbers of elements in the universes of $A$ and $B$,
respectively.
The rules are derived from the experimental findings, physiotherapy
consultancy and collaboration, leading joint hypothesis (interpretation of
control of human movements) and knowledge implying relations between the
angles of joints [33]. The output is calculated based on input variables as
illustrated in Fig. 7, and defined by Eqn. 9.

$\begin{array}[]{llllll}IF~{}P.E_{W}~{}is~{}...~{}AND/OR~{}P.E_{E}~{}is...~{}THEN~{}P.E_{S}~{}is~{}...\\\
IF~{}O.E_{W}~{}is...~{}AND/OR~{}O.E_{E}~{}is...~{}THEN~{}O.E_{S}~{}is~{}...\\\
IF~{}AV.E_{W}~{}is...~{}AND/OR~{}AV.E_{E}~{}is...~{}THEN\\\
~{}~{}~{}~{}AV.E_{S}~{}is~{}...\\\
IF~{}T.E_{H}~{}is...~{}AND/OR~{}T.E_{S}~{}is...~{}THEN~{}...\\\
IF~{}T.E_{C}~{}is...~{}AND/OR~{}T.E_{R}~{}is...~{}THEN~{}...\end{array}$ (9)

Figure 7: Surface Viewer of the fuzzy inference system based on the entire
span of the inputs and output in the ReHabGame.

##  6 Results and Discussion

The system calculates the orientation of joints (shoulder, elbow, wrist and
hand) using inverse kinematics. The posture of the body is also controlled
during the game through the position and orientation of the spine to Shoulder,
neck to head using fuzzy logic and rules. Mamdani’s fuzzy inference method is
coded in Unity3D through a fuzzy-plugin, and the fuzzy inference process is
applied at the level of individual rules. The contribution of each fuzzy set
inferred is considered individually, and the final crisp control action is
obtained by taking a calculus (an average) over a concrete fresh
characteristic value from each one of inputs. If the input from every
kinematic rule is ”Very Good”, with at least one input belongs to the “Good”
category, then the execution of the current iteration is ”Very Good”. If the
input from every rule is ”Good”, ”Very Good”, with at least one input belongs
to the “Good” category, then the execution of that iteration is ”Good”. The
developmental results show that the system allows limb movement analyses in
the joint space which can be used to define define performance indicators that
quantify patient residual motor capabilities and estimate the results of the
therapy. The ReHabGame uses experience gained from exploration of a data set
to improve the performance or predictive ability and provides superior
predictions to guide the development of improved clinical protocols.

##  7 Conclusion

The increasing availability of ’off-the-shelf’ gaming peripherals in the
consumer market has provided greater opportunities for research into non-
gaming or serious gaming applications. The ReHabGame developed in the current
work is one example of this fusion of different technologies via relatively
cheap acquisition of the Kinect Xbox, Mya armband, and Rudder Pedal devices. A
fuzzy logic solution incorporated in a virtual environment is facilitated by
the Unity 3D game engine. The proposed system aims to address kinematic
activity of the upper and lower limbs which at the same time provides
requirements to cater tasks to the needs of the patient. Various constraints,
inputs and rules are employed by the player or the therapist through the menu
of ReHabgame and depend on the level of the desired outcome and motor skills
that need to be attained. The simulation of the upper limb motions through the
robotic avatar is performed according to the position of the terminal
trajectory using the IK algorithm and dynamics method. The intelligent system
can halt game progression and game difficulty until the player achieves a
desired outcome. The system offers the possibility to provide a personalised,
autonomously-learnt rehabilitation programme for patients with neuromuscular
disorders by performing various facilitation movements.

## References

  * [1] D. T. Lai, R. K. Begg, and M. Palaniswami, “Computational intelligence in gait research: a perspective on current applications and future challenges,” IEEE Transactions on Information Technology in Biomedicine, vol. 13, no. 5, pp. 687–702, 2009.
  * [2] S. Q. Xie and P. K. Jamwal, “An iterative fuzzy controller for pneumatic muscle driven rehabilitation robot,” Expert Systems with Applications, vol. 38, no. 7, pp. 8128–8137, 2011.
  * [3] F. Kavalier, “Video games and health,” Bmj, vol. 331, pp. 122–3, 2005.
  * [4] J. H. Lilly, Fuzzy control and identification.  John Wiley & Sons, 2011.
  * [5] P. Czekalski, “Evolution-fuzzy rule based system with parameterized consequences,” International Journal of Applied Mathematics and Computer Science, vol. 16, no. 3, p. 373, 2006.
  * [6] M. H. F. Zarandi, J. Jouzdani, and M. F. Zarandi, “Reinforcement learning for fuzzy control with linguistic states,” Journal of Uncertain Systems, vol. 2, no. 1, pp. 54–66, 2008.
  * [7] L. A. Zadeh, “The role of fuzzy logic in modeling, identification and control,” 1994.
  * [8] B. R. Gaines, “The learning of perceptual-motor skills by men and machines and its relationship to training,” Instructional Science, vol. 1, no. 3, pp. 263–312, 1972.
  * [9] C. MacDonald, Z. Moussavi, and T. Sarkodie-Gyan, “Formation of an internal model of environment dynamics during upper limb reaching movements: A fuzzy approach,” in 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4862–4865, IEEE, 2007.
  * [10] H. Singh, M. M. Gupta, T. Meitzler, Z.-G. Hou, K. K. Garg, A. M. Solo, and L. A. Zadeh, “Real-life applications of fuzzy logic.,” Adv. Fuzzy Systems, vol. 2013, pp. 581879–1, 2013.
  * [11] A. Nawrocka, M. Nawrocki, and A. Kot, “Fuzzy logic controller for rehabilitation robot manipulator,” in Control Conference (ICCC), 2014 15th International Carpathian, pp. 379–382, IEEE, 2014.
  * [12] D. Skoda, P. Kutilek, V. Socha, J. Schlenker, A. Stefek, and J. Kalina, “The estimation of the joint angles of upper limb during walking using fuzzy logic system and relation maps,” in Applied Machine Intelligence and Informatics (SAMI), 2015 IEEE 13th International Symposium on, pp. 267–272, IEEE, 2015.
  * [13] J. H. Cho, H. S. Woo, H. J. Lee, and C. S. Kim, “Control configuration for upper limb rehabilitation robotic systems,” in Control, Automation and Systems (ICCAS), 2015 15th International Conference on, pp. 1500–1502, IEEE, 2015.
  * [14] R. Huq, R. Wang, E. Lu, D. Hébert, H. Lacheray, and A. Mihailidis, “Development of a fuzzy logic based intelligent system for autonomous guidance of post-stroke rehabilitation exercise,” in Rehabilitation Robotics (ICORR), 2013 IEEE International Conference on, pp. 1–8, IEEE, 2013\.
  * [15] M.-K. Chang and T.-H. Yuan, “Experimental implementations of adaptive self-organizing fuzzy slide mode control to a 3-dof rehabilitation robot,” in Innovative Computing Information and Control, 2008. ICICIC’08. 3rd International Conference on, pp. 503–503, IEEE, 2008.
  * [16] A. Karime, M. Eid, J. M. Alja’am, A. El Saddik, and W. Gueaieb, “A fuzzy-based adaptive rehabilitation framework for home-based wrist training,” IEEE Transactions on Instrumentation and Measurement, vol. 63, no. 1, pp. 135–144, 2014.
  * [17] V. Fernandez-Cervantes, E. Stroulia, L. E. Oliva, F. Gonzalez, and C. Castillo, “Serious games: Rehabilitation fuzzy grammar for exercise and therapy compliance,” in Games Entertainment Media Conference (GEM), 2015 IEEE, pp. 1–8, IEEE, 2015.
  * [18] M. Pirovano, R. Mainetti, G. Baud-Bovy, P. L. Lanzi, and N. A. Borghese, “Self-adaptive games for rehabilitation at home,” in 2012 IEEE Conference on Computational Intelligence and Games (CIG), pp. 179–186, IEEE, 2012.
  * [19] S. Liu, Z. Shen, M. J. McKeown, C. Leung, and C. Miao, “A fuzzy logic based parkinson’s disease risk predictor,” in 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1624–1631, IEEE, 2014.
  * [20] V. Grecu, N. Dumitru, and L. Grecu, “Analysis of human arm joints and extension of the study to robot manipulator,” in Proceedings of the International MultiConference of Engineers and Computer Scientists, vol. 2, pp. 18–20, 2009.
  * [21] E. Papaleo, L. Zollo, S. Sterzi, and E. Guglielmelli, “An inverse kinematics algorithm for upper-limb joint reconstruction during robot-aided motor therapy,” in 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 1983–1988, IEEE, 2012\.
  * [22] D. Tolani and N. I. Badler, “Real-time inverse kinematics of the human arm,” Presence: Teleoperators & Virtual Environments, vol. 5, no. 4, pp. 393–401, 1996.
  * [23] D. J. Lura, The Creation of a Robotics Based Human Upper Body Model for Predictive Simulation of Prostheses Performance.  PhD thesis, University of South Florida, 2012.
  * [24] C. Welman, Inverse kinematics and geometric constraints for articulated figure manipulation.  PhD thesis, Simon Fraser University, 1993.
  * [25] S. S. Esfahlani and T. Thompson, “Intelligent physiotherapy through procedural content generation,” in Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference, 2016.
  * [26] L.-X. Wang, “A course in fuzzy systems and control prentice hall,” Facsimile edition, 1997.
  * [27] M.-K. Chang, “An adaptive self-organizing fuzzy sliding mode controller for a 2-dof rehabilitation robot actuated by pneumatic muscle actuators,” Control Engineering Practice, vol. 18, no. 1, pp. 13–22, 2010.
  * [28] A. Barghout, A. Alamri, M. Eid, and A. El Saddik, “Haptic rehabilitation exercises performance evaluation using automated inference systems,” International Journal of Advanced Media and Communication, vol. 3, no. 1-2, pp. 197–214, 2009.
  * [29] E. Czogala and J. Leski, Fuzzy and neuro-fuzzy intelligent systems, vol. 47.  Physica, 2012.
  * [30] O. Cordón, F. Herrera, and A. Peregrín, “Applicability of the fuzzy operators in the design of fuzzy logic controllers,” Fuzzy sets and systems, vol. 86, no. 1, pp. 15–41, 1997.
  * [31] R. Alcalá, J. Casillas, O. Cordón, F. Herrera, and I. Zwir, “Techniques for learning and tuning fuzzy rule-based systems for linguistic modeling and their application,” Knowledge Engineering Systems, Techniques and Applications, vol. 3, pp. 889–941, 1999.
  * [32] E. D. Bigler, “Mild traumatic brain injury: the elusive timing of “recovery”,” Neuroscience letters, vol. 509, no. 1, pp. 1–4, 2012\.
  * [33] N. Dounskaia, “Control of human limb movements: the leading joint hypothesis and its practical applications,” Exercise and sport sciences reviews, vol. 38, no. 4, p. 201, 2010.
  * [34] W. Siler and J. J. Buckley, Fuzzy expert systems and fuzzy reasoning.  John Wiley & Sons, 2005.